When Bad Things Happen to Good Projects* HP's project managers knew what could go wrong with their ERP rollout. They just didn't plan for so much of it to happen at once. By CIO Staff, Apr 2, 2007 8:00 AM PT It goes against human nature to always expect the worst. But with IT projects, pessimismotherwise known as contingency planningis the only way to keep small technology problems from becoming full- blown business disasters. Too bad no one can bring themselves to do enough of it. Christina Hanger had little reason to be pessimistic in May 2004, when she was moving one of Hewlett- Packard's biggest North American divisions onto a centralized ERP system from SAP. As the leader of an IT consolidation project rooted in HP's acquisition of Compaq two years earlier, Hanger, HP's senior vice president of Americas operations and IT, had an unbroken record of success migrating five product groups within the two former companies onto one of two SAP systems. Hanger had every reason to believe that the sixth would go well too. Even so, she knew to be prepared for problems. At approximately $7.5 billion in annual revenue, the division involved with this latest project, Industry Standard Servers (ISS), is much larger than any of the others that Hanger had migrated to SAP to that point. So Hanger took the contingency plan that her team had developed for the other five migrations and adjusted it to accommodate the ISS division's larger sales volume. She planned for three weeks of IT snafus, mostly focused on what might happen as a result of tweaking a legacy order- entry system to work with the new SAP system. The contingency plan addressed business impacts too. But the plan wasn't pessimistic enough. Starting when the system went live at the beginning ofJune and continuing throughout the rest of the month, as many as 20 percent of customer orders for servers stopped dead in their tracks between the legacy order-entry system and the SAP system. As IT problems go, this wasn't too big: Some data modeling issues between the legacy system and the SAP system prevented the SAP system from processing some orders for customized products. These programming errors were fixed within four or five weeks. But Hanger and her business colleagues from the ISS division who were on the project steering committee never envisioned the degree to which these programming glitches would affect the business. Orders began to backlog quickly and HP did not have enough manual workarounds to keep servers flowing fast enough to meet customer demand. Angry customers picked up the phone and called HPor worse, archcompetitors Dell and IBM. In a commodity market such as servers, customer loyalty is built upon a company's ability to configure products to order and get them delivered on time. HP could do neither for much of the summer. In a third-quarter conference call on Aug. 12, HP Chairman and CEO Carly Fiorina pegged the financial impact at $160 million: a $120 million order backlog that resulted in $40 million in lost revenue. That's more than the cost of the project itself, estimated to be $30 million. Problem Investigation At HP, Hanger says her team tested the connections between the legacy front-end ordering system and the SAP system. And the connections worked fine for orders that did not involve any custom configuration, as well as for some custom orders. But Hanger's team was unable to adequately test * Modified and shortened by Lian Qi 1 orders that could be configured by customers because the product marketing team had not fully scoped the breadth of configurations that customers would want. When the system went live, some of these custom configurations went through and others did not. Those that didn't got spat out into a dead zone, sitting idle until they could be entered manually. "We had customers wanting a little more flexibility in how they purchased and configured their servers," she says. "And we did not always have the data modeled correctly [for that to happen]." Could Hanger have tested more configurations prior to going live? Yes. Could the team have envisioned all of them? Probably not. But that wasn't the only problem. Hanger's team trained the customer service representatives who would be using the new system two weeks prior to going live. All representatives were required to pass a test showing that they could enter orders without making errors. But when the system went live, the representatives had trouble remembering all the training and were flustered, compounding the number of dropped orders. Hanger's team offered refresher training two weeks into the rollout, but by then, too many orders had fallen out to catch up. "We might have improved things if we had started giving them refresher training a week into the rollout instead of two weeks," she says. Things only got worse when customer demand for configure-to-order systems spiked by 35 percent in June, beyond what HP's demand forecast models predicted. Instead, Hanger's business contingency plan called for normal demand through the summer. "We had planned for three weeks of extra inventory at a 50/50 split between standard servers and configure-to-order servers," she says. The factory in Omaha that had set aside some of its lines to handle the three weeks of orders quickly became overwhelmed. HP rushed to get a second factory in Europe online to help take up the slack two-to-three weeks later, but by then, it was too late. Once the orders backed up, the only way HP could respond was by speeding up deliveries. Depending on the size of the product and distance shipped, the extra cost ranged between 35 percent and 40 percent, according to HP, and gained the company a few days on orders that had already been delayed by weeks. Analysts commented that probably the company culture had not allowed for much active involvement of employees, and as a result the problems which surfaced ultimately became more and more difficult to overcome. The company seemed to have ignored valuable suggestions from employees. Media reports claimed that many HP insiders knew the project, code-named Fusion, had huge risks despite the company' s expertise with SAP migrations. The reports said that the company sales staff had warned that it was not enough on the company's part to choose the traditionally slowest quarter for the rollout. In addition, many Vice-presidents across HP had left the company to join rival firms, which must have affected project implementation. A survey of employees at HP revealed that there was ample distrust of upper management and they were perceived as being overpaid and inefficient. Some employees pointed to a cultural divide within the company which they felt was a matter of serious concern leading to non co-operation between the IT management team and the business team. The Contingency Plan That Wasn't The headlines all claimed an IT disaster, but in fact, HP's disaster resulted from a few relatively small problems in IT that snowballed into a much bigger problem for the business: the inability to cope with the order backlog. However, if ClOs want to stop being held liable for hundreds of millions of dollars in losses for relatively small lT problems, they have to convince business leaders of the vastly increased risks that major enterprise software projects pose to businesses with high-volume supply chains. They must use that awareness to buildwith full support and cooperation from the businessa business contingency plan for IT projects that is as robust as the project plans they create for the new software. In retrospect, Gilles Bouchard, HP CIO and executive VP of global operations, says he should have had additional manufacturing capacity ready before the ERP rollout. In December 2003, Bouchard became CIO and executive vice president of global operations, one of those rare ClOs who also runs the supply chain. Besides emphasizing business contingency planning more strongly, he is in the process of reorganizing the operations and IT groups of HP's businesses. Bouchard replicated his dual role at the regional level too. Hanger runs both IT and operations for the Americas. The more consolidated approach should improve communication between IT and the business, he believes, and make it easier to identify how IT projects affect operations. One message that needs to be communicated more strongly within HP and within every company these days, Bouchard believes, is the message that is implicit in his dual role: "There is big leverage between IT and the business processes when you deal with a large supply chain," he says. "Just looking at contingency planning from an IT point of view would be a big mistake. It has to be looked at from an integrated view of IT processes and the business. Here's what Bouchard would have done differently. He would have expanded the contingency plan to cover five or six weeks. Instead of trying to prevent IT problems that were too small and rolled up in too many strange combinations, it would have been easier to bring the backup factory in Europe online earlier and stockpile more generic servers. By traditional IT contingency planning standards, HP had already gone beyond the call of duty. Most |T contingency planning focuses on preparing extra code rather than extra products. Convention says that to ensure a problem-free transition to a new system, there should be a redundant system ready to handle things if the rollout goes sour. At the very least, there should be a "rollback" strategy to go back to the old system if there is an issue. But both Bouchard and Hanger maintain that neither strategy would have worked in HP's case, considering the scale of its server business. "If we had had two systems running, that would have meant that every supplier would have had an order for us in the old system and the new system," says Hanger. "When it was received into the manufacturing line, it would have had to be received twice and then [be reconciled] twice." Says Bouchard: "You would have created a backlog of orders because of the cumbersomeness of the duplication effort." Probably, what HP should also have done was to create a plan for taking orders and shipping products that assumed the IT system it had planned to use didn't exist. "Contingency planning is not about IT," says AMR Research's Swanton. "It's [about] having people who are watching what's happening, able to detect if there's a problem and working out some simple manual way around it until you're ready to work with the system. If that takes a team working a bunch of overtime, fine. It will be a lot less disruptive than losing sales." Taking your order-to-cash process back to the 19505 requires a mind-set change among all the people involved in that process, from customer service representatives to warehouse clerks, because it will feel silly and unnecessarysort of like imagining what will happen if the sun doesn't come up tomorrow