Project Management in Practice: Facebook Risks Interruption to Move a Terabyte

Working on the bleeding edge of innovation is standard procedure for Facebook. To do so however, speed is critical to their operation, and the combination of speed and innovation brings with it high risk. But Facebook is used to handling risk. For example, a recent project involved a multimillion dollar effort to move a terabyte of data from a near-capacity data center to a new, higher-capacity data warehouse by the end of the year, only a hundred days away, at the time. A terabyte (that is, a trillion bytes, or a million megabytes) is equivalent to 250 billion Likes on Facebooka lot of data! The project involved two phases: building and outfitting the new warehouse, and then transferring the data. The new data warehouse was designed so the servers could handle four times as much data as the current ones, and the processors and software were upgraded as well, with the result that the new data warehouse could hold eight times more data and move and manage it more efficiently, all of which represented a savings of millions of dollars in energy costs. Given the short timeline and the importance of the hardware and software working together without a hitch, the project team took many steps to reduce the risks. First, they set clear expectations with both the vendors and internal stakeholders up front so everyone could fit their objectives into those of Facebook's. Also, they conducted round-the-clock testing of the hardware, the software, and the ability of both to work together to deliver the speed, volume, and accuracy Facebook was depending on. To transfer the data to the new warehouse, they had a choice between loading the data onto the equipment before physically moving it to the warehouse (but risking lost or damaged equipment in the move), versus moving and checking the equipment first, and then flowing the data directly to the new site (but risking a network outage or a site crash disrupting their entire website). They took the risk of the latter, but planned multiple risk avoidance steps. First they had to calculate how long it would take to flow the terabyte of data, assuming no network failures or power outagesthree weeks! But there was still a risk that the data flow would use too much network capacity and affect the website. To avoid this, the team built a customer application to throttle the data by limiting and monitoring the bandwidth throughout the entire 3-week data flow. They also performed constant error-checking and data-level corrections to keep the flow synchronized, and alert the team if problems arose. Their up-front detailed planning, constant monitoring, and risk avoidance measures paid off in a successful data move to the new warehouse, on time with no delays or downtime. Source: S. F. Gale, A Closer Look, PM Network, Vol. 24.