Working Around Defects

Your system test team has a well researched and documented test plan. A rich set of software workloads have been developed to really stress the system. You have you hardware, infrastructure installed, the software deployed and the workloads fired up and running. Development has been timely in responding to defects from your team and problem turnaround is pretty good. The test effort starts to hit a few defects as you scale the workload up across lots of servers in the test environment. The problems begin to be a real drag on test progress. You cannot seem to get through an overnight test without hitting some of the errors. Development comes back after much analysis and says, Your workload stresses the system more than a real customer ever would, it is not realistic. You go back and forth but neither team is buying the other's arguments. The date for product shipment looms so everyone decides that we'll risk the exception. Everything else with the product seems good and those pesky system test defects can be attributed to unrealistic system stress workloads. We're golden, or at least gilded. Well, everything is looking good for a few months after product shipment. Customers are happy and we're selling lots of systems. However, a couple customer incidents occur one day. Investigation reveals that they both have scaled their REAL workloads and number of servers up to a point that no one really thought possible, or realistic. Lo and behold, the root cause is that they're running the system software ... to hard. Huh, and they said our test wasn't realistic.

Assignment: Analyze the situation. Based on your analysis, what do you think the system test team should have done that could have prevented the customer incidents?