Measuring ROI for e-Learning: A Case Study In 1995, General Electric started a corporate wide quality initiative, Six Sigma Quality. The company developed two company specific Six Sigma quality processes to improve or design products/services. The quality processes are sophisticated statistical-driven business roadmaps and knowledge systems that require extensive training and coaching to apply to specific job functions. The quality processes have proved to be highly effective in identifying areas critical to quality (CTQs) and to exceed customer expectations. A company-wide project tracking database that records all project information has demonstrated the cost savings and revenue generating impact. To assist employee job performance using the Six Sigma tool, in 1998, the company invested $3+ million on an online EPSS, Six Sigma Quality Coach (SSQC), which allows all employees worldwide to access the business and quality process at any time. The rationale for building an online EPSS stemmed from the need to more efficiently support the quality processes. The existing system of one-on-one coaching was overburdened, created information bottlenecks. In this system, employees had to contact designated quality leaders, Black Belts or Master Black Belts for individualized coaching, waiting for a reply that could take up to four weeks. As a result, project cycle times were adversely affected. The SSQC is a personalized, intranet-based, performance support and knowledge management system. Employees can search information needed by job functions and by business division. An ROI project was conducted in 1999, one year after the deployment, to measure the effectiveness of the online EPSS. Suppose you are conducting such a project, how would you do the following? 1. Based on your understanding of the ROI system and the functions of SSQC, which type of control group would you use to start the ROI project? 2. What data sources would you be looking into? Explain. 3. What key measures would you choose to measure the ROI, and why? 4. What analyses would you be conducting on the data you collect? Note: there are no right or wrong answers to the questions, participation is most important.Identify Control Group Methods: An Exercise Instructions: Suppose you are required to evaluate and measure the HRD interventions listed below. For each situation, identify the control group type that is appropriate and most efficient to conduct evaluations for levels II, III, and IV, including return on investment (ROI) measurement. Situation Control Group Type O An organization is planning to launch a pilot program to test the feasibility and effectiveness of a large scale organization wide HRD intervention planned O A leadership development program involving all members of middle management teams with a built-in evaluation plan O An voluntary online performance support system implemented 7 month ago in a large multinational company O A new e-learning course that was migrated from instructor -led training (ILT); the ILT course has been offered many times in the organization 1 A supervisory skills training conducted in 7 out of 12 locations in a large organization A company recently offered an online training which never offered in the ILT format. Determine the effectiveness and ROI for a online EPSS for bank tellers in all 170 branches in a banking system O Evaluating courses currently delivered online as opposed to those previously delivered in classrooms O Evaluating a customer service training to all call center employees from the beginning to the end O An organization determine to measure all HRD programs for the 5 yearsModule 3 Assignment ROI Worksheet: Transition to Web-based Learning Introduction The following worksheet is a real world consulting project of calculating ROI for a transition from instructor-led training (ILT) to e-learning. The company is a well-known and fast growing fast food restaurant chain in the U.S. Let's call it Fast-to-Go, Inc. to hide its real name. Due to the high turnover rate among employees and managers, the company has to maintain constant training to keep up with the competitions. However, the high cost of ILT and related travel plus additional factors, e.g., September 11, and the nature of franchise system, the company is considering transitioning the learning programs from ILT to e-learning. To assist the decision-making project, the predictive ROI measurement was conducted. Note that the worksheet only includes the U.S. domestic retail stores. The analytical framework is to compare the costs of training between the two different delivery modes. And then use the standard ROI formula: ROI = (Total benefit - Total Cost) / Total Cost x 100% = Net Benefit/Total Cost x 100%. In this case particularly: Net Benefit = Cost saving due to e-learning compared with ILT Total Cost = Total cost of e-learning, including learning time and other opportunity cost. The worksheet should be able to guide you through the variables and factors need to be considered when conducting an ROI measurement project. You may easily reach your own conclusion about the projected ROI given the clearly identified cost and benefit factors. This sample project also demonstrate the complexity of ROI measurement, although the ROI formula is seemingly easy and straightforward.Retail Manager Training FY 2002 FY 2003 FY 2004 A. Existing stores 6,000 7,000 8,000 B. New stores 900 1,000 1,000 C. Managers per store (average) 3.5 3.5 3.5 D. New managers due to 3, 150 3,500 3,850 expansion [B x C] E. Training of current managers 4,000 4,000 4,000 (backlog) 1 F. New managers due to turnover 6,300 7,350 8,400 [A x C x 30%] G. Total managers needing 13,450 14,850 15,900 training [D + E + F] H. Average daily manager $85 $85 $85 compensation 2 1. Number of class days per 11 5 2 manager 3 J. Number of travel days per 2.5 1.5 1 manager K. Total travel days [G x J] 33,625 37, 125 15,900 L. Travel time cost [K x H] $2,858, 125 $1,893,375 $1,351,500 M. Time in class per manager 11 5 2 N. Total time in class [G x M] 147,950 74,250 31,800 O. Class time cost [N x H] $12,575,750 $6,311,250 $2,703,000 P. Self-study time [(11-M) x .7515 0 4.5 6.75 Q. Self-study cost [P x H] $0 $5,680, 125 $9, 122,625 R. Total learning cost (transition $15,433,875 $13,884,750 $13, 177, 125 to web-based) [L + 0 + Q] S. Total learning cost (without $15,433,875 $20,827, 125 $22,299,750 transition to web-based) [G x 16.5] x H T. Savings due to transition to 0 $6,942,375 $9, 122,625 web-based training [S - R] U. Total investment in e-learning $2,566,000 $1,054,000 $86,000 1. Assumes backlog of 12,000 untrained managers 2. Based on average salary of $25,000 for manager ($12.50/hr.) and $20,000 for shift leader ($10.00/hr) for 8 hours; no benefits or salary increases included. 3. Based on assumption of 11 days in FY2002, 5 days in FY2003, 2 days FY2004. 4. Based on assumption of .5 travel days for each class 5. Efficiency factor of .75 used as multiplierCrew Training FY 2002 FY 2003 FY 2004 A. Stores 6,000 7,000 8,000 B. Total crew members 125,000 145,000 160,000 C. Anticipated turnover rate2 100% 90% 80% D. Total crew members needing 250,000 290,000 320,000 training E. Average crew wage: $8 per hour $8 $8 $8 F. Training time to proficiency 20 20 20 G. Cost of turnover - before $32,500,00 |$37,700,000 $41,600,000 transition [($100 x B) + (B x Ex 0 F) 13 H. Cost of turnover - during 32,500,000 $33,930,000 $33,280,000 transition [($100 x B x C) + (B x C X Ex F) ] Benefit of reduced turnover 0 $3,770,000 $8,320,000 1. Based on 10 crew members per Baskin-Robbins; 25 per Dunkin Donuts; 15 per Togos 2. Assumed improvement due to more crew member access to learning and better trained managers 3. Based on assumption of $100 per new hire acquisition cost and incremental training cost Instructor Costs FY 2002 FY 2003 FY 2004 # of instructors 16 12 8 Average instructor compensation $100,000 $100,000 $100,000 Total instructor cost $1,600,000 $1,200,000 $800,000 Savings due to transition to web- 0 $400,000 $800,000 based training 1. Estimated fully loaded compensation; no salary increases