Analytics mindset – Final Project 4

Gamification

Part 1: 

Background 

You are the chief technology officer (CTO) of an international bank. A key component of your job is to manage risk within the bank related to information technology (IT). Banks face significant regulatory oversight and must have well-functioning internal controls to prevent and detect any problems related to IT. Within the IT area, data security and privacy are high-risk areas. As such, you must design and implement internal controls to reduce risk. One key preventive internal control that your bank has implemented is employee training. 

As part of this control, all bank employees must complete regular IT trainings. The feedback you have received about past trainings is that they are tedious and boring. You are concerned that employees may not engage fully in the trainings and, thus, the control is not helping reduce the risk of an IT security incident. If there is a significant IT security incident on your watch, you are likely to lose your job. 

At a recent conference, you heard of a new way to increase interest in and learning from training: gamification of training. Gamification is the application of gaming techniques — like using points, badges, leaderboards, stories, etc. — to non-game scenarios. As one gamified vendor representative explained, “We take traditional training courses and make them more fun by making them a game. Your employees will engage at a deeper level, learning significantly more than in any traditional training session, and have fun while doing it!” 

The possibility of making IT security training more interesting has perked your interest. You need to make a recommendation to the rest of the executive team about whether you will purchase and implement gamified training for your next wave of IT security training or go with a traditional training module. To help you make an informed decision, you reach out to a friend at another bank who recently implemented a gamified IT security training module at her bank. She sends you a data file and memo for you to analyze to help inform your decision. 

Before you go any further, you remember your training about the importance of using an analytics mindset. You decide to review the training material before continuing. 

Implementing an analytics mindset 

Having and using an analytics mindset are critical in accounting and business. An analytics mindset is the ability to:

Ask the right questions

Extract, transform and load relevant data

Apply appropriate data analytics techniques

Interpret and share the results with stakeholders

In this setting, using an analytics mindset means using data to inform your decision, rather than going with your “gut feeling,” another person’s recommendation or using another way of deciding. Given that you have data from a similar bank, it makes sense to see what you can learn (and recognize what you cannot learn) by using their data. 

Required

As a CTO, there are many things you need to consider when choosing the best IT security training program for your employees. Develop a list of questions (at least five) for which you want answers to make the best decision about whether you should implement a gamified training model. 

Review the memo and descriptions of the data sent from your friend in the appendix. Also, review the data and consider the following:

Which questions that you generated in the first requirement can you answer and or not by using the data? 

What additional data would you need to answer the questions you developed? 

What are the limitations of the data provided by your friend? 

Prepare a recommendation for the rest of your organization’s executives about whether your organization should use gamified training. This will be included in the final memo.  

Use Tableau to create two stories (see on next page) that can be sent to everyone before the meeting. Give thought to how you will display your analyses so that it is understandable and convincing. 

Make sure your deliverable clearly states the problem, your recommendation, the reasons supporting your recommendation, and any key questions and issues that you were not able to address (and what you would need to address them).

1. Simple Story

 

Tab – Knowledge test – simple

Tab – Learning objectives – simple

Tab – Satisfaction – simple

 

 

Tab – Rank – simple

 

 

 

 

 

1. Advanced story 

 

See videos in Learn.

   
 

Appendix

Memo from friend

Our bank recently decided to try a gamified IT training model. Before providing the training, we sent a survey to a number of our employees to test their IT security knowledge (this group has not done any recent IT training). We received 325 usable responses from this group (Group 1). We then had all employees of the bank complete the gamified training. Afterward, we asked all employees to fill out a survey. We received 531 usable responses from this group (Group 2). For Group 2, we asked the same questions we used to measure IT security knowledge as we did with Group 1. We also asked Group 2 numerous questions about how much these employees enjoyed the training, how they rated it, etc. A full description of the questions and data fields in the Excel file is included below. 

A few notes about the data file:

Any time a field is blank, it means there is no response for that question from the employee. Be careful as you import data to make certain that the values reflect that they are missing rather than showing the value as zero.

It may be obvious, but there is no data about Group 1’s satisfaction with the training because they had not yet completed the training.   

The data does not include personally identifiable information, like an email address, so the data between the two groups cannot be linked for an employee who participated in both surveys. You might consider how this could influence the interpretation of your results. 

Here’s a description of the data in the Excel file. 

ID – a randomly generated unique identifier for each employee response in the data set. 

ReceivedTraining – a dummy variable that equals “Yes” if the employee filled out the survey after completing the gamified training and “No” if the employee did not participate in the gamified training. 

TotalKnowledge – the percentage score of the employee on the IT security knowledge test. Scores can range from 0.00000 (missed every question) to 1.00000 (answered every question correctly). 

The next data fields measure how employees who completed the gamified training scored on the bank’s learning objectives. Each question was measured on a seven-point scale with 1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, 4 = neither agree nor disagree, 5 = somewhat agree, 6 = agree and 7 = strongly agree.

BetterPerform – the answer to: “I can better perform my job because of this training.”

ContentNeeded – the answer to: “This program provided the training content that I needed for my job.”

UnderstandResponsibilities – the answer to: “After the training, I feel proficient in the following areas: I understand my responsibilities for protecting information.”

ApplyTechniques – the answer to: “After the training, I feel proficient in the following areas: I can apply the risk management techniques used in protecting information.”

KnowImportance – the answer to: “After the training, I feel proficient in the following areas: I know the reputational importance of effective information security and the consequences of information being lost or stolen.” 

The next data fields all used the same basic question: “Please compare the most recently completed gamified training that used an interactive, game-style approach with your last training experience that did not use this approach. Please rate which was better using the following dimensions … .”

Employees could select any number on a seven-point scale with responses anchored at 1 = gamified training, 4 = they were the same and 7 = traditional training. Employees rated their satisfaction with the training based on the following words:

Enjoyable

Interesting

Fun

Informative

Boring

Waste of time

The next data fields contain rankings of different types of training. Employees were asked to “Please provide a rank ordering of what you would prefer for future training.” A ranking of 1 was the most preferred, followed by 2, 3, 4 and 5 being the least preferred. The types of training that were ranked include the following (with description):

RankGamified – online training using an interactive, game-style approach

RankOnlineVoice – online training using mostly written materials with voice-over (e.g., PowerPoint presentation with a narrator)

RankWritten – online training containing only written material

RankLecture – in-person training with a traditional approach

RankOther – other, please describe

 

Analytics mindset

 

Gamification

 

Part 2: 

 

To provide additional evidence about how a gamified training approach compares with more traditional training approaches, you conduct an experiment. Experiments have several advantages over surveys (like the data you analyzed in Part 1). In a true experiment, you can randomly assign participants to do different things (called conditions in experimental talk). Randomly assigning participants creates equal comparisons because any unique factors should be represented equally in each group. For example, assume in Part 1 that the group that did not receive gamified training included the least intelligent and least motivated people in the company. When comparing the responses, you wouldn’t know whether differences in the performance of the gamified training were caused by the training or the differences in motivation and intelligence of the groups. If you can randomly assign participants to either complete the training or not complete the training, then there should be an approximately equal number of intelligent and motivated people in each group (especially as you have larger groups participate), so any differences should be caused by the training and not other factors. 

 

In the experiment, there are three different groups:

 

Group 1 received no training at all. 

 

Group 2 received a traditional training course that did not use gamification. 

 

Group 3 received a gamified training course. 

 

Participants in all three groups answered the same knowledge questions as in Part 1. Participants in Groups 2 and 3 also answered the satisfaction questions related to enjoyable, interesting, fun, informative, boring and waste of time. In the experiment, these questions were measured on a seven-point scale with 1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, 4 = neither agree nor disagree, 5 = somewhat agree, 6 = agree and 7 = strongly agree.

 

In the experiment, there is one additional field labeled TypeOfTraining, which lists whether they received “No training”, “Non-gamified training” or “Gamified training.”

 

Required

 

Brainstorm the advantages and disadvantages for using an experiment versus a survey. What are the strengths and weaknesses of each approach?

 

Prepare a recommendation for the rest of your organization’s executives about whether or not you should use gamified training based on the experimental data. You will include this in your final memo. 

 

Use Tableau to create two stories that can be sent to everyone before the meeting. Give thought to how you will display your analysis so that it is understandable and convincing. 

 

Make sure your deliverable clearly states the problem, your recommendation, the reasons supporting your recommendation, and any key questions and issues that you were not able to address (and what you would need to address them). Also, consider the following questions:

 

Do you reach a different conclusion based on the experimental data versus the survey data? 

 

Which data do you think is more convincing?   

 

What evidence supports your recommendation and does not support your recommendation?  Why do you believe the evidence supporting your recommendation is more convincing than the evidence that does not support your recommendation? 

 

1. Simple Story

 

Tab – Knowledge – simple

 

Tab – Satisfaction differences – simple

 

2.  
   • Advanced Story
    

  

 

 

Tab – Satisfaction – advanced (see video)

 

 

 
 

Part 3: 

 

Prepare a brief, two-page memo (before tables and figures) which includes:

 

a recommendation for the rest of your organization’s executives about whether your organization should use gamified training, based on the survey data (Part 1 Q3).

 

recommendation for the rest of your organization’s executives about whether or not you should use gamified training based on the experimental data (Part 2 Q2)

 

your overall recommendation about whether gamified training is worth it.

 

Consider the following questions when writing your memo.

 

Do you reach a different conclusion based on the experimental data versus the survey data? 

 

Which data do you think is more convincing?   

 

What evidence supports your recommendation and does not support your recommendation?

 

 Why do you believe the evidence supporting your recommendation is more convincing than the evidence that does not support your recommendation?