Dr. Agars was recently hired by the state of California as a senior research analyst in the Department of Corrections (DoC). His degree was in industrial and organizational (I/O) psychology, with a minor in forensic psychology, while his undergraduate degree was in criminal justice. Thus, his boss knew that Dr. Agars had some expertise in psychological testing as well as the job duties of a parole agent. The DoC was recently mandated by the state legislature to develop a certification exam for parole agents. Corrections in the state of California are a $5.2 billion industry, by far the largest in the United States. A large part of the reason for the exorbitant cost is that 66% of the 125,000 annual parolees from the state’s 33 prisons are reincarcerated before their three-year parole is up. That’s more than twice the national average. It’s not all that surprising, however, because 75% of parolees have drug or alcohol problems, 50% are illiterate, and 80% have no job when they get out. Thus, the goal of the state legislature in passing the certification requirement was to hire additional parole agents (who typically have a load of between 80 and 100 ex-convicts at any one time) to work more closely with current felons to prepare them for their eventual release from prison. This, the legislature hopes, will dramatically reduce the number of reincarcerated felons, thus more than making up for the cost of new parole agents. However, there is no way of knowing if the current parole agents are qualified to perform these additional functions, hence, the new certification requirement. 

The Department of Corrections has close to 1,000 applicants for the parole agent job each year. Thus, the DoC typically offered the civil service exam for parole agents four times a year. However, with the certification prerequisite, there will be an additional requirement that individuals not only pass the civil service exam to be a parole agent, they must also pass a new certification exam. In addition, current parole agents will need to take and pass the as yet to be implemented certification exam. The State Personnel Board has hundreds of multiple-choice test items that it has given to tens of thousands of job applicants over the last 20 years (when it started its electronic scoring procedures) for the job of parole agent. The DoC would like to create a computerized version of the certification test that could be offered on an as-needed basis. Professor Agars knows a little about computer-based testing, but is by no means an expert. However, given the DoC was interested in continuous testing and there were a lot of questions and data to get things started, it seemed to Dr. Agars that using item response theory (IRT) to create a computer adaptive test (CAT) would be a logical choice. Use of IRT would allow the test to be tailored (or adapted) to each test taker. In addition, because each individual would, in a sense, have his or her own exam with a different mixture of questions, unlike the state civil service exam, problems with cheating and remembering items would be minimized. Thus, applicants who had already passed the state civil service exam for parole agent could come into a testing center at a designated time and take the certification exam. Finally, using IRT to create a CAT version of the certification exam would also allow individuals currently in the position to take the certification exam multiple times over a short time period until they passed. Thus, it seemed using IRT to create a CAT version of the certification exam was a logical choice. When Dr. Agars presented the idea to his boss, not only was his boss excited about the idea, he wanted to know if he could also do the same thing (i.e., use IRT to create a CAT) for the civil service exam for parole agents. Suddenly, Dr. Agars was beginning to wonder what he had gotten himself into. 

Questions 

1. Does it appear that IRT is a viable option for creating a written exam (CAT or paper-and-pencil) in this situation? 

2. Given the changing nature of the job of parole agent, should Dr. Agars be using questions from prior civil service exams for the selection of new parole agents? 

3. Are there any unique issues concerning the use of IRT for certification and/or licensing exams? If so, what are they? 

4. Would the development and use of a test using IRT procedures be any different for the civil service exam (which typically rank orders job applicants) and the certification exam (which typically sets a pass point and those above are “certified” while those falling below the pass point are not “certified”)? 

5. Based on the information presented in the case study, does it appear that a new certification exam is the answer to the state’s reincarceration problem? What unique information do you think it will provide? 

6. What would be the advantage of using IRT methods over CTTIA procedures to develop the certification test in this instance? 

7. Should the applicants and current incumbents be treated any differently in this situation?