Elena, a first-year educational measurement graduate student, vaguely remembered a discussion of item response theory (IRT) in her undergraduate tests and measurements class, but never thought that she might actually conduct such a study one day. The whole concept of IRT seemed so complex and appeared to require a level of mathematical sophistication that was well beyond her. In addition, the item response function (IRF) graphs she remembered seemed like the apparent random lines she recalled seeing on her father’s oscilloscope when she was a child. How could she possibly understand all of it, let alone help a professor conduct such a study using IRT? 

However, she had recently agreed to serve as a paid graduate research assistant for Professor Koshino in the college of education. Professor Koshino was contracted to help a large local school district evaluate the English competency exit exam it had recently developed and administered to seniors in the district’s four high schools. The district graduated more than 2,500 students each year from its four high schools combined. Not surprisingly, students varied widely in their English ability, both within and across schools. Given the large sample sizes and wide ability ranges, Professor Koshino decided that IRT would be a good way to examine items on the test to determine which items should be kept and which should be revised or discarded. However, Professor Koshino was no expert in IRT. He was hoping he could just turn over the analysis part of the project to Elena and some other graduate students in the educational measurement PhD program. However, Elena and the other graduate students were feeling rather uncomfortable trying to apply what little they had learned about IRT so far to this very real life situation. It seemed time to sit down and have a frank discussion with Professor Koshino.

Questions 

1. What would be the advantages and disadvantages of using CTTIA in this situation instead of IRT? 

2. What would be the advantages and disadvantages of using the 1- PL IRT model? The 2-PL IRT model? The 3-PL IRT model? 

3. Where should Elena start to “get up to speed” with the IRT procedures? 

4. Should the four high schools be analyzed separately or together? 

5. What should Elena and Professor Koshino be focusing on in their IRT computer printouts? 

6. What advantages are there to examining the item response functions (IRFs) in this situation?