Caleb knew he had to pay a visit to Dr. Zavala, the instructor of his psychometrics course. Caleb’s grade on the first exam was, shall we say, less than impressive. While his performance on the multiple choice section was nothing spectacular, the brief essay section of the exam had really tripped him up. Caleb had received only 5 out of 10 points on the first essay question, and 6 out of 10 points on the second and third essay questions.

When Caleb walked up to his instructor’s office, he was happy to see the door was open and no other students were in the office. Dr. Zavala had been willing to go over the exam with Caleb, question by question. Unfortunately, Caleb was unable to detect any particular pattern in the multiple-choice items that he’d gotten wrong. He certainly had missed quite a few, but it did not seem that the items he’d gotten wrong were all from the same chapter or lecture. Rather, the items he’d gotten wrong were from content throughout the topics covered in the course. Caleb did have one major question for the instructor when it came to the essay questions. 

“Dr. Zavala, how come you don’t provide several options for the essay questions? My history professor allows us to choose three out of five essay questions.” 

“I’d like you to tell me Caleb,” replied the professor. “The primary reason I don’t like to offer an option between essay questions has to do with reliability, and classical test theory. Given what we’ve discussed in class, can you hazard a guess as to why I don’t like to provide a choice between questions?”

Caleb thought for a few minutes before responding. “I understand that my actual test score is called an observed score, and that is due to my true score plus error.”

“Correct”, interjected Dr. Zavala.

“So are you saying that my score is due to error?” asked Caleb.

“Well, in part” chuckled Dr. Zavala. “But that is true for just about every test. What I want you to do is focus on the true score. It’s because of the true score that I don’t provide students a choice between test questions.”

Caleb responded, “I know that the true score is my theoretical expected number of items correct on the test. Basically, if I took the test an infinite number of times, it would be the average of my observed scores. Although a true score is what we really want to measure, the observed score on any one administration of the test is only an approximation of the true score because of the error on the test.”

Dr. Zavala smiled. “You know more than your exam grade might indicate. You just revealed what I was trying to get at.”

Caleb looked confused. “I did?”

“You stated the true score is your expected number of items correct on that test. So, if…” 

Caleb cut him off. “Oh, I get it. If you provided a choice between items, there would actually be different tests, depending upon which items a student chose.” 

“Exactly,” responded Dr. Zavala, “and your true score would actually differ, depending upon which items you chose. I’ll admit that giving students options on which test questions to answer can raise their grades. But that’s not the purpose of testing. I also believe there is a certain content domain that I want to ensure students master. I then pick items that assess that entire content domain. I don’t want students picking items from just part of that content. I want to make sure that all students in your class are tested on the same measure, so I don’t provide students an option between items.”

Questions 

1. Caleb was unable to detect any pattern in the content of multiple choice items he had gotten wrong. Does this challenge the reliability of the test in any way?

2. Explain why Caleb’s performance on the essay items might provide some small evidence of reliability for the exam.

3. Caleb appears to have a good understand of the concept of a true score. How would you define error according to Classical Test Theory?

4. Explain why, according to Classical Test Theory, a student’s true score might differ depending upon which items among several alternatives the student answered.

5. How convincing is Dr. Zavala’s argument for not allowing students to choose among several alternative items? Explain your opinion.