Jaime was rightly proud of the midterm exam he created for the course in which he served as teaching assistant, PSY 451: Introduction to Forensic Psychology. The instructor, Dr. Dan Kellemen, had asked him to create a free-response test that could be completed within the 1.5-hour class session. Jaime had given a lot of thought to proper test construction techniques in the creation of the test. He carefully went over the topics Dr. Dan wanted covered, and he considered the relative importance of these various topics. Jaime considered a variety of free-response options for the test, but in the end decided to modify Dr. Dan’s usual approach of two to three essay questions. Similar to the midterm exam Dr. Dan had used the previous semester, Jaime’s test was organized around three large “questions.” However, Jaime used these broad questions only to introduce the topic and to help students focus on the sub-items that followed. Under each of the three broad questions, Jaime created between three and four sub-items labelled (a), (b), (c), and so on. It was the responses to these items that were to be graded for the exam. Following each of these sub-items, Jaime recorded the number of points that a student could possibly receive for that item. Items on more important topics received a higher number of points. Overall, the exam contained ten items. 

Before showing the test to Dr. Dan, Jaime produced responses to each item to ensure that the questions were, in fact, capable of being answered. Based on this exercise, Jaime had to revise a couple of the items. Much to Jaime’s delight, Dr. Dan had been noticeably impressed with the quality of the exam and had not made a single modification. During the administration of the exam, a few students asked for the usual types of clarification, but no one indicated any major difficulties in understanding the items on the exam. Now that the students had completed their midterm, Jaime’s next responsibility was to score them. When it came to scoring, Jaime was once again a man with a plan. To be fair to everyone, he had asked students to record only their student identification numbers on their blue books, rather than their names. He also planned on scoring every student’s response to the first item, before scoring even a single response to the second item. Jaime also decided to assign a single, holistic score to each item, rather than assigning separate scores based on content, clarity, originality, and so forth. Even so, Jaime was surprised how long it took to score all of the exams in the class. 

Back in class the next day, Jaime was excited to return the exams to the students. It was, in many ways, the final step in the lifespan of his first exam. Later that day during office hours, he received a visit from Juan, a student in the class. Juan demanded to know why he received a much lower score than another student who, he claimed, had provided similar correct answers. As evidence, Juan produced both his own and another student’s blue books. On question after question, the content of each response was similar, yet Jaime had given the other student a higher score. “How could that be?” Jaime nearly wondered aloud. Stalling, Jaime informed Juan that he’d examine both blue books and would provide a decision at the next class session. In reviewing both blue books more carefully, Jaime realized that he had to agree with Juan on one point—both blue books contained similar quality of information in response to the items. However, Juan’s responses were characterized by poor grammar, spelling, and a general lack of organization. Still, the answers were there, if one searched for them sufficiently. “What role should writing ability play in determination of this grade?” wondered Jaime. On the one hand, it seemed irrelevant to knowledge of forensics. On the other hand, wouldn’t clear communication play a major role in the job of forensic psychologist? Luckily, perhaps, he was “just” the TA—he’d need to seek the advice of Dr. Dan on this one. 

Questions 

1. Jaime followed a number of recommended steps for test development. For each of the following, explain how it assists in the development of a quality test of maximal performance:

a. Consideration of the various weighting of topics

b. Consideration of appropriate response formats

c. Creation of sub-items, rather than fewer, larger essay questions

d. Specification of the number of points assigned to an item

e. Creation of ideal responses to items prior to test administration 

2. Jaime also followed a number of recommended steps to score this essay test. For each of the following, explain how it helps improve reliability:

a. Recording of student identification numbers rather than names on blue books

b. Scoring one item at a time for all respondents, before proceeding to the next item

c. Using holistic scores for an item, rather than using multiple sub-scores for an item

3. Jaime unwittingly included writing ability in his scoring. Is writing ability an appropriate test component for a university class in forensic psychology? Explain.