Alexius, a fifth-year doctoral student, had agreed to sit on the committee that was restructuring the doctoral comprehensive exams for his department. It seemed every year students complained about the long essays they had to write and professors complained about having to read and grade the essays with little guidance. Therefore, a committee of mostly full professors in the department was formed to explore the possibility of having a new two-part multiple-choice comprehensive exam. The first part would be a 250-question multiple-choice exam covering several general areas (e.g., history and systems, statistics, and research methods). The second part of the test would also have 250 multiple-choice questions, but in the student’s area of concentration (e.g., social, cognitive, clinical, or I/O psychology). Thus, the test would consist of 500 multiple-choice questions in all. While a common standard of 80% correct or 90% correct could be used to set the pass point, the committee did not feel that was wise, as they knew the questions on the test would change each year. In addition, while all students in a given year took the same general portion of the test, students in different concentrations took different area-specific tests. Thus, it was felt a new pass point should be established each year for each test segment. Several of the counseling professors on the committee sat on the state licensing board for counseling psychology, and they used a similar procedure to set the pass point for the professional licensing exam for counseling psychologists in their state for the written multiple-choice portion of the exam. 

Because the number of students who took the comprehensive exams in a given year was relatively small (i.e., usually less than 20 students), an empirical strategy for setting the cutoff score did not seem feasible. However, the committee was uncomfortable with using a purely judgmental procedure for setting the cutoff score. In addition, multiple cutoff scores had to be set, one for the general portion of the test and a separate cutoff score for each specific test. The committee chair, who also happened to be the department chairperson, asked Alexius to provide the committee with a proposal of how best to set the cutoff scores for each portion of the test. Alexius felt a bit overwhelmed. Here were all these professors in the department, many who had been there 30 years or more, and they were asking him for recommendations on how to set cutoff scores for the tests. Yes, he had just taken his comprehensive exams the year before, but that was under the old system when you had to write six or eight long essays, not this multiple-choice format. In some ways, he thought this was probably better than having to write all those questions. Therefore, Alexius went back to his notes from his applied psychological measurement class and started a literature search on the “best practices” for setting cutoff scores in such situations. 

Questions 

1. If you were Alexius, where would you start your search for “best practices” for setting cutoff scores on a graduate comprehensive examination? 

2. While a purely empirical method for setting the cutoff scores seems unrealistic given the small sample sizes, what things could Alexius do to make his judgmental procedures more empirical?

3. Who are the likely SMEs for setting the cutoff scores for the general test? The area-specific tests? 

4. Is there a problem with the same individuals writing the questions and also helping to set the cutoff scores on the test they created? 

5. Would information from past “pass rates” be of any use to the committee given that the format is being changed?