“It’ll be easy money.” So said the director of faculty development, when trying to convince Dr. Patricia Lonergan to present a brief overview on test development to a group of interested faculty. She’d earn $200 for delivering a two-hour lecture. The words still rang in her ears. “It’ll be easy money.” So then, what could have gone so wrong? 

Patricia had taught a graduate-level course in test construction for several years, but she knew it would be a challenge to condense a semester’s worth of material into a two-hour faculty seminar. Recognizing that most faculty were interested primarily in how to improve the development of their own tests, Patricia decided to limit her lecture to a quick overview of the concepts of reliability and validity, followed by a lengthy discussion of item-writing tips, and ending with simple procedures for computing item statistics. Though she had lectured many times before on these subjects, Patricia realized she was more than a little anxious about presenting before a group of her peers. She had only been out of graduate school for three years, and she knew that her colleagues at her university had reputations as great teachers. Perhaps it was for this reason that she decided to bring along a lengthy handout that would help the faculty participants remember her main points. Reflecting back on the experience, Patricia was grateful that she’d brought the handout—otherwise, most of her points would never have been heard at all. 

The talk had gotten off to a fairly good start. The conference room in which she gave the presentation was surprisingly full—nearly 20 faculty members were in attendance, most of whom she had never met. These faculty members seemed to follow most of what she reviewed about the importance of developing reliable and valid exams. Most of the participants nodded in agreement to her points, and she noticed that several jotted down a few notes. Somewhat disappointingly, however, no one seemed to contribute his or her own thoughts about these topics. 

Then came what Patricia considered the “meat” of her presentation—a discussion of item-writing tips. In referring to the handout, she asked the faculty participants to alternate taking turns reading aloud the recommendations for writing the first type of items she planned on discussing, multiple-choice questions. A few tips were read. Then Patricia noticed that, simultaneously, several faculty members raised their hands to contribute to the discussion. “Great,” thought Patricia, “now we’ll get some insights into people’s own approaches to creation of these items.” But no. Instead, that’s when things started to go terribly wrong. Patricia first selected a faculty member from the philosophy department. Clearing his throat, he asked, “Why should we waste our time discussing these so-called objective items. Everyone knows they serve no good academic purpose.” A professor across from him concurred, saying, “Unless we abandon our dependence on these types of items, we’ll never prepare our students for the real world. Life doesn’t provide multiple-choice options.” Looking at Patricia, a third faculty member accusatorily asked, “If you call these selected response items ‘objective,’ what does that say about your opinion of essays? Are you saying they are subjective?” Several other faculty members added their own comments in support of these individuals. 

Just when Patricia was trying to formulate a response—any response—several other faculty members began taking issue with the comments of their colleagues. “I’m tired of this rhetoric about the importance of testing through writing,” quipped a member of the chemistry department. Actually, now that Patricia thought about it, he hadn’t used the word “rhetoric,” but something a bit more colorful. At any rate, several other faculty members then chimed in their agreement with the chemist. A few complained about the large size of their classes and the perceived difficulty in using free-response exams. The next few minutes were something of a blur to her. Suffice it to say, however, that a heated discussion erupted, and little of her talk went as planned. Try as she might, her colleagues seemed a lot more interested in airing their opinions about the appropriateness of certain item formats rather than her recommendations for writing better test items. Returning to her office, Patricia reflected once more on those words that had got her into this in the first place. “It’ll be easy money.”

Questions 

1. Do certain item formats prepare students for the “real world” better than others? Why or why not? 

2. What are likely some of the arguments put forth by those who reject selected-response testing in universities? To what degree do you feel these arguments are valid? 

3. In a university setting, why might some departments likely champion free-response item formats while other departments prefer selected-response formats? 

4. What role does politics play in the choice of adoption of item formats in a college classroom? 

5. What role should practical concerns (such as class size) play in the determination of item formats? 

6. Can selected-response items assess higher-level cognitive objectives? 

7. What item formats do you prefer to be tested with? Why? 

8. Based on your own personal experience and observations, in what ways does the choice of item format influence student test preparation?