The time had come, at last. After countless years as a student, Janie had been teaching her first undergraduate college course—an introductory statistics course—for nearly five weeks. Now the time had finally arrived to create her very first exam. Sure, she’d had plenty of experience on the test-taker side of the table, but now it was her turn to create a test of her own. She had serious criticisms of some of the tests her own professors had administered to her over the years, and she was determined to do better. Janie wanted to ensure that the statistics test was fair by ensuring that the test assessed knowledge proportionate to what was covered in the course. The only trouble was, determining what was actually covered in the course was a little trickier than she had thought it would be.
Because she had only lectured on the first four chapters of the textbook, she thought she’d have a fairly clearly defined domain. The material covered so far included

(a) A general introduction to statistics;

(b) A chapter on frequency distributions that emphasized the interpretation and development of graphs and tables;

(c) A chapter on measures of central tendency, including the mean, median, and mode; and, finally,

(d) A chapter on measures of variability, including the range, standard deviation, and variance. 

The latter two chapters seemed more important than either of the first two chapters. Janie wondered whether it would be best to create more items on these latter chapters, or if students might expect that each chapter would be tested equally. Her stomach began to be tied into knots when she thought about the prospect of having to create a lot of items from the first chapter, which seemed to provide little information of any real substance.

Suddenly, another thought came to her. The test really shouldn’t be drawn just from the assigned textbook readings. She recognized that the course content was actually composed of three elements: content that had been presented during lecture only, content that had been part of the assigned textbook readings only, and, finally, content that had been presented in both her lecture and the assigned readings. Each of these components of course content was important, although she recognized an implicit pecking order of importance of the material: the material that she lectured on and was presented in the readings was most crucial to a good understanding of course concepts, while material presented only in lecture would likely come next in importance, and the content that was presented only in the assigned readings was somewhat less important. 

Janie was also concerned about the types of items she should use in testing. On the one hand, she wanted to ensure students could apply their learning through use of computational problems, but, on the other hand, she felt that introductory statistics should emphasize understanding of these foundational concepts above anything else. How, then, could she ensure not only that she covered the domain appropriately but also that the right types of items were used to assess exactly the type of learning she hoped to promote? This question was no easier to answer when Janie realized that students would have only about 50 minutes to complete the test. One thing seemed certain—she was quickly developing a greater respect for the professors who had constructed all those exams she had taken throughout the years.

Questions 

1. Describe the first few steps you would take in defining the content domain that would comprise Janie’s statistics exam.

2. Would you recommend that Janie write the same number of items for each of the textbook chapters? Why or why not?

3. What percentage of items should be written for each of the following?

a. Content presented both in readings and in lecture

b. Content presented in lecture only

c. Content presented in the assigned readings only On what basis are you making your recommendations?

4. What types of items should Janie use to assess student learning in the statistics class? Should all of the course content be assessed using the same item format?