Linda, a second-year master’s student, had agreed to help out the department of psychology with its outcomes assessment process. In exchange for her work on the project, the department chair agreed to let Linda use some of the data collected for the outcomes assessment project for her master’s thesis. Linda had decided to investigate whether students’ attitudes toward statistics were related to performance on a comprehensive statistics exam. Therefore, Linda needed to construct a 100-item statistical knowledge test. She gathered old exams, study guides, and items from professors in the department who taught undergraduate and graduate statistics classes. She went through piles of statistics books, study guides, and test banks of test items to draft items for the test. Some professors had even agreed to write items for her. 

After several months of pulling together items and going through multiple revisions from the department outcomes assessment committee, Linda was finally ready to pilot test her assessment device. She was able to get 21 current graduate students in the program to take her 100-item statistics knowledge test. Some of the items seemed to be working for her, while others clearly needed revision. Table 13.3 displays the item analysis results for the first five items from her assessment device. Answer the questions that follow, based on the item analysis results reported in Table 13.3.

Questions 

1. How did students seem to do based on the five items presented in Table 13.3? 

2. Based only on the information presented in Table 13.3, what revisions should Linda make to each item? 

3. Do you have a concern that Linda had 100 items but only 21 subjects? What problems might this cause in interpreting her item analysis results? 

4. Why do you think 0.000s are printed for all the non-keyed entries in item 5, as well as for some options in the other items? 

5. Which item would you say is the “best” item? Why? 

6. Are there any items Linda should simply just throw out (i.e., they are just not worth spending the time revising)? 

7. What additional information would be helpful in evaluating the test items? 

8. Is there a problem with using graduate students during the pilot testing phase if the test will eventually be used as an outcomes assessment device for undergraduates?

Table 13.3

Transcribed Image Text:

Quest. No. 1 N Item Statistics 5 Prop. Correct 0.381 0.667 0.143 0.857 1.000 Biser. 0.238 0.353 0.238 0.614 0.000 Point Biser. 0.187 0.272 0.153 0.396 0.000 Alt. ABCDAMUDARCA ABCDABCD Alternative Statistics Prop. Endorsing 0.238 0.143 0.381 0.238 0.238 0.000 0.095 0.667 0.810 0.143 0.048 0.000 0.857 0.095 0.048 0.000 0.000 1.000 0.000 0.000 Biser. -0.106 -0.227 0.238 -0.022 -0.615 0.000 0.366 0.353 -0.392 0.238 0.533 0.000 0.614 -0.770 -0.084 0.000 0.000 0.000 0.000 0.000 Point Biser. -0.077 -0.146 0.187 -0.016 -0.447 0.000 0.211 0.272 -0.271 0.153 0.248 0.000 0.396 -0.444 -0.039 0.000 0.000 0.000 0.000 0.000 Key