An educational software company says that their math software dramatically increased student test scores, based on a study in which the Pebblebush school district offered the software for purchase to students, and then state-wide standardized test scores of the students who bought and used the software were compared with those who didn’t have it. Then, Leafstone School District ran their own study, in which two of the four 8th grade general math teachers gave the software to all their students and the other two teachers did not give out the software. This study found that students who got the software did a little better on the same state-wide standardized test than those who did not get it. Finally, Rocklawn School District reported that they found using the software actually harmed learning. They based this on their study, in which all students who were identified as struggling in math were given the software, and then were compared to the other students as controls, on the same statewide standardized test.

a. What problems can you identify in the studies?

b. Assuming that the effectiveness of the software doesn’t differ across the three school districts, how can the differences in the study designs explain the differences in results?