You and your team work as educational researchers. You want to know the effect of having...

Transcribed Image Text:

You and your team work as educational researchers. You want to know the effect of having in-class pizza parties on 5th grade student achievement on end-of-year testing results (T = 0 means no pizza parties and T = 1 means they had pizza parties). Your prior theory/causal story is that pizza parties encourage students to continue o learn and not grow weary of the testing grind; students who do not have pizza parties grow weary of test prep and perform worse on the actual test. Your team uses two approaches to find the magnitude of the treatment effect: 1. You survey 1,000 classes in a large number of schools and find those that have had pizza parties (T= 1) and those who have not (T = 0). You then compare the average test scores across the two groups. 2. You randomly assign 1,000 classes to either treatment or control. Those that are treated have pizza parties weekly while those that are not never have one. You compare the average test scores across the two groups. For the observational design (1), your observed treatment effect has a p-value of .0001 with a 95% confidence interval of [8,17]. For the experiment (2), your difference in means has a p-value of .9 with a confidence interval of [-2,1]. Our pre-determined "surprise" threshold is .05 and the null hypothesis for both p-values is that the ATT is zero. A. (5 Pts.) In a sentence or two, explain the results of the observational study (1) in terms of the p- value and 95% confidence interval. What is the implied sign and magnitude of the ATT? What can we say about the value of the ATT for this study? B. (5 Pts.) In a sentence or two, explain the results of the randomized experiment (2) in terms of the p-value and 95% confidence interval. What is the implied sign and magnitude of the ATT? What can we say about the value of the ATT for this study? C. (10 Pts.) Update your causal story relating to pizza parties and test scores appealing to these results. Think carefully about how the results for the randomized experiment differ from the results for the observational study. Are there any common inferential errors which could explain this result? D. (10 pts.) You present the results of your study at a conference and one attendee is underwhelmed. She believes that the results of your experiment are still too noisy and the result should be large and confidently positive. You have some additional money in your research coffers. What action can you take to deal with the editor's concerns? Why? E. (10 pts.) A school booster (who has taken an intro stats class but didn't attend all of the lectures) looks at these results and says "These studies provide definitive proof that pizza parties have zero effect on test scores! Instead of spending money making sure that students have fun lunches in class, we should spend that money on a new hockey rink for the hockey team." In a sentence or two, give one reason why the results of these studies do not support this claim. You and your team work as educational researchers. You want to know the effect of having in-class pizza parties on 5th grade student achievement on end-of-year testing results (T = 0 means no pizza parties and T = 1 means they had pizza parties). Your prior theory/causal story is that pizza parties encourage students to continue o learn and not grow weary of the testing grind; students who do not have pizza parties grow weary of test prep and perform worse on the actual test. Your team uses two approaches to find the magnitude of the treatment effect: 1. You survey 1,000 classes in a large number of schools and find those that have had pizza parties (T= 1) and those who have not (T = 0). You then compare the average test scores across the two groups. 2. You randomly assign 1,000 classes to either treatment or control. Those that are treated have pizza parties weekly while those that are not never have one. You compare the average test scores across the two groups. For the observational design (1), your observed treatment effect has a p-value of .0001 with a 95% confidence interval of [8,17]. For the experiment (2), your difference in means has a p-value of .9 with a confidence interval of [-2,1]. Our pre-determined "surprise" threshold is .05 and the null hypothesis for both p-values is that the ATT is zero. A. (5 Pts.) In a sentence or two, explain the results of the observational study (1) in terms of the p- value and 95% confidence interval. What is the implied sign and magnitude of the ATT? What can we say about the value of the ATT for this study? B. (5 Pts.) In a sentence or two, explain the results of the randomized experiment (2) in terms of the p-value and 95% confidence interval. What is the implied sign and magnitude of the ATT? What can we say about the value of the ATT for this study? C. (10 Pts.) Update your causal story relating to pizza parties and test scores appealing to these results. Think carefully about how the results for the randomized experiment differ from the results for the observational study. Are there any common inferential errors which could explain this result? D. (10 pts.) You present the results of your study at a conference and one attendee is underwhelmed. She believes that the results of your experiment are still too noisy and the result should be large and confidently positive. You have some additional money in your research coffers. What action can you take to deal with the editor's concerns? Why? E. (10 pts.) A school booster (who has taken an intro stats class but didn't attend all of the lectures) looks at these results and says "These studies provide definitive proof that pizza parties have zero effect on test scores! Instead of spending money making sure that students have fun lunches in class, we should spend that money on a new hockey rink for the hockey team." In a sentence or two, give one reason why the results of these studies do not support this claim.