An insightful comic (xkcd.com/882) illustrates the multiple testing problem (or multiple comparisons problem). Essentially, the scientists keep testing whether a different color jelly bean (a candy) causes acne (a skin condition), until they finally find p < 0.05 and reject the null hypothesis of "no effect" at a 5% level. This is essentially the same as if they keep computing a 95% CI for the effect of each color jelly bean, until they find a CI that does not contain zero. For a 95% CI, this happens roughly 5% of the time, or 1 in 20 datasets; knowing this, the comic shows them testing 20 different colors. The multiple testing problem is essentially that if you keep trying enough times, eventually you'll get a "false positive": the data show a non-zero effect, even though the true effect is zero. As an analogy: even though there's a full moon less than 5% of all nights, as long as you keep looking up at the sky every night, eventually you'll see a full moon.

Discussion Question 3.2 (jellybean solution?). Consider the jelly bean comic from xkcd.com/882, discussed above. a) Would it help to use a 99% CI instead of 95%? Explain why, why not, or how much it might help. b) Would it be even better to use a 100% CI? Explain why or why not.