Suppose that 2% of the modifications proposed to improve browsing on a Web site actually do improve customers’ experience. The other 98% have no effect. Now imagine testing 100 newly proposed modifications. It is quick and easy to measure the shopping behavior of hundreds of customers on a busy Web site, so each test will use a large sample that allows the test to detect real improvements. The tests use independent samples, and the level of significance is α = 0.05.

(a) Of the 100 tests, how many would you expect to reject the null hypothesis that claims the modification provides no improvement?

(b) If the tests that find significant improvements are carefully replicated, how many would you expect to again demonstrate a significant improvement?

(c) Do these results suggest an explanation for why scientific discoveries often cannot be replicated?