# Question

Refer to the list of warnings on pages 527–528. Explain which ones should be of concern if the sample size(s) for a test are small.

From this discussion, you should realize that you can’t simply rely on news reports to determine what to conclude from the results of studies. In particular, you should heed the following warnings:

1. If the word significant is used to try to convince you that there is an important effect or relationship, determine if the word is being used in the usual sense or in the statistical sense only.

2. If a study is based on a very large sample size, relationships found to be statistically significant may not have much practical importance.

3. If you read that “no difference” or “no relationship” has been found in a study, try to determine the sample size used. Unless the sample size was large, remember that an important relationship may well exist in the population but that not enough data were collected to detect it. In other words, the test could have had very low power.

4. If possible, learn what confidence interval accompanies the hypothesis test, if any. Even then you can be misled into concluding that there is no effect when there really is, but at least you will have more information about the magnitude of the possible difference or relationship.

5. Try to determine whether the test was one- sided or two-sided. If a test is one-sided, as in Case Study 24.1, and details aren’t reported, you could be misled into thinking there would be no significant difference in a two-sided test, when in fact there was one in the direction opposite to that hypothesized.

6. Remember that the decision to do a one-sided test must be made before looking at the data, based on the research question. Using the same data to both generate and test the hypotheses is cheating. A one- sided test done that way will have a p- value smaller than it should, making it easier to reject the null hypothesis.

7. Beware of multiple testing and multiple comparisons. Sometimes researchers perform a multitude of tests, and the reports focus on those that achieved statistical significance. If all of the null hypotheses tested are true, then over the long run, about 1 in 20 tests should achieve statistical significance just by chance. Beware of reports in which it is evident that many tests were conducted, but in which results of only one or two are presented as “significant.”

From this discussion, you should realize that you can’t simply rely on news reports to determine what to conclude from the results of studies. In particular, you should heed the following warnings:

1. If the word significant is used to try to convince you that there is an important effect or relationship, determine if the word is being used in the usual sense or in the statistical sense only.

2. If a study is based on a very large sample size, relationships found to be statistically significant may not have much practical importance.

3. If you read that “no difference” or “no relationship” has been found in a study, try to determine the sample size used. Unless the sample size was large, remember that an important relationship may well exist in the population but that not enough data were collected to detect it. In other words, the test could have had very low power.

4. If possible, learn what confidence interval accompanies the hypothesis test, if any. Even then you can be misled into concluding that there is no effect when there really is, but at least you will have more information about the magnitude of the possible difference or relationship.

5. Try to determine whether the test was one- sided or two-sided. If a test is one-sided, as in Case Study 24.1, and details aren’t reported, you could be misled into thinking there would be no significant difference in a two-sided test, when in fact there was one in the direction opposite to that hypothesized.

6. Remember that the decision to do a one-sided test must be made before looking at the data, based on the research question. Using the same data to both generate and test the hypotheses is cheating. A one- sided test done that way will have a p- value smaller than it should, making it easier to reject the null hypothesis.

7. Beware of multiple testing and multiple comparisons. Sometimes researchers perform a multitude of tests, and the reports focus on those that achieved statistical significance. If all of the null hypotheses tested are true, then over the long run, about 1 in 20 tests should achieve statistical significance just by chance. Beware of reports in which it is evident that many tests were conducted, but in which results of only one or two are presented as “significant.”

## Answer to relevant Questions

Suppose that you were to read the following news story: “Researchers compared a new drug to a placebo for treating high blood pressure, and it seemed to work. But the re-searchers were concerned because they found that ...Refer to the previous exercise. a. Do you think the alternative hypothesis used by the researchers was one-sided or two-sided? Explain. b. Do you think the researchers would be justified in specifying a one-sided alternative ...In Example 13.2, we revisited data from Case Study 6.3, regarding testing to see if there was a relationship between gender and driving after drinking. We found that we could not rule out chance as an explanation for the ...An article in Science (Mann, 11 November 1994) describes two approaches used to try to determine how well programs to improve public schools have worked. The first approach was taken by an economist named Eric Hanushek. Here ...A standard protocol for the treatment of all participants that must be strictly followed. Exercises 12 to 21: Explain the main ethical issue of concern in each of the following. Discuss what, if anything should have been ...Post your question

0