The presence of statistical significance in marketing research analysis can be deceptive. Statistical significance does not necessarily mean that the difference has any practical significance. In addition to large sample sizes, there are many potential sources of error that can create problems for researchers when identifying statistically significant differences.

Generally, two kinds of error affect the validity of statistical measurements. Random error introduces error variance, but as it occurs randomly across respondents, it does not add statistical bias to the data. Systematic error is consistent across respondents, creating a bias within the data that may or may not be known. Typically, the causes for these kinds of error fall into two categories: sampling error, arising in the process of building a respondent pool; and measurement error, arising from the way the questionnaire is constructed.

Sampling Error 

Three major sources of sampling error include under coverage, nonresponse, and self-selection.

1. Under coverage—Under coverage occurs when a certain segment of the population is not adequately represented.

2. Nonresponse—Nonresponse error is a result of portions of the population being unwilling to participate in research projects.

3. Self-selection—Self-selection can result from respondents having control over survey completion. For instance, participants in an online survey panel might get bored and opt out before the survey is over.

Measurement Error 

The following six types of measurement error can result in random or systematic error.

1. Question interpretation—Respondents may interpret vague or ambiguously worded questions differently.

2. Respondent assumptions—Regardless of the way a question is worded, respondents still bring personal assumptions to the table, including any number of varying external factors influencing their understanding of the question.

3. Question order—Respondents might answer a question differently depending on where it falls in the survey, as their opinions might be influenced by their thoughts on surrounding questions.

4. Method variance—Researchers must be aware of potential errors introduced by the method used to deliver the survey.

5. Attribute wording—The way in which survey attributes are described may elicit different answers from respondents.

6. Omitting important questions—Systematic error most commonly results from inadequate coverage of critical variables within the question battery. Absent variables can significantly affect the results of data analysis.

Questions

1. Of the potential causes for error described above, which do you think would be easiest to identify? Hardest?
Explain your reasoning.

2. Can you think of any ways that could help researchers determine whether occurrences of statistical significance in their results have managerial significance?