You don’t know what you don’t know. This truism often lurks in my mind as I am designing a questionnaire. I feel it is my job to ask just the right questions so that my clients can obtain the information they need. But I can’t be sure if I have hit the mark if I do not know how respondents will understand and interpret my questions. Perhaps the best way to illustrate my concern about the unknown is to ask you to imagine what would happen if:

■ Respondents don’t see an answer that reflects their thinking in a list of response options to particular question.

■ A listing of attributes does not include all the factors that would be relevant to the client’s objectives.

■ Respondents think a term means something different than what you intended.

■ Respondents completely misunderstand the thrust of the question.

■ You just didn’t think of something and leave out what would be an important component to the study.

Well, the simple answer is that the results you deliver to clients would either be incomplete or based on suboptimal data that you thought was just fine. Researchers currently use a variety of methods to try to avoid these problems, including having research colleagues review the draft questionnaire; conducting qualitative research before moving on to the quantitative research; and conducting and monitoring pretest interviews. Each of those methods can help. But even when we use all three methods, our questions can still be off-target, incomplete, or worded in a way that is not effective because there is still something that we don’t know.

Research colleagues who help us with their expert review can often see gaps and help fill them in. But we and our colleagues often have very different life experiences from the subjects who will be completing the survey, so we cannot think like them or imagine every situation that might apply to them.

Figure 1:

Questions

1. Can cognitive interviewing help reduce error? If so, what type, random or systematic?

2. What are some of the benefits of cognitive interviewing?

3. One scholar recommends three rounds of cognitive interviews, each with 10 interviews. What are the advantages and disadvantages of this process? Isn’t one round enough?

Transcribed Image Text:

a. Trans fat content b. Total fat content c. Mono- and poly- saturated fat content d. Saturated fat content e. Omega-3 content f. Omega-6 content Not at all Important 1 1 1 1 1 1 Not too Important 2 222 2 NNN 2 2 Somewhat Important 333 333 Very Important 4 4 4 4 4 Not Sure 5 5 5 50 50 50