Could you help me identify all of the non-sampling error issues? In the following scenario:

I listened to a sales pitch from a consultant regarding diversity training for my organization. I made a deal with the consultant that my company could receive the diversity training for a sample of management and professional employees of my company for free as long as we'd be willing to have the results used in promotional materials (if the training is successful).

The consultant mentioned that the diversity training is particularly effective if the individuals participating are volunteers, rather than being "forced" to attend the training program. So I agreed to start the evaluation by taking the first 40 volunteers for the program. (Strangely, it turned out that over half of the first 40 volunteers were either working in the Human Resources Department of the company even though HR actually makes up about 10% of our workforce).

This training program was conducted off-site for a period of a full work week. This was done at a very nice hotel, in order to increase the level of volunteering. When they were considering volunteering, the potential participants were told that, after the training was concluded on Friday afternoon, they would be permitted to have family members or significant others join them for the weekend for a little "rest and relaxation" before returning to work on Monday, with the company picking up the tab for the hotel. Most of the 40 volunteers involved took advantage of this family offer.

When they came back, I was really interested in how this diversity training might impact the volunteers, so once the trainees returned to work, I spent some time observing their interactions with others. I think maybe the trainees noticed that I happened to be around a bit more often than usual, but this was "unobtrusive" observation. After all, they should always be on their "good behavior" when they are at work.

The consultant and I puzzled about the appropriate outcome measure for this study. Did the training make a difference? What could we measure? We decided that the tone of interactions of the volunteers with others in the workplace was as good an outcome measure as any. But we wanted a comparison group, so we asked each training program participant to nominate a co-worker in the company who was as similar as possible to himself or herself with respect to demographic factors (race, age, gender, education level, hierarchical level, and type of position). We used these "non-volunteers" as our control group. We then tracked, for a full week, who the volunteers were choosing to interact with and the tones of those interactions. We sent forms to these coworkers of the volunteers and asked the coworkers to rate the general nature of their interactions with the volunteers, including the level of positivity and professionalism in the interaction. We also watched to see who the "control match," non-volunteers were interacting the most with at work, and we sent the same surveys to the coworkers of these non-volunteers.

The coworkers rated the tones of their interactions with each focal individual (either a training "volunteer" or comparison "non-volunteer" group) with respect to how positive and how professional that interaction was. The coworkers were asked, specifically, to comment on if they felt like their interactions with each subject "promoted an aura of synergistic inertia in professionalism."

After the data were collected, I separated the surveys into two piles and calculated scores for how positive and professional the interactions of each group were (one pile was the volunteers who were trained and the control pile were their 'matches' who were not). On making the statistical comparison, I didn't see much difference between the "professionalism" interaction scores of the trainee volunteers and the control non-volunteers who were not trained. But I was very gratified to discover that the trainee group was rated much more highly on the positivity of the tones of their interactions with other employees than the non-volunteer, control group members. In fact, the significance level (p-value) was 0.008, so I am sure that there is NO possibility of error in these findings. I am ready to call the program a smashing success and advise that we make this kind of training mandatory throughout the company. I think the consultant will probably be happy to hear of my conclusions regarding the training too. She'll be able to use this good news to promote her services to other organizations.