Reflecting concerns that standardized testing was unfair, the psychology department at South East State University (SESU) decided six years ago to eliminate the requirement that prospective students submit Graduate Record Examination (GRE) scores for admission to its popular master’s program. Although hired as an assistant professor only a year ago, Dr. Lisa Span already found herself questioning whether this policy was a wise decision. Dr. Span’s main concern was the quality of the graduate students in her classes. Although a number of students were highly talented academically, others appeared unable to grasp theoretical concepts, and even seemed incapable of abstract thinking. Although Dr. Span had repeatedly questioned whether these problems were a consequence of her own teaching style, her concerns with the ability of the department’s graduate students were mirrored by similar concerns expressed by other faculty in the department. In an effort to understand how students were selected for the MA program, Dr. Span investigated the criteria used by the selection committee. She found that, in the absence of GRE scores, the selection committee relied heavily on three aspects of the student’s application file: grade point average (GPA) as an undergraduate, the student’s one-page statement of purpose, and three letters of recommendation. 

Sensing an opportunity, Dr. Span decided to determine the validity of the selection system. The department agreed to give her access to the application files of all 110 students admitted and enrolled in the MA program since the GRE was discarded from the department’s grad student selection process. She was also able to obtain the GPAs these students had amassed while grad students at SESU. Although undergraduate GPA and graduate GPA were easy variables to enter into a data file, the same could not be said for the statement of purpose and the three letters of recommendation. In the end, Dr. Span decided to code the statement of purpose on a four point scale. This scale reflected a number of characteristics, including writing ability, ability to convincingly communicate the desire to attend graduate school, undergraduate involvement in research, and relevant work experience. The three letters of recommendation were coded as a single score of 0, 1, 2, or 3, reflecting the number of letters submitted that had only positive things to say about the applicant. Thus, if an applicant received two letters that said only positive things, and one that included at least one negative comment, the applicant received a score of “2” for this predictor. While inputting the data, Dr. Span realized that the vast majority of applicants received a score of “3” on this predictor. 

With much anticipation, Dr. Span ran the multiple regression analysis examining the ability of the set of predictors to explain graduate GPA. The resulting multiple correlation coefficient was R = .31. Stunned, Dr. Span realized that, combined, these three predictors explained very little of the performance of graduate students. She was particularly surprised that two of the three selection tests—undergraduate GPA and letters of recommendation—received very low beta weights. Indeed, the zero-order correlation between undergraduate GPA and graduate GPA was an astonishingly low r = .13. By chance, Dr. Span discovered that a student’s application file contained a GRE reporting form. Opening several additional application files, she found that some other students had also reported their GRE scores to SESU. Although students were not required to do so, and although they were not considered in admission, some applicants had seemingly taken the GRE for other schools and had reported their scores to SESU as well. In all, Dr. Span was able to find GRE scores for 67 students. After quickly inputting these data into her file, she was pleased to see that the correlation between GRE scores and graduate GPA was r = .36. Convinced that the GRE should be reinstated as a requirement for graduate application to the department, Dr. Span prepared a report to her colleagues. 

Questions

1. What percentage of variance in graduate GPA is being explained by the current entrance criteria? 

2. Is there any reason to expect that letters of recommendation would have a low criterion-related validity, even before conducting the statistical analysis? Explain. 

3. Which of the current entrance criteria likely has the greatest criterion-related validity? How can you tell from the given information? 

4. Would it be appropriate to conclude that undergraduate GPA is unrelated to graduate GPA in the psychology master’s program at SESU? Why or why not? (Note: You may want to review the issues we discussed in Module 7 in answering this question.) 

5. Would it be appropriate to conclude that GRE scores will act as a better predictor of graduate GPA for future graduate students at SESU than the current entrance criteria? Explain.