The Educational Testing Service identified differential item functioning (DIF) on an analogy question from the SAT exam. The question was an analogy that asked: “Strawberry: Red as (a) peach: ripe, (b) leather: brown, (c) grass: green, (d) orange: round, or (e) lemon: yellow.” The test question demonstrated DIF against Hispanic test takers in that they were more familiar with lemons that are green, not yellow. As a result, they were more likely to select option (c) instead of the correct answer, option (e). The procedure ETS used to determine DIF was a statistic developed by psychometricians at ETS called the Delta statistic, which compares how difficult different groups found the item. In addition to the DIF analysis, ETS also gathered a panel of experts who were the ones who identified that Hispanic examinees would be more likely to associate lemons with being green and not yellow. Thus, this item was ultimately discarded as the question was intended to assess one’s knowledge of analogies, but for at least one group, Hispanics, it was more an assessment of one’s knowledge of different fruit colors. 

Questions

1. What additional information might use of IRT procedures for DIF analysis provide that are not available with use of the Delta statistic? 

2. Psychometricians at ETS no doubt know a lot about IRT procedures. In fact, ETS psychometricians were among the earlier pioneers in IRT research and development. Why, then, do you think they opted not to use IRT procedures to assess DIF in this instance? 

3. As noted in the module overview, in order to perform IRT DIF procedures, item parameters need to be linked or equated across groups. Do you think such equating would also be required if other statistics, such as ETS’s Delta procedure, are used to establish DIF? 

4. If you plotted the item response functions (i.e., IRFs) for the different ethnic groups noted in this example, what do you think you are likely to see? 

5. Could this item be revised instead of simply being discarded? If so, how?