Does guessing matter? Differences between ability estimates from 2PL and 3PL IRT models in case of guessing

• Authors: Tomasz Żółtak , Grzegorz Golonka
• Languages of publication: EN

Abstracts

EN

Modern approaches to measuring cognitive ability and testing knowledge frequently use multiple-choice items. These can be simply and rapidly scored without problems associated with rater subjectivity. Nevertheless, multiple-choice tests are often criticized owing to their vulnerability to guessing. In this paper the impact of guessing was examined using simulation. Ability estimates were obtained from the two IRT models commonly used for binary-scored items: the two-parameter logistic model and the three-parameter logistic model. The latter approach explicitly models guessing, whilst the former does not. Rather counter-intuitively, little difference was identified for point estimates of ability from the 2PLM and 3PLM. Nevertheless, it should be noted that difficulty and discrimination parameters are severely downwardly biased if a 2PLM is used to calibrate data generated by processes involving guessing. Estimated standard errors for ability estimates also differ considerably between these models.

Keywords

EN

ability estimates; guessing; IRT; 2PLM; 3PLM

• Publisher: Instytut Badań Edukacyjnych
• Journal: Edukacja
• Year: 2015
• Issue: 3(134)
• Pages: 63–76

Dates

published

2015-09-30

Contributors

author

Tomasz Żółtak

• Educational Research Institute

author

Grzegorz Golonka

• Educational Research Institute

References

• Barton, M. A. and Lord, F. M. (1981). An upper asymptote for the three-parameter logistic item-response model. Princeton: Educational Testing Service. Retrieved from: http://files.eric.ed.gov/fulltext/ED207996.pdf.
• Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord and M. R. Novick (eds.), Statistical theories of mental test scores (chapters 17–20). Reading: Addison–Wesley.
• Brown, C., Templin, J. and Cohen, A. (2014). Comparing the two- and three-parameter logistic models via likelihood ratio tests a commonly misunderstood problem. Applied Psychological Measurement, 0146621614563326. doi: 10.1177/0146621614563326
• Espinosa, M. P. and Gardeazabal, J. (2010). Optimal correction for guessing in multiple-choice tests. Journal of Mathematical Psychology, 54(5), 415–425.
• Hambelton, R. K. (1982). Item response theory: the three-parameter logistic model. CSE Report No. 220. Los Angeles: University of California.
• Han, K. T. (2012). Fixing the c parameter in the three-parameter logistic model. Practical Assessment, Research & Evaluation, 17(1), 1–24.
• Lord, F. M. (1974). Estimates of latent ability and item parameters when there are omitted responses. Psychometrika, 39(2), 247–264.
• San Martín, E. S., Pino, G. del and De Boeck, P. D. (2006). IRT models for ability-based guessing. Applied Psychological Measurement, 30(3), 183–203.
• Shrock, S. A. and Coscarelli, W. C. (2008). Criterion-referenced test development: technical and legal guidelines for corporate training (3rd ed.). San Francisco: Wiley.
• Woods, C. M. (2008). Consequences of ignoring guessing when estimating the latent density in item response theory. Applied Psychological Measurement, 32(5), 371–384.
• Yen, W. M. (1981). Using simulated results to choose a latent trait model. Applied Psychologial Measurement, 5(2), 245–262.
• Zimmerman, D. W. and Williams, R. H. (1997). Properties of the Spearman correction for attenuation for normal and realistic non-normal distributions. Applied Psychological Measurement, 21(3), 253–270.

Notes

http://www.edukacja.ibe.edu.pl/images/numery/2015/3-4-zoltak-golonka-does-guessing-matter.pdf

Identifiers

ISSN

0239-6858