Automated Essay Scoring Effect on Test Equating Errors in Mixed-format Test

Temel Bulgular & Veri:The goal of this study was to score the constructed-response items in mixed-format tests automatically with different test/training data rates and to investigate the indirect effect of these scores on test equating compared with human raters.

💡

Sonuç & Katkı:It was concluded that automated scoring can be applied because it is convenient in terms of equating.

🔬Tür:Araştırma Makalesi

📍Kapsam:Genişletilmiş Akademik Örneklem

⚙️Değişkenler:Test equating · Automated scoring · Classical test theory

Özet

Scoring constructed-response items can be highly difficult, time-consuming, and costly in practice. Improvements in computer technology have enabled automated scoring of constructed-response items. However, the application of automated scoring without an investigation of test equating can lead to serious problems. The goal of this study was to score the constructed-response items in mixed-format tests automatically with different test/training data rates and to investigate the indirect effect of these scores on test equating compared with human raters. Bidirectional long-short term memory (BLSTM) was selected as the automated scoring method for the best performance. During the test equating process, methods based on classical test theory and item response theory were utilized. In most of the equating methods, errors of the equating resulting from automated scoring were close to the errors occurring in equating processes conducted by human raters. It was concluded that automated scoring can be applied because it is convenient in terms of equating.

Abstract

Scoring constructed-response items can be highly difficult, time-consuming, and costly in practice. Improvements in computer technology have enabled automated scoring of constructed-response items. However, the application of automated scoring without an investigation of test equating can lead to serious problems. The goal of this study was to score the constructed-response items in mixed-format tests automatically with different test/training data rates and to investigate the indirect effect of these scores on test equating compared with human raters. Bidirectional long-short term memory (BLSTM) was selected as the automated scoring method for the best performance. During the test equating process, methods based on classical test theory and item response theory were utilized. In most of the equating methods, errors of the equating resulting from automated scoring were close to the errors occurring in equating processes conducted by human raters. It was concluded that automated scoring can be applied because it is convenient in terms of equating.

Anahtar kelimeler

Test equating Automated scoring Classical test theory Item response theory Mixed-format tests automated scoring classical test theory item response theory mixed format tests

🔖

Bu Makaleye Atıf Yap

UYSAL, İ., DOĞAN, N. (2022). Automated Essay Scoring Effect on Test Equating Errors in Mixed-format Test. International Journal of Assessment Tools in Education, Cilt: 8 - Sayı: 2, 222–238. https://ajindex.com/articles/540053

📄 Tam Metin PDF Okuyucu

Yeni Sekmede Aç ↗

↗Yeni Sekme

📄

PDF görüntüleyici hazırlanıyor...

Kaynakça (10)

Adesiji, K. M., Agbonifo, O. C., Adesuyi, A. T., & Olabode, O. (2016). Development of an automated descriptive text-based scoring system. British Journal of Mathematics & Computer Science, 19(4), 1-14. https://doi.org/10.9734/BJMCS/2016/27558
Albano, A. D. (2016). equate: An R package for observed-score linking and equating. Journal of Statistical Software, 74(8), 1-36. https://doi.org/10.18637/jss.v074.i08
Almond, R. G. (2014). Using automated essay scores as an anchor when equating constructed-response writing tests. International Journal of Testing, 14(1), 73 91. https://doi.org/10.1080/15305058.2013.816309
Angoff, W. H. (1984). Scales, norms and equivalent scores. Educational Testing Service. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® v.2. Journal of Technology, Learning, and Assessment, 4(3), 1-30. http://www.jtla.org.
Barendse, M. T., Oort, F. J., & Timmerman, M. E. (2015). Using exploratory factor analysis to determine the dimensionality of discrete responses. Structural Equation Modeling: A Multidisciplinary Journal, 22(1), 87 101. https://doi.org/10.1080/10705511.2014.934850
Bartlett, M. S. (1950). Tests of significance in factor analysis. British Journal of Psychology, 3(2), 77-85. https://doi.org/10.1111/j.2044-8317.1950.tb00285.x
Chen, H., Xu, J., & He, B. (2014). Automated essay scoring by capturing relative writing quality. The Computer Journal, 57(9), 1318-1330. https://doi.org/10.1093/comjnl/bxt117
Cliff, N. (1993). Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin, 114(3), 494-509. https://doi.org/10.1037/0033-2909.114.3.494
Cliff, N. (1996). Ordinal methods for behavioral data analysis. Routledge. Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. The Journal of Applied Psychology, 78(1), 98 104. https://doi.org/10.1037/0021 9010.78.1.98 Creswell, J. W. (2012). Educational research: Planning, conducting and evaluating quantitative and qualitative research (4th ed.). Pearson. Deng, W., & Monfils, R. (2017). Long-term impact of valid case criterion on capturing population-level growth under Item Response Theory equating (Research Report 17-17). Educational Testing Service. https://doi.org/10.1002/ets2.12144
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 2-18. https://doi.org/10.1037/a0024338 Gonzalez,