SavedAdd your journal

Journals / Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi / 2026 / Volume: 14 Issue: 3

Ability Estimation with Polytomous Items in Computerized Multistage Tests

Hasibe Yahsi Sarı(HACETTEPE ÜNİVERSİTESİ)0000-0002-0451-6034
Hülya Kelecioğlu(HACETTEPE ÜNİVERSİTESİ)0000-0002-0741-9934
Pages
—
Publication Date
2023-09-30
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

It was found that as the number of categories and test length increase, the mean absolute bias and RMSE values decrease, while the correlation values increase.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The aim of the study is to examine how the ability estimations of individuals change under different conditions in tests consisting of polytomous items in an computerized multistage test environment.
📊
Key Findings & Data:In the study, 108 (3x3x6x2=108) conditions were examined consisting of three categories (3, 4 and 5), three test lengths (10, 20 and 30), six panel designs (1-2, 1-2-2, 1-3, 1-3-3, 1-4 and 1-4-4) and two routing methods (Maximum Fisher Information (MFI) and Random).
💡
Conclusion & Impact:It was found that as the number of categories and test length increase, the mean absolute bias and RMSE values decrease, while the correlation values increase.
🔬Design:Lab & Field Experiment
📍Scope:Comprehensive Sample
⚙️Variables:Computerized multistage tests · polytomous items · routing methods

Abstract

The aim of the study is to examine how the ability estimations of individuals change under different conditions in tests consisting of polytomous items in an computerized multistage test environment. The research is a simulation study. In the study, 108 (3x3x6x2=108) conditions were examined consisting of three categories (3, 4 and 5), three test lengths (10, 20 and 30), six panel designs (1-2, 1-2-2, 1-3, 1-3-3, 1-4 and 1-4-4) and two routing methods (Maximum Fisher Information (MFI) and Random). Simulations and analyses were carried out in the mstR package in R program, with a pool of 200 items, 1000 people and 100 replications (e.g., iterations). As the outcomes of the research, mean absolute bias, RMSE and correlation values were calculated. It was found that as the number of categories and test length increase, the mean absolute bias and RMSE values decrease, while the correlation values increase. In terms of routing methods, although MFI and random methods have similar tendencies, MFI gives better results. There is a similarity between the panel designs in terms of results.

🔖

Cite This Article

Sarı, H. Y., Kelecioğlu, H. (2023). Ability Estimation with Polytomous Items in Computerized Multistage Tests. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, Volume: 14 Issue: 3. https://doi.org/10.21031/epod.1056079

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (26)

  1. Chen, L-Y. (2010). An investigation of the optimal test design for multi-stage test using the generalized partial credit model. [Doctoral dissertation, The University of Texas]. UT Electronic Theses and Dissertations. Retrieved from https://repositories.lib.utexas.edu/handle/2152/ETD-UT-2010-12-344
  2. Donoghue, J. R. (1994). An empirical examination of the IRT information of polytomously scored reading items under the generalized partial credit model. Journal of Educational Measurement, 31(4), 295-311. https://doi.org/10.1111/j.1745-3984.1994.tb00448.x
  3. Dodd, B. G., De Ayala, R. J., & Koch, W. R. (1995). Computerized Adaptive Testing with Polytomous Items. Applied Psychological Measurement, 19(1), 5-22. https://doi.org/10.1177/014662169501900103
  4. Embretson, S. E., & Reise, S. P. (2013). Item response theory. Psychology Press.
  5. Han, K. T. (2007). WinGen: Windows software that generates item response theory parameters and item responses. Applied Psychological Measurement, 31(5), 457-459. DOI: 10.1177/0146621607299271
  6. Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, 26(2), 44-52. https://doi.org/10.1111/j.1745-3992.2007.00093.x
  7. ILOG. (2006). ILOG CPLEX 10.0 [User’s manual]. Paris, France: ILOG S.A. Retrieved from https://www.lix.polytechnique.fr/~liberti/teaching/xct/cplex/usrcplex.pdf
  8. Kim, J., Chung, H., & Dodd, B. G. (2010, May). Comparing routing methods in the multistage test based on the partial credit model [Conference presentation]. In AERA, Denver, CO.
  9. Kim, J., Chung, H., Park, R., & Dodd, B. G. (2013). A comparison of panel designs with routing methods in the multistage test with the partial credit model. Behavior Research Methods, 45, 1087–1098. https://doi.org/10.3758/s13428-013-0316-3
  10. Luecht, R. M., & Nungester, R. J. (1998). Some practical examples of computer‐adaptive sequential testing. Journal of Educational Measurement, 35(3), 229-249. https://doi.org/10.1111/j.1745-3984.1998.tb00537.x
  11. Luecht, R. M. (2000, April). Implementing the computer-adaptive sequential testing (CAST) framework to mass produce high quality computer-adaptive and mastery tests. [Conference presentation]. In NCME, New Orleans, LA. Retrieved from https://eric.ed.gov/?id=ED442823
  12. Macken-Ruiz, C. L. (2008). A comparison of multi-stage and computerized adaptive tests based on the generalized partial credit model (Publication No. 3328282) [Doctoral dissertation, The University of Texas]. ProQuest Dissertations Publishing. Retrieved from https://www.proquest.com/docview/304482829?pq-origsite=gscholar&fromopenview=true
  13. Magis, D., Yan, D., von Davier, A., & Magis, M. D. (2018). Package ‘mstR’. Retrieved from https://cran.r-project.org/web/packages/mstR/mstR.pdf
  14. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149-174. https://doi.org/10.1007/BF02296272
  15. Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992(1), i–30. https://doi.org/10.1002/j.2333-8504.1992.tb01436.x
  16. Öztürk, N. B. (2019). How the Length and Characteristics of Routing Module Affect Ability Estimation in ca-MST?. Universal Journal of Educational Research, 7(1), 164-170. doi: 10.13189/ujer.2019.070121
  17. R Core Team. (2018). R: A language and environment for statistical computing: R foundation for statistical computing.
  18. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34 (17). Retrieved from https://psycnet.apa.org/record/1972-04809-001
  19. Sari, H. I., & Raborn, A. (2018). What Information Works Best?: A Comparison of Routing Methods. Applied psychological measurement, 42(6), 499-515. https://doi.org/10.1177/0146621617752990
  20. Sari, H.I., Yahsi Sari, H., & Huggins Manley, A.C. (2016). Computer adaptive multistage testing: Practical issues, challenges and principles. Journal of Measurement and Evaluation in Education and Psychology, 7(2), 388-406. https://doi.org/10.21031/epod.280183
  21. Weiss, D. J. (1982). Improving Measurement Quality and Efficiency with Adaptive Testing. Applied Psychological Measurement, 6(4), 473–492. https://doi.org/10.1177/014662168200600408
  22. Weiss, D. J. (1983). Latent trait theory and adaptive testing. In Weiss D. J. (Ed.), New horizons in testing (pp. 5-7). Academic Press.
  23. Zenisky, A. L. (2004). Evaluating the effects of several multi-stage testing design variables on selected psychometric outcomes for certification and licensure assessment (Publication No. 5710) [Doctoral dissertation, University of Massachusetts Amherst]. UMass Amherst Libraries. https://scholarworks.umass.edu/dissertations_1/5710
  24. Zenisky A., Hambleton R.K.,& Luecht R.M. (2009) Multistage Testing: Issues, Designs, and Research. In: van der Linden W., Glas C. (eds) Elements of Adaptive Testing. Springer.
  25. Zurovac, J., Cook, T. D., Deke, J., Finucane, M. M., Chaplin, D., Coopersmith, J. S.,. & Forrow, L. V. (2021). Absolute and Relative Bias in Eight Common Observational Study Designs: Evidence from a Meta-analysis. https://arxiv.org/ftp/arxiv/papers/2111/2111.06941.pdf
  26. Zurovac, J., Cook, T. D., Deke, J., Finucane, M. M., Chaplin, D., Coopersmith, J. S.,. & Forrow, L. V. (2021). Absolute and Relative Bias in Eight Common Observational Study Designs: Evidence from a Meta-analysis. https://arxiv.org/ftp/arxiv/papers/2111/2111.06941.pdf