SavedAdd your journal

Journals / International Journal of Assessment Tools in Education / 2021 / Cilt: 8 - Sayı: 1

The Effect of Item Pools of Different Strengths on the Test Results of Computerized-Adaptive Testing

Fatih Kezer
Pages
145–155
Publication Date
2022-10-18
DOI
—
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

At the end of the study, it was found that tests run by item pools with parameter b in the range that matches the ability level end up with fewer items and have a more accurate stimation.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:Item response theory provides various important advantages for exams carried out or to be carried out digitally.
📊
Key Findings & Data:To this end, eight different 2000-people item pools were designed in simulation which consist of 500 items with ability scores and varying difficulty levels.
💡
Conclusion & Impact:At the end of the study, it was found that tests run by item pools with parameter b in the range that matches the ability level end up with fewer items and have a more accurate stimation.
🔬Design:Lab & Field Experiment
📍Scope:2000 Verileri
⚙️Variables:Adaptive test · Item pool · Item difficulty

Abstract

Item response theory provides various important advantages for exams carried out or to be carried out digitally. For computerized adaptive tests to be able to make valid and reliable predictions supported by IRT, good quality item pools should be used. This study examines how adaptive test applications vary in item pools which consist of items with varying difficulty levels. Within the scope of the study, the impact of items was examined where the parameter b differentiates while the parameters a and c are kept in fixed range. To this end, eight different 2000-people item pools were designed in simulation which consist of 500 items with ability scores and varying difficulty levels. As a result of CAT simulations, RMSD, BIAS and test lengths were examined. At the end of the study, it was found that tests run by item pools with parameter b in the range that matches the ability level end up with fewer items and have a more accurate stimation. When parameter b takes value in a narrower range, estimation of ability for extreme ability values that are not consistent with parameter b required more items. It was difficult to make accurate estimations for individuals with high ability levels especially in test applications conducted with an item pool that consists of easy items, and for individuals with low ability levels in test applications conducted with an item pool consisting of difficult items.

Özet

Item response theory provides various important advantages for exams carried out or to be carried out digitally. For computerized adaptive tests to be able to make valid and reliable predictions supported by IRT, good quality item pools should be used. This study examines how adaptive test applications vary in item pools which consist of items with varying difficulty levels. Within the scope of the study, the impact of items was examined where the parameter b differentiates while the parameters a and c are kept in fixed range. To this end, eight different 2000-people item pools were designed in simulation which consist of 500 items with ability scores and varying difficulty levels. As a result of CAT simulations, RMSD, BIAS and test lengths were examined. At the end of the study, it was found that tests run by item pools with parameter b in the range that matches the ability level end up with fewer items and have a more accurate stimation. When parameter b takes value in a narrower range, estimation of ability for extreme ability values that are not consistent with parameter b required more items. It was difficult to make accurate estimations for individuals with high ability levels especially in test applications conducted with an item pool that consists of easy items, and for individuals with low ability levels in test applications conducted with an item pool consisting of difficult items.

🔖

Cite This Article

Kezer, F. (2022). The Effect of Item Pools of Different Strengths on the Test Results of Computerized-Adaptive Testing. International Journal of Assessment Tools in Education, Cilt: 8 - Sayı: 1, 145–155. https://ajindex.com/articles/540083

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (11)

  1. Adams, R. (2005). Reliabilty as a measurement design effect. Studies in Educational Evaluation, 31, 162–172.
  2. Baker, F. B., & Kim, S. H. (2004). Item response theory: Parameter estimation techniques. Marcel Bekker Inc. Boyd, A. M., Dodd, B. & Fitzpatrick, S. (2013). A comparison of exposure control procedures in cat systems based on different measurement models for testlets. Applied Measurement in Education, 26(2), 113-115.
  3. Brown, J. M., & Weiss, D. J. (1977). An adaptive testing strategy for achievement test batteries (Research Rep. No. 77-6).
  4. University of Minnesota, Department of Psychology, Psychometric Methods Program. Bulut, O., & Kan, A. (2012) Application of computerized adaptive testing to entrance examination for graduate studies in Turkey. Egitim Arastirmalari-Eurasian Journal of Educational Research, 49, 61–80.
  5. Chang, H. H. (2014). Psychometrics behind computerized adaptive testing. Psychometrika, 1-20.
  6. Cömert, M. (2008). Bireye uyarlanmış bilgisayar destekli ölçme ve değerlendirme yazılımı geliştirilmesi [Computer-aided assessment and evaluation analysis adapted to the individual] [Unpublished master’s thesis]. Bahçeşehir University. Crocker, L., & Algina, J. (1986). Introduction classical and modern test theory. Harcourt Brace Javonovich College Publishers.
  7. Dodd, B. G., Koch, W. R., & Ayala, R. J. (1993). Computerized adaptive testing using the partial credit model: effects of item pool characteristics and different stopping rules. Educational and Psychological Measurement, 53(1), 61-77.
  8. Eggen, T. J. H. M., & Verschoor, A. J. (2006). Optimal testing with easy or difficult items in computerized adaptive testing. Applied Psychologgical Measurement, 30(5), 379-393.
  9. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates. Georgiadou, E., Triantafillou, E., & Economides, A. A. (2006). Evaluation parameters for computer adaptive testing. British Journal of Educational Techonology, 37(2), 261–278.
  10. Glas, C. A., & Linden, W. (2003). Computerized adaptive testing with item cloning. Applied Psychological Measurement, 27(4), 247–261.
  11. Hambleton, R. K., & Swaminathan, H. (1989). Item response teory: Principles and applications. Kluwer Nijhoff Publishing. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage Publications Inc. Iseri, A. I. (2002). Assessment of students' mathematics achievement through computer adaptive testing procedures [Unpublished doctoral