SavedAdd your journal

Journals / International Journal of Assessment Tools in Education / 2026 / Volume: 11 Issue: 2

Investigation of a multistage adaptive test based on test assembly methods

Ebru Doğruöz(ÇANKIRI KARATEKİN ÜNİVERSİTESİ)0000-0001-6572-274X
Hülya Kelecioğlu(HACETTEPE ÜNİVERSİTESİ, EĞİTİM FAKÜLTESİ)0000-0002-0741-9934
Pages
—
Publication Date
2024-06-20
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

According to the results obtained from the MST simulation based on the top-down test assembly method, mean RMSE values reduced when the module length increased and when the panel pattern changed from 1-2 to 1-2-2 and 1-2-3 for MST applied to small and large samples.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:In this research, multistage adaptive tests (MST) were compared according to sample size, panel pattern and module length for top-down and bottom-up test assembly methods.
📊
Key Findings & Data:Within the scope of the research, data from PISA 2015 were used and simulation studies were conducted according to the parameters estimated from these data.
💡
Conclusion & Impact:According to the results obtained from the MST simulation based on the top-down test assembly method, mean RMSE values reduced when the module length increased and when the panel pattern changed from 1-2 to 1-2-2 and 1-2-3 for MST applied to small and large samples.
🔬Design:Research Article
📍Scope:2015 Verileri
⚙️Variables:Adaptive multistage test · Test assembly · PISA

Abstract

In this research, multistage adaptive tests (MST) were compared according to sample size, panel pattern and module length for top-down and bottom-up test assembly methods. Within the scope of the research, data from PISA 2015 were used and simulation studies were conducted according to the parameters estimated from these data. Analysis results for each condition were compared in terms of mean RMSE and bias. According to the results obtained from the MST simulation based on the top-down test assembly method, mean RMSE values reduced when the module length increased and when the panel pattern changed from 1-2 to 1-2-2 and 1-2-3 for MST applied to small and large samples. Within the scope of the research, data from PISA 2015 were used and simulation studies were conducted using the parameters estimated from these data. Analysis results for each condition were compared in terms of mean RMSE and bias.

Özet

In this research, multistage adaptive tests (MST) were compared according to sample size, panel pattern and module length for top-down and bottom-up test assembly methods. Within the scope of the research, data from PISA 2015 were used and simulation studies were conducted according to the parameters estimated from these data. Analysis results for each condition were compared in terms of mean RMSE and bias. According to the results obtained from the MST simulation based on the top-down test assembly method, mean RMSE values reduced when the module length increased and when the panel pattern changed from 1-2 to 1-2-2 and 1-2-3 for MST applied to small and large samples. Within the scope of the research, data from PISA 2015 were used and simulation studies were conducted using the parameters estimated from these data. Analysis results for each condition were compared in terms of mean RMSE and bias.

🔖

Cite This Article

Doğruöz, E., Kelecioğlu, H. (2024). Investigation of a multistage adaptive test based on test assembly methods. International Journal of Assessment Tools in Education, Volume: 11 Issue: 2. https://doi.org/10.21449/ijate.1268614

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (58)

  1. American Institute of Certified Public Accountants. (2019, February 18). CPA exam structure. https://www.aicpa.org/becomeacpa/cpaexam/examinationcontent.html
  2. Belov, D.I. (2016). Review of modern methods for automated test assembly and item pool analysis. Law School Admission Council Research Report 16-01 March 2016, LSAC Research Report Series, 23 pages, https://www.lsac.org/docs/default-source/research-(lsac-resources)/rr-16-01.pdf
  3. Breithaupt, K., Ariel, A., & Veldkamp, B. (2005). Automated simultaneous assembly for multistage testing. International Journal of Testing, 5(3), 319 330. https://doi.org/10.1207/s15327574ijt05038
  4. Breithaupt, K., & Hare, D.R. (2007). Automated simultaneous assembly of multistage testlets for a high-stakes licensing examination. Educational and Psychological Measurement, 67(1), 5-20. https://doi.org/10.1177/0013164406288162
  5. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. CBS College Publishing.
  6. Dallas, A. (2014). The effects of routing and scoring within a computer adaptive multi-stage framework [Unpublished Doctoral Dissertation]. The University of North Carolina.
  7. Davis, L.L., & Dodd, B.G. (2003). Item exposure constraints for testlets in the verbal reasoning section of the MCAT. Applied Psychological Measurement, 27(5), 335 356. https://doi.org/10.1177/0146621603256804
  8. Educational Testing Service. (2018, February 18). Computer-delivered GRE general test content and structure. http://www.ets.org/gre/revised%5Cgeneral/about/content/computer/
  9. Hambleton, R.K., & Xing, D. (2006). Optimal and nonoptimal computer-based test designs for making pass-fail decisions. Applied Measurement in Education, 19(3), 221-239. https://doi.org/10.1207/s15324818ame1903_4
  10. Hembry, I.F. (2014). Operational characteristics of mixed format multistage tests using the 3PL testlet response theory model [Unpublished Doctoral Dissertation]. University of Texas at Austin.
  11. Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, 26(2), 44 52. https://doi.org/10.1111/j.1745 3992.2007.00093.x
  12. Hogan, J., Thornton, N., Diaz-Hoffmann, L., Mohadjer, L., Krenzke, T., Li, J. & Khorramdel, L. (2016, July 5,). US program for the international assessment of adult competencies (PIAAC) 2012/2014: Main study and national supplement technical report (NCES 2016-036REV). U.S. Department of Education. National Center for Education Statistics. https://nces.ed.gov/pubs2016/2016036 rev.pdf
  13. Jodoin, M.G., Zenisky, A., & Hambleton, R.K. (2006). Comparison of the psychometric properties of several computer-based test designs for credentialing exams with multiple purposes. Applied Measurement in Education, 19(3), 203 220. http://doi.org/10.1207/s15324818ame1903_3
  14. Khorramdel, L., Pokropek, A., & van Rijn, P. (2020). Special Topic: Establishing comparability and measurement invariance in large-scale assessments, part I. Psychological Test and Assessment Modeling, 62(1), 3 10. https://www.psychologie aktuell.com/fileadmin/Redaktion/Journale/ptam-2020-1/01_Khorramdel.pdf
  15. Kim, S., Moses, T., & You, H. (2015). A comparison of IRT proficiency estimation methods under adaptive multistage testing. Journal of Educational Measurement, 52(1), 70-79. https://doi.org/10.1111/jedm.12063
  16. Kim, J., Chung, H., Dodd, B.G., & Park, R. (2012). Panel design variations in the multistage test using the mixed-format tests. Educational and Psychological Measurement, 72(4), 574-588. https://doi.org/10.1177/0013164411428977
  17. Kirsch, I., & Lennon, M.L. (2017). PIAAC: A new design for a new era. Large-scale Assessments in Education, 5, 11. https://doi.org/10.1186/s40536-017-0046-6
  18. Lord, F.M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates, Inc.
  19. Luecht, R. (2000). Implementing the CAST framework to mass produce high quality computer adaptive and mastery tests. Paper presented at the Annual Meeting of the National Council on Measurement in Education (NCME), New Orleans, LA.
  20. Luecht, R.M. (2006). Designing tests for pass-fail decisions using item response theory. In S. Downing & T. Haladyna (Eds.), Handbook of test development, 575-596. Lawrence Erlbaum Associates.
  21. Luecht, R., Brumfield, T., & Breithaupt, K. (2006). A testlet assembly design for adaptive multistage tests. Applied Measurement in Education, 19(3), 189 202. https://doi.org/10.1207/s15324818ame1903_2
  22. Luecht, R.M., & Nungester, R.J. (1998). Some practical examples of computer-adaptive sequential testing. Journal of Educational Measurement, 35(3), 229 249. https://www.learntechlib.org/p/87698/.
  23. Luo, X. (2019). Automated test assembly with mixed-ınteger programming: The effects of modeling approaches and solvers. Journal of Educational Measurement, 57(4), 547-565. https://doi.org/10.1111/jedm.12262
  24. Luo, X., & Kim, D. (2018). A top-down approach to designing the computerized adaptive multistage test. Journal of Educational Measurement, 55(2), 243 263. https://doi.org/10.1111/jedm.12174
  25. Lynn Chen, L.Y. (2010). An investigation of the optimal test design for multi-stage test using the generalized partial credit model [Unpublished Doctoral Dissertation]. The University of Texas at Austin.
  26. OECD (2015). PISA 2015 technical report. http://www.oecd.org/pisa/sitedocument/PISA-2015-Technical-Report-Chapter-1-Programme-for-International-Student-Assessment-an-Overview.pdf
  27. OECD (2017). PISA 2015 technical report. http://www.oecd.org/pisa/sitedocument/PISA-2015-Technical-Report-Chapter-9-Scaling-PISA-Data.pdf
  28. Papadimitriou, C.H., & Steiglitz, K. (1982). Combinatorial optimization: Algorithms and complexity. Prentice-Hall.
  29. Park R. (2015). Investigating the impact of a mixed-format item pool on optimal test designs for multistage testing [Unpublished doctoral dissertation]. University of Texas, Austin.
  30. Patsula, L.N. (1999). A comparison of computerized-adaptive testing and multi-stage testing [Unpublished doctoral dissertation]. University of Massachusetts at Amherst.
  31. Pihlainen, K.A.I., Santtila, M., Häkkinen, K., & Kyröläinen, H. (2018). Associations of physical fitness and body composition characteristics with simulated military task performance. The Journal of Strength & Conditioning Research, 32(4), 1089-1098. https://doi.org/10.1519/jsc.0000000000001921
  32. Sari, H.İ. (2016). Examining content control in adaptive tests: Computerized adaptive testing vs. computerized multistage testing [Unpublished doctoral dissertation]. University of Florida.
  33. Sari, H.I., & Raborn, A. (2018). What information works best? A comparison of routing methods. Applied psychological measurement, 42(6), 499 515. https://doi.org/10.1177/0146621617752990
  34. Şahin Kürşad, M., Çokluk-bökeoglu, Ö. & Çıkrıkçı, N. (2022). The study of the effect of item parameter drift on ability estimation obtained from adaptive testing under different conditions. International Journal of Assessment Tools in Education, 9(3), 654-681. https://doi.org/10.21449/ijate.1070848
  35. Theunissen, T.J.J.M. (1985). Binary programming and test design. Psychometrika, 50(4), 411-420. https://link.springer.com/article/10.1007/BF02296260
  36. Tian, C. (2018). Comparison of four stopping rules in computerized adaptive testing and examination of their application to on-the-fly multistage testing [Unpublished master dissertation]. University of Illinois.
  37. van der Linden, W.J., & Glas, C.A.W. (2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13, 35 53. https://doi.org/10.1207/s15324818ame1301_2
  38. van der Linden, W.J. (2005). Linear models of optimal test design. Springer.
  39. van der Linden, W.J., & Boekkooi-Timminga, E. (1989). A maximin model for IRT-based test design with practical constraints. Psychometrika, 54(2), 237 247. https://link.springer.com/article/10.1007/BF02294518
  40. Veldkamp, B.P. (1999). Multiple-objective test assembly problems. Journal of Educational Measurement, 36, 253-66. http://www.jstor.org/stable/1435157
  41. Veldkamp, B.P., Matteucci, M., & de Jong, M.G. (2013). Uncertainties in the item parameter estimates and robust automated test assembly. Applied Psychological Measurement, 37, 123-139. https://doi.org/10.1177/0146621612469825
  42. Wang, K. (2017). Fair comparison of the performance of computerized adaptive testing and multistage adaptive testing [Unpublished doctoral dissertation]. Michigan State University.
  43. Wise, S.L., & Kingsbury, G.G. (2000). Practical issues in developing and maintaining a computerized adaptive testing program. Psicologica, 21, 135 155. https://www.uv.es/revispsi/articulos1y2.00/wise.pdf
  44. Xiao, J., & Bulut, O. (2022). Item selection with collaborative filtering in on-the-fly multistage adaptive testing. Applied Psychological Measurement, 46(8), 690 704. https://doi.org/10.1177/01466216221124089
  45. Xing, D., & Hambleton, R.K. (2004). Impact of test design, item quality, and item bank size on the psychometric properties of computer-based credentialing examinations. Educational and Psychological Measurement, 64, 5-21. https://doi.org/10.1177/0013164403258393
  46. Xu, L., Wang, S., Cai, Y., & Tu, D. (2021). The automated test assembly and routing rule for multistage adaptive testing with multidimensional item response theory. Journal of Educational Measurement, 58, 538-563. https://doi.org/10.1111/jedm.12305
  47. Yan, D., Lewis, C., & von Davier, A. (2014). Overview of computerized multistage tests. In D. Yan, A.A. von Davier, & C. Lewis (Eds.). Computerized Multistage Testing: Theory and Applications, 3-20. Chapman & Hall.
  48. Yan, D., von Davier, A.A., & Lewis, C. (Eds.). (2014). Computerized Multistage Testing: Theory and Applications (1st ed.). Chapman and Hall/CRC. https://doi.org/10.1201/b16858
  49. Yang, L. (2016). Enhancing item pool utilization when designing multistage computerized adaptive tests [Unpublished doctoral dissertation]. Michigan State University.
  50. Zenisky, A. (2004). Evaluating the effects of several multistage testing design variables on selected psychometric outcomes for certification and licensure assessment [Unpublished doctoral dissertation]. University of Massachusetts at Amherst.
  51. Zenisky, A., & Hambleton, R. (2014). Multistage test designs: Moving research results into practice. In Yan, D., Von Davier, A., & Lewis, C. (Eds.), Computerized Multistage Testing: Theory and Applications, 21-36. Chapman & Hall.
  52. Zenisky, A., Hambleton, R.K. & Luecht, R.M. (2010). Multistage testing: Issues, designs and research. In: der Linden, W.J. & Glas, C.A.W. (Eds.). Elements of Adaptive Testing. 355-372. Springer.
  53. Zenisky, A.L., Sireci, S.G., Martone, A., Baldwin, P., & Lam, W. (2009). Massachusetts adult proficiency tests technical manual supplement: 2008-2009. Center for Educational Assessment Research. http://www.umass.edu/remp/docs/MAPTTMSupp7-09 final.pdf
  54. Zheng, Y. (2014). New methods of online calibration for item bank replenishment [Unpublished Doctoral Dissertation]. University of Illinois at Urbana-Champaign.
  55. Zheng, Y., & Chang, H.-H. (2015). On-the-Fly assembled multistage adaptive testing. Applied Psychological Measurement, 39(2), 104 118. https://doi.org/10.1177/0146621614544519
  56. Zheng, Y., Nozawa, Y., Gao, X., & Chang, H. (2012). Multistage adaptive testing for a large-scale classification test: the designs, heuristic assembly, and comparison with other testing modes. Paper presented at the Annual Meeting of the National Council on Measurement in Education (NCME) (ACT Research Reports 2012-6). Vancouver, British Columbia, Canada.
  57. Zheng, Y., Nozawa, Y., Zhu, R., & Gao, X. (2016). Automated top-down heuristic assembly of a classification multistage test. Int. J. Quantitative Research in Education, 3(4), 242-265. https://doi.org/10.1504/IJQRE.2016.082387
  58. Zimowski, M.F., Muraki, E., Mislevy, R.J., & Bock, R.D. (1996). BILOG-MG: Multiple-group IRT analysis and test maintenance for binary items. [Computer software]. Scientific Software International.