KaydedilenlerDerginizi ekleyin

Dergiler / Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi / 2022 / Volume: 13 Issue: 4

Analyzing the Interaction of Item Position Effect and Student Characteristics within Explanatory IRT Models

Sinem Demirkol(REPUBLIC OF TÜRKİYE MINISTRY OF NATIONAL EDUCATION)0000-0002-9526-6156
Hülya Kelecioğlu(HACETTEPE UNIVERSITY)0000-0002-0741-9934
Sayfa
—
Yayın Tarihi
2022-12-25
🏛️Bu makalenin bibliyografik verileri ve tam metin bağlantısı DergiPark üzerinden sağlanmıştır.
DergiPark Kaydı ↗
✨

Yapay Zeka Hızlı Bakış & Çıkarımlar

AI Insight
⚡ TL;DR(1 Cümlede Hızlı Çıkarım)

In addition, male students are more affected by item position than female students.

📌 3 Maddede Bu Makale

🎯
Amaç & Hipotez:The location of the same item in different positions among booklets leads to a biased estimation of item parameters.
📊
Temel Bulgular & Veri:In the study, the items in the PISA 2015 reading domain are used.
💡
Sonuç & Katkı:In addition, male students are more affected by item position than female students.
🔬Tür:Modelleme & Sayısal Simülasyon
📍Kapsam:2015 Verileri
⚙️Değişkenler:Item position effect · Explanatory IRT models · student characteristics reading domain

Abstract

The location of the same item in different positions among booklets leads to a biased estimation of item parameters. This undesirable effect on the probability of answering the items correctly is referred as the item position effect. The purpose of this study is to examine the items that are more sensitive to the item position effect and to investigate the student characteristics related to the item position effect. In the study, the items in the PISA 2015 reading domain are used. The study group consists of 2418 students who responded to the items in the reading domain from PISA 2015 Turkey Sample. Explanatory IRT models are used in the analysis of the research. According to the results, 42% of the items are affected by the item position. The most important characteristic related to item position is the SES level of students. In addition, male students are more affected by item position than female students.

🔖

Bu Makaleye Atıf Yap

Demirkol, S., Kelecioğlu, H. (2022). Analyzing the Interaction of Item Position Effect and Student Characteristics within Explanatory IRT Models. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, Volume: 13 Issue: 4. https://doi.org/10.21031/epod.1126368

📄 Tam Metin PDF Okuyucu

📄

PDF görüntüleyici hazırlanıyor...

Kaynakça (52)

  1. Albano, A. D. (2013). Multilevel modeling of item position effects. Journal of Educational Measurement, 50(4), 408–426. https://doi.org/10.1111/jedm.12026
  2. Alexandrowicz, R., & Matschinger, H. (2008). Estimating item location effects by means of a generalized logistic regression model. Psychology Science Quarterly, 50(1), 64-74. https://www.psychologie-aktuell.com/fileadmin/download/PschologyScience/1-2008/06_Alexandrowicz.pdf
  3. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME]. (2014). Standards for educational and psychological testing. American Psychological Association.
  4. Angoff, W. H. (1971). Scales, norms and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). American Council on Education.
  5. Atar, B. (2011). An application of descriptive and explanatory item response models to TIMSS 2007 Turkey mathematics data. Education and Science, 36(159), 256-259. http://egitimvebilim.ted.org.tr/index.php/EB/article/view/811
  6. Atar, B., & Cobanoglu Aktan, D. (2013). Person explanatory item response theory analysis: Latent regression two parameter logistic model. Education and Science, 38(168), 59–68. http://egitimvebilim.ted.org.tr/index.php/EB/article/view/942
  7. Bates, D., Maechler, M., Bokler, B., & Walker, S. (2014). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
  8. Berger, V. F., Munz, D. C., Smouse, A. D., & Angelino, H. (1969). The effects of item difficulty sequencing and anxiety reaction type on aptitude test performance. Journal of Psychology, 71(2), 253–258. https://doi.org/10.1080/00223980.1969.10543091
  9. Brennan, R. L. (1992). The context of context effects. Applied Measurement in Education, 5(3), 225–264. https://doi.org/10.1207/s15324818ame0503_4
  10. Briggs, D. C. (2008). Using explanatory item response models to analyze group differences in science achievement. Applied Measurement in Education, 21(2), 89-118. https://doi.org/10.1080/08957340801926086
  11. Bulut, O. (2021). eirm: Explanatory item response modeling for dichotomous and polytomous item responses, R package (version 0.3.0) [Computer software]. http://CRAN.R-project.org/package=eirm
  12. Bulut, O., Quo, O., & Gierl, M. (2017). A structural equation modeling approach for examining position effects in large‑scale assessments. Large Scale in Assessments in Education, 5(8), 1-20. https://doi.org/10.1186/s40536-017-0042-x
  13. Butler, J., & Adams, R. J. (2007). The impact of differential investment of student effort on the outcomes of international studies. Journal of Applied Measurement, 8(3), 279–304. https://pubmed.ncbi.nlm.nih.gov/17804895/
  14. Christiansen, A., & Janssen, R. (2020). Item position effects in listening but not in reading in the European Survey of Language Competences. Educational Assessment, Evaluation and Accountability, 33(3), 49–69. https://doi.org/10.1007/s11092-020-09335-7
  15. Cronbach, L. J. (1984). Essentials of psychological testing. Harper and Row.
  16. Davey, T., & Lee, Y. H. (2011). Potential impact of context effects on the scoring and equating of the multistage GRE revised general test (RR-11-26). ETS.
  17. De Boeck, P., & Wilson, M. (2004). Explanatory item response models: A generalized linear and nonlinear approach. Statistics for Social Science and Public Policy. Springer.
  18. Debeer, D., & Janssen, R. (2013). Modeling item-position effects within an IRT framework. Journal of Educational Measurement, 50(2), 164-185. https://doi.org/10.1111/jedm.12009
  19. DeMars, C. (2016). Madde tepki kuramı (Çev. Ed. H. Kelecioğlu). Nobel.
  20. Demirkol, S., & Kelecioğlu, H. (2022). Investigating the effect of item position on person and item parameters: PISA 2015 Turkey sample. Journal of Measurement and Evaluation in Education and Psychology, 13(1), 69-85. https://doi.org/10.21031/epod.958576
  21. Duncan, G. J., & Magnuson, K. A. (2005). Can family socioeconomic resources account for racial and ethnic test score gaps? The Future of Children, 15(1), 35-54. https://doi.org/10.1353/foc.2005.0004
  22. Eklöf, H. (2007). Test-taking motivation and mathematics performance in TIMSS 2003. International Journal of Testing, 7(3), 311–326. https://doi.org/10.1080/15305050701438074
  23. Embretson, S. E., & Reise, S. P. (2000). Item response theory (1st ed.). Psychology Press. https://doi.org/10.4324/9781410605269
  24. Finn, B. (2015). Measuring motivation in low-stakes assessments (RR-15–19). Educational Testing Service. https://doi.org/10.1002/ets2.12067
  25. Grandy, J. (1987). Characteristics of examinees who leave questions unanswered on the GRE general test under rights-only scoring (RR-87-38). Educational Testing Service. https://doi.org/10.1002/j.2330-8516.1987.tb00242.x
  26. Guertin, W. H. (1954). The effect of instructions and item order on the arithmetic subtest of the Wechsler- Bellevue. Journal of Genetic Psychology, 85(1), 79–83. https://doi.org/10.1080/00221325.1954.10532863
  27. Hahne, J. (2008). Analyzing position effects within reasoning items using the LLTM for structurally incomplete data. Psychology Science Quarterly, 50(3), 379-390. http://www.fyhe.com.au/past_papers/2006/Papers/Taylor.pdf
  28. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Theory and applications. Kluwer-Nijhoff.
  29. Hartig, J., & Buchholz, J. (2012). A multilevel item response model for item position effects and individual persistence. Psychological Test and Assessment Modeling, 54(4), 418-431. https://www.proquest.com/openview/075b5103d499407933e3c62cca521618/1?pq-origsite=gscholar&cbl=43472
  30. Hohensinn, C., Kubinger, K., Reif, M., Schleicher, E., & Khorramdel, L. (2011). Analysing item position effects due to test booklet design within large-scale assessment. Educational Research and Evaluation, 17(6), 497-509. https://doi.org/10.1080/13803611.2011.632668
  31. Kingston, N. M., & Dorans, N. J. (1982). The effect of the position of an item within a test on item responding behavior: An analysis based on item response theory (RR-79-12bP). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.1982.tb01308.x
  32. Kingston, N. M., & Dorans, N. J. (1984). Item location effects and their implications for IRT equating and adaptive testing. Applied Psychological Measurement, 8(2), 147–154. https://doi.org/10.1177/014662168400800202
  33. Leary, L. F., & Dorans, N. J. (1985). Implications for altering the context in which test items appear: A historical perspective on an immediate concern. Review of Educational Research, 55(3), 387–413. https://www.jstor.org/stable/1170392
  34. MacNicol, K. (1956). Effects of varying order of item difficulty in an unspeeded verbal test. Unpublished manuscript, Educational Testing Service.
  35. Meyers, J. L., Miller, G. E., & Way, W. D. (2009). Item position and item difficulty change in an IRT-based common item equating design. Applied Measurement in Education, 22(1), 38-60. https://doi.org/10.1080/08957340802558342
  36. McKeachie, W. J., Pollie, D., & Speisman, J. (1955). Relieving anxiety in classroom examinations. Journal of Abnormal and Social Psychology, 50(1), 93-98. https://doi.org/10.1037/h0046560
  37. Mollenkopf, W. G. (1950). An experimental study of the effects on item-analysis data of changing item placement and test time limit. Psychometrika, 15(3), 291–315. https://doi.org/10.1007/BF02289044
  38. Nagy, G., Nagengast, B., Frey, A., Becker, M., & Rose, N. (2018). A multilevel study of position effects in PISA achievement tests: Student- and school-level predictors in the German tracked school system. Assessment in Education: Principles, Policy & Practice, 26(4), 422–443. https://doi.org/10.1080/0969594X.2018.1449100
  39. Organisation for Economic Co-operation and Development [OECD]. (2017). PISA 2015 technical report. OECD. https://www.oecd.org/pisa/data/2015-technical-report/
  40. Pomplun, M., & Ritchie, T. (2004). An investigation of context effects for item randomization within testlets. Journal of Educational Computing Research, 30(3), 243–254. https://doi.org/10.2190/Y4FU-45V7-74UN-HW4T
  41. Qian, J. (2014). An investigation of position effects in large-scale writing assessments. Applied Psychological Measurement, 38(7), 518–534. https://doi.org/10.1177/0146621614534312
  42. Rose, N., Nagy, G., Nagengast, B., Frey, A., & Becker, M. (2019). Modeling multiple item context effects with generalized linear mixed models. Frontiers in Psychology, 10(248), 1-13. https://doi.org/10.3389/fpsyg.2019.00248
  43. Sirin, S. R. (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75(3), 417–453. https://doi.org/10.3102/00346543075003417
  44. Smouse, A. D., & Munz, D. C. (1968). The effects of anxiety and item difficulty sequence on achievement testing scores. Journal of Psychology, 68(2), 181-184. https://doi.org/10.1080/00223980.1968.10543421 Stanley, J. C. (1961). Studying status vs. manipulating variables. In R. O. Collier & S. M. Elam (Eds.), Research design and analysis. Phi Delta Kappan.
  45. Taylor, J. A. (2005). Poverty and student achievement. Multicultural Education, 12(4), 53-55. https://www.proquest.com/openview/3c9d66c77504e21370c0b718cf66e27f/1?pq-origsite=gscholar&cbl=33246
  46. Towle, N. J., & Merrill, P. F. (1975). Effects of anxiety type and item-difficulty sequencing on mathematics test performance. Journal of Educational Measurement, 12(4), 241–249. https://www.jstor.org/stable/1434151 Weirich, S., Hecht, M., Penk, C., Roppelt, A., & Böhme, K. (2017). Item position effects are moderated by changes in test-taking effort. Applied Psychological Measurement, 41(2), 115–129. https://doi.org/10.1177/0146621616676791
  47. Whitely, E., & Dawis, R. (1976). The influence of test context on item difficulty. Educational and Psychological Measurement, 36(2), 329–337. https://doi.org/10.1177/001316447603600211
  48. Wise, L. L., Chia, W. J., & Park, R. (1989, March 27-31). Item position effects for test of word knowledge and arithmetic reasoning. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
  49. Wise, S. L., & DeMars, C. E. (2005). Examinee motivation in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17. https://doi.org/10.1207/s15326977ea1001_1
  50. Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163-183. https://doi.org/10.1207/s15324818ame1802_2
  51. Wu, Q., Debeer, D. Buchholz, J., Hartig, J., & Janssen, R. (2019). Predictors of individual performance changes related to item positions in PISA assessments. Large Scale Assessment in Education, 7(5), 1-20. https://doi.org/10.1186/s40536-019-0073-6
  52. Yen, W. M. (1980). The extent, causes and importance of context effects on item parameters for two latent trait models. Journal of Educational Measurement, 17(4), 297–311. http://www.jstor.org/stable/1434871