KaydedilenlerDerginizi ekleyin

Dergiler / International Journal of Assessment Tools in Education / 2019 / Cilt: 6 - Sayı: 3

On the Statistical and Heuristic Difficulty Estimates of a High Stakes Test in Iran

Ali Darabi BAZVAND
Alireza AHMADI
Shiela KHEİRZADEH
Sayfa
330–343
Yayın Tarihi
2022-10-18
DOI
—
🏛️Bu makalenin bibliyografik verileri ve tam metin bağlantısı DergiPark üzerinden sağlanmıştır.
DergiPark Kaydı ↗
✨

Yapay Zeka Hızlı Bakış & Çıkarımlar

AI Insight
⚡ TL;DR(1 Cümlede Hızlı Çıkarım)

Therefore, it can be concluded that students’ perceptions of item difficulty might be a better estimate of test difficulty and a combination of test takers’ perceptions and statistical difficulty might provide a better picture of item difficulty in assessment contexts.

📌 3 Maddede Bu Makale

🎯
Amaç & Hipotez:The findings of previous research into the compatibility of stakeholders’ perceptions with statistical estimations of item difficulty are not seemingly consistent.
📊
Temel Bulgular & Veri:Results indicate that, 1) the whole test along with its sub-tests is difficult which might lead to test invalidity; 2) the respondents’ ratings of the total test in terms of difficulty level are almost convergent with the difficulty values indicated by IRT and CTT, except for the two subtests where students underestimated the difficulty values, and 3) CTT difficulty estimates are convergent with IRT difficulty estimates.
💡
Sonuç & Katkı:Therefore, it can be concluded that students’ perceptions of item difficulty might be a better estimate of test difficulty and a combination of test takers’ perceptions and statistical difficulty might provide a better picture of item difficulty in assessment contexts.
🔬Tür:Araştırma Makalesi
📍Kapsam:Genişletilmiş Akademik Örneklem
⚙️Değişkenler:Classical true score theory · Heuristic difficulty · High stakes test

Özet

The findings of previous research into the compatibility of stakeholders’ perceptions with statistical estimations of item difficulty are not seemingly consistent. Furthermore, most research shows that teachers’ estimation of item difficulty is not reliable since they tend to overestimate the difficulty of easy items and underestimate the difficulty of difficult items. Therefore, the present study aims to analyze a high stakes test in terms of heuristic (test takers’ standpoint) and statistical difficulty (CTT and IRT) and investigate the extent to which the findings from the two perspectives converge. Results indicate that, 1) the whole test along with its sub-tests is difficult which might lead to test invalidity; 2) the respondents’ ratings of the total test in terms of difficulty level are almost convergent with the difficulty values indicated by IRT and CTT, except for the two subtests where students underestimated the difficulty values, and 3) CTT difficulty estimates are convergent with IRT difficulty estimates. Therefore, it can be concluded that students’ perceptions of item difficulty might be a better estimate of test difficulty and a combination of test takers’ perceptions and statistical difficulty might provide a better picture of item difficulty in assessment contexts.

Abstract

The findings of previous research into the compatibility of stakeholders’ perceptions with statistical estimations of item difficulty are not seemingly consistent. Furthermore, most research shows that teachers’ estimation of item difficulty is not reliable since they tend to overestimate the difficulty of easy items and underestimate the difficulty of difficult items. Therefore, the present study aims to analyze a high stakes test in terms of heuristic (test takers’ standpoint) and statistical difficulty (CTT and IRT) and investigate the extent to which the findings from the two perspectives converge. Results indicate that, 1) the whole test along with its sub-tests is difficult which might lead to test invalidity; 2) the respondents’ ratings of the total test in terms of difficulty level are almost convergent with the difficulty values indicated by IRT and CTT, except for the two subtests where students underestimated the difficulty values, and 3) CTT difficulty estimates are convergent with IRT difficulty estimates. Therefore, it can be concluded that students’ perceptions of item difficulty might be a better estimate of test difficulty and a combination of test takers’ perceptions and statistical difficulty might provide a better picture of item difficulty in assessment contexts.

🔖

Bu Makaleye Atıf Yap

BAZVAND, A. D., AHMADI, A., KHEİRZADEH, S. (2022). On the Statistical and Heuristic Difficulty Estimates of a High Stakes Test in Iran. International Journal of Assessment Tools in Education, Cilt: 6 - Sayı: 3, 330–343. https://ajindex.com/articles/540181

📄 Tam Metin PDF Okuyucu

📄

PDF görüntüleyici hazırlanıyor...

Kaynakça (11)

  1. Alderson, J. C. (1993). Judgments in language testing. In D. Douglas & C. Chapelle (eds.), A new decade of language testing (pp. 46–57). Arlington. VA: TESOL. Apostolou, E. (2010). Comparing perceived and actual task and text difficulty in the assessment of listening comprehension. In Lancaster University Postgraduate Conference in Linguistics & Language Teaching (pp. 26-47).
  2. Bachman, L. (2002). Some reflections on task-based language performance assessment. Language Testing, 19, 453–476.
  3. Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford university press. Baker, F. (2001).
  4. The basics of item response theory., College Park: ERIC Clearinghouse on Assessment and Evaluation, University of Maryland. Bejar, I. (1983). Subject matter experts’ assessment of item statistics. Applied Psychological Measurement, 7, 303–310
  5. Bereby-Meijer, Y., Meijer, J., & Flascher, O. M. (2002). Prospect theory analysis of guessing in multiple choice tests. Journal of Behavioral Decision Making, 15, 313–327.
  6. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord. & M. R. Novick (Eds.), statistical theories of mental test scores. Reading, MA: Addison-Wesley. Brindley, G., & Slatyer, H. (2002). Exploring task difficulty in ESL listening assessment. Language Testing, 19, 369-394.
  7. Brown, S., & Glasner, A. (1999). Assessment matters in higher education. Buckingham: SRHE and Open University Press.
  8. Chapelle, C. A., Enright, M. K., & Jamieson, J. (2010). Does an argument‐based approach to validity make a difference?. Educational Measurement: Issues and Practice, 29, 3-13.
  9. Cizek, G. J. (2012). Defining and distinguishing validity: Interpretations of score meaning and justifications of test use. Psychological Methods, 17, 31. Conejo, R., Guzmán, E., Perez-De-La-Cruz, J. L., & Barros, B. (2014). An empirical study on the quantitative notion of task difficulty. Expert Systems with Applications, 41, 594-606.
  10. Elder, C., Iwashita, N., & McNamara, T. (2002). Estimating the difficulty of oral proficiency tasks: what does the test-taker have to offer?. Language Testing, 19, 347-368.
  11. Embretson, S. E., & Reise, S. (2000). Item response theory for psychologists. Mahwah: Erlbaum. Farhady, H. (1998). A critical review of the English section of the BA and MA University Entrance Examination. In the Proceedings of the conference on MA tests in Iran, Ministry of Culture and Higher Education, Center for Educational Evaluation. Teh