KaydedilenlerDerginizi ekleyin

Dergiler / IOJET / 2020 / Cilt: 7 - Sayı: 3

A RASCH ANALYSIS OF RATER BEHAVIOUR IN SPEAKING ASSESSMENT

Murat POLAT
Dergi
IOJET
Sayfa
1126–1141
Yayın Tarihi
2022-10-18
DOI
—
🏛️Bu makalenin bibliyografik verileri ve tam metin bağlantısı DergiPark üzerinden sağlanmıştır.
DergiPark Kaydı ↗
✨

Yapay Zeka Hızlı Bakış & Çıkarımlar

AI Insight
⚡ TL;DR(1 Cümlede Hızlı Çıkarım)

The results of the present research showed thatgraders had significant score differences among each other and some of them scored tooleniently or too stringently that might affect learners’ speaking grades significantly.

📌 3 Maddede Bu Makale

🎯
Amaç & Hipotez:The assessment of speaking skills in foreign language testing has always had some pros(testing learners’ speaking skills doubles the validity of any language test) and cons (many testrelevant/irrelevant variables interfere) since it is a multi-dimensional process.
📊
Temel Bulgular & Veri:In the meantime,exploring grader behaviours while scoring learners’ speaking skills is necessary not only forinter/intra-rater reliability estimations but also for identifying the potential stringent and lenientgraders in the rater-group to act accordingly to settle the best matches for graders when pairedrater-scorings or cross-marking-gradings are preferred for increasing the objectivity.
💡
Sonuç & Katkı:The results of the present research showed thatgraders had significant score differences among each other and some of them scored tooleniently or too stringently that might affect learners’ speaking grades significantly.
🔬Tür:Modelleme & Sayısal Simülasyon
📍Kapsam:2019 Verileri
⚙️Değişkenler:Alan Parametreleri

Abstract

The assessment of speaking skills in foreign language testing has always had some pros(testing learners’ speaking skills doubles the validity of any language test) and cons (many testrelevant/irrelevant variables interfere) since it is a multi-dimensional process. In the meantime,exploring grader behaviours while scoring learners’ speaking skills is necessary not only forinter/intra-rater reliability estimations but also for identifying the potential stringent and lenientgraders in the rater-group to act accordingly to settle the best matches for graders when pairedrater-scorings or cross-marking-gradings are preferred for increasing the objectivity. In thisexploratory study, which was implemented in 2019, 6 expert speaking graders scored 24English language learners’ speaking interviews from their video recordings including anindividual and a pair discussion task for each student. A Rasch model in which MFRM (ManyFaceted Rasch Measurement) was utilised to explore the scoring behaviours of the expertgraders in terms of stringency and find out if their grading habits significantly affect languagelearners’ overall speaking performances. The results of the present research showed thatgraders had significant score differences among each other and some of them scored tooleniently or too stringently that might affect learners’ speaking grades significantly.

🔖

Bu Makaleye Atıf Yap

POLAT, M. (2022). A RASCH ANALYSIS OF RATER BEHAVIOUR IN SPEAKING ASSESSMENT. IOJET, Cilt: 7 - Sayı: 3, 1126–1141. https://ajindex.com/articles/542221

📄 Tam Metin PDF Okuyucu

📄

PDF görüntüleyici hazırlanıyor...

Kaynakça (14)

  1. Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge, UK: Cambridge University Press.
  2. Brown, A. (1995). The effect of rater variables in the development of an occupation- specific language performance test. Language Testing, 12, 1-15.
  3. Cohen, A. D. (1994). Assessing language ability in the classroom. (2nd ed.) Boston, MA: Heinle & Heinle. Congdon, P.J., & McQueen, J. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37(2), 163–178.
  4. Di Nisio, R. (2010). Measuring school learning through Rasch Analysis: the interpretation of results. Procedia - Social and Behavioural Sciences, Volume 9, 2010, Pages 373-377. https://doi.org/10.1016/j.sbspro.2010.12.167
  5. Ducasse, A., & Brown, A. (2009). Assessing paired orals: Raters’ orientation to interaction. Language Testing, 26(3), 423–443. https://doi.org/10.1177/0265532209104669
  6. Eckes, T. (2009). On common ground? How raters perceive scoring criteria in oral proficiency testing. In A. Brown & K. Hill (Eds.), Tasks and criteria in performance assessment: Proceedings of the 28th Language Testing Research Colloquium (pp. 43–73).
  7. Frankfurt, Germany: Lang. Engelhard, G. (2002). Monitoring raters in performance assessments. In G. Tindal & T. M. Haladyna (Eds.), Large-scale assessment programs for all students: Validity, technical adequacy, and implementation (pp. 261–287).
  8. Mahwah, NJ: Erlbaum. Fulcher, G. (2003). Testing Second Language Speaking. London: Pearson Education Limited. Hubbard, C., Gilbert, S., & Pidcock, J. (2006). Assessment processes in speaking tests: A pilot verbal protocol study. Research Notes, 24, 14–19.
  9. Kubinger, K. D. (2005). Psychological test calibration using the Rasch model: Some critical suggestions on traditional approaches. International Journal of Testing, 5, 377–394.
  10. Koizumi, R., Kaneko, E., Setoguchi, R., Innami, Y., & Naganuma, N. (2019). Examination of CEFR-J spoken interaction tasks using many-facet Rasch measurement and generalizability theory. Language Testing and Assessment 8(2), 1-33.
  11. Lane, S., & Stone, C.A. (2006). Performance Assessment. In R. L. Brennan (Ed.): Educational Measurement (pp 387-431).
  12. Wesport, CT: ACE/Praeger. Linacre, J.M. (2002). Optimizing Rating Scale Category Effectiveness. Journal of Applied Measurement, 3, 85-106.
  13. Linacre, J.M., & Wright, B.D. (2002). Construction of Measures from Many-Facet Data. Journal of Applied Measurement, 3, 484-509.
  14. Lumley, T. (2002). Assessment criteria i