SavedAdd your journal

Journals / IOJET / 2020 / Cilt: 7 - Sayı: 3

A RASCH ANALYSIS OF RATER BEHAVIOUR IN SPEAKING ASSESSMENT

Murat POLAT
Journal
IOJET
Pages
1126–1141
Publication Date
2022-10-18
DOI
—
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

A Rasch model in which MFRM (ManyFaceted Rasch Measurement) was utilised to explore the scoring behaviours of the expertgraders in terms of stringency and find out if their grading habits significantly affect languagelearners’ overall speaking performances.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The assessment of speaking skills in foreign language testing has always had some pros(testing learners’ speaking skills doubles the validity of any language test) and cons (many testrelevant/irrelevant variables interfere) since it is a multi-dimensional process.
📊
Key Findings & Data:In the meantime,exploring grader behaviours while scoring learners’ speaking skills is necessary not only forinter/intra-rater reliability estimations but also for identifying the potential stringent and lenientgraders in the rater-group to act accordingly to settle the best matches for graders when pairedrater-scorings or cross-marking-gradings are preferred for increasing the objectivity.
💡
Conclusion & Impact:A Rasch model in which MFRM (ManyFaceted Rasch Measurement) was utilised to explore the scoring behaviours of the expertgraders in terms of stringency and find out if their grading habits significantly affect languagelearners’ overall speaking performances.
🔬Design:Modeling & Simulation
📍Scope:2019 Verileri
⚙️Variables:Domain Parameters

Abstract

The assessment of speaking skills in foreign language testing has always had some pros(testing learners’ speaking skills doubles the validity of any language test) and cons (many testrelevant/irrelevant variables interfere) since it is a multi-dimensional process. In the meantime,exploring grader behaviours while scoring learners’ speaking skills is necessary not only forinter/intra-rater reliability estimations but also for identifying the potential stringent and lenientgraders in the rater-group to act accordingly to settle the best matches for graders when pairedrater-scorings or cross-marking-gradings are preferred for increasing the objectivity. In thisexploratory study, which was implemented in 2019, 6 expert speaking graders scored 24English language learners’ speaking interviews from their video recordings including anindividual and a pair discussion task for each student. A Rasch model in which MFRM (ManyFaceted Rasch Measurement) was utilised to explore the scoring behaviours of the expertgraders in terms of stringency and find out if their grading habits significantly affect languagelearners’ overall speaking performances. The results of the present research showed thatgraders had significant score differences among each other and some of them scored tooleniently or too stringently that might affect learners’ speaking grades significantly.

🔖

Cite This Article

POLAT, M. (2022). A RASCH ANALYSIS OF RATER BEHAVIOUR IN SPEAKING ASSESSMENT. IOJET, Cilt: 7 - Sayı: 3, 1126–1141. https://ajindex.com/articles/542221

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (14)

  1. Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge, UK: Cambridge University Press.
  2. Brown, A. (1995). The effect of rater variables in the development of an occupation- specific language performance test. Language Testing, 12, 1-15.
  3. Cohen, A. D. (1994). Assessing language ability in the classroom. (2nd ed.) Boston, MA: Heinle & Heinle. Congdon, P.J., & McQueen, J. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37(2), 163–178.
  4. Di Nisio, R. (2010). Measuring school learning through Rasch Analysis: the interpretation of results. Procedia - Social and Behavioural Sciences, Volume 9, 2010, Pages 373-377. https://doi.org/10.1016/j.sbspro.2010.12.167
  5. Ducasse, A., & Brown, A. (2009). Assessing paired orals: Raters’ orientation to interaction. Language Testing, 26(3), 423–443. https://doi.org/10.1177/0265532209104669
  6. Eckes, T. (2009). On common ground? How raters perceive scoring criteria in oral proficiency testing. In A. Brown & K. Hill (Eds.), Tasks and criteria in performance assessment: Proceedings of the 28th Language Testing Research Colloquium (pp. 43–73).
  7. Frankfurt, Germany: Lang. Engelhard, G. (2002). Monitoring raters in performance assessments. In G. Tindal & T. M. Haladyna (Eds.), Large-scale assessment programs for all students: Validity, technical adequacy, and implementation (pp. 261–287).
  8. Mahwah, NJ: Erlbaum. Fulcher, G. (2003). Testing Second Language Speaking. London: Pearson Education Limited. Hubbard, C., Gilbert, S., & Pidcock, J. (2006). Assessment processes in speaking tests: A pilot verbal protocol study. Research Notes, 24, 14–19.
  9. Kubinger, K. D. (2005). Psychological test calibration using the Rasch model: Some critical suggestions on traditional approaches. International Journal of Testing, 5, 377–394.
  10. Koizumi, R., Kaneko, E., Setoguchi, R., Innami, Y., & Naganuma, N. (2019). Examination of CEFR-J spoken interaction tasks using many-facet Rasch measurement and generalizability theory. Language Testing and Assessment 8(2), 1-33.
  11. Lane, S., & Stone, C.A. (2006). Performance Assessment. In R. L. Brennan (Ed.): Educational Measurement (pp 387-431).
  12. Wesport, CT: ACE/Praeger. Linacre, J.M. (2002). Optimizing Rating Scale Category Effectiveness. Journal of Applied Measurement, 3, 85-106.
  13. Linacre, J.M., & Wright, B.D. (2002). Construction of Measures from Many-Facet Data. Journal of Applied Measurement, 3, 484-509.
  14. Lumley, T. (2002). Assessment criteria i