SavedAdd your journal

Journals / Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi / 2020 / Cilt: 11 - Sayı: 2

Rating Performance among Raters of Different Experience Through Multi-Facet Rasch Measurement (MFRM) Model

Muhamad Firdaus Bin MOHD NOH
Mohd Effendi Ewan Bin MOHD MATORE
Pages
147–162
Publication Date
2022-10-12
DOI
—
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

Independent t-test indicated that therewas a significant difference in ratings provided by the inexperienced and the experienced raters, t-value = -0.96,df = 28, p

📌 3 Key Takeaways

🎯
Objective & Hypothesis:One’s experience can greatly contribute to a diversified rating performance in educational scoring.
📊
Key Findings & Data:The aim of the study is to examine raters’ rating performance inassessing oral tests among lower secondary school students using Multi-facet Rasch Measurement (MFRM) modelindicated by raters’ severity.
💡
Conclusion & Impact:Independent t-test indicated that therewas a significant difference in ratings provided by the inexperienced and the experienced raters, t-value = -0.96,df = 28, p
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Domain Parameters

Abstract

One’s experience can greatly contribute to a diversified rating performance in educational scoring. Heterogeneousratings can negatively affect examinees’ results. The aim of the study is to examine raters’ rating performance inassessing oral tests among lower secondary school students using Multi-facet Rasch Measurement (MFRM) modelindicated by raters’ severity. Respondents are thirty English Language teachers clustered into two groups basedon their rating experience in high-stakes assessment. The respondents listened to ten examinees’ recorded answersof three oral test items and provided their ratings. Instruments include items, examinees’ answers, scoring rubric,and scoring sheet used to appraise examinees’ competence in three domains which are vocabulary, grammar, andcommunicative competence. MFRM analysis showed that raters exhibited diversity in their severity level with chisquareχ2=2.661. Raters’ severity measures ranged from 2.13 to -1.45 logits. Independent t-test indicated that therewas a significant difference in ratings provided by the inexperienced and the experienced raters, t-value = -0.96,df = 28, p

🔖

Cite This Article

NOH, M. F. B. M., MATORE, M. E. E. B. M. (2022). Rating Performance among Raters of Different Experience Through Multi-Facet Rasch Measurement (MFRM) Model. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, Cilt: 11 - Sayı: 2, 147–162. https://ajindex.com/articles/497184

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (8)

  1. Ahmadi Shirazi, M. (2019). For a greater good: Bias analysis in writing assessment. SAGE Open, 9(1), 1–14. https://doi.org/10.1177/2158244018822377
  2. Albano, A. D., & Rodrigues, M. (2018). Item development research and practice. Handbook of Accessible Instruction and Testing Practices: Issues, Innovations, and Applications, 181–198. https://doi.org/10.1007/978-3-319-71126-3_12
  3. Alp, P., Epner, A., & Pajupuu, H. (2018). The influence of rater empathy, age and experience on writing performance assessment. Linguistics Beyond And Within, 3(2017), 7–19. Retrieved from https://www.ceeol.com/search/article-detail?id=716601
  4. Anthony, L., & Miriam, S. (2019). Drill in English Skills Practice: CEFR-Alligned Curriculum. Selangor: Oxford Fajar Attali, Y. (2016). A comparison of newly-traind and experienced raters on a standardized writing assessment. Language Testing, 33(1), 99–115. https://doi.org/10.1177/0265532215582283
  5. Bond, T. G., & Fox, C. M. (2015). Applying the Rasch Model Fundamental Measurement in the Human Sciences. New Jersey: Lawrence Erlbaum Associates. Creswell, J. W., & Creswell, J. D. (2018). Research Design: Qualitative, Quantitative, And Mixed Methods Approaches (5th ed.). California: Sage Publications. Davis, L. (2016). The influence of training and experience on rater performance in scoring spoken language. Language Testing, 33(1), 117–135. https://doi.org/10.1177/0265532215582282
  6. Eckes, T. (2015). Introduction to Many-Facet Rasch Measurement: Analyzing and Evaluating Rater-Mediated Assessments (2nd ed.). Peter Lang. Eckes, T. (2019). Implications for rater-mediated language assessment. In Aryadoust, V., & Raquel, M. (Eds.), Quantitative Data Analysis for Language Assessment Volume I: Fundamental Techniques (pp. 153-175). London & New York: Routledge. Engelhard, G., & Wind, S. A. (2018). Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments. Routledge. New York & London: Routledge. https://doi.org/10.1017/CBO9781107415324.004
  7. Fisher, J. W. P. 2007. Rating scale instrument quality criteria. Rasch Measurement Transactions 21(1): 1095. Govindasamy, P., Salazar, M. D. C., Lerner, J., & Green, K. E. (2019). Assessing the reliability of the framework for equitable and effective teaching with the many-facet rasch model. Frontiers in Psychology, 10(June), 1–10. https://doi.org/10.3389/fpsyg.2019.01363
  8. Haladyna, T. M., & Rodrigues, M. C. (2013). Developing and Validating Test. New York: Routledge. Huang, L., Kubelec,