Rating Performance among Raters of Different Experience Through Multi-Facet Rasch Measurement (MFRM) Model

Abstract

One’s experience can greatly contribute to a diversified rating performance in educational scoring. Heterogeneousratings can negatively affect examinees’ results. The aim of the study is to examine raters’ rating performance inassessing oral tests among lower secondary school students using Multi-facet Rasch Measurement (MFRM) modelindicated by raters’ severity. Respondents are thirty English Language teachers clustered into two groups basedon their rating experience in high-stakes assessment. The respondents listened to ten examinees’ recorded answersof three oral test items and provided their ratings. Instruments include items, examinees’ answers, scoring rubric,and scoring sheet used to appraise examinees’ competence in three domains which are vocabulary, grammar, andcommunicative competence. MFRM analysis showed that raters exhibited diversity in their severity level with chisquareχ2=2.661. Raters’ severity measures ranged from 2.13 to -1.45 logits. Independent t-test indicated that therewas a significant difference in ratings provided by the inexperienced and the experienced raters, t-value = -0.96,df = 28, p