SavedAdd your journal

Journals / International Journal of Assessment Tools in Education / 2026 / Volume: 13 Issue: 3

The impact of rubric layout on L2 English writing scores and raters’ decision-making

Özgür Şahan(University of Southampton)0000-0002-6948-0423
Turgay Han(ORDU ÜNİVERSİTESİ)0000-0002-9196-0618
Pages
—
Publication Date
2026-09-02
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

These findings suggest that the placement of rubric categories can affect raters’ focus and scoring, emphasizing the importance of careful rubric design to ensure consistent assessment outcomes.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:This study examined the influence of rubric layout on ratings assigned to L2 English essays and raters’ decision-making process in the context of Turkish higher education.
📊
Key Findings & Data:Raters scored 24 essays in two rounds with the same analytic rubric presented in standard and reverse layouts, responded to a criteria importance survey at six time points throughout the study, and provided written score explanations to justify their ratings.
💡
Conclusion & Impact:These findings suggest that the placement of rubric categories can affect raters’ focus and scoring, emphasizing the importance of careful rubric design to ensure consistent assessment outcomes.
🔬Design:Survey & Empirical Study
📍Scope:Comprehensive Sample
⚙️Variables:Primacy effect · Analytic scoring · Rater decision-making

Abstract

This study examined the influence of rubric layout on ratings assigned to L2 English essays and raters’ decision-making process in the context of Turkish higher education. Raters scored 24 essays in two rounds with the same analytic rubric presented in standard and reverse layouts, responded to a criteria importance survey at six time points throughout the study, and provided written score explanations to justify their ratings. The two scoring rounds were conducted eight weeks apart, with the rubric categories reversed in the second round. A two-way repeated measures ANOVA revealed significant differences in scores between the two rounds. Multivariate generalizability analyses indicated that the rubric layout influenced variability in scores across raters. Content analysis of the written score explanations revealed a primacy effect, with raters placing greater emphasis on organization and grammar when these categories appeared earlier. However, raters consistently prioritized content and organization when justifying their scores. These findings suggest that the placement of rubric categories can affect raters’ focus and scoring, emphasizing the importance of careful rubric design to ensure consistent assessment outcomes.

Özet

This study examined the influence of rubric layout on ratings assigned to L2 English essays and raters’ decision-making process in the context of Turkish higher education. Raters scored 24 essays in two rounds with the same analytic rubric presented in standard and reverse layouts, responded to a criteria importance survey at six time points throughout the study, and provided written score explanations to justify their ratings. The two scoring rounds were conducted eight weeks apart, with the rubric categories reversed in the second round. A two-way repeated measures ANOVA revealed significant differences in scores between the two rounds. Multivariate generalizability analyses indicated that the rubric layout influenced variability in scores across raters. Content analysis of the written score explanations revealed a primacy effect, with raters placing greater emphasis on organization and grammar when these categories appeared earlier. However, raters consistently prioritized content and organization when justifying their scores. These findings suggest that the placement of rubric categories can affect raters’ focus and scoring, emphasizing the importance of careful rubric design to ensure consistent assessment outcomes.

🔖

Cite This Article

Şahan, Ö., Han, T. (2026). The impact of rubric layout on L2 English writing scores and raters’ decision-making. International Journal of Assessment Tools in Education, Volume: 13 Issue: 3. https://doi.org/10.21449/ijate.1807948

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (38)

  1. Ahmadi, A., & Tavazoei, M. (2025). Rater training through eye-tracking: A case-study of a novice rater. Teaching English Language, 19(2), 511-539. https://doi.org/10.22132/tel.2025.472698.1668
  2. Ballard, L. (2017). The effects of primacy on rater cognition: An eye-tracking study. [Unpublished doctoral dissertation]. Michigan State University. https://d.lib.msu.edu/etd/4632
  3. Bacha, N. (2001). Writing evaluation: What can analytic versus holistic essay scoring tell us? System, 29(3), 371-383. https://doi.org/10.1016/S0346-251X(01)00025-2
  4. Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing, 12(2), 86-107. https://doi.org/10.1016/j.asw.2007.07.001
  5. Barkaoui, K. (2010). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54 74. https://doi.org/10.1080/15434300903464418
  6. Brennan, R.L. (2001). Generalizability theory. Springer. https://doi.org/10.1007/978-1-4757-3456-0
  7. Crano, W.D. (1977). Primacy versus recency in retention of information and opinion change. The Journal of Social Psychology, 101(1), 87-96. https://doi.org/10.1080/00224545.1977.9923987
  8. Cresswell, J.W. (2011). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). PHI Learning Private.
  9. Cumming, A. (1990). Expertise in evaluating second language composition. Language Testing, 7(1), 31-51. https://doi.org/10.1177/02655322900070010
  10. Cumming, A., Kantor, R., & Powers, D. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86(1), 67-96. https://doi.org/10.1111/1540-4781.00137
  11. DeRemer, M.L. (1998). Writing assessment: Raters’ elaboration of the rating task. Assessing Writing, 5(1), 7-29. https://doi.org/10.1016/S1075-2935(99)80003-8
  12. Diab, R., & Balaa, L. (2012). Developing detailed rubrics for assessing critique writing: Impact on EFL university students’ performance and attitudes. TESOL Journal, 2(1), 52 72. https://doi.org/10.5054/tj.2011.244132
  13. Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155-185. https://doi.org/10.1177/0265532207086780
  14. Eckes, T. (2012). Operational rater types in writing assessment: Linking rater cognition to rater behavior. Language Assessment Quarterly, 9(3), 270-292. https://doi.org/10.1080/15434303.2011.649381
  15. Forgas, J.P. (2011). Can negative affect eliminate the power of first impressions? Affective influences on primacy and recency effects in impression formation. Journal of Experimental Social Psychology, 47(2), 425-429. http://doi.org/10.1016/j.jesp.2010.11.005
  16. Gay, L.R., Mills, G.E., & Airasian, P. (2009). Educational research: Competencies for analysis and applications (9th ed.). Pearson Education Inc.
  17. Goodwin, S. (2016). A Many-Facet Rasch analysis comparing essay rater behavior on an academic English reading/writing test used for two purposes. Assessing Writing, 30, 21 31. https://doi.org/10.1016/j.asw.2016.07.004
  18. Goulden, N.R. (1994). Relationship of analytic and holistic methods to raters’ scores for speeches. The Journal of Research and Development in Education, 27(2), 73-82.
  19. Heidari, N., Ghanbari, N., & Abbasi, A. (2022). Raters’ perceptions of rating scales criteria and its effect on the process and outcome of their rating. Language Testing in Asia, 12(1). https://doi.org/10.1186/s40468-022-00168-3
  20. Huang, J. (2011). Generalizability theory as evidence of concerns about fairness in large‐scale ESL writing assessments. TESOL Journal, 2(4), 423-443. https://doi.org/10.5054/tj.2011.269751
  21. Jeong, H. (2019). Writing scale effects on raters: An exploratory study. Language Testing in Asia, 9(20), 1-19. https://doi.org/10.1186/s40468-019-0097-4
  22. Kim, H. (2020). Effects of rating criteria order on the halo effect in L2 writing assessment: A many-facet Rasch measurement analysis. Language Testing in Asia, 10(1), Article 16. https://doi.org/10.1186/s40468-020-00115-0
  23. Lumley, T. (2005). Assessing second language writing: The rater’s perspective. Peter Lang.
  24. Panadero, E., Delgado, P., Zamorano, D., Pinedo, L., Fernández-Ortube, A., & Barrenetxea-Mínguez, L. (2025). Putting excellence first: How rubric performance level order and feedback type influence students’ reading patterns and task performance. Learning and Instruction, 99, Article 102168. https://doi.org/10.1016/j.learninstruc.2025.102168
  25. Perkins, K. (1983). On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly, 17(4), 651 671. https://doi.org/10.2307/3586618
  26. Plakans, L. (2013). Writing scale development and use within a language program. TESOL Journal, 4(1), 151-163. https://doi.org/10.1002/tesj.66
  27. Raimes, A. (1990). The TOEFL test of written English: Causes for concern. TESOL Quarterly, 24(3), 427-442. https://doi.org/10.2307/3587228
  28. Rezaei, A.R., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15(1), 18-39. https://doi.org/10.1016/j.asw.2010.01.003
  29. Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A premier. Sage.
  30. Song, B., & Caruso, I. (1996). Do English and ESL faculty differ in evaluating the essays of native English speaking, and ESL students? Journal of Second Language Writing, 5(2), 163 182. https://doi.org/10.1016/S1060-3743(96)90023-5
  31. Şahan, Ö. (2019). The impact of rater experience and essay quality on the variability of EFL writing scores. In S. Papageorgiou & K.M. Bailey (Eds.), Global perspectives on language assessment: Research, theory, and practice (pp. 32-46). Routledge. https://doi.org/10.4324/9780429437922
  32. Şahan, Ö., & Razı, S. (2020). Do experience and text quality matter for raters’ decision-making behaviors? Language Testing, 37(3), 311-332. https://doi.org/10.1177/0265532219900228
  33. Tulving, E. (2008). On the law of primacy. In M.A. Gluck, J.R. Anderson, & S.M. Kosslyn (Eds.), Memory and mind: A festschrift for Gordon H. Bower (pp. 31-48). Lawrence Erlbaum Associates Publishers.
  34. Vaughan, C. (1991). Holistic assessment: What goes on in the raters’ minds? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 11-126). Ablex.
  35. Weigle, S.C. (2002). Assessing writing. Cambridge University Press. https://doi.org/10.1017/CBO9780511732997
  36. Winke, P., & Lim, H. (2015). ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study. Assessing Writing, 25, 38-54. https://doi.org/10.1016/j.asw.2015.05.002
  37. Wolfe, E.W., Kao, C., & Ranney, M. (1998). Cognitive differences in proficient and nonproficient essay scorers. Written Communication, 15(4), 465-492. https://doi.org/10.1177/0741088398015004002
  38. Yan, T., Cernat, A., & Keusch, F. (2025). The impact of scale direction on data quality. Survey Research Methods, 19(2), 175-186. https://doi.org/10.18148/srm/2025.v19i2.8384