SavedAdd your journal

Journals / Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi / 2021 / Cilt: 12 - Sayı: 2

Investigating the Effect of Rater Training on Differential Rater Function in Assessing Academic Writing Skills of Higher Education Students

Mehmet ŞATA
İsmail KARAKAYA
Pages
163–181
Publication Date
2021-06-30
DOI
—
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

The findings of the study revealed that the given rater training was effective on differential rater function, and suggestions based on these results were presented.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:This study aimed to examine the effect of rater training on the differential rater function (rater error) in the process of assessing the academic writing skills of higher education students.
📊
Key Findings & Data:The study group of the research consisted of 45 raters, of whom 22 came from experimental, and 23 came from control groups.
💡
Conclusion & Impact:The findings of the study revealed that the given rater training was effective on differential rater function, and suggestions based on these results were presented.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Academic writing · many facet Rasch model · rater training

Abstract

This study aimed to examine the effect of rater training on the differential rater function (rater error) in the process of assessing the academic writing skills of higher education students. The study was conducted with a pre-test and post-test control group quasi-experimental design. The study group of the research consisted of 45 raters, of whom 22 came from experimental, and 23 came from control groups. The raters were pre-service teachers who did not participate in any rater training before, and it was investigated that they had similar experiences in assessment. The data were collected using an analytical rubric developed by the researchers and an opinion-based writing task prepared by the International English Language Testing System (IELTS). Within the scope of the research, the compositions of 39 students that were written in a foreign language (English) were assessed. Many Facet Rasch Model was used for the analysis of the data, and this analysis was conducted under the Fully Crossed Design. The findings of the study revealed that the given rater training was effective on differential rater function, and suggestions based on these results were presented.

🔖

Cite This Article

ŞATA, M., KARAKAYA, İ. (2021). Investigating the Effect of Rater Training on Differential Rater Function in Assessing Academic Writing Skills of Higher Education Students. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, Cilt: 12 - Sayı: 2, 163–181. https://ajindex.com/articles/218020

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (12)

  1. Aryadoust, V. (2016). Understanding the growth of ESL paragraph writing skills and its relationships with linguistic features. Educational Psychology, 36(10), 1742-1770. https://doi.org/10.1080/01443410.2014.950946
  2. Attali, Y., Bridgeman, B., & Trapani, C. (2010). Performance of a generic approach in automated essay scoring. Journal of Technology, Learning, and Assessment, 10(3), 1-16. Retrieved from https://ejournals.bc.edu/ojs/index.php/jtla/article/view/1603
  3. Baştürk, M. (2012). İkinci dil öğrenme algılarının belirlenmesi: Balıkesir örneği. Balikesir University Journal of Social Sciences Institute, 15(28-1), 251-270. Retrieved from http://dspace.balikesir.edu.tr/xmlui/handle/20.500.12462/4594
  4. Bayat, N. (2014). Öğretmen adaylarının eleştirel düşünme düzeyleri ile akademik yazma başarıları arasındaki ilişki. Eğitim ve Bilim, 39(173), 155-168. Retrieved from http://eb.ted.org.tr/index.php/EB/article/view/2333
  5. Bernardin, H. J. & Pence, E. C. (1980). Effects of rater training: New response sets and decreasing accuracy. Journal ofApplied Psychology, 65, 60-66. https://doi.org/10.1037/0021-9010.65.1.60
  6. Bernardin, H. J., & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6(2), 205-212. Retrieved from https://journals.aom.org/doi/abs/10.5465/amr.1981.4287782
  7. Bijani, H. (2018). Investigating the validity of oral assessment rater training program: A mixed-methods study of raters’ perceptions and attitudes before and after training. Cogent Education, 5(1), 1-20. https://doi.org/10.1080/2331186X.2018.1460901
  8. Bitchener, J., Young, S., & Cameron, D. (2005). The effect of different types of corrective feedback on ESL students. Journal of Second Language Writing, 14, 191–205. https://doi.org/10.1016/j.jslw.2005.08.001
  9. Bond, T., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences. New York and London: Routledge. https://doi.org/10.4324/9781315814698
  10. Brennan, R.L., Gao, X., & Colton, D.A. (1995). Generalizability analyses of work key listening and writing tests. Educational and Psychological Measurement, 55(2), 157-176. https://doi.org/10.1177/0013164495055002001
  11. Brijmohan, A. (2016). A many-facet Rasch measurement analysis to explore rater effects and rater training in medical school admissions. (Doktora Tezi). Retrieved from http://www.proquest.com/
  12. Brookhart, S. M. (2013). How to create and use rubrics for formative assessment and grading. Alexandria, Virginia: ASCD. Brown, H. D. (2007). Teaching