Home-Grown Automated Essay Scoring in the Literature Classroom: A Solution for Managing the Crowd?

Abstract

Managing crowded classes in terms of classroom assessment is a difficult task due tothe amount of time which needs to be devoted to providing feedback to studentproducts. In this respect, the present study aimed to develop an automated essayscoring environment as a potential means to overcome this problem. Secondarily, thestudy aimed to test if automatically-given scores would correlate with the scores givenby a human rater. A quantitative research design employing a machine learningapproach was preferred to meet the aims of the study. The data set to be used formachine learning consisted of 160 scored literary analysis essays written in an EnglishLiterature course, each essay analyzing a theme in a given literary work. To train theautomated scoring model, LightSide software was used. First, textual features wereextracted and filtered. Then, Logistic Regression, SMO, SVO, Logistic Tree and NaïveBayes text classification algorithms were tested by using 10-Fold Cross-Validation toreach the most accurate model. To see if the scores given by the computer correlatedwith the scores given by the human rater, Spearman’s Rank Order CorrelationCoefficient was calculated. The results showed that none of the algorithms weresufficiently accurate in terms of the scores of the essays within the data set. It was alsoseen that the scores given by the computer were not significantly correlated with thescores given by the human rater. The findings implied that the size of the datacollected in an authentic classroom environment was too small for classificationalgorithms in terms of automated essay scoring for classroom assessment.