Investigating a new method for standardising essay marking using levelsbased mark schemes

Abstract

Standardisation is a procedure used by Awarding Organisations tomaximise marking reliability, by teaching examiners to consistently judgescripts using a mark scheme. However, research shows that people are betterat comparing two objects than judging each object individually.Consequently, Oxford, Cambridge and RSA (OCR, a UK awardingorganisation) proposed investigating a new procedure, involving rankingessays, where essay quality is judged in comparison to other essays. Thisstudy investigated the marking reliability yielded by traditionalstandardisation and ranking standardisation. The study entailed a markingexperiment followed by examiners completing a questionnaire. In the controlcondition live procedures were emulated as authentically as possible withinthe confines of a study. The experimental condition involved ranking thequality of essays from the best to the worst and then assigning marks. Aftereach standardisation procedure the examiners marked 50 essays from an ASHistory unit. All participants experienced both procedures, and markingreliability was measured. Additionally, the participants’ questionnaireresponses were analysed to gain an insight into examiners’ experience. It isconcluded that the Ranking Procedure is unsuitable for use in publicexaminations in its current form. The Traditional Procedure producedstatistically significantly more reliable marking, whilst the Ranking Procedureinvolved a complex decision-making process. However, the RankingProcedure produced slightly more reliable marking at the extremities of themark range, where previous research has shown that marking tends to be lessreliable.