Grading exam essays using comparative judgment is “significantly” quicker and still as reliable as double-marking them, Ofqual has found.
On Tuesday the exams watchdog published findings from research it conducted over the past five years into improving the quality of examiner marking at GCSE, AS and A-levels.
It found paired comparison, also known as comparative judgment, in which examiners judge pairs of essays against one another until all are ranked and then graded, is as reliable and takes as long as traditional marking and another method called “rank ordering”.
Rank ordering is simply where examiners use example essays provided by the exam board as “anchors” for the grade a paper should get, and is not the same as comparative judgment.
As well as being as reliable as traditional marking and rank ordering, Ofqual also found paired comparison is “significantly quicker” than double marking, without losing any reliability.
“Paired comparison can match the predictive accuracy of double marking or two examiners rank ordering with significantly less time invested,” said the Marking reliability studies 2017 report.
The report follows pledges from education secretary Damian Hinds to help teachers strip away workload tasks that don’t “add value”, including triple marking, in the face of a persistent teacher-retention crisis.
Daisy Christodoulou, director of education at No More Marking, which promotes comparative judgment in schools, said the new research proved the method could transform workload for teachers.
“This shows that comparative judgment is amazingly reliable and efficient,” she said.
The method also allows for more “holistic” judgment about essays rather than trying to stick to a mark scheme which some teachers struggle with, she added.
However, the Ofqual report warned further research would be needed if paired comparison was used on a mass scale, to find out the point at which the method “starts to deteriorate and the whole model fit starts to collapse”.
Paula Goddard, a fellow of the Chartered Institute of Educational Assessors, warned paired comparison “may be faster” than double marking, but “it still hasn’t been used for a full-sized examination cohort” of 10,000 entries.
Examiners only looked at 60 extended essays for AS history papers in Ofqual’s research.
But if the paired comparison model could be made to work at scale, then Ofqual’s finding could hold “great” possibilities for the marking system in future, said Goddard.
Ofqual’s report concluded there is “scope to further investigate alternatives to marking”.
It added careful thought would need to be given to ensuring “transparency” as to how the rank-order placing was determined for each individual candidate.
EEF to study impact of marking on science teacher retention
A new study funded by the Education Endowment Foundation and the Wellcome Trust will find out if reducing time-consuming marking practices as well as other strategies can improve retention of science teachers.
Science teachers, senior leaders and governors at 140 secondary schools will take part in a four-year trial of Leadership Lite, a training programme developed by the Carmel Education Trust, which has three schools in the north east.
The National Foundation for Educational Research will evaluate the training, which will focus on four strands: marking and assessment, leadership of change, quality assurance and planning and classroom practice.
The focus on science coincides with the latest initial teacher training recruitment figures, in which the government hit only 47 per cent of its physics trainee target, down from 68 per cent the year before.
Another EEF study will see 80 early career chemistry teachers be paired with a mentor as a way of improving retention.
Sir Kevan Collins, chief executive of the EEF, said teacher workload is “undoubtedly one of the biggest factors” affecting teacher wellbeing and therefore retention.
“Teachers spend hours each week on time-consuming marking, but there’s little evidence to tell us whether these strategies have any impact on pupil attainment.”
Your thoughts