Teachers at high-performing schools mark the work of their lowest-performing pupils more harshly in their SATs assessments, suggests a new study of primary marking.
Meanwhile, the writing of high-flying 11-year-olds who attend the lowest-performing primaries are marked “too conservatively”, which could mean that bright pupils get lower marks than they deserve.
The findings are part of a new pilot into comparative judgment marking – a system in which teachers compare two pieces of work and decide which is better. In conventional frameworks, each piece is judged on its own.
Under comparative judgment, once teachers have compared each essay enough times, a computer puts them in rank order and standardises the grades.
More than 1,600 teachers from 199 schools took part in the study led by No More Marking, a company that produces comparative judgment software. The writing portfolios of more than 8,500 year 6 pupils were judged.
The study used statistical tests to judge the reliability of the method, and found it produced “objective and highly reliable results”.
However, the report also revealed interesting insights into the way schools differed in their interpretation of key stage 2 writing standards.
After teachers ranked the 8,500 pieces of marked work, researchers created a national scale of writing split into the three standards: “greater depth”, “expected standard”, and “working towards”.
The scores were then compared against those given by schools in the previous year.
Teachers from highly performing schools were found to be “relatively harsh” in their judgment of their low-attaining pupils.
By contrast, poorly performing schools marked their high-achievers more “conservatively”. This was an unexpected finding for the researchers, who had theorised that such children might be over-marked.
Dr Chris Wheadon, No More Marking’s founder, said the finding backed up concerns that “we are very bad at absolute judgments”.
“If teachers have never seen what [a piece of writing at] ‘greater depth’ looks like, they don’t know they are looking at it.
“Teachers see so little writing from schools elsewhere in the country, they don’t know the child’s work they are looking at is that good.”
When asked to explain the harsh marking of high-attaining children in lower-performing schools, Wheadon said he believed there was “a reluctance to say that work is really good and fear of the moderator”.
While teachers marked most of their own pupil’s work, every fifth judgment included work from other schools. The report says that teachers who took part benefited from a “wide exposure to a range of writing and writing approaches across the country”.
The study also encountered some anomalies. For instance, some pupils received much higher or lower marks in the comparative judgment process than their results based on the interim frameworks.
The report said this was usually because pupils had written very good pieces that missed out key parts of the framework, or “very formulaic” pieces that nevertheless ticked many of the framework’s boxes.
However, researchers admitted the project added to the workload of year 6 teachers and that “long-term success of a comparative judgment approach depends on it replacing existing approaches rather than being an addition to the process.
“Overall, however, this project shows that teacher assessment of writing can be used to produce objective, highly reliable results … [and] cannot be undermined by schools seeking to improve their writing results for accountability purposes.”
Your thoughts