Progress scores for grammar schools are likely to be overstated due to unreliable SATs scores, amid fears they could be used to make a case for more selection.
Analysis of Progress 8 results by Education Datalab, and shared exclusively with Schools Week, reveal that pupils at grammar schools last year made more progress in their GCSEs compared with similar pupils in non-selective schools – scoring around a third of a grade higher across eight GCSEs.
The average progress score for grammar schools was +0.34, which is well above the government’s floor target of -0.5 and ahead of the average for secondary moderns, which Education Datalab calculated was -0.11.
The government is under pressure to produce evidence supporting its proposals for allowing more grammar schools to open, and could seize on any positive findings in January’s school league tables – which will include Progress 8 scores for the first time.
But Dr Rebecca Allen (pictured), director of Education Datalab, said Progress 8 scores in grammar schools were overstated.
Allen compared pupils’ key stage 2 scores with whether or not they passed the 11-plus, finding that a SATs score of 5.2 was the score above which most pupils attended a grammar.
But the research also found large numbers of pupils with lower SATs scores also passing the test and entering grammar school. In some cases, children got into a grammar school with a SATs score as low as 4.5.
A deeper dive into the statistics found that these pupils with lower key stage 2 scores later achieved “incredibly high” progress scores in their GCSEs. Allen explained this was because their key stage 2 score “almost certainly understates their academic ability at age 11. Some of the progress made during primary school is then unfairly attributed to their grammar school.”
She also found the opposite was true for pupils with high key stage 2 results who failed the 11-plus, suggesting their SATs scores were over-estimated.
“Some of the progress at their secondary modern is therefore unfairly attributed to their primary school.”
She said the combination caused grammar school performance to be overstated, while secondary moderns were understated.
Rob Coe, director of Durham University’s centre for evaluation and monitoring, which produces 11-plus tests, highlighted the unreliability of key stage 2 scores.
He said the findings showed that Progress 8 results as part of the grammar debate must be interpreted “with caution”.
“There is a bias – grammar schools have the decks stacked in their favour.”
He also highlighted an Ofqual study in 2013 that found that the accuracy of key stage 2 test scores was about 85 per cent for English, 87 per cent for science and 90 per cent for maths.
Ofqual said this was a “substantial improvement” compared with the late 1990s when the tests were introduced.
But Allen said that SATs scores could be impacted by additional factors – such as pupils having a bad day, or striking it lucky on questions – which she called “noise”.
Private tuition might also explain some of the difference between SAT and 11-plus scores.
A report by the Sutton Trust published last week found the private tuition market was worth £2 billion a year, with nearly one in five pupils who received tutoring doing so for the 11-plus.
One head told Schools Week that if grammars were found to be genuinely adding value, it would be “horrific” for social justice.
“That would mean pupils who were level 4 in ability, but trained to do the 11-plus because their parents could afford it, then go on to grammar schools and make more progress. That’s so unfair.”
The government has now proposed that grammar schools will take a proportion of pupils from lower-income households to “ensure that selective education is not reserved for those who . . . pay for tuition to pass the test”.
Your thoughts