The school progress measures were a step in the right direction, but in their current form they can only partially correct for intake ability, writes Tom Perry
As we have known for decades, and the data clearly shows, raw attainment scores such as schools’ GCSE results say more about schools’ intakes than their performance. The progress measures – despite numerous limitations – were a step in the right direction.
Progress 8 compares pupils’ KS4 scores only to those with the same KS2 scores. This enables like-for-like comparisons between pupils and helps level the playing field for schools.
But what happens when the KS2 scores we use to form KS4 expectations are unreliable? The conventional answer is that some schools will be “lucky”, taking more pupils who underperformed at an earlier stage and caught up. The hope is that these errors will cancel out when looking at whole cohorts.
My research, recently published in Research Papers in Education, shows that this is not the case.
If we have imperfect measures of prior attainment, we get an incomplete correction for ability
If we have imperfect measures of prior attainment, we get an incomplete correction for ability. We end up with some middle ground between the original KS4 scores – which are strongly correlated with intake prior attainment – and a perfect school value-added score for which intake ability doesn’t matter.
We have known that value-added measures are correlated with ability ever since Professor Stephen Gorard found a “surprising correlation” between them and attainment in the first ever English school value-added measure back in 2004. What was harder to understand was why.
Let me introduce you to some monsters which explain the problem.
First, the dreaded “shrinking expectations” (technically, regression attenuation bias).
Progress 8 expectations rely on the relationship between KS2 and KS4 scores. As error increases, this relationship breaks down as pupils of different ability levels get mixed up. KS2 measurement error produces a serious ability bias, complete with a ‘phantom’ grammar school effect
With normal levels of error, we end up somewhere in the middle – a twilight zone if you will – where the relationship moderately breaks down and the expectations shrink a little to the national average.
Next, we encounter the terrifying “phantom” effect, which is about as exciting as statistical terminology gets.
The handy thing about averages is that a lot of the pupil errors cancel out. A relationship that has broken somewhat in the pupil scores will only break down a little in the school averages.
This means that the school-level KS2-KS4 relationship is just what is left over from an incomplete correction of the pupil scores. Researchers have known about this for some time and have been wrestling with how to measure the effect of pupil clustering by ability – at its most extreme in grammar schools – without falling for phantoms (i.e. results caused by statistical error).
Ability bias is inevitable to some extent
What we didn’t know was the extent to which this affects the progress scores, until now.
Ability bias is inevitable to some extent: measures are not perfectly reliable and, crucially, these measures only correct for pupil scores. Should we be scared? I used reliability estimates based on Ofqual research, ran simulations using the National Pupil Database and found that KS2 measurement error produces a serious ability bias, complete with a “phantom grammar school effect” which is eerily similar to that seen in the actual data.
Like most stories, we now need a plucky protagonist and a plan.
The solution is simple, we adjust for school average prior attainment as well as prior attainment on a pupil-by-pupil basis.
This is in keeping with the clear principle behind the progress measures: schools should be judged by the progress their pupils make rather than the starting points of their pupils. We just need to add that schools should not be advantaged or disadvantaged by the average prior attainment of their intake any more than that of individual pupils.
Now all we need is that plucky protagonist. But beware! There are some politics lurking…
Tom Perry is a teaching fellow at the University of Birmingham, and research manager at CUREE
Your thoughts