School accountability will always pose problems. It’s impossible to have a system that can’t be manipulated, but it is possible to design one that limits cynical practices.
Sadly, our headline measures incentivise precisely such practices. And coupled with an inspection system fraught with issues around subjectivity and reliability, you begin to see why so many who join the profession with excitement fall out of love with it so quickly.
If we are brutally honest, attainment trumps progress in the eyes of most parents; they want to know the proportions passing English and maths as a proxy for quality. Very few take any interest in Attainment 8, and while more look at Progress 8 (P8), too many don’t properly understand it. This alone calls into question its utility.
P8 was Michael Gove’s attempt to force schools into adopting the ‘academic’ curriculum he preferred. In truth, it is easily manipulated by some, while others are held to ransom by it.
In October 2023, I ran P8 though our AI tool in Lighthouse and asked it to tell me which contextual factors caused it to increase or decrease and by what amount.
The analysis doesn’t identify whether the relationship is causal, but it does open up key questions about the appropriateness of the P8 measure.
So, for example, Ofsted judgements are closely linked to P8 scores. Without a more detailed analysis of admissions linked to Ofsted judgments over time, we can only rely on anecdotal evidence to decide whether that’s because judgments are sound, or because a judgment’s impact on admissions mean some schools thrrive while others find it harder to improve outcomes.
In terms of this analysis, we plugged in the standard contextual factors around schools, such as school size, FSM, admissions criteria, region, gender of school, faith context and key stage 2 scaled score on entry.
It is clear that Progress 8 is a deeply flawed measure
The contextual factors are based on school averages rather than national pupil averages. Moreover, we removed SEN from the analysis because of the complexity that surrounds it and the bluntness with which conclusions could be derived. It requires its’ own separate analysis.
What we found is that the top five factors related to an increase in P8 scores are (in order):
- fewer children eligible for free school meals (7.7 per cent or less)
- an Ofsted rating of ‘outstanding’
- selective admissions
- key stage 2 average point score on entry above 107.3
- a girl-only intake
One way or another, these all relate back to admissions.
The DfE have always claimed that P8 is equalised across the system as they band every child based on their key stage 2 prior attainment data.
But what they have never factored in is what happens when you group children together (for example, larger cohorts of more able children together in one school). There is a clear advantage to this. School leaders know this intuitively, hence the gerrymandering of admission criteria.
On the converse, the top five factors relating to a decrease in P8 scores (in order):
- an Ofsted rating of ‘special measures’
- a mixed-gender intake
- an Ofsted rating of ‘serious weakness’
- an Ofsted rating of ‘requires improvement’
- more children eligible for free school meals (23.4 per cent or higher)
One other notable discrepancy is location. Being in London increases P8, but being in the north east or the north west are respectively the eight and tenth most impactful factors in P8 reduction.
It is clear (and has been since its inception) that P8 is a deeply flawed measure. We need a more broadly inclusive progress measure that allows for more curriculum autonomy.
More broadly, we need better use of accountability data from the DfE. There’s no reason not to provide robust contextualisation and the ability to compare similar schools’ relative achievements.
To break down the barriers to opportunity, government needs a clear idea of what constructive accountability can look like and how to harness data to that end.
After two years without any progress measure, now is an auspicious time to create a fairer and more accessible key stage 4 progress score.
This is entirely achievable, but the clock is ticking.
For a full breakdown of the contextual factors most likely to affect Progress 8 scores and their effect sizes, click here.
Your thoughts