Reducing the attainment gap has been a priority for several recent governments. Today’s release of new data on secondary school attainment outcomes for 2023/24 should provide a key metric for judging their progress on this priority. Sadly, the measure just isn’t fit for that purpose.
Disadvantaged pupils are identified by their eligibility for free school meals (FSM). Any eligible pupil who has claimed FSM at any point in the last six years (known as FSM6) is included in the measurement of the index.
That index was introduced in 2015. But rather than using actual attainment outcomes, it uses a ranked approach calculated as the difference in the average rank of performance in GCSEs of non-disadvantaged pupils and disadvantaged pupils.
This makes the data difficult to interpret for any given year. However, we can glean insights from the trend over time.
Figure 1 shows that this trend was generally downward between 2010/11 when the disadvantage gap was at its widest and 2019/20, the first year of the Covid pandemic, when it reached its joint lowest level.
From then, however, the index has widened each year up to 2022/23 (albeit when KS4 outcomes were determined differently due to the pandemic).
Grading has now fully returned to pre-pandemic standards, resulting in national results that were similar to pre-pandemic. Today’s 2023/24 data shows the attainment gap remains broadly around the same level as the previous year, confirming that much of the progress made in the first half of the last decade has been reversed.
On the face of it then, things are getting worse for disadvantaged pupils. But how much worse?
The short answer is: we simply don’t know. Because the index compares the average rankings for disadvantage pupils and non-disadvantage pupils, we can’t tell from the published data whether actual attainment outcomes are deteriorating or by how much.
We do know that changes in the secondary school population will be affecting these numbers. There has been an enormous increase in the number of FSM-eligible pupils in recent years – an almost-doubling from 438,000 in 2018/19 to 828,000 in 2023/24.
We also know that this dramatic increase is due to a few key factors. The pandemic, of course, and the subsequent cost-of-living crisis. Overall secondary school pupil numbers have themselves increased rapidly too (albeit less rapidly than FSM-eligible pupils). And then, we know that the transitional arrangements to smooth the roll-out of Universal Credit have also inflated the figures.
This matters because, as we explained in an earlier report, a pupil who newly becomes FSM-eligible will on average have lower attainment compared to pupils who are not eligible for FSM but have higher (albeit closer) attainment to those pupils who are already FSM-eligible.
Therefore, as FSM-eligible pupil numbers increase, the average attainment for disadvantaged pupils as a whole is likely to appear to improve. Average attainment for the non-disadvantaged group also increases due to relatively lower-attaining newly disadvantaged pupils leaving this group. Quite what this means for the average rankings is anybody’s guess.
In summary, the attainment gap measure is highly sensitive to changing patterns of disadvantage and it is not possible to understand from published data what impact changes in composition are having on the measurement of the gap.
That means we can’t really tell whether any observed changes are being driven by the changing composition of the disadvantaged group, economic conditions or changes in the relative attainment of disadvantaged and non-disadvantaged pupils.
This is clearly not helpful for making evidence-based policy to close the gap and break down barriers to opportunity. Nor does it help to hold the government to account for progress on this central mission.
NFER has been calling for change since our 2023 report, which recommended the government:
- explore the feasibility of introducing a ‘continuity measure’ of disadvantage based on the underlying eligibility criteria for FSM, which would remove the effect of the transitional arrangements, and
- consider replacing the current rank-based attainment gap measure with a simpler metric based on average point scores.
These recommendations only grow more important as governments and their priorities change.
Your thoughts