Free schools came top as the best-performing “type” of school in this year’s Progress 8 tables, but were their scores really good enough to assert that with confidence? Becky Allen explains…
Today appeared to be a good day for free schools, with mainstream free schools achieving a Progress 8 score of +0.24 across the sector. This means students in their schools achieved a quarter of a grade better than expected in each subject. A great result, surely? Or is it?
Alongside guidance on the performance tables, the government said that the number of these schools with year 11 pupils – at 76 schools – is “too small to allow robust conclusions to be drawn about their performance”. You may ask why – surely a school’s Progress 8 is a thing we know with certainty?
GCSE exams are imperfect measures of true pupil attainment
For a long time now, governments have reported confidence intervals around school performance statistics. These are intended to capture the idea that the students who actually attended the school and received those grades are just a sample of the population of students who potentially could have attended the school. Given the rules around how students get allocated to schools, this is quite a strange idea, but it does give us a sense of how secure we should feel about Progress 8 as a measure of ‘true’ school quality. So, for example, Bedford Free School will report their Progress 8 was +0.4, but we should feel much more secure in saying their Progress 8 is somewhere between +0.13 (i.e. quite good) and +0.66 (i.e. fantastic).
We can take this uncertainty about each of the 76 free schools’ results and perform a little simulation to find out how secure we should feel about the +0.24 for the free school sector. We do this by simulating school Progress 8 scores, given the published government confidence intervals on these, and taking the average across the 76 free schools 10,000 times. When we do this, it allows us to say that the free school sector’s Progress 8 is most likely somewhere between +0.21 and +0.27.
I wouldn’t want to go as far as saying this “proves” free schools are better schools. GCSE exams are imperfect measures of true pupil attainment and judging all pupils from their key stage 2 starting points can be systematically unfair on some schools. However, given the way we choose to measure school performance, I think it is reasonable to say that free schools had a successful exams season last summer and we should congratulate them for it.
Rebecca Allen is a professor of education at UCL Institute of Education. She is also co-founder of the teacher survey app Teacher Tapp and is an associate research fellow at FFT Education Datalab