Amanda Spielman has pledged to continue ‘inspecting the inspectors’ as part of a new Ofsted research drive after publishing a landmark reliability study.
The education watchdog has this morning published the findings of its reliability study on short inspections – where two inspectors were sent to the same school to see if they gave the same judgment.
The report found that inspectors agreed on the outcomes in 22 of 24 short inspections (92 per cent), which the watchdog said was a “positive view of inspector consistency”.
Speaking exclusively to Schools Week, Spielman, Ofsted’s chief inspector, said the findings are just the start of her forensic testing push to “really make sure” that Ofsted “contributes to system-level improvement”.
We want to be as constructive a force as we possibly can
“We want to make sure we understand [the role we fulfil in the system], test that, and be as constructive a force as we possibly can be in the system.”
Writing in an online blog today, Spielman added the watchdog should “routinely be looking at issues of consistency and reliability”.
“And even more importantly, we should be looking at the validity of inspection: is inspection succeeding in measuring what it is intended to measure?”
Under 10 per cent of inspections unreliable
Inspectors visited 26 primary schools rated as ‘good’ by Ofsted. A total of 22 of the 24 cases that were analysed as part of the study resulted in the same outcome from both inspectors – either the school remained ‘good’, or there was a need for conversion to a full inspection.
In the two case where the judgment differed, one was put down to “subjective interpretation” of the same evidence, while the other was said to be caused by the “study design”, rather than differing conclusions.
Two cases did not feature in the final analysis. One was dropped after a safeguarding issue was discovered, while the other was dropped because it was “clear” one inspector had influenced the other.
Spielman told Schools Week the findings “give us some level of reassurance that inspectors are applying the short inspection construct reasonably consistently”.
But she added that while the watchdog can do a “great deal” to minimise the impact of bias in human judgments, “we are unlikely to ever reach a position where perfect consistency can be guaranteed”.
“If it was perfect it would actually get a bit worrying – if it was too mechanistic. That would suggest it’s all a bit too narrow – there’s not enough space for inspectors to dig in.”
Inspecting the inspectors
Alan Passingham, one of the report’s authors, told Schools Week the additional scrutiny on inspectors’ decisions was “incredibly powerful”, adding they “found it good professional development”.
“They often inspect on their own, there’s no way to test themselves on ‘are we doing the right thing?’.”
However the report stated that views varied. While some inspectors found it “reassuring” to come to the same conclusion as a colleague, others said the reliability focus was “professionally uncomfortable” and found not “having ownership of the inspection frustrating”.
Passingham said there was potential for the reliability checks to be “utilised” as part of inspector training.
Spielman added: “I think it is [a useful thing for them to go through]. I don’t think anybody should get isolated in their practice and interesting thing to reflect on.”
Spielman said the report – the first reliability study into the whole inspection process – is “just a first step towards a continuing programme of research into inspection”.
The report states that a further roll-out of the reliability study across a larger sample of primary schools with more independent observers “may be warranted”.
The report concluded there are four factors associated with reliability, including the “triangulation” of senior leaders’ views against evidence collected during the inspection, and the level of inspector training, although Ofsted said the latter required “further study”.
The report stated there is a need to further investigate these individual factors.
Spielman added: “We are beginning to shape up what this research programme should look like. But this is not a quick hit in which everything is sorted at once: rather, it will be a steady process in which questions are addressed systematically.”