Ofsted had to beef up safeguards during its double inspection reliability study after inspectors, who were supposed to independently come to final judgments, were found to be discussing their findings, according to a report released by the inspectorate today.
The education watchdog has this morning published the findings of its tests on the reliability of short inspections.
Schools Week first reported the pilot in February 2015, and had an exclusive preview of the findings.
The tests involved two inspectors visiting 26 primary schools and delivering an independent verdict of whether the school should remain good, or needed a full inspection.
In 22 of the 24 cases analysed (two were dropped from the final analysis) inspectors came to the same conclusion.
Inspectors were doing minor things that could prejudice judgments
However the report reveals an earlier pilot of the reliability study in summer 2015 – involving seven schools – found some inspectors had “spoken frequently” during their inspections.
In two instances, inspectors came up with “single shared outcome”, as opposed to separate decisions.
The report explains how in each case one inspector had identified the need to continue on to a full inspection, then discussed the findings with their colleague, before the pair subsequently reached the same decision.
Schools Week has previously revealed that these findings were understood to have been raised in an interim report of the study – which will not be published.
However the final report, published today, does include a summary of the interim review findings.
Alan Passingham, one of the report’s authors, told Schools Week that inspectors “didn’t quite get what we were trying to achieve” in the earlier pilot.
“They were doing little things, such as a wave to each other – minor things that could prejudice judgments.
“We took that on board and included protocols [in the final study] to ensure less of that kind of behaviour was going on.”
In the second round of reliability tests, inspectors had to complete summary evaluation forms and place them in sealed brown envelopes handed to their boss before meeting to agree a final outcome.
Inspectors were also given different roles, with one required to “concede authority” to another if their inspections overlapped.
And independent observers were appointed to oversee some inspections, following discussions with Ofsted’s expert panel of advisors.
Passingham added: “We can’t be absolutely sure we’ve managed to nullify [all problems], but there is enough evidence to reassure us that if there was interaction it was likely to be unintentional.”
The two inspectors did still meet the headteacher at the same time – at the start of the inspection – to develop lines of enquiry.
The report stated this was likely to introduce an “element of confirmation bias” to the study, but added this was done to avoid additional burden on the school.