Two months into the job and Amanda Spielman, Ofsted’s new chief inspector, has shot out of the traps as you would expect – with research in hand.
Criticised by the education select committee for lacking passion, Spielman is well-known for being a data-nerd. As chair of qualifications regulator Ofqual, she zeroed in on making decisions based on lengthy research programmes and hard numbers rather than hunches.
She hopes to do the same at Ofsted, and says this week’s publication of a study on the reliability of inspectors is just a start.
“I can take no credit for it,” she admits, as the work originally started two years ago, “but I was impressed by the thorough analysis of the work that went into the design.”
It is the start of a thorough look at inspection, in an attempt to understand how – or whether – it really works.
“This is the beginning of structuring a programme so we can pick off the most important questions to answer,” she says.
The reliability of Ofsted inspectors – that is, the extent to which they come to the same conclusion – is found in the report to be pretty high, at slightly over 90 per cent. But Spielman is more worried about validity – that is, the extent to which inspections accurately measure the quality of a school.
“Validity is the single biggest thing we should be focusing on in the research programme going forward,” she says. “It’s the same construct when you’re testing a qualification. Are you measuring what you intend? Are you measuring what you think you’re measuring? And is it in ways that are efficient, useful and constructive?”
Validity is the single biggest thing we should be focusing on
This will come as a relief for those headteachers who have long wondered whether Ofsted is over-reliant on data, or if its grades are too dependent on the whims of individual inspectors.
Last year, Schools Week reported that a “robo-inspector”, created using an algorithm from Arbor Education, was able to reliably predict when inspectors would award a ‘good’ rating, purely by using a school’s performance data. However, ‘inadequate’ and ‘outstanding’ ratings were handed out on a less predictable basis.
Spielman believes that no test should ever be perfect – “I would be worried if reliability were 100 per cent,” she says of the study this week, suggesting that such high scores could indicate cheating.
Furthermore, she cautions against viewing inspection as a mechanistic process, and wants to go beyond a formula that says ‘if this is found then that outcome is given’.
This is particularly important regarding Ofsted’s guidance on qualifications gaming and off-rolling. Sean Harford, the inspectorate’s director of education, has recently written to inspectors telling them they should be looking for any evidence of schools putting pupils in for inappropriate qualifications purely to garnish performance data, or of their taking poorly performing pupils off roll during their final year so they aren’t included in published exam results.
But Spielman says no specific action should be required if an inspector finds evidence of such behaviours.
“They look at the published data, take it into account and interpret it, and use it to inform the rest of the inspection work they do,” she says. “So if there was some data suggesting significant off-rolling, then it would become one of the lines of inquiry inspectors would pursue in discussion with management.
“They would understand what was happening, and that could feed into other judgments as well. It feeds into leadership and management, for example.”
Nor do schools need to be concerned by the announcement last week that Ofsted would help police the planned introduction of compulsory sex and relationship education across all schools, including academies, which were previously exempt.
Ofsted already looks at such provision as part of their personal development, behaviour and welfare judgment, she contends.
“The best way of looking at these changes is to say that they give us a stronger basis for commenting on what we see. It gives us a stronger position to criticise or take it into account in an overall judgment, but it’s not something new for us.”
The leadership and management judgment is the thing that really reflects the quality of the whole leadership of a school
What is newer for inspectors – another concern of Spielman’s – is the extent to which changes to GCSE and A-level examinations over the next few years are playing havoc with the usefulness of school data. Where teachers could previously estimate pupil outcomes with reasonable accuracy, the complete overhaul of grading – including GCSEs moving to a numerical instead of a lettered grade system – makes it almost impossible for teachers to make predictions.
It is also likely that schools will have more volatile results over the next few years, and they could see results dropping in some subjects. This isn’t due to any changes in teaching quality, but to other aspects of the exam regulation system, which Spielman knows only too well from her time at Ofqual.
To that end, she has asked Ofsted’s data and insight team to draw up a list of inferences that inspectors will and will not be able to make from data from next year.
“We know that with the new grading, nobody at the moment has a sense of what grades look like in terms of standard of work. So there’s more uncertainty than usual at school level. We are trying to make sure that inspectors are also getting to grips with it, and are in as good a position as possible to make use of the new data and minimise the likelihood of it being misinterpreted.
“You’re dealing with grades, distributions and things you’ve never seen before. Sometimes the more dangerous thing is for people to think they can see a trend when it’s not actually telling them anything.
“It’s helpful to be really clear about this. It’s about making sure that people have good advice on what represents a significantly different dataset. That’s not necessarily something that you want every inspector to individually try to work out.”
Hers is a very different approach to that of her predecessor, Sir Michael Wilshaw, who never gave an inch on Ofsted’s abilities to inspect – going so far as to brand academic reports questioning their reliability as “tosh and nonsense”.
In contrast to Sir Michael, Spielman wants her inspectors to consider the entire leadership and management teams at a school, rather than just headteachers.
“The leadership and management judgment is the thing that really reflects the quality of the whole leadership of a school, not just a named top leader but a group. We all know that leadership is actually a set of people, not just a single person. The ‘management’ word is quite important as well.”
There is however one team which she still has to win over: although Ofsted is an independent body, its priorities are set by government ministers.
Still, Spielman is nonchalant about dealing with the government: “I am onto minister number 12,” she laughs.
But does she think she’ll be able to stand up to them?
She purses her lips: “I think I’ll be precisely as steely as I always was. I don’t think I’ve changed my spots.”
Your thoughts