Ofsted tests AI to review parents’ and pupils’ survey responses

Ofsted’s plans to use artificial intelligence to help summarise survey responses from parents and pupils have sparked mixed reactions.

A government document reveals the watchdog is testing the new technology, which is aimed at reducing inspector workload and providing earlier insights into emerging issues.

Former senior inspectors have welcomed the move to cut down on inspector overtime, while union leaders have urged caution, warning the technology requires “utmost care”.

Currently, inspectors must manually review and summarise free-text survey responses gathered during an inspection, which the government said can be “high in volume or in word count”.

One former HMI said inspectors could receive as few as 10 and as many as 1,000 survey responses for an inspection.

The new AI tool would allow them to submit survey responses as an Excel spreadsheet to a web interface, where they would be processed by a large language model (LLM).

The LLM is being tested on free-text responses to Ofsted’s “parent view” survey, completed by parents of children at state schools, and “FES learner survey”, carried out by FE and skills students.

It processes responses using a series of prompts, and has three main functions: “summarisation, identification of themes and identification of safeguarding concerns”.

AI ‘supports human decision-making’

The government document stressed the LLM “does not make decisions, judgments, or replace part of the decision-making process”, but instead “supports human decision-making”.

It would provide “concise summaries”, “key themes”, and “prevalence scores for each theme and performing safeguarding-related classification”.

All employees would be given online training.

The LLM could also be “extended in the future to ingest other types of inspection survey… where appropriate use cases are identified”.

Ofsted said it would not be rolled out “until we are confident it is safe and helps the inspection process”.

A recent survey by union FDA, which represent in-house His Majesty’s inspectors (HMI), revealed 98 per cent were working beyond their contracted hours.

Three out of five reported working more than 10 extra hours a week, partly because of the time it took to analyse survey responses.

Reduction in overtime?

The government document admitted this often took place “outside core hours”.

It hoped the LLM would reduce “work outside of core hours, and contribute to increase in wellbeing of inspectors”.

It could also help to improve quality and consistency, allowing inspectors to identify key lines of enquiry earlier and reveal issues that would otherwise be “time-consuming to identify, or not evident”.

One former HMI, who wanted to remain anonymous, told Schools Week it typically took about an hour-and-a-half for lead inspectors to “sift through” survey responses on Tuesday nights.

Outcomes from this process can strongly affect the direction the inspection takes on day two.

“Anything to ease inspector workload, it’s a massive welcome change,” she said. “It is freeing us up to be inspectors rather than to do the admin.”

Personal data

Matt Newman, FDA national officer for Ofsted, said the union was broadly supportive of AI initiatives, “particularly when used to enhance wellbeing and reduce excessive workloads”.

“But we will need to work closely with Ofsted to ensure that these tools are not detrimental in the longer term to our members and ultimately to parents and children.”

Survey responses could contain sensitive personal data, but the LLM is prompted to provide “broad summaries and… avoid naming individuals and any identifiable information”.

Testing has reportedly shown the LLM “accurately follows these instructions”.

Meanwhile, a preliminary investigation into potential social bias found the LLM “did not negatively target specific groups of people with protected characteristics”.

Diversity issues

However, “on occasions it appeared that the output did not always reflect the diversity of responses when summarising into themes”.

“Prompt engineering work” has reportedly improved this, and further testing will take place this summer.

Pepe Di’Iasio, the general secretary of the Association of School and College Leaders, said the union believed some “will feel uneasy about the use of AI to analyse survey results, particularly given [this] admission” on diversity.

“While we understand this is a potential time-saving measure, the fact that free-text responses can be ‘high in volume or in word count’ surely points to the fact that these are nuanced views on important education matters that require careful consideration.”

He stressed Ofsted must take “utmost care” in developing the tool, and “provide reassurance that human oversight will remain paramount”.

Paul Whiteman, the general secretary of the leaders’ union NAHT,  said: “These responses could form part of a school’s overall Ofsted judgment which can have an enormous impact on not just the school but an individual headteacher’s career, so accuracy must be taken incredibly seriously.”

Meanwhile Colin Richards, a former HMI, queried whether AI could pick up “potentially significant comments made by a small number of respondents”.

Adrian Gray, another former HMI, said it it might “strike fear into the hearts of inspectors who are already scarred by earlier IT failures”.

Ofsted is currently considering replacing its electronic evidence gathering (EEG) system, which has been hit with ongoing glitches.

“On the other hand, inspectors do have too little time and this may help ease the pressures.”