Inspectors without subject expertise struggle to reliably assess the quality of lessons and pupils’ workbooks in secondary schools, according to a new analysis from Ofsted.
Ahead of the launch of the inspectorate’s new education inspection framework in September, Ofsted has published research into the validity and reliability of its inspection methods.
The first set of reports, published today, focused on lesson visits and scrutiny of pupils’ work. It found inspectors – all of whom were experienced Her Majesty’s Inspectors (HMIs) -in secondary schools could make consistent and reliable judgements about behaviour in lessons, but were less reliable when it came to assessing curriculum, teaching methods and scrutiny of workbooks.
Ofsted is now planning to develop subject-specific guidance for inspectors in all subjects, in collaboration with “expert groups”.
Under its new framework, Ofsted will move from checking data to inspecting the curriculum, including deep dives that include lessons visits and workbook scrutiny.
“Reliability is higher at a primary school than at a secondary school level. This is probably due to the fact that subject knowledge required of secondary school pupils is deeper and more specific than it is at primary school level,” the report on workbook scrutiny said.
“Subject matter expertise is therefore likely to be beneficial for workbook scrutiny in secondary school, which is why were are producing detailed subject guidance and training for inspectors.”
A pilot involving nine HMIs found that all were confident in their assessment for subjects that were in their area of expertise, but only two thirds (six of nine) were confident when scrutinising books outside of their expertise.
Six of the nine found it easy to arrive at a judgement for a subject in their own area of expertise, and five said the same for judgements outside of their expertise.
Concerns have been raised about the importance of subject knowledge for inspectors in the new framework previously. Stephen Tierney, chair of the Headteachers’ Roundtable, questioned the reliability of workbook scrutiny in an article for Schools Week in January, and warned inspectors “may well not have any expert knowledge and will basically be having a bit of a punt about what they see”.
However, in a commentary on the research, Ofsted’s deputy director for research and evaluation Daniel Muijs said the judgement of inspectors must be “central” to inspection, as relying only on various indicators and scales can lead to “a range of unintended consequences, such as standardization and potential gaming”.
On lesson observations, the report found that in primary schools inspectors had a “reasonable level of reliability” – with curriculum, teaching and behaviour domain statistics to be above 0.6.
However curriculum and teaching statistics for the secondary school sample achieved just a moderate reliability level.
The research also found that two HMIs paired together for lesson observations made more consistent judgements than when an HMI was paired with an additional inspector, and noted the “uniqueness” of the judgements that need to be made about curriculum mean “further investigation is perhaps warranted before decisions are made on their validity”.
It also noted that workbook scrutiny may be difficult to implement in special schools and schools that use “alternative methodologies” like Montessori schools.
The report acknowledged that scrutinising workbooks will also be more challenging in modern foreign languages when much classroom activity is spoken rather than written, and emphasised the importance of inspectors also speaking with teachers and observing lessons.
The amount of work in workbooks at the beginning of the academic year “may not be sufficient for inspectors to make a valid and reliable judgment”, the report also stated. But this would not be an issue if “workbooks from the last few months of the previous year were also available for the pupils”.
Your thoughts