The Education Endowment Foundation (EEF) has defended the research methodology used in its philosophy for children (P4C) report, which came under public criticism this week.
The education charity released figures suggesting that philosophy sessions for pupils as young as 9 and 10 could improve maths and reading progress by an average of two months, with an even greater change for disadvantaged children.
The findings were based on a randomised controlled trial of 3,159 primary pupils across 48 schools delivered by the Society for the Advancement of Philosophical Enquiry and Reflection in Education.
Stephen Gorard, professor in the school of education at Durham University who led the research, said the results were “not huge, but given that it is a relevant achievement of the intervention, it seems to be a success”.
However, critics said the research did not include a “statistical significance” check that would ensure the positive results were truly being caused by the philosophy intervention and not by chance.
Camilla Nevill (pictured), evaluation manager at the EEF, told Schools Week that the P4C report indicated that the findings only had “moderate security”.
She said: “As with all EEF-funded studies, P4C was assessed by an independent evaluation team.
“Though evaluators use a variety of statistical approaches, all teams are encouraged to consider the possibility that a result occurred by chance.
“All studies are awarded a security rating as part of the peer-review process, which takes into account a range of factors including the design of the study and the size of the trial.
“Our padlock ratings help teachers and school leaders decide how much weight they should place on the evidence. The P4C report received three out of five padlocks, indicating moderate security.
“In this case, we believe the report identifies a promising, low-cost approach that schools could consider.”
Reacting to EEF’s response, Coventry-based maths teacher Andrew Old, a regular writer for Schools Week and public critic of the report, said the absence of a statistical significance test, “which would usually be standard in this type of research”, was a cause for concern.
“Given that it seems entirely possible that the results are completely by chance and therefore show nothing, then to give P4C three padlocks, one wonders what would have less than three.
“The fact, however, that it is three out of five suggests that the report is something that we should believe, to some extent, but perhaps still doubt.”
He added that his problem was with the lack of information put into the public domain about the results’ tentative nature.
“It was not released as a provisional finding for further research, it was reported as evidence that philosophy is good for maths and English.”
The EEF has confirmed to Schools Week that it is planning to share the data from all studies through a UK data archive, 12 months after publication.
“This will allow researchers to explore the data and perform their own analysis,” Ms Nevill said.
“In the meantime, the results from this study will be incorporated into the Teaching and Learning Toolkit, which summarises results from multiple trials into recommendations for schools.”
As most EEF research appears to be 3 padlocks or less, I was surprised as an end user that most of their research is only moderately reliable and its up to us whether to try it or not. I was even more surprised to find that different evaluators (and it looks like the EEF themselves) are able to present the same findings in different ways, because evaluators other that Prof Gorard use significance tests. How can we properly compare across EEF reports? Given the issues of regression to mean (besides confidence intervals) raised by Andrew Old and others it would be good for Schools Week to get an independent expert view on this, as there are important points of confidence in EEF reporting that we should be aware of when making important decisions for our schools.