Most trials carried out by a major educational research charity “don’t tell us anything” about whether an intervention had an impact on pupil learning, researchers have warned.
The Education Endowment Foundation (EEF) was founded in 2011 under Michael Gove, the former education secretary, with a £125 million grant to fund research into boosting attainment for disadvantaged pupils.
But a new study by Loughborough University, which analysed 82 randomised control trials commissioned by the EEF, found 55 per cent produced results that were inconclusive.
The EEF, which spends about £500,000 a trial, said even if its trials did not provide conclusive evidence, they still helped to expose programmes that falsely claimed to boost results.
Researcher Hugues Lortie-Forgues told Schools Week the average effect size, which measures an intervention’s impact on pupils, was just 0.06 standard deviation for EEF trials. Usually, a 0.4 effect size is needed to demonstrate impact, he said.
In many of these very expensive trials, the conclusion doesn’t tell us anything
“The goal of research is to learn new things, but in many of these very expensive trials, the conclusion doesn’t tell us anything.”
A trial was uninformative if findings meant the intervention could be “either effective or ineffective”, said the report. Whether the trial was uninformative depended on the effect size and how precisely the effect size was calculated.
Instead the EEF should test interventions on a much smaller-scale to ensure the intervention was “really promising” before carrying out a large-scale and expensive RCT, Lortie-Forgues said.
It should also ensure the intervention was implemented properly and the pupil sample size was large.
But Stephen Fraser, the deputy chief executive of the EEF, said about half its published projects revealed a positive impact on pupil attainment.
Interventions proven not to have an effect also helped senior leaders “avoid wasting scarce time and money where it’s unlikely to make much difference”.
Past EEF trials have evaluated whether chalk slates or banning grades could boost pupil attainment.
The research also looked at the National Centre on Education and the Economy in the US, finding a similar proportion of trials were uninformative.
An inconclusive finding is as valid as one that finds a positive or negative outcome. When a trial begins, it’s not known what the outcome will be. It’s wrong, therefore, to dismiss trials because they don’t lead to a firm conclusion.
As welcome as this article is, it still doesn’t touch on two major problems with education research. First, teachers’ prior training will affect results. In an analysis of a 1997 Michigan reading initiative, Standerford found that such changes as teachers actually implemented were seldom inconsistent with their prior preferences or beliefs about the best way to teach reading. In 2006, Anghileri came to a similar conclusion about the inconsistency of the implementation of the National Numeracy Strategy in Cambridge schools.
The second is ecological validity. What works in controlled trials may not work at all well when implemented on a larger scale. In his 2013 inaugural lecture at Durham’s CEM, Robert Coe admitted “. . .we do not know how to get large groups of teachers and schools to implement these interventions in ways that are faithful, effective and sustainable. . .It is now a rare thing, in my experience, to meet any teacher in any school inEngland who would not claim to be doing Assessment for Learning. And yet, the evidence presented above suggests that during the fifteen years of this intensive intervention to promote AfL, despite its near universal adoption and strong research evidence of substantial impact on attainment, there has been no (or at best limited) effect on learning outcomes nationally.”
I’m the researcher mentioned in this article. Just to avoid misunderstandings, I noted that some of the quotes are not entirely accurate. For example, I would not say that “Usually, a 0.4 effect size is needed to demonstrate impact”. Most EEF trials can demonstrate the impact of interventions producing effect sizes smaller than 0.4. Regardless, the article correctly reported the key finding: most trials – 55% of EEF trials in our analysis – were not able to determine whether the intervention tested should be implemented at scale or whether its use should be discontinued.
Link to the study: https://pure.york.ac.uk/portal/en/publications/most-rigorous-largescale-educational-rcts-are-uninformative(c59efeb6-152d-440d-a3b8-eaf8ba05b9c1).html
Since this news article was written the paper has been published with very different results: only 40% are uninformative. But no matter the change in results, the authors conclude the same. How does that work?