Most EEF trials 'don't tell us anything', say researchers

Most trials carried out by a major educational research charity “don’t tell us anything” about whether an intervention had an impact on pupil learning, researchers have warned.

The Education Endowment Foundation (EEF) was founded in 2011 under Michael Gove, the former education secretary, with a £125 million grant to fund research into boosting attainment for disadvantaged pupils.

But a new study by Loughborough University, which analysed 82 randomised control trials commissioned by the EEF, found 55 per cent produced results that were inconclusive.

The EEF, which spends about £500,000 a trial, said even if its trials did not provide conclusive evidence, they still helped to expose programmes that falsely claimed to boost results.

Researcher Hugues Lortie-Forgues told Schools Week the average effect size, which measures an intervention’s impact on pupils, was just 0.06 standard deviation for EEF trials. Usually, a 0.4 effect size is needed to demonstrate impact, he said.

In many of these very expensive trials, the conclusion doesn’t tell us anything

“The goal of research is to learn new things, but in many of these very expensive trials, the conclusion doesn’t tell us anything.”

A trial was uninformative if findings meant the intervention could be “either effective or ineffective”, said the report. Whether the trial was uninformative depended on the effect size and how precisely the effect size was calculated.

Instead the EEF should test interventions on a much smaller-scale to ensure the intervention was “really promising” before carrying out a large-scale and expensive RCT, Lortie-Forgues said.

It should also ensure the intervention was implemented properly and the pupil sample size was large.

But Stephen Fraser, the deputy chief executive of the EEF, said about half its published projects revealed a positive impact on pupil attainment.

Interventions proven not to have an effect also helped senior leaders “avoid wasting scarce time and money where it’s unlikely to make much difference”.

Past EEF trials have evaluated whether chalk slates or banning grades could boost pupil attainment.

The research also looked at the National Centre on Education and the Economy in the US, finding a similar proportion of trials were uninformative.