How do you know which interventions will actually work in your school? You need to look beyond the effect size, says Jonathan Haslam
I’ve learned a lot from these three numbers: +0.76, +0.11 and +0.26.
They are all effect sizes from trials run by the Education Endowment Foundation. An effect size is a way of quantifying the difference between two groups, so, in these cases, a measure of the impact of an intervention.
The first (+0.76) comes from a trial of self-regulated learning and was the largest ever seen in an EEF project. A scale-up trial was understandably commissioned, and after years of waiting the results were published earlier this month. It shows a much lower impact, an effect size of +0.11.
The intervention is based on self-regulated strategy development (SRSD), put together in the US in the Nineties. This is a writing process model in which pupils are encouraged to plan, draft, edit, and revise their writing and is summarised in the project’s name, IPEELL, which stands for introduction, point, explain, ending, links, and language. The strategy provides a clear structure to assist writers and can be used for most genres, including narrative writing. The intervention also involves “memorable experiences” to act as a stimulus for pupils’ writing.
However, there were important differences between the original trial and the scale-up. The latest trial was bigger (more than 2,500 pupils as against just over 250) and larger trials generally have smaller effects. In this latest trial, the intervention used teacher trainers who had never seen IPEELL delivered, whereas the trainer in the original trial was an SRSD developer from the US. The second trial also measured the average impact on all pupils, while the first looked only at pupils with low prior attainment. This seems to be important: in this latest trial, pupils with low prior attainment who used IPEELL for two years made more progress in writing (that third effect size, +0.26) than pupils who did not.
What does it all mean? Does the IPEELL intervention “work” or not?
The original trial showed that the approach can have a big impact on children who are struggling with their writing. The study was robust and well-structured. The second suggests that how you implement the approach can affect its effectiveness – it is effective if you have read about the approach and then use it in the classroom, but not as effective as if you have been coached by experts. And it doesn’t seem to make as much impact with higher-achieving pupils. In the latest trial, there is also a concern that concentrating on writing negatively affected pupil performance in reading, spelling and maths; a warning to take care with implementation and make sure it’s not taking the focus away from other areas.
More generally, these numbers reiterate the importance of looking beyond the headline figures of effect sizes. Posts on social media for “high-impact educational strategies to use today” – “collective teacher efficacy – effect size +1.57” are unhelpful. That leap forward is as much as students improve in the whole of secondary school. It seems, to say the least, unlikely that there is a simple strategy that you can start using “today” that will have such an incredible impact. Only by looking at what happened in the studies that obtained these enormous effect sizes (with whom, where, how many, for how long, etc) might you begin to understand whether that impact is achievable in real life (spoiler: still probably not!).
What does this mean for schools trying to implement approaches that have research evidence to support them?
Don’t take the magnitude of effect sizes at face value. Huge effect sizes (more than +1.00) are most likely the results of poor experiments, but a small size (+0.10) from a large, well-run trial might still be educationally important. Compare like with like.
Delve into the details. Do the results come from schools like yours and for pupils like yours? Who delivered the approach – developers, researchers, or teachers – and could you replicate it?
Focus on implementation. Whatever approach you select, what do you need to do to give it the best chance of success?
Evaluate for yourself. Look into ways of obtaining objective measures of impact, and reflect on how well the implementation went.
Those three numbers helped me to go behind the headline findings and gain a deeper understanding of the potential of the approach, and of the way that research can help to support improvement in schools.
Your thoughts