On Monday around 100 education researchers and users of research gathered at the Royal Statistical Society to celebrate the centenary of the first-ever randomised controlled trial (RCT) in education.
It may come as a surprise that the origins of the RCT span at least as far back in education as in medical research. There were a few examples of random allocation in medicine in the 1920s and 1930s, but in education there was one in 1919 (although whether it was truly randomised was challenged on the day!), carried out by Robert Alexander Cummins of Iowa.
News from 1919
Cummins’s trial concluded that reducing the length of lessons over time in US grades 3 to 8 resulted in improved performance due to pupils’ “more intensive application […] in their effort to get as much done as possible”. However, the correct analysis techniques for his data had not yet been invented, so we should take his conclusion with a large pinch of salt!
Experimentation in schools remains under threat.
At the centenary event, as well as reflecting on the ebbs and flows of education trials over the past century, presentations covered recent improvements, such as independent evaluation and so-called “open science” practices employed to improve the replicability of research. We also heard about what RCTs mean for teachers and school leaders.
Scope for improvement
However, the event was not all about celebration and self-congratulation. The researchers, whose business it is to evaluate, did not hold back when invited to reflect on the practices of their community. Hugues Lortie-Forgues presented a paper claiming education RCTs are often uninformative (as covered in Schools Week earlier this year), and the debate focused on improving the interventions themselves as well as trial design. One suggestion was to fully regain the tradition of scientific enquiry, testing out individual elements of the theory of change before moving to the RCT phase.
For example, many interventions involve Continuing Professional Development (CPD) programmes. For these to improve outcomes for children, several steps must happen successfully. Firstly, sufficient numbers of teachers must attend and engage with the CPD sessions and then employ what they have learned in the classroom with sufficient fidelity. Any adjustment in practice must be effective, ie. children must learn more as a result, and this must lead finally to an improvement in the specific assessment chosen for the evaluation.
Without unpicking each of these steps using a variety of research methods beforehand, it is understandable that many interventions fail to demonstrate an effect when evaluated using an RCT.
Breakfast club evidence
Although much of Monday’s event concerned such null results, there are many examples of positive RCT outcomes that have influenced schools directly. Breakfast clubs are a good example of education policy being directly influenced by the positive results of a trial. When the evaluation showed improvements at key stage one, a £26 million DfE investment followed.
Of course, null results are very useful too. For example, two trials of In:tuition, a school-based programme aimed at reducing teenage drinking, showed no effect on primary outcomes. Drinkaware, which funds the programme, is currently reviewing how aspects of it may be repurposed rather than continuing to promote it to schools.
Funding concerns
But despite positive developments and huge support given by teachers, experimentation in schools remains under threat. There is a risk that, due to the explosion in funding for RCTs, evaluators could find it increasingly difficult to recruit schools to take part. Especially given continuing workload concerns, ceasing research activity is an easy target.
Another risk for the future is political expediency. It is easy for politicians and civil servants to assume a null result implies a problematic research design and to use this as an excuse to halt funding. A concerted effort by the funding community, researchers and teachers alike should help to mitigate such risk in future, and preserve the education RCT for another hundred years.
Insights from the centenary event can be found on Twitter using the hashtag #EducationRCTs100
Your thoughts