A whole-school improvement programme which has been used at over 4,000 schools across England has a “detrimental effect on learning”, according to a damning two-year study.
The Education Endowment Foundation (EEF) conducted a randomised controlled trial of 134 primary schools to assess the impact of the Achievement for All (AfA) programme at key stage 2.
The trial, which cost over £900,000, took place between 2016-18 and found children in treatment schools made two months’ less progress in key stage 2 reading and maths, compared with children in control schools.
The findings rank AfA as the joint worst-performing of more than 100 projects reviewed by EEF since 2011, with only three other projects earning an impact rating of negative two months.
Of these it is the only one to have the highest possible evidence strength of five – which indicates EEF “have very high confidence in its findings”.
The EEF found the programme “had a detrimental effect on learning” and advised schools delivering the system to “evaluate whether it is having the intended impact”.
AfA was founded in 2011 by Professor Sonia Blandford and its trained coaches deliver bespoke two-year programmes to schools through 12 half-day coaching sessions a year which focus on leadership, learning, parental engagement and wider outcomes.
It also focuses on improving the outcomes for a target group of children – commonly the lowest 20 per cent of attainers.
The not-for-profit organisation’s website states its “proven ability to accelerate academic progress closes the gap for all ages and stages, regardless of their background, challenge or need”.
However, EEF’s evaluation concluded these target children, on average, made two months’ less progress in reading than children in schools who did not receive the programme – while children eligible for free school meals made two months’ less progress in maths.
Blandford pointed out that disadvantaged pupils within the AfA trial schools still “achieved above national expectations, which was our key aim in the intervention”.
She added “it was an error to agree to a trial that attempted to evaluate the effectiveness of our broad and yet bespoke approach through the narrow lens of two school improvement parameters”.
The founder also said seven out of every 10 participating schools continued with the programme and in the two years since the trial ended changes had been made to further support schools as part of the programme’s “continuous development”.
During the trial, the flexibility of the programme divided opinion. While some schools found it beneficial, others reported it made it “difficult to grasp what the programme was”.
Nevertheless, the evaluation states the findings are “unlikely to be a result of schools failing to effectively implement” the programme, as in some cases “better implementation… may even have been associated with more negative impacts”.
The EEF’s findings are in contrast to glowing reviews the programme has previously attained, most notably from the Department for Education in 2013. A national evaluation commissioned by the government found AfA had a significant impact upon progress in English and maths among pupils with SEND who made “significantly greater progress” than the nation average.
Elsewhere it stated the programme led to “significant improvements in positive relationships, and reductions in bullying and behaviour problems among pupils when compared to those in non-AfA schools”.
It is interesting to see our two evaluation studies portrayed in this way. The first evaluation was of AFA conducted under ideal conditions, LAs were better funded, schools were given additional funding to work through organisational change (school improvement) and the effects for children with SEND were better than similar children in comparison schools. We call this an efficacy trial – it really tests whether the principles work under ideal conditions.
The later study was done under very different conditions. Schools had to fork our the money themselves when budgets were tight. LA had been through a decade of cuts and many of the support staff had been made redundant or not replaced. Some of the large quangos that supported schools had also disappeared. We were interested then in whether AFA worked in the real world under less than ideal conditions – was it effective? The trial was set up as a randomised control trial with schools at the onset intending to use the intervention (Intention to treat model). Those not allocated to the trial arm (using AfA) were allowed to do whatever they wanted in the interim (treatment as usual) and of course they would seek out other means of school improvement. What we found then was not that pupils with SEND did not make progress, but that they made less progress if the school bought into AFA than if they did some other school improvement programme or just carried on as normal.