How you can dampen the effects of publication bias

Too much research literature is skewed towards positive representations, says Baz Ramaiah

Imagine that you’re planning a holiday. A quick survey of websites show nothing but positive reviews for your chosen hotel, so you commit your cash and book. It turns out to be half-building site, half-bus station. How did this happen?

Knowing is half the battle

It is then that you learn that the listings websites have a tacit policy of publishing only positive accounts. Seasoned holiday-makers know this, and no longer bother to try to post negative experiences because they realise there is no point.

This dilemma is similar to the one that teachers, senior leaders and education policymakers face with “publication bias”, the tendency to only publish studies with positive findings. Studies that show negative findings are systematically rejected because they are typically judged to have been poorly designed. The result is that the literature is skewed towards positive representations of the efficacy of given treatments and programmes, with no countervailing evidence to challenge it.

This selective approach can have serious consequences. Landmark research in biomedical science has shown the way that publication bias in scientific journals can lead to costly and dangerous practices as negative findings about drug X linger unpublished on the lab floor, while positive ones are widely disseminated and used by policymakers to issue guidelines recommending that very drug. Clinicians, in turn, use these guidelines to prescribe said drug. This results in wasted resources on a drug of questionable efficacy. The waste is colossal.

What about in education? Two recent meta-reviews of academic studies have provided consistent evidence of publication bias in educational research journals. Polanin, Tanner-Smith and Hennessy (2016) surveyed 6,392 studies (through grouped meta-analyses) published between 1986 and 2013 and found that published studies were much more likely than unpublished ones to have positive results. They report a large magnitude of impact for their studied programmes.

In an even broader study, Chow & Ekholm (2018) selected 240 studies from 20 major journals of educational research, and compared them with available studies unpublished in scientific journals. Again, the identified published studies reported a stronger impact for studied interventions and were much more likely to report positive findings.

This practice creates dangers down the line: teachers may be using ineffective instructional practices; school leaders may introduce interventions without any sense of the pitfalls; teacher-trainers may be misleading a whole generation of early career teachers. All of this while being ostensibly “evidence-based”.

Undoubtedly, evidence-based practice is the future for teaching, but evidence is only valuable if it is comprehensive and unbiased. Pending major changes to how journals do their screening, what can the conscientious consumer of educational research do to dampen the effects of publication bias?

First, good systematic reviews often include estimates of the extent of publication bias. Resist the urge to gloss over these passages as they are vital in moderating your confidence in the finding.

Second, when you do encounter useful studies with a negative finding, tell people about it, by bleat or tweet, to counter the wider dissemination of positive findings over negative ones.

Finally, be mindful that there are already reasons to be more sanguine on the subject. The Education Endowment Foundation (EEF) has committed to publishing results from all its studies, independent of outcome. This is a bold stride in the right direction, but nonetheless represents a small proportion of the total literature. Our expectations in this area should be uniformly high. With that in mind, EEF would benefit from calculating and including indices of publication bias in its systematic reviews.

While findings on phenomena such as publication bias might be dispiriting, we should always have in mind the slogan of evidence-based practitioners and the Saturday morning TV icon GI Joe: “Knowing is half the battle.” The more we understand about the scientific process, the better equipped we are to interpret its results. And, of course, to book hotels.