Avoid 'uncritical use' of PISA scores, researchers warn ministers

Ministers should avoid “uncritical use” of PISA scores because the tests ignore important elements of what English schools teach, new research has found.

A new report for Cambridge Assessment by Dr Matthew Carroll and Dr Tom Benton found the focus on essay-writing skills in the GCSE English curriculum is not mirrored in PISA tests, meaning that the international study ignores “some of the key skills that schools are trying to teach”.

The PISA tests, sat every three years, are held in high esteem by the government. In 2015, the Department for Education even went so far as to say that its reforms would be judged “by reference to international tables of student attainment, such as Pisa.”

The study compared performance in the Programme for International Student Assessment (PISA) and GCSEs, and found that “an individual’s performance in PISA is likely to depend upon their choices with regard to what they study”. It warned ministers should not use PISA results “without careful consideration of exactly what is being measured”.

Despite the government’s backing, the tests are not without their detractors. Even Nick Gibb, the schools minister has accused the Organisation for Economic Co-operation and Development (OECD), which publishes the international rankings, of being “too political” and pushing a “progressive” curriculum.

The last tests, sat in 2015, found that English pupils had not improved their PISA scores for almost a decade.

Today’s research, published by Cambridge Assessment, a department of the University of Cambridge and the parent organisation of exam board OCR, come as the latest round of PISA testing in schools is drawing to a close. Results will be published in around a year’s time.

Carroll and Benton say that understanding the links between performances on the two assessments is important, “not least because it helps us understand the extent to which performance in PISA reflects the success or otherwise of the UK’s school system”.

For example, it was found that the correlation between performance in the PISA reading test and GCSE English is “not particularly high”, and in fact, performance in PISA reading tests “is at least as closely aligned with achievement in GCSE science (correlation of 0.69) as it is with GCSE English (0.68)”.

Analysis of published PISA questions also found that several asked students to read and interpret tables or figures of scientific information, a skill that is more often a feature of GCSE science than GCSE English.

Although GCSE English forms a “fundamental part” of the way school performance is judged in England, with a “strong focus” on essay-writing skills, this differs when it comes to international tests.

“It is clear that when it comes to judging the performance of our education system as a whole using PISA, essay-writing is not measured,” the researchers say.

“Thus the performance of England’s education system is judged whilst ignoring some of the key skills that schools are trying to teach.”

Studying separate sciences rather than combined sciences also appeared to give pupils an advantage in PISA maths tests.

Even among pupils with “equally good attainment in maths GCSE”, those who studied separate sciences performed “much better” than those who studied combined sciences.