Dr Ian Jones, Lecturer, Mathematics Education Centre, Loughborough University
What are you working on?
We [Dr Jones, Chris Wheadon from No More Marking, Sara Humphries of Ofqual and Matthew Inglis, also from Loughborough University] have just published “Fifty years of A-level mathematics: Have standards changed?”
Our research looked at exam papers over the last 50 years to see if standards had got worse. We used 66 exam papers from the 1960s, 1990s and 2012 as a sample.
It was different to most studies that look at similar changes in that it went back so far. Usually studies compare exams over the last five or ten years.
What is interesting about it?
This study used comparative judgment more robustly than has been done previously.
One way of doing this was that we made sure all the tests were in the same format so it was not obvious what year they were from, as an exam from the 1960s looks very old in comparison to today’s papers. This involved typesetting the exam questions, and rewriting all the answers by hand. All marks and comments were removed from each of the papers so that this did not influence those making the comparative judgment.
Most studies use experienced examiners, who are the experts, but who bring knowledge and expectations of qualifications and surrounding debates that might affect their objectivity. We used PhD maths researchers instead, from different countries, who are unfamiliar with A-levels and the debates around standards.
Previous research has mostly been conducted by exam boards and tends to be interested in grade boundaries, and they therefore choose candidate close to boundaries. We used candidates from the middle of grades so we could be sure that, say, a grade A candidate really was representative.
What do you hope its impact will be?
This study is more likely to be useful for policymakers, rather than school leaders. So I would hope that it would inform the debates that go on about A-level maths.
The main impact is the method itself – we have refined a technique for comparing standards, across time or say across exam boards, which can be applied to any subject. Ofqual used this approach to compare maths and science GCSEs from different exam boards in 2015.
What do the findings show?
The fact there has been no decline in standards since the 1990s is the big surprise. Perhaps people can now worry less that standards continue to decline – there is no evidence they have changed since the 1990s – and focus more confidently on making sure A-levels are fit for the purposes they need to serve nowadays.
Is there any other research that you would recommend?
This study is one of a programme of research into how comparative judgement can improve the assessment of maths. Here we applied it to standards, but we have also used it for the direct assessment of students.
Maths exams tend to be dominated by short, piecemeal questions that can be marked reliably. But we need to assess mathematical performance too – we want the maths equivalent of a creative writing assessment (imagine languages were only assessed using grammar and spelling tests, which would be the analogous situation).
Comparative judgment enables mathematical performance to be examined: students can write extended pieces of creative mathematical work and, unlike traditional practices based on mark schemes, comparative judgment lets us assess them reliably.
You can read more of Dr Jones’s publications at: http://homepages.lboro.ac.uk/~maij/
Your thoughts