Moderation of practical assessments at scale is now a reality

The promise of collaboration is central to the mission of many multi-academy trusts. When it comes to assessing subjective skills in arts, practical, performance and technical subjects, this ideal often clashes with the reality of logistics.

In theory, the collective expertise of teachers working across multiple schools should lead to leaner processes and better outcomes. But in practice, each school operates with its own curriculum and within its own context.

This leaves many leaders in a quandary: How do you standardise assessment across a dozen different schools without causing immense disruption and additional workload?

Traditional moderation and standardisation procedures are a significant pain point for trusts. They typically require in-person meetings, often spanning whole days, bringing together teachers from different schools to pore over criterion-referenced rubrics.

This process is time-consuming, expensive in terms of disruption, cover and travel, and rarely produces the consistency of judgment that trust leaders crave. It is a logistical and cultural challenge that often impedes the very collaboration it aims to foster.

This challenge is particularly acute at key stages 2 and 3, a crucial period for developing a holistic range of skills. While declarative knowledge is important, employers and educators alike are increasingly recognising the value of attributes like creativity, critical thinking and communication. Capturing these human skills in a meaningful way has to be the next frontier in assessment.

But what if the solution wasn’t more meetings but a fundamentally different and inherently digital methodology?

Moderation time was slashed from days to minutes

Our work has shown that a new approach is not only possible but highly effective. The key is to combine the power of human professional judgment with the efficiency of AI-supported technology.

The methodology is called adaptive comparative judgment. Instead of assessing work against complex rubrics, teachers use a digital platform to simply compare two anonymised pieces of student work. This can be anything from a design project to a piece of written composition, an audio file or a recording of a performance.

The teacher’s job is simply to decide which is better. And the role of AI is to make this process scalable trust-wide. The machine learning algorithm acts as a powerful facilitator, intelligently selecting the most informative pairs for comparison.

This means teachers from different schools can participate at their convenience, from their own classrooms.

The AI’s ability to rapidly process these human judgments and build a statistically robust rank order of all student work means that the process creates consistent standards across the entire trust without a single moderation meeting.

This ‘human-in-the-loop’ model directly addresses the inherently human elements of creativity and judgment, while making the assessment process fast, reliable and valid. The computer handles the repetitive, data-heavy tasks, while a person makes the nuanced decisions that require experience, insight or intuition.

A recent research pilot in a large trust produced remarkable findings. We proved that it is possible to achieve consistently high reliability scores of over 85 per cent, demonstrating that this methodology provides a trustworthy, holistic picture of a learner’s abilities.

Better still, the time and resources typically associated with standardisation and moderation were slashed from days to minutes. And teachers valued the process for another reason too: it gave them a window into the standards of their peers across the trust, fostering a deeper, shared, tacit understanding of what constitutes quality.

For forward-thinking trust leaders, this technology provides a powerful solution to a long-standing problem. It enables them to gain a trusted, holistic view of pupil, department, and school performance across subjective curriculum areas for the first time.

In short, it turns collaborative assessment into a practical reality, without causing disruption.

We are now preparing to take this research to the next level. If you are working in a trust and interested in being part of the next phase of this groundbreaking work, we would be delighted to hear from you.

We believe this research has the potential to fundamentally change how we assess a learner’s true potential at scale, making the invisible visible.

Register your interest here