A multi-million pound policy piloted in thousands of primary schools this year could be scrapped over concerns regarding its comparability.

A study, commissioned by the Department for Education (DfE), is understood to have found that tests from the three chosen providers for baseline assessments of reception children cannot easily be compared – putting the policy’s future in doubt.

Under the controversial baseline assessment scheme, primary schools must, from September, use the tests if they want to be assessed on pupil progress at key stage 2 (KS2), as opposed to pupil attainment.

Three approved baseline assessment providers – Early Excellence, Durham University’s Centre for Evaluation and Monitoring (CEM) and the National Foundation for Educational Research (NFER) – began rolling out their assessments at the start of this academic year.

The Standards and Testing Agency ordered a study late last year to check the “robustness” of the assessments by each provider.

If the assessments are not adequately comparable, the progress of pupils in each school cannot be compared against each other on a national scale, putting paid to planned performance measures expected to be used to highlight underperforming schools.

Government officials would not confirm the content of the study but stated its results would be published “in due course”.

A mooted alternative to baseline test is the introduction of “school readiness” checks – an option said to be preferred by No 10. Such checks assess children’s ability to hold pencils or read basic words and have been widely debated in the US.

Schools Week understands the DfE, as well as Downing Street, is keen to explore how early years providers can make sure children are “school ready” when starting reception.

The prime minister’s office was approached for official comment but did not respond.

In an email sent to providers and seen by Schools Week, the department said the extension of the baseline assessments was dependent on the outcome of the report and a decision by ministers.

It added: “Our expectation was that a final decision would be taken towards the end of January. However… we are still awaiting our ministers’ decision.”

Schools Week understands the study, which was carried out by the Scottish Qualifications Authority, will be released within the next month.

The email warns providers that any costs incurred in preparation for the 2016/17 academic year are “undertaken at [their] own risk” and the department “will not accept any liability for any costs [the providers] may incur as a result of this work” should their contracts not be extended.

Early Excellence said it would not speculate before the report’s publication. NFER would also not be drawn on its position if the report finds the tests are incomparable.

CEM director Rob Coe said: “It is a bit of a mess isn’t it? The problem is that if they do find the tests to be incompatible and if they find the comparability hasn’t worked out, where does that leave the policy? I’m not quite sure.”

He said CEM was happy with its tests and given all providers had met DfE criteria there ought to be reasonable compatibility and comparability between providers, adding: “If that hasn’t happened, I guess something has gone wrong somewhere.”

Baseline assessments had a troubled start. Before its introduction, 80 signatories, including children’s author Philip Pullman, urged the government to scrap the tests.

In July, a report published by the DfE and written by NFER said there was “some evidence” to suggest schools might game the assessments. Schools with lower results at the baseline could, in theory, show more progress by the end of KS2.

In November, problems arose when “anomalies” were discovered in Early Excellence’s literacy scores.

This month, a survey of 1,000 teachers by the UCL Institute of Education found 60 per cent did not think the tests accurately reflected child attainment and the tests were dubbed “inaccurate, invalid and unreliable” by teacher unions.

Schools taking part in baseline tests have costs reimbursed by the DfE.

Early Excellence was chosen by more than 12,000 schools. Its tests cost an initial £85, plus £3.10 per pupil. Assuming each of those schools has just one reception class of 30 pupils, the DfE’s paybill to Early Excellence will be more than £2.1 million.