The chair of the Education Funding Agency’s advisory board has cast doubt on plans to convert all schools into academies and questioned proposals for more grammar schools.

Les Walton, who also chairs the Northern Education Trust, told a fringe event organised by teaching unions at the Labour Party Conference that the government had to be “sensible” about what could be achieved in the next five years.

Walton’s comments, which he insisted were based on his personal views and not those of the EFA, follow several other indications the government could be attempting to row back on previous pledges to achieve full academisation in the near future.

An original plan to force all schools to become academies by 2022 was abandoned by Nicky Morgan earlier this year, but her successor Justine Greening has said she still expects to see schools opting to convert of their own accord as ministers focus on those which are “struggling”.

Walton, who was first appointed by Ed Balls to chair the Young People’s Learning Agency in 2009 and then re-appointed by Michael Gove to lead the advisory board of its successor, the EFA, in 2012, said the schools sector had to look at the “trajectory” of the academies programme.

“When I was appointed by Ed Balls I was asked to take over 221 academies. There’s now 5,000. There’s still 17,000 to go. Therefore if we continue with that trajectory, we probably won’t get to 22,000 in the next five years,” he said.

“I think we’ve just got to be sensible about what can be achieved and what can’t be achieved. “

Walton also appeared to criticise plans unveiled recently by the prime minister to create new grammar schools, adding that his own experience of being in a grammar school, secondary modern and “one of the first comprehensive schools” gave him an unusual insight.

“I hated my grammar school,” he said. “The trouble is, everybody who talks about grammar schools who wants them back are the ones who achieved well at the grammar schools. So we’ve got to start talking to people like me who didn’t do very well.

“We’ve got to get to the reality of what grammar schools were like. When grammar schools were at their peak in the 50s, you weren’t allowed to be homosexual, you would’ve been still able to be killed if you were a witch and we lost the Suez Canal. Life is not the same.”

He also raised concerns about proposals to force independent schools and universities to run schools.

“We can learn from the private sector, we can learn from the independent sector, we can learn from universities, but why have we got such low self-esteem that as schools we can’t learn from ourselves?” he said.