Politics is a brutal game, and nothing quite epitomises that than seeing an ex-minister for the first time after a reshuffle.

Damian Hinds, a man who until two months ago ran a department of thousands of people, greets me personally in the reception area of parliament’s Portcullis House.

Gone are the advisers who once flanked him on every visit. Gone are the press officers who would tap their watches when he (regularly) spent too much time answering journalists’ questions.

The MP for East Hampshire is carrying his own bag again, and he looks intensely relaxed.

I don’t think, in hindsight, that this judgment was wrong

The former education secretary has booked a meeting room for us because his new office is still not ready for visitors. Secretaries of state lose their departmental offices the moment they are sacked, and finding space to set up camp on the packed parliamentary estate isn’t easy.

“Everything is different in politics at the moment,” he muses. For him, “it’s a quieter time, a time to catch up with some other things”.

“You never leave the constituency issues. You can’t, and you shouldn’t. But when you’re not doing a government job any more, yes, there’s an opportunity to do more of everything else.”

Was he expecting to be reshuffled out of his job by Boris Johnson?

“I didn’t know. You never know,” he says. “I had hoped I wouldn’t be. I loved the job I was doing, and I would have been more than happy to continue serving. But it’s in the nature of this line of work. There is fairly limited job security, and that’s part of the deal. You understand it when you come into politics.”

With no government majority, no serious cash boosts forthcoming from the Treasury and Brexit hanging over everything like a dark cloud, Hinds’s ability to actually get anything done was substantially limited.

I remark that it must have been frustrating to watch Gavin Williamson, his successor, announce a massive school-funding package within weeks of starting in the role, a settlement Hinds and his team had lobbied for during his entire tenure at the DfE.

“We were in a difficult time, because of the majority in parliament, because we’re in this period relative to Brexit and having not quite got over the line and needing to get over the line and all that political energy that took up. And it was a difficult time in relation to timing of the spending review that had been expected in the autumn, but based on leaving the European Union at the end of March.

“All those things are interconnected and yeah, of course that’s a frustrating thing to work with.”

Is there anything he regrets not doing, or not doing differently?

“Anyone who says there’s nothing you could’ve done better is a big old fibber or self-deluded. But honestly you can only do what you do with the benefit of the knowledge you have at the time, and I don’t actually look back and think, ‘Oh I wish I’d done this’. As I’ve just set out, there are some circumstances I certainly wish had been different.”

Languages is one area where more work is needed, he admits. “We’ve got an issue with the number of kids taking up languages, and there has been since 2004, particularly with, say, German, and I would’ve loved to have been able to do more on that front.

“Also character and resilience, which I think is such a fundamental part of social mobility and general preparation for life. I would’ve liked to have been able to develop our approach there further.”

When historians write about his time as education secretary (we ask), what will they say?

Hinds hopes they’ll describe him as someone “who was totally committed, totally dedicated to the job, and understood the commitment of everybody working in education”.

Although Hinds will probably be best remembered for his work on teacher recruitment and retention (his strategy received unprecedented levels of support from the unions and the sector more widely), he also earned plaudits for setting in motion significant reforms to school accountability.

On his watch, coasting and floor standards were scrapped, and efforts to hold academy trusts to account were stepped up.

Given he identified this need for greater accountability for academies, does he understand why the sector sometimes gets a bad rap?

“I think I do understand why there is an issue,” he says. “I also think it’s important to talk about it and get things in perspective. The academy sector is more transparent, it’s more open to probing than the maintained sector is.

“And yes, there have been problems with some schools that are academies, and when you’ve got 50 per cent of kids going to academies, it’s a very big part of the sector. But there have also historically been problems with local authority schools you were less likely to know about.”

While Hinds believes academies have overall been “very beneficial” for education, he says some people will always “want to find faults”.

“You find fault in one individual place and you use that to extrapolate out to a system, and that’s just not an extrapolation you can make or you should make,” he insists.

The former education secretary hails reforms to relationships and sex education, which were completed on his watch, as “an enormous step forward,” and defends himself against criticism that by leaving the decision on when to teach pupils about LGBT issues up to heads, he left them vulnerable to attacks from parents and other members of their communities.

“We needed to have a programme which works for everybody. That’s what happens when you make something mandatory,” he tells me. “And what we brought forward had the support of Stonewall and the Catholic Education Service. That’s a strong coalition of support.”

He also doesn’t regret his decision not to visit Anderton Park, the Birmingham school at the centre of bitter protests over its equalities teaching, claiming it could have attracted more protesters at the school if he got involved.

“You have to make judgments on these occasions, and you could argue it both ways, but my judgment, and I have no reason to think in hindsight that this judgment was wrong, was that going there would be more likely to have a negative impact than a positive one.”

So what next for Damian Hinds, a politician once seen as a rising star of his party and tipped as a future prime minister?

He says he will maintain his interest in social mobility, an area where there’s “still an awful lot of work to do.” He is also interested in continuing to look into the education technology sector.

“The longer I went on at the DfE, I realised we’d only really started in a tiny way to scratch the surface. It’s a huge sector. And often questions about technology in education get ridiculously simplified, to whether you’re pro or anti.

“Now if you force me to answer that, I’m anti, right? I don’t want kids spending all day on diversionary, soft-focus electronica. But that would be a really dumb, blinkered view to take, when you consider all the potential benefits of properly deployed and properly delineated and controlled use of technology. So I think that’s a hugely interesting area.”

There’s still an awful lot to do on social mobility

He is unsure if regulation is what is needed, but he believes there is space for “something in the middle, between 21,000 potential buyers and hundreds, probably thousands, of potential sellers where the two don’t really know that much about each other”.

“There is definitely a role for improving knowledge, intelligence about what’s available, and perhaps for schools to share with each other their experiences.”

He looks wistful as he talks about work started at the DfE but never completed. Was he gutted to lose the job?

“I was very sad,” he admits after a pause. “But you also know that in this life, in this world and in particular in this political life, nothing is for ever, and, you know, things happen.”