Unemployed ex-schools minister says ‘there’s a lot of schools that will see who I used to represent … and won’t even give me an interview’