Before joining the ODA, Mr Armitt was chief executive of Network Rail (2001-07) and before that was boss of Costain (1997-2001).
At City & Guilds he will succeed Michael Howell, who said: "On behalf of the trustees and council, I am delighted that John has agreed to accept this nomination. John is the perfect candidate to support City & Guilds on its exciting path of growth and diversification, while raising the profile of vocational education both nationally and internationally."
Mr Armitt said: "I am delighted to be nominated as chairman of City & Guilds. I am a long time admirer of the organisation and look forward to working with [director general] Chris Jones and his team to continue the outstanding progress made in recent years. I believe in the importance of vocational education in supporting UK plc - a workforce that is fit for purpose is central to the achievement of sustainable economic growth. At this critical juncture for the UK economy, City & Guilds plays a vital role in supporting employers' continued investment in their staff."