Mr Hoare, a former city banker, said of the island: “Most people go there for sailing for two weeks a year. There’s a sailing club that is one of the best in the world, where there’s champagne.
“But just within inches, there are people who live in a ghetto and we’ve allowed it to happen.”
He added: “They think of it as holiday land. But it is shocking. It’s a ghetto; there has been inbreeding.”
Mr Hoare also said coastal towns were often ignored in terms of poverty and educational under-performance.
Last night, the Isle of Wight’s council leader, Jonathan Bacon, said he would contact Education Secretary Justine Greening to seek an explanation from Mr Hoare for his comments.
He said: “David Hoare’s comments about ‘inbreeding’ and ‘ghettos’ on the Isle of Wight are truly offensive to the people of the Isle of Wight and bear no relation to the facts.”
Mr Hoare later apologised for “any upset or offence”, but said he was concerned about the “unacceptably poor performance of schools on the Isle of Wight over many years”.
He added: “Those who know me will realise that I am passionate about improving outcomes for children from our most disadvantaged communities and my comments were made in this context.
“It is important that we draw attention to low educational standards, especially among low-income white British communities in our coastal areas, so that collective action is taken to improve the situation.”
“I welcome the efforts that are being made, supported by Hampshire County Council, to improve school performance on the island.”
But Coun Bacon said standards on the island had risen to such an extent that its “improvement services” were now rated as effective by Ofsted itself.
He added: “We now have only one school on the Island that is rated as inadequate, in line with national averages.”
The independent councillor blamed government cuts for the isle’s “austerity agenda” which he said had taken no account of the geography of an area.
The council, which is dominated by independent and Conservative members, has had to cut more than £60m from its budget in the last six years and expects to save £34m more by 2020.
Coun Bacon said: “It would be refreshing if the Ofsted chairman was able to make a positive contribution to improving education on the Isle of Wight rather than making facile and offensive comments.”
An Ofsted spokesman said: “The chairman was expressing his personal views.”