The London mayor said it was “completely wrong” if people could not make themselves understood in English to NHS staff.
Mr Johnson said failing to learn English was a “wasted opportunity” for immigrants and it was a “great shame” that not everyone took steps to master the language.
His comments came after Ukip leader Nigel Farage claimed there is a problem with foreign GPs not speaking good English despite all NHS doctors having to pass a language test.
Speaking on LBC Radio, Mr Johnson said: “I think everybody in London, everybody who comes to work in our economy, should be able to speak English.”
The Tory mayor said there were some communities where people did not feel the need to learn the country’s language.
“You go to Tower Hamlets, places like that, you can find people who have been there for several generations who still don’t speak English. One of the reasons is that the media is so massive now, so disparate, so diffuse, so diverse that people can be tuned in to their own communities and not feel the need to learn the common language of this city and this country.
“I think that is a great, great shame, it is a huge wasted opportunity for them and I think everybody in this country - particularly people working in our public services - should speak English.”
At the weekend Mr Farage told Sky News “people out there are talking about” GPs not speaking good enough English but the NHS careers website clearly states that registered doctors must pass a test under the International English Language Testing System.
Mr Farage made his claim as he defended Ukip’s policy to stop people who do not speak good English working in the NHS.
Mr Johnson said: “I’m amazed by reports that people cannot make themselves understood in English in this country to people working in the NHS. That is completely wrong. I’m sure NHS managers will be taking steps to sort it out.
“I think that we should have a culture in this country that if you come here, you do as the Romans do, you learn English and you speak English.”
He condemned the multicultural policies that resulted in a “Balkanisation” of British society as children were taught in their native languages rather than English.
“I think that’s a disastrous approach. They should be learning in English,” he said, adding that it was “not a difficult language”.
He said: “I don’t want to be hostile to speakers of other languages, other languages are beautiful things and people should learn other languages. But this is a country that happens to speak English.”