The Work and Pensions Secretary accused Labour of telling "a pack of lies" about the reforms.
Although Tory London Mayor Boris Johnson had compared it to "Kosovo-style social cleansing", Mr Duncan Smith insisted Mr Johnson now backed the policy.
And he issued a warning to landlords that the "game is over" on high rents, vowing to force those charged to benefits claimants down to commercial levels.
State help for rents is to be capped – at 400 for a four-bed property – as part of an effort to slash 1.8bn a year from the "out-of-control" housing benefit bill.
Local Housing Allowance rates will also be cut and payments cut by 10 per cent for people who have been on jobseeker's allowance for more than a year in changes that have alarmed anti-poverty group.
Shadow Work and Pensions Secretary Douglas Alexander this week said the "potentially devastating" reforms risked displacing families and increasing homelessness. He told MPs London would "look very different in the years ahead" – echoing previous warnings from critics that the less well off would be forced out of the centre of the capital.
"What he said to Labour politicians who were whipping up fear about this – by the way a pack of lies... is that what they were saying, 'social cleansing', is an outrage and is not going to happen.
"800,000 living in social housing in central London alone are unaffected by this, this is not Paris," he told BBC1's Andrew Marr Show.
"What we are dealing with is people stuck in houses they cannot afford.
If they go back to work they lose their housing benefit and have to move.
"Is it a fairness to someone locked into an expensive house in central London knowing they can't take that job because the moment they take that job they will lose their house? That's not fair.
Asked how many altogether could be forced out of their homes, he said: "It will be a very low number of thousands and even probably less and even within their area.
Key to keeping that number down would be forcing lower rents, he said.
"Because we are 40 per cent of the rental market, we drove all the rental prices up in places like London so that if you are a working person on low or marginal income you can't afford to live in central London any more – you have to be on benefits or incredibly wealthy."
"I am going to be pushing and telling the landlords 'the game is over, these rents are coming down'.
Mr Duncan Smith also said that parents who refused to take up work would not be spared having their benefits stopped but could receive help.
Under a new "claimant contract" unemployed people who persistently turn down or refuse to apply for jobs will lose their 65-a-week Job Seeker's Allowance for up to three years.
Labour's candidate for the London mayoral election in 2012, Ken Livingstone, said: "Iain Duncan Smith's revelation that Boris Johnson met him last week and confirmed his wholehearted support for the Government's housing benefit cuts leaves London totally leaderless on an issue that threatens huge problems across London.
"Two Tories agree and thousands suffer. If Boris Johnson thought he could get away with saying one thing in public while saying something completely different behind closed doors he has been found out.
He said thousands of families in London faced eviction and poverty as a result of the cuts.