Iain Duncan Smith believes the UK could trigger the formal process to leave the EU before Christmas.
The former cabinet minister insisted it is a “ridiculous pipe dream” to suggest Brexit will not happen following June’s referendum.
He added he expects Prime Minister Theresa May to begin the two-year process of negotiating the UK’s withdrawal by invoking Article 50 of the Lisbon Treaty “if not before Christmas, fairly soon afterwards”.
Mrs May has made clear she does not intend to trigger Article 50 before the end of 2016, with legal challenges over Brexit also set to be heard by the High Court in October.
Speaking to Pienaar’s Politics on BBC Radio 5 Live, Mr Duncan Smith said: “The truth is we’re leaving, the British people voted for this and as a democrat you have to stand by what they voted.
“As Theresa May made clear from the Cabinet meeting the other day... the migration process is about taking control of your borders, control of your laws which allows you to set your overseas treaties.
“So that process will now engage and I’d expect Article 50, as our international legal obligation, to be invoked fairly soon - I’d think if not before Christmas, fairly soon afterwards - and that process then begins.”
The former work and pensions secretary also backed keeping the target to reduce net migration to the tens of thousands per year.
Mr Duncan Smith said: “I’m with Theresa on this, I’m completely with Theresa on this.
“Our present immigration system is that we have the front door shut and the back door basically wide open and that’s why you simply can’t control migration properly.”
He added: “I think it is wholly possible to both have a target and do it through the idea of a proper, controlled work permit system.”