Lord Heseltine, who was an early supporter of devolution, was speaking during an event at the Transport for the North (TfN) annual conference, which was held online today.
And he said England was behind many other countries in handing out powers from the centre, but that there was no political will to do more.
Downing Street has insisted on a number of occasions that the Prime Minister is still committed to the levelling up programme promised at the election, including full devolution for all parts of the North.
But Lord Heseltine, who made the point that devolution is not purely about the North, said: “I’m after something the Government has abandoned, they have lost interest in devolution.
“And you see it in every announcement that you get. The mere fact that you see these transport announcements is because the Department for Transport is back in control.”
He said: “I want to see devolution throughout England, I want to see proper boundaries, proper powers.”
But he said getting rid of district councils would be a logical next step, to achieve “bigger economies and massive efficiencies” through unitary councils which could then bid for devolution with directly-elected mayors.
He said that would “make it look like what every other advanced economy has been doing for decades”.
And he added: “We are behind the curve”
“We are now facing one of the greatest economic challenges hidden away behind Covid,” he said.
“That's bad enough, but behind it is Brexit. And Brexit is a serious, devastating threat to this country's prosperity.
“And in order to lead the fight back, you need local people, local enthusiasm, local partnerships, local knowledge, and local determination, and Whitehall cannot provide any of that.”
But De Montfort University Associate Professor in Local Politics and Public Policy, Arianna Giovannini, said the current way devolution was handled had not been assessed.
“We really need to acknowledge that devolution in the North and across England remains a profoundly unfinished business that really lacks a clear framework and direction of travel,” she said.
“We know that parts of the North have benefited from devolution, but the process has really developed in fits and starts.”
And she said the concept of signing devolution deals with the Government was also flawed.
“This is problematic because again, while devolution was aimed at bridging existing divides, the competitive nature of devolution deal making is now creating, actually, winners and losers of devolution,” she said.
“I think that so far, what's been lacking is a clear roadmap for devolution, and also willingness on the part of central Government to develop a long term strategy for devolution, that does not serve only the short term - and sometimes political interest of the centre.”
And she said more than “warm words” were needed when “trust between different levels of Government is really something that is missing at the moment.”
She added: “Without genuine devolution, there is not going to be a levelling up, the two agendas cannot be cannot be taken forward if they continue to be controlled - almost unilaterally - from the centre.”
A Ministry of Housing, Communities and Local Government spokesperson said: “We’re levelling up and empowering all areas of the country by devolving money, resources and control away from Westminster.
”We intend to bring forward the Devolution and Local Recovery White Paper in due course.”