The Conservative grandee told an online debate that mayors like Tracy Brabin and Dan Jarvis needed to "get off their butts" and start being more vocal about central government dominating decision-making.
And he said the current "fragmented" approach to devolution in Yorkshire was a poor substitute for a single mayor for the region of 5.3 million people, a proposal already rejected by the Government last year.
The majority of council leaders as well as business leaders and unions backed a 'One Yorkshire' devolution proposal. But after the Government said the region was too big and diverse for such a deal, devolution deals have been signed in West Yorkshire and the Sheffield City Region.
Lord Heseltine, who was converted to the cause of regional devolution in 1981 when he was appointed “Minister for Merseyside” by Margaret Thatcher in the wake of the Toxteth riots, told the event organised by think-tank Policy Yorkshire: "My appetite is for a mayor of Yorkshire.
"I would give my eye teeth to travel the world saying 'I represent Yorkshire', and actually because of the way in which the mayoralties have been designed, we've got fragmented situations in many places, or places with boundaries which make no real sense, in terms of economic viability.
"So, yes, we have West Yorkshire, we have South Yorkshire but what about North Yorkshire and what about Hull. So I believe that the real agenda would be Yorkshire, a Greater Yorkshire. I have believed that from the earliest days of working on this devolution agenda."
The Government is due to start talks with civic leaders in North Yorkshire and Hull and the East Riding about devolution deals for these areas.
But Lord Heseltine said he did not believe there was a public appetite for devolution. He said: "There's no confidence in the body politic in this country, and turnout in local elections is derisory because people actually don't believe it makes any difference.
"By and large the results reflect people's judgment on national government. It's the same in by-elections, it's the same in local government elections.
"So I don't expect anybody to be saying 'what we really want is mayoral authorities', because they don't have any place in the polity of public administration full stop, whether it's national or local."
Praising the enthusiasm of Tracy Brabin, who became West Yorkshire mayor this month, he said the existing metro mayors had achieved some successes.
And he said: "That shows what we want is someone who is identified with a set of locally relevant policies, with support from those people who can make contributions in the public sector, third sector, local areas in the private sector. They need a leader to bring them together."
The Conservative peer spoke about how the Government's enthusiasm for devolution had cooled in recent months.
He added: "So, I'm afraid my advice to the mayors collectively was get off your butts, start saying it loud and clear, because this government is taking you for a ride.
"They're talking about levelling up and all these schemes that they announced will be added up as a package and said 'look what we did to level up'.
"But actually, all they are doing is what governments have always done, at times of employment they spend money on transport, if they have trouble with a school they all do some new schools policy, it's the same old business."
Ms Brabin joined the event after a meeting with her nine other metro mayors and spoke about the potential of her office for West Yorkshire.
She said: “Let’s not forget we’ve got [fellow metro mayors] Steve Rotheram, Andy Burnham, now myself, into South Yorkshire with Dan Jarvis, that thread, and that desire to deliver for our communities with very similar challenges. I think that’s going to be really exciting going forward.”