Full election coverage »

In a bid to restore his battered authority, Mr Brown took to the TV studios to insist he understands voters' worries over rising prices and economic uncertainty, saying: "I feel the hurt they feel".

He said he had the "convictions and ideas" to get Britain through the current economic downturn and pledged to get out of Westminster "far more often" to go around the country to listen to voters' concerns.

The Prime Minister's attempt to launch a fightback came against a backdrop of open speculation among Labour MPs about his future as party leader.

Foreign Secretary David Miliband denied he was plotting a bid to unseat Brown, telling the BBC he was "the right man to take us forward into the next general election".

But Manchester Blackley MP Graham Stringer said ministers and backbenchers were already discussing a possible challenge to the Prime Minister.

And Blyth Valley MP Ronnie Campbell said: "If Gordon Brown can't handle it, he should get out and get somebody in who can handle it. He has to decide that."

Shadow cabinet member Alan Duncan said Mr Brown was now "a spent force" who would fight the next election in a "much weaker position" after leading Labour to its worst showing at the ballot box for 40 years, with a 24% share of the vote.

Mr Brown's problems deepened today as former minister Frank Field warned he was ready to restart a backbench rebellion against the abolition of the 10p tax rate, which ministers have blamed for the drubbing administered by voters this week.

The Prime Minister acknowledged he had made "mistakes" over the 10p rate and allowing speculation about a snap election to get out of hand last year.

But he told BBC1's Andrew Marr Show: "Of course we can recover from this position and I will tell you how.

"First of all by sorting out the immediate problem with the economy and showing people we can come through, as we have in the past, very difficult economic times.

"Secondly by showing people we have a vision of the future that will carry the country - optimistically in my view - into its next phase."

Assessing Labour's ballot-box mauling, which culminated in Boris Johnson's defeat of Ken Livingstone in the London mayoral election, Mr Brown said: "I feel responsible. There are no excuses on my part at all."

Mr Brown said he did not believe "many MPs" were discussing his future as leader, and dismissed suggestions he might stand down before the election expected in 2010, saying: "There's still a job to do."

"I'm resolute and determined, and I've got convictions and ideas, and I'm not going to be put off by a few days' headlines from the job that I'm determined to do for this country," he said.

Asked if Labour was the underdog, after finishing 20 points behind the Tories and being beaten into third place by Liberal Democrats on Thursday, he replied: "If we are the underdog, we are certainly fighting and we are fighting hard."

Mr Brown did not make any announcements of specific policy changes to woo back voters.

But he told Sky News' Sunday Live: "You will be seeing in the next few weeks as we publish proposals for the legislative programme ahead that we have got the right ideas for how we make a better future for the country."

With Labour MPs feeling nervous about the safety of their seats, the PM is coming under pressure from all sides of the party to refocus his appeal to voters.

Influential backbencher Jon Cruddas warned that the party leadership had taken working-class voters for granted and now had to win them back.

The Dagenham MP told the Sunday Mirror: "Our people are abandoning us, we're sinking fast, and no amount of hand-wringing and promises of 'listening and learning' from election night will change that... It's not too late to change - but choose change we must."

Former minister Peter Kilfoyle told BBC1's The Politics Show: "We need to refocus our efforts on those people who have deserted us, people who feel as though we have left them out."

And Gower MP Michael Caton said the 10p tax fiasco showed Labour must be ready to direct future tax hikes at the wealthy, rather than the poorest in society.

"We should prove we have listened to what was told us on the doorsteps in this election and start talking about really tackling poverty, and that, I think, is going to have to mean that the richest people in society contribute more," he told Radio 4's World This Weekend.

Labour's deputy leader Harriet Harman warned colleagues against "navel-gazing" and inward-looking debate over the leadership, insisting that Mr Brown was the best person to lead Britain through the current economic turbulence.

And left-winger Diane Abbott - one of the MPs who threatened to rebel over the 10p tax issue - said it would be "madness" to replace the leader.

"We have got to stick with Gordon. Any changes that have to be made have to be made under Gordon," she said.

Mr Duncan told Sky News: "I think if that was a fightback, Gordon Brown is now in deeper water." And Liberal Democrat home affairs spokesman Chris Huhne said it was clear the Labour Party was "in some turmoil".

Mr Brown now faces the potential of a renewed revolt over the compensation package for those who lost out when the 10p income tax rate was abolished, with Mr Field threatening to table a parliamentary motion on Tuesday.

And further challenges just a few weeks down the line include the May 22 by-election in Crewe & Nantwich - for which Labour has selected Gwyneth Dunwoody's daughter Tamsin as its candidate - and a possible rebellion over 42-day pre-charge detention for terror suspects.

Environment Secretary Hilary Benn told Sky News: "It was a bad day and a bad night on Thursday. Let's not mess about, we all know that is the case.

"But I genuinely believe there is everything to play for.

"There's up to two years before the next general election, and while Thursday was an opportunity, if you like, for people to have a referendum on the Government and send us a message - and they certainly did that - the next general election will be people voting about who they want to be Prime Minister and who they want to govern the country.

"I think that there's a real fight between now and then and there will be a real choice."

Mr Benn said he had "no idea" if there would be a Cabinet reshuffle, adding that Mr Brown was "absolutely the right man to lead us into the future".