Licking his wounds: Miliband Snr walks away from Ed's front bench

In a letter to the chairman of his constituency Labour Party, Mr Miliband said he would remain as MP for South Shields.

But he said Labour would be more likely to be able to make "a fresh start" under its new leader if he moved on to the backbenches.

"I genuinely believe that I can best serve Ed, the party and the country from a new position," he wrote.

In his letter to Alan Donnelly, Mr Miliband said his continued presence as a member of the shadow cabinet would be a "distraction" from his brother's leadership of the party.

And he said he wanted to "recharge my political and intellectual batteries" after 16 years in positions in and around the top of politics and to have more time to spend with sons Isaac and Jacob and partner Louise.

His dramatic announcement comes as nominations close for the election of 19 members of the new Labour shadow cabinet, and just four days after David lost out on the party's leadership by a narrow 1.3% margin to his younger brother.

Mr Miliband said he was confident that stepping back from the frontline was "the right decision for Ed, for the party and for me and the family".

Speculation over David's future overshadowed Labour's annual conference in Manchester this week, with expectations growing as the days went by that he would decline to serve in his brother's top team.

Much of the press coverage of Ed's crucial first leader's speech yesterday was dominated by a critical comment from David picked up by a TV microphone.

In his letter today, David made clear his concern that if he remained in the shadow cabinet, Labour's efforts to supplant the coalition Government would be seen through the prism of their personal relationship.

"As I said at party conference on Monday, we now have a great responsibility to unite behind Ed's leadership," he wrote.

"With his strong speech yesterday, Ed has already started the fightback. On the day that nominations close for the Shadow Cabinet, I think it right to explain to you and party members why I think I can best support him from the backbenches.

"The party needs a fresh start from its new leader, and I think that is more likely to be achieved if I make a fresh start.

"This has not been an easy decision but having thought it through and discussed it with family and friends I am absolutely confident it is the right decision for Ed, for the party and for me and the family."

He added: "This is now Ed's party to lead and he needs to be able to do so as free as possible from distraction.

"Any new leader needs time and space to set his or her own direction, priorities and policies. I believe this will be harder if there is constant comparison with my comments and position as a member of the shadow cabinet.

"This is because of the simple fact that Ed is my brother who has just defeated me for the leadership. I genuinely fear perpetual, distracting and destructive attempts to find division where there is none and splits where they don't exist, all to the detriment of the party's cause.

"Ed needs a free hand but also an open field."

Shadow Welsh secretary Peter Hain, who voted for Ed in the leadership ballot, said the former foreign secretary had left the door open for an eventual return to senior political office.

"I'm sad about it because he's a big figure not only in the Labour party but also in British politics, a great asset to us," said Mr Hain.

"But he's been very magnanimous about it. He will support his brother Ed as leader. He will also support the party in fighting the Conservative-Liberal Democrat Government and he's going to recharge his batteries.

"The good thing is after recharging his batteries, he's left open that he might come back, like William Hague did after his sabbatical."

Shadow home secretary Alan Johnson, who was one of David's most prominent backers in the leadership race after deciding not to stand himself, said: "David has made his decision in the interests of his brother and the Labour Party.

"To remain in the shadow cabinet would invite constant scrutiny of their relationship and endless discussion of every nuance, however trivial.

"This decision will make it easier for Ed to flourish as leader whilst allowing David to pursue other political interests as well as working tirelessly for his South Shields constituents."

Mr Miliband also made clear that he had personal reasons for wanting a break from the pressures of frontline politics.

He said he hoped to develop the politics of community organising that he promoted through his Movement for Change during the leadership campaign.

"I want to recharge my political and intellectual batteries to be of greater service to the party and the country," he said.

"I have spent 16 years in or around the top of politics in one capacity or another. There's a world out there that I have touched but about which I want to know more - from education to the environment to foreign policy.

"I think I can best make a contribution to the election and success of the next Labour government under Ed's leadership by devoting myself to understanding better the new challenges and new ideas that will dominate the next couple of decades, and figuring out how to put our values into practice."

And he said that he wanted to give more time to his children - aged five and two - and his partner.

"I have essentially been a Cabinet minister for the whole of Isaac and Jacob's lives," he wrote.

"That is tough for me and tough for them. One happy consequence of the leadership election will be more time with Louise and the boys."

Mr Miliband - who spent the day with his family in their north London home after leaving Manchester after Ed's speech yesterday - told Mr Donnelly: "You know how important public service and politics are to me. My job as MP for South Shields is precious. So is my commitment to the Labour Party...

"I have been touched by how many party members came up to me at Conference asking me to campaign for them in the upcoming Scottish, Welsh and local council elections, and I look forward to doing so. But I genuinely believe that I can best serve Ed, the party and the country from a new position."

Mr Miliband said he was convinced he had made the correct decision to stand down from the shadow cabinet.

"After the result was announced on Saturday I had my own instincts about what was the right thing to do, but I thought that it was right to take three or four days to make sure that the decision was the right one," he told the BBC.

"I am absolutely certain that for the party and the country, which has driven me in every political decision I have taken, this is the right thing to do."

He said the way the media picked up on his exchange with deputy leader Harriet Harman when she applauded Ed Miliband's declaration that the Iraq war was wrong, showed how difficult it would have been if he had remained.

"I think that raising a wry eyebrow with Harriet yesterday shows the dangers that can come," he said.

"I think that it is really important that no one is able to use anything, the merest sneeze or body language or comment to divert from the really important task that the party has, that Ed has, and that I want to support him in doing."

Conservative Party chairman Baroness Warsi said: "David Miliband was a leading architect of New Labour. The fact that he doesn't want a place in Ed Miliband's shadow cabinet speaks volumes about the direction in which the new leader is taking Labour.

"After being elected by the unions, this is further evidence that Ed Miliband is vacating the centre ground of British politics."

Former home secretary David Blunkett said: "I believe that David Miliband is a man of honour. He has made the right decision for himself, his party and his brother.

"I promise him that it is possible to make a contribution outside cabinet and shadow cabinet, to still be heard where it matters and to think the unthinkable. I look forward to working with him from the backbenches in that role."