Seeing red: David Miliband in dig at brother over Iraq

The Doncaster North MP criticised large slices of New Labour's record in office in his first full conference speech as leader but said a "new generation" hungry for change offered optimism in contrast to the "miserable, pessimistic view" of the coalition.

He sought to reassure those concerned about the union backing which swept him by power by criticising "overblown rhetoric" on strikes and saying Labour must stomach some unsavoury cuts proposed by the coalition in order to tackle the deficit – but also warned against "making a bad situation worse" by cutting too fast.

To those who have branded him "Red Ed", he said: "Come off it. Let's start to have a grown-up debate in this country about who we are and where we want to go and what kind of country we want to leave for our kids."

But last night his attempt to restore party unity after the leadership contest was undermined when defeated brother David was caught on tape making a barbed comment to the party's deputy leader Harriet Harman as tensions within the Miliband family over Ed's win appeared to spill over.

Following a key passage where Ed condemned the invasion of Iraq as "wrong", his elder brother was said to have turned to Mrs Harman sitting next to him and said: "You voted for it, why are you clapping?" She replied: "I am clapping because as you know, I am supporting him."

Although Shadow Foreign Secretary David Miliband – whose wife Louise was spotted in floods of tears backstage following his own well-received address the previous day – praised the speech afterwards, he was reported to be heading back to London amid mounting speculation that he will not stand for the Shadow Cabinet, a decision he must make by 5pm today.

The incident took the gloss off his brother's big speech, in which he signalled a clean break from the era of Tony Blair and Gordon Brown as he sought to quell fears he would lead the party in a lurch to the left after defeating David Miliband courtesy of union backing.

He said the party must "always stand up for the mainstream majority" and fight for the centre ground, but criticised swathes of New Labour's record including on Iraq, City regulation, immigration, tuition fees and Gordon Brown's hubristic claim to have abolished boom and bust.

In a bid to ease fears he was in hock to the unions, he called for an end to "overblown rhetoric about waves of irresponsible strikes" saying he would have "no truck" with such action – but won union applause for supporting protection for agency workers, family-friendly working hours and the living wage. Business groups voiced concern at his attack on excessive pay and support for a banking levy.

On the key issue of the economy, the leader insisted he was "serious about reducing our deficit" and warned Labour had four years to restore its economic credibility. "There will be some cuts the coalition does that we won't like as a party but we will have to support," he said.

But he accused the Government of being "irresponsible" in not giving equal emphasis to a plan for growth, and even questioned Mr Camaron "patriotism" by suggesting his drive to eliminate the structural deficit within one parliament risked damaging the country which the next generation will inherit.

"True patriotism is about reducing the debt burden we pass on to our kids," he said. "But Mr Camaron, true patriotism is also about building an economy and a society fit for our kids to work and live in."

David Miliband praised the speech, which was warmly received in the conference hall, but the ITS News footage later emerged of his exchange with Ms Harman.

During the leadership campaign, Ed Miliband described going to war in Iraq as a mistake, but his brother has defended the decision.