Lord Adonis said there needed to be a "dramatic increase" in the number of academy students going to university, especially elite institutions.
This is the "biggest single development, above all others, that will transform social mobility in this country", he said.
He cited the example of Mossbourne Community Academy in east London which has helped 10 of its students to win Oxbridge places this year, saying there was "no reason" why others cannot do it.
Lord Adonis warned that the academies programme must expand rapidly because there are "still too many under- performing schools".
"We need to see many hundreds more academies and we need to see them over quite a short period of time," he said.
"Time will not wait. Every year that we leave these schools which are seriously under-performing, not delivering the goods, that is another generation of kids who are failed and its another big handicap to social mobility."
Lord Adonis, who was a pioneer of the academies programme under Tony Blair's Labour government, also repeated his calls for every university to sponsor an academy.
"Universities are always by definition the most successful and highest-aspiration education institutions in the country," he said.
Speaking afterwards, he added: "The Government needs to be in dialogue with every university about sponsoring an academy and be prepared to help facilitate it."
At the moment a "handful" of institutions, including Nottingham University and University College London, have agreed to academy sponsorship.
Academies are semi-independent state schools, which receiving funding direct from Government. The programme was established under Mr Blair to transform underperforming secondaries in poor areas.
The coalition Government extended the programme to allow any primary, secondary or special school in England to apply for the freedoms. There are currently around 400 academies.