However the Education Secretary told the Commons that the current system was “broken” and needed to be reformed.
He said that instead of new E-Bacc certificates, GCSEs would be reformed in English, maths, the sciences, history and geography.
Mr Gove also confirmed plans to reform secondary school league tables in an attempt to stop schools focusing on pupils on the C/D borderline.
Labour’s Shadow Education Secretary Stephen Twigg said this was a “humiliating climbdown” and told Mr Gove the words “GCSE” and “fiasco” go hand-in-hand with this Government.
He asked Mr Gove to apologise to pupils and parents for having “done down their hard work on GCSEs.”
The u-turn comes following widespread opposition and pressure from within the coalition from the Liberal Democrats.
Mr Gove told MPs that instead of new qualifications, GCSEs will be reformed, with exams taken at the end of the course, rather than in modules, extended questions and less internal assessment.
He also confirmed that he will not be pressing ahead with plans to hand each of the core EBC subjects to a single exam board - a move he had previously argued was essential to prevent boards “dumbing down” standards to attract more schools.
“Last September we outlined plans for changes to GCSE qualifications designed to address the grade inflation, dumbing down and loss of rigour in those examinations,” Mr Gove told the Commons.
“We have consulted on those proposals and there is now a consensus that the system needs to change. But one of the proposals I put forward was a bridge too far.
“My idea that we end the competition between exam boards to offer GCSEs in core academic qualifications and have just one - wholly new - exam in each subject was just one reform too many at this time.
“The exam regulator Ofqual - which has done such a great job in recent months upholding standards - was clear that there were significant risks in trying to both strengthen qualifications and end competition in a large part of the exams market.
“So, I have decided not to make the best the enemy of the good.”
They will still be “universal qualifications”, taken by the vast majority of pupils.
The Education Secretary also outlined plans to revamp league tables.
Under the current system, schools are ranked on the proportion of pupils achieving at least five C grades, including English and maths.
Mr Gove said that this measure had encouraged schools to “choose exams based on how easy they are to pass”, to concentrate on just five subjects and to focus heavily on pupils on the C/D grade borderline.
In future, there will be two measures, Mr Gove said.
The percentage of pupils achieving a set threshold in English and maths, and an average points score showing how much progress every student makes between the end of primary school and GCSE level.
“The average point score measure will reflect pupils’ achievement across a wide range of eight subjects,” he said.
“As well as English and maths, it will measure how well pupils perform in at least three subjects from the English Baccalaureate - sciences, history, geography, languages - and Computer Science, and in three additional subjects, whether those are arts subjects, academic subjects or high-quality vocational qualifications.
“This measure will incentivise schools to offer a broad, balanced curriculum, with high-quality teaching and high achievement across the board.”
Today’s announcement was hailed by one union leader as a victory for those who campaigned against plans to ditch GCSEs.
Christine Blower, general secretary of the National Union of Teachers (NUT), said: “This is a victory for all those who have campaigned against this ill thought-out reform to GCSEs.
“The Education Secretary must now learn a lesson from this fiasco and consult with those who know far more than he appears to do about education.”
Brian Lightman, general secretary of the Association of School and College Leaders (ASCL), said it was “encouraging” that Mr Gove had listened to those who had urged him to think again.
He added: “We have never believed that GCSE is beyond repair and have been urging the Government for many months not to abandon it.”
Mr Gove originally wanted to introduce EBCs in England in English, maths, science, foreign languages, history and geography.
Each of the core subjects would have been handed to a single examination board.
Mr Twigg added: “We have to focus on the standards and move beyond the shambles. There should be a cross-party consensus on a future plan for the next generation of qualifications. Based on the best available expert evidence. Not the back of an envelope. Will he do things differently this time?”