The 29-year-old Briton tumbled to the tarmac on Tuesday’s fourth stage, damaging his wrist.
He started the fifth stage from Ypres to Arenberg Porte du Hainaut, which took place in torrential rain that slickened the roads.
Froome crashed early on the route before a second crash with around 70-kilometres remaining saw him end his defence.
He consulted Team Sky doctor Alan Farrell before taking his place in the back of the Team Sky car.
The stage commemorates 100 years since the start of World War One and features many of the cobbles used in the Paris-Roubaix one-day race ‘the Hell of the North’.
Wet weather forced race organisers to remove two of the nine cobbled sections, but Froome’s falls came prior even to the first section of cobbles he had been dreading since the route was announced last autumn.
The sad sight of Froome, dominant in winning the 2013 Tour, grimacing by the roadside was reminiscent of Sir Bradley Wiggins’ withdrawal with a broken collarbone in the first week of the 2011 Tour.
The Tour last took to the cobbles in 2010 in a stage Lance Armstrong predicted would be carnage.
So it proved as Geraint Thomas finished second to Thor Hushovd and Frank Schleck was among those to crash out.
That took place in the dry, but the downpours caused treacherous conditions even prior to the first section of cobbles.
Froome fell on his left side during June’s Criterium du Dauphine and again on Tuesday, and then on his right.
The Team Sky rider was dragged back to the main pack after 42km as seven riders, including world time-trial champion Tony Martin (Omega Pharma-QuickStep), were up ahead.
The frenetic pace and difficult conditions set-up a compelling day’s racing, with Froome a victim of misfortune as a further tumble proved too much and he exited the Tour.
His team-mate Bernhard Eisel had also crashed and was waiting for Froome, but the Briton clambered gingerly into the rear of the Team Sky sports car.
Team Sky principal Sir Dave Brailsford said prior to the start that Richie Porte would be a protected rider, like Froome, as an insurance policy.
It is likely the Australian will now be Team Sky’s leader, although Thomas could also be given a chance to prove his potential.
Froome’s fate again highlighted Brailsford’s call to omit Wiggins, the 2012 winner, from Team Sky’s squad.
Wiggins was ninth in Paris-Roubaix in April and won May’s Tour of California but was not selected.
The first cobbled section was due after 87km of racing, with organisers removing sector seven and five - two sections totalling 2.4km - reducing the stage distance to 152.5km, 3km less than originally planned.
Froome joins Mark Cavendish in exiting the Tour after the Manxman’s crash on stage one to Harrogate.
Cavendish had shoulder surgery on Wednesday which rules him out of the Commonwealth Games.
Thomas and Simon Yates (Orica-GreenEdge) were the remaining Britons.
Froome was not the only rider to be unseated on a difficult day, with Marcel Kittel (Giant-Shimano), the winner of three of the opening four stages, also tumbling.
Froome’s supremacy in the mountains earned him the 100th Tour title in 2013, but a duel with two-time winner Alberto Contador will now have to wait until 2015, at least.
Contador’s Tinkoff-Saxo team wrote on Twitter: “@chrisfroome we wish you a swift recovery #TDF. We were looking forward to the battle in the mountains @letour”