The billionaire businessman said he felt there was “definitely a friendly star above our family” as they had to scramble behind large boulders during the dramatic incident.
He told the Press Association the team of climbers were picked up by a helicopter after reaching the summit as they agreed the path back down was not worth the risk.
The 68-year-old, along with Sam and his nephew Noah Devereux, reached the top of the Alpine peak on the French-Italian border on Saturday after completing the 33-day Virgin Strive Challenge – a 2,000km trek across Western Europe.
A core team of 13 hiked, climbed, cycled and even used sea kayaks to complete their journey from Sardinia to the top of Mont Blanc – raising money for the charity Big Change which helps disadvantaged young people in the UK.
In an interview with the Press Association after completing the climb, Sir Richard said: “God forbid, a few seconds earlier it would not have been as pretty – so I’m very thankful we got through it.
“There’s definitely a friendly star above our family and it keeps a watchful eye over us. Three seconds earlier and we would all certainly have been killed.”
Sir Richard added: “I have never come so terrifyingly close to losing my son and other team-mates and it all happened in a matter of seconds.
“A side of a cliff had broken away from the mountain further up and seconds later huge boulders the size of small cars were bouncing towards us. Rocks rained down on us from every angle.
“I am not exaggerating when I say four of our team were seconds away from certain death.
“We were all quite emotional and we were completely shattered and exhausted.”
Sam Branson captured the terrifying moment the team passed through the area known as “The Gully of Death” on camera as he admitted he was glad to see his children the day before.
“Obviously being attached to my dad was scary enough going up because I love him,” he said.
“But there’s that added element of concern outside which is yourself and the whole team to think about.
“We were hiding in fear, not knowing whether the rocks were falling directly on us or to the side – there was an overwhelming array of emotions.”
Noah, who had watched events unfold from above, thought Sam and Sir Richard had been killed in the rockfall.
He said: “I was screaming ‘Sam! Sam!’ but we didn’t hear anything back.
“It was just the sound of the rockfall just slowly dissipating which was quite an eerie sound itself and then just complete silence.
“I saw Sam and Richard’s head appear above the crest of ridge behind me and that’s probably one of the greatest senses of relief I’ve ever experienced.”
Sir Richard’s daughter Holly, who was not told about their traumatic experience until the team were back in Chamonix, said: “I was told ‘you are going to see them a bit quicker than you thought’ and I hung up the phone and I did start crying.”