Clifford had been serving eight years in jail for historic sex offences.
He collapsed in his cell last Thursday while trying to clean it, then again the following day, according to reports.
The 74-year-old was said to be "in a bad way" after collapsing twice at Littlehey Prison in Cambridgeshire, where he was serving an eight-year jail sentence for historic sex offences.
A spokesman for the Ministry of Justice confirmed Clifford died in hospital on December 10.
"As with all deaths in custody, there will be an investigation by the independent Prisons and Probation Ombudsman," the spokesman said.
"Our condolences are with Mr Clifford's family at this difficult time."