Fifteen NHS trusts have terminated their contracts with Healthcare Environmental Services (HES) and a new provider was put in place from yesterday morning.
The update has been given by health minister Stephen Barclay, who pledged that the Government would "ensure lessons are learnt" in a written statement to parliament.
It follows reports of tonnes excess clinical waste piling up at HES sites, including in Normanton, West Yorkshire, and enforcement action being launched by the Environment Agency.
Mr Barclay said the facilities management company Mitie had taken over NHS contracts after trusts formally ditched HES as their provider at 4pm on Sunday.
The statement said: "In parallel, the Department of Health and Social Care, the Cabinet Office, NHS Improvement and the affected trusts have negotiated a new contract with Mitie to step in and replace this service.
"This contract was enacted, following the termination of the contract with HES, and Mitie have been fully operational across all affected trust sites from Monday morning."
Mr Barclay said the Government was formally notified by the Environment Agency on July 31 of concerns over too much waste being stored at some HES sites.
He said: "While the waste was stored securely, it was not being processed and disposed of within the correct regulatory timescales. At no point has there been an impact on public health or any delay to the ability of the NHS to carry out operations."
The Environment Agency partially closed HES's Normanton site on October 3 following breaches of the environmental permit.
The health regulator NHS Improvement (NHSI) then gave the company 48 hours to show how it was complying with its contractual obligations.
Mr Barclay said: "NHSI concluded that HES failed to demonstrate that they were operating within their contractual limits.
"Consequently, 15 NHS Trusts served termination notices to HES formally to terminate their contracts at 4pm on Sunday, October 7."
The Environment Agency has placed restrictions on waste being accepted at the HES Normanton site and could take similar action at other sites, including one in Newcastle.
The statement said: "It is the company’s responsibility to clear its sites and operate legally."
Mr Barclay stressed that NHS services would operate as normal and there was no risk to patients or public health.
He added: "The Government is working with the Environment Agency and NHS to ensure lessons are learnt, and we are reviewing how contracts will be awarded in the future."