Pontefract South representative George Ayre stood down as a deputy cabinet member after leading a failed coup to unseat Coun Box last month.
In his first interview since a 'no confidence' motion was defeated at the Labour group's last meeting, Coun Ayre says that the leader hasn't taken Ofsted's findings "seriously enough".
The education regulator rated the service inadequate in a damning report in July, and concluded that vulnerable youngsters had been left at risk.
Inspectors also said that "senior leaders across the council....have not tackled the serious and widespread failures across children’s services".
Coun Ayre said: "As councillors we are corporate parents and we have a moral obligation to safeguard the children of Wakefield.
"We cannot stand by while Coun Box continues to display a lackadaisical approach to children's services.
"He seems to put more energy into clinging onto power than dealing with the issues at hand."
Opponents of Coun Box claim he was warned about problems within children's services two years ago, though this is denied by the Labour group.
Coun Ayre insisted that he does not want to be the next leader of the local authority but said he believed a change was required at the top.
He added: "This is about protecting the vulnerable children of Wakefield which in my opinion, Coun Box doesn't take seriously enough, and is the reason I put the vote of 'no confidence' forward.
"After 20 years it's time he resigned. We need a new energetic leader with the drive and enthusiasm to resolve the issues we face and protect our most vulnerable."
Fellow councillors Michelle Collins and Jack Hemingway also resigned their deputy Cabinet portfolios last month after the 'no confidence' motion failed.
And Backbencher Steve Tulley, who has also been critical of Coun Box, stood down from his position on the Police and Crime Panel.
Coun Tulley said: "Everything we're trying to do now with children's services is like closing the stable door after the horse has bolted.
"If we need more social workers now, we needed them two years ago when we were warned about it.
"We can't just sweep it under the carpet. Children have been left potentially at risk and that's completely unacceptable."
In response, Coun Box said: "Urgent and decisive action has been taken to address the concerns raised by Ofsted.
"The council is committed to ensuring the best possible services are provided for our children and we are working hard to make sure that happens.
"I am continuing to work closely with the commissioner, the chief executive, the Cabinet member responsible for children’s services and the chair of scrutiny to ensure the best outcome as quickly as possible."
Labour's chief whip, Coun Richard Forster said: "The Labour group confirmed their continued support for Coun Peter Box in September, through a democratic process, by voting to back him in his role as leader of the council."
Had the 'no confidence' motion succeeded, the vote would have been advisory rather than binding, meaning Coun Box would have been under no obligation to resign.