A new letter released by Ofsted on Friday said progress within the service, which was placed in special measures last summer, was too slow and that chances were being missed to protect vulnerable children.
Councillor Nadeem Ahmed, Tory group leader, said that the report was "not a good one" for the council, and that the "time had come" for Coun Box, who has led the authority since 1998, to stand down.
Coun Ahmed submitted a motion at a full council meeting last October stating 'no confidence' in the leader, though this was foiled by the Labour group.
Speaking after the publication of the new report, Coun Ahmed said: "Clearly the pace of improvement has not been at the rate that the council had expected.
"There are some positives in there, but from my own experience of teaching and Ofsted, they're not doing as well as they should be.
"They've put plenty of resources in there, so I think the problems are with leadership.
"I think if I was Coun Box I'd consider my position. I think he might consider it, and I'd be surprised if he doesn't."
Coun Ahmed claimed there was "not much leadership" behind Coun Box on the Labour benches, and said that issues within children's services pointed to problems within its management.
But he added: "He's had a good innings as leader. We all have good times and bad times in politics, but I think his time's up now.
"I think he knows that as well."
Responding to Ofsted's report in a statement on Friday, the council's corporate director for children and young people, Beate Wagner, said: "We are acutely aware of the significant challenges we face and welcome Ofsted’s feedback, which supports our own findings.
"The inspectors make it clear that we absolutely know what needs to be done and they agree that it will take time to fully implement the changes given the scale of improvement needed.
"We, like Ofsted, want to see an increasing difference made as a result of the actions we have taken. However, we also know that change is not sustainable without the firm foundations we have been building.
"We must, and are already making, our response to risk more robust, and steadily improving consistency.
"We know this is still not good enough and it is a clear priority for us.
"Ofsted identified areas where positive changes had been made, but where these were new and have not had chance to be fully embedded.
"This is encouraging and we should increasingly be able see the positive difference these new arrangements are making to our children and their families."
Local Democracy Reporting Service