The local authority's 11 Tory councillors will put forward a motion at next week's council meeting, requesting Coun Box resign "immediately".
Coun Box, who has led the authority since 1998, has been under pressure since Wakefield's children's services was rated inadequate by Ofsted in July, and told it was leaving vulnerable youngsters at risk.
An unsuccessful coup by members of his own party was seen off in September, but Coun Box has faced sustained pressure on his leadership ever since.
The Conservative group has not called on him to stand down at any point during the saga, until now.
Speaking on Tuesday, Tory leader Nadeem Ahmed said that he would have continued to work with Coun Box to turn children's services around had Labour been united.
But he said that the party's divisions were now undermining the council's efforts to remedy the problem and that the leader's "time has come".
Coun Ahmed said: "Initially we saw this row as an internal matter for the Labour party, and not something for us to get involved in.
"I didn't say anything at the last council meeting because I saw it as a party matter. But now the situation has come to a head.
"There's too much internal conflict now for Peter Box to continue. The party is split into two camps.
"In any line of business you need people to be united otherwise you're not going to be productive.
"This is now getting in the way of council business and it's now going to be a lot more difficult to turn around children's services, which is their achilles heel at the moment."
Coun Ahmed cited the council's problems with recruiting extra social workers and said that political strife would discourage good applicants from coming forward.
He also claimed that Labour rebels wanting a change of leadership had approached him privately, but said he was unsure how many may support him in voting in favour of the motion on October 24.
"If you're a social worker, it's going to be easier to work in Leeds or Kirklees at the moment," he said.
"At moments like this, your job as a leader is to hold onto high calibre people and that's so important now.
"In politics everyone's time comes. Mine will come eventually. It's his time now.
"I will continue to support the chief executive and the social workers, and I'll support a leader that supports that. I won't support incompetence."
The Labour group declined to comment on the contents of the motion.
The motion to be debated at full council, put forward by Coun Ahmed, in full:
This Council notes: The damning OFSTED report into Wakefield Council’s children’s services department, that has confirmed serious and widespread failures that have left vulnerable young people at risk.
A Commissioner is now deciding whether the government should take full control of the department.
The lack of political leadership and direction from the leader of the council.
The growing number of councillors from the controlling Labour Group, who are resigning from key positions in Coun Box’s administration, confirming publicly their collective lack of confidence in him as leader of the council.
This council further notes: The continuing embarrassment to the council, both locally and nationally of this ongoing political saga.
That those vulnerable young people who are still seriously at risk, are not being best served or represented during this crisis.
That this ongoing instability has led to the paralysis of proper leadership and governance at Wakefield Council.
That this council: has no confidence in the leader of the council, calls on him to resign immediately, in the best interests of those vulnerable young people at risk that he has so badly let down.