The party was in no danger of losing its large majority in the district, but has seen its number of representatives shrink slightly to 49.
They lost seats in Knottingley to the Liberal Democrats, who saw a councillor elected to the authority for the first time since 2011, and to independent candidates in Hemsworth and Airedale and Ferry Fryston.
Long-serving councillors Glenn Burton, Yvonne Crewe and Glyn Lloyd were the casualties.
The Conservatives held the four seats they were defending, but narrowly failed to defeat Labour in a handful of key marginals.
In response to the results, Coun Box said: "It has been a disappointing night for us, without doubt.
"I believe it's down to Brexit. The truth of the matter is that no matter how good a local authority thinks it is, national politics will always play its part in local elections.
"Hopefully by next year, Brexit will be finished and we can talk about local issues a little bit more than we have this year."
He added: "Politics is an unforgiving business. All of those who've lost tonight I class as friends, and it's a great shame."
The Liberal Democrats sprung a major surprise in Knottingley, which has been Labour territory for decades.
Tom Gordon, 25, took the seat with a majority of more than 1,200.
Mr Gordon said: "I grew up in this town and I never thought Knottingley would ever elect any other party than the red rosette.
"Tonight all that has changed. "
The Conservatives now hold 11 of the council's 63 seats, with independent candidates having two, and the Liberal Democrats one. Labour have the remaining 49.
The turnout was 26.98 per cent, only marginally higher than last year's local elections in Wakefield.
Local Democracy Reporting Service