The Secretary of State will have the final say on the future of Norwood House, following a meeting at County Hall on Monday.
Campaigners had hoped the former school, described as one of the best Georgian buildings in the country, would be turned into a national museum and community venue.
Instead it will be turned into seven self-contained office suites, with 25 parking spaces.
Coun Kate Gray was the only one to vote against the move. She said: "I can understand why they need a tenant but what I do find sad is the fact the three main rooms, the drawing room, the dining room and the library, really quite important rooms, will not now be available for the public to see.
"Also every day on Norwood, there are many car parking spaces available and I think it's short-sighted to destroy the curtilage of a listed building, simply for more parking when there's other spaces available.
"It's ironic that we are celebrating the rich history of Beverley with a new heritage trail and this wonderful example of Georgian architecture is taken out of the public domain."
Conservationists, including the Beverley and District Civic Society claimed the proposals could cause "substantial and irreversible damage to the fabric and character of the building and its immediate landscape environment."
However English Heritage backed the proposal, which will bring the building, which has been empty for years and is on the buildings at risk register, back into use. It has been damaged by a fire and wet and dry rot. The decision to sell Norwood House was taken last March by the council's cabinet.