McCarthy & Stone wants to build 51 apartments on the site of Westwood Park care home, on Langholm Close, which closed five years ago, and is attracting anti-social behaviour and vandalism.
East Riding Council's eastern area planning sub committee had deferred a decision for a site visit.
Ward councillor Linda Johnson told a meeting on Monday she was concerned about a building which at three storeys high would be taller than anything else built on the edge of the Hurn or the Westwood, with Juliet balconies "which will look out of place".
Although people were "desperate" for the site to be cleared, because of the anti-social behaviour it attracts, she said residents felt it was "too much".
An agent representing neighbours said the numbers in the new development would increase the local population by three times.
He said: "They accept that it's an appropriate location and the site needs to be redeveloped, but they can't accept the size of the development that causes harm to the conservation area, dominates and overlooks existing dwellings."
However a resident spoke in favour saying she would hate to see the site developed for million pound houses and said she thought McCarthy & Stone would do a "good job".
She said the site was originally a council-owned care home, and following that again a care home and traffic "would be exactly the same".
She added: "It would be ideal for independent living for elderly people to walk into town, it's not far from amenities and services."
Councillors voted in favour however after hearing that the developers wanted to be a "good neighbour".
Councillor Denis Healy, who was unable to vote as he was at home isolating, expressed the concerns of neighbours, but said he'd spoken to McCarthy & Stone and they were "very positive about engaging with residents".
He said he and Coun Johnson would be happy as ward councillors to facilitate dialogue with the developers.
He said he would be writing to residents and McCarthy & Stone about setting up a monthly meeting.
The plans have been deferred for the signing of a legal agreement and will then be approved.