Police, Fire and Crime Commissioner Mr Allott made comments in an interview with BBC Radio York earlier this month where he called on women to be “streetwise” and said Ms Everard should never have submitted to her arrest by serving Met Police Officer Wayne Couzens.
Ms Everard, 33, was kidnapped by Couzens on her walk home in Clapham, south London. He then raped and murdered her. Last month he was sentenced to life in prison.
Mr Allott appeared before the Police and Crime Panel on Thursday to explain his remarks.
The panel is made up of councillors from across York and North Yorkshire, and has a role in holding the PFCC to account, but has no legal powers to strip him of his position or force his resignation.
Over 1000 official complaints against Mr Allott’s remarks have been registered by both his office and the panel.
Council members, including several from Mr Allott’s own Conservative party, queued in the meeting to implore him to resign.
When Mr Allott doubled down on his refusal to resign, a motion of no confidence in him was called by York councillor Darryl Smalley.
The motion passed with 11 votes for the expression of no confidence, and none against.
Before the vote of no confidence, Allott re-iterated an apology to the panel for his remarks, but said he was confident he could “rebuild trust” from the public.
He said: “I can see that tensions are running high within two weeks of that interview.
“If everyone resigned who made a mistake in an interview, nothing would ever get done in the country.
“The issue is whether I can regain or not. I genuinely think I can regain trust.”
Panel leader Carl Les, who is also the leader of North Yorkshire County Council, said: “I don't see how it addresses the issue, the volume, the anger, the hurt and the fears raised by your remarks by your apologies or your retraction.
“The overall view by a big margin is we have lost confidence in you.
“Only you can judge the damage done to your position, and only you can resign, we cannot make you. We can only make recommendations and there’s a frustration with that.”