Sally Bercow said many Conservative MPs still could not accept he won the contest to take the Speaker's chair when Michael Martin was forced out in 2009.
She branded rows between Tory figures and Mr Bercow "ridiculous" and insisted her husband was doing a good job, adding: "Sod the whispering campaign".
In the latest bust-up senior Conservative backbencher Mark Pritchard swore at the Speaker in a behind-the-scenes confrontation in the Commons.
Mr Pritchard reportedly told Mr Bercow "you are not f****** royalty, Mr Speaker" during a confrontation in a Westminster corridor after the MP, who was refused permission to ask a Commons question, would not stand aside to let him pass by.
The MP later said he regretted using "colourful language" but not the fact he had taken a stand against the Speaker.
Mrs Bercow said: "The bottom line is a lot of Tories, I'm afraid, still can't accept that my husband won the speakership fair and square and is doing a very good job and he did it without Tory support.
"He only had 20-odd Tory MPs backing him and as a result they have got no hold over him and they don't like it.
"There will always be a whispering campaign but if you look at John and what he has done in the job, he has introduced a lot of urgent questions, basically he is on the side of backbenchers.
"He is not on the side of Labour MPs, he is not on the side of Tory MPs, he is on the side of backbenchers and that is what he is trying to do." She added: "I say sod the whispering campaign."
Mr Bercow was a Tory MP before becoming Speaker, but was viewed as a maverick by many of his colleagues in the party. He was elected Speaker with strong Labour and Liberal Democrat support.