Speaking at an event in Sheffield hosted by the campaign group Momentum, Mr Williamson also said he had responded to the resignations of Labour MPs last week by singing the 1980s hit by Kool and the Gang - ‘Celebration’.
The Derby North MP suggested that the singing was likely to have been heard by one of the Labour defectors Joan Ryan, as her parliamentary office is near his.
In a video posted by Sheffield Momentum and passed to The Yorkshire Post, he can be seen telling the meeting: “The party that has done more to stand up to racism is now being demonised as a racist, bigoted party.
“I have got to say I think our party's response has been partly responsible for that because in my opinion… we have backed off too much, we have given too much ground, we have been too apologetic.”
Ms Ryan quit Labour last week following the earlier resignations of seven of her colleagues, to form a new Independent Group of MPs.
Speaking at the time she blamed a “culture of anti-Semitism” within Labour for her departure.
However, a party source insisted that Labour was taking anti-Semitism extremely seriously and was constantly strengthening its internal processes.
Watch the full video here.