Lord Storey, the party’s education spokesman, said the Deputy Prime Minister is “not a popular person” among voters in his home city of Liverpool.
The peer stressed he found it “very difficult” as Mr Clegg is a “nice guy” with principles who passionately believes in what he is doing.
The comments will heap further pressure on the Lib Dem leader who has faced disastrous local and European election results as well as a bid to undermine his leadership from peer Lord Oakeshott.
Lord Storey also admitted the party had not just lost councillors in last month’s local elections but entire political machines, including fundraisers and organisers.
He told BBC One’s Sunday Politics: “Knocking on doors in Liverpool, I have to tell you that Nick Clegg is not a popular person.
“And some might use the word toxic.
“I find this very, very difficult because I know Nick very well and I see a principled person who passionately believes in what he’s doing, and he’s a nice guy.”
He went on: “In parts of the country, take Liverpool as an example, where we had 60 plus councillors, we’re down to three.”