Speaking to BBC Radio 4’s Today programme Mr Mann said “vast numbers of Labour voters are in danger of walking away” if the party positions itself as supporting remaining in the EU in any upcoming General Election.
The 59-year-old, who announced on Sunday he would be standing down from the Commons after 18 years representing Bassetlaw, said: “That’s not a good recipe for winning a General Election and in the North of England, Labour will pay a huge price.”
Mr Mann, who is set to take up a full-time post as the Government's anti-Semitism tsar, said: “Every Labour MP, every Labour activist in the North and the Midlands, knocking on doors listening to the voters, is hearing what is going on.
“The Labour voters who voted Brexit in vast numbers in the North and the Midlands reject the direction that Jeremy Corbyn is taking the Labour Party and the anger is building up - and it’s building up very rapidly.”
Mr Mann said a “huge swathe of voters” would feel deserted by the party if Mr Corbyn backed the 48 per cent of voters who opted to remain in the EU in the 2016 referendum.
He rejected the suggestion that the majority of voters in Labour heartlands had voted to remain.
He said: “In my constituency, in the most Labour wards, it was the most Brexit.
“The vast majority in the Labour heartlands voted Brexit. And they are increasingly despairing seeing what’s going on in Parliament and the role that the Labour Party is playing, and the fact the Labour Party won’t even have an election, frankly they’re scratching their heads and wondering what is going on and what is the Labour Party for?”
Mr Mann previously said he could not campaign for Mr Corbyn knowing he could become Prime Minister, and told The Sunday Times he would "never forgive" his left-wing leader for allowing the party to be "hijacked" by anti-Semites.