Mr Jarvis, who recently announced he will not stand for a second term as the South Yorkshire Mayor, said he is “incredibly grateful” to have taken the decision to stand down from the mayoralty himself, but also detailed incidents on the constituency campaign trail, where he feared being “involved in a physical altercation.”
The MP for Barnsley Central told a Brighton fringe event: “In the end, the big fight will be about who's going to secure that the next government.
“I want to do my bit from parliament, supporting the leader hopefully on the front bench, if he offers me a job, that will be for him.”
The former Army officer has served in the Commons since 2011 when he took his seat in a by-election, but remembers the 2019 General Election campaign as “grim” and “tough”, amid frustrations from the public that saw him labelled a “traitor”.
Seats across Yorkshire and the North of England changed hands from Labour to the Conservative over that campaign, with some voters backing the Tories for the first time in their lives, as the Red Wall turned blue.
Remembering one incident where he was confronted by a “big burly bloke” in his slippers, Mr Jarvis said of the 2019 campaign: “I can recall that at least four occasions in the Barnsley Central constituency, where I thought that I would basically be involved in a physical altercation, because there were numerous occasions where very angry people came out and confronted me.
“I remember on one occasion, I just stuck a leaflet through somebody's letterbox, and then walked away down the drive, and this big burly bloke came knocking it down his driveway, in his slippers - I was a bit relieved that he was in his slippers - that gave me something of advantage to get to continue my flight.
“But he basically sort of squared up to me, called a traitor, said I'd let him down.
“He said that he traditionally voted Labour, as has his family, but would be ashamed to vote Labour.
“It wasn't that he wouldn't vote Labour, but he couldn't bring himself to vote for Labour”.
As traditional Labour seats swapped to the Conservatives with a swell of support for Prime Minister Boris Johnson, Mr Jarvis says he thinks Labour needs to work to show that traditional Labour voters “can place their faith and trust” back in the party.
He explained: “In the end, it's about trust. It's about people in those Red Wall, heartland seats, thinking that they can place their faith and trust in us, thinking that our instincts and our values are in line with their own.
“And when it comes to really important decisions, like national security, like being patriotic in terms of support for the armed forces, these are the kind of things that my constituents and those heartland communities really take seriously and really believe in, and we can never afford to be in a situation where they think that we're questionable when it comes to those particular issues.”