Writing on the Labour List website, Mr Jarvis sets out how the slow decline of Labour's traditional base was accelerated by Brexit and the party's unpopular leadership.
Barnsley Central MP, who is also the elected Sheffield City Region mayor, saw his majority slashed from 15,546 in 2017 to 3,571 at last week's General Election.
The Brexit Party received more than 11,000 votes while Mr Jarvis got 10,000 fewer than the near 25,000 he received in 2017.
The former paratrooper said Nigel Farage's party came a "worryingly close second" , adding: "Nearly one in three people – in the town still home to the National Union of Mineworkers – supported a political outfit headed by a man who once said he “supported Thatcher’s reforms of the economy”.
He wrote: "They threw the kitchen sink at us – visits from Nigel Farage with national media entourage in tow, a full wraparound our local paper, a plane flying a party banner, a dozen mailings to households, publicity across the borough and countless targeted Facebook ads."
Mr Jarvis said that he won due to the activists who "rolled up their sleeves and eked out the votes, house by house".
He said: "On four occasions, I was confronted by constituents displaying such visceral anger that I was prepared for a physical altercation. I count myself fortunate that it never quite reached that point. They included an ex-miner who said Labour was no longer for the working class and a veteran who said our party’s frontbench supported terrorists."
He added that activists were regularly verbally abused while talking to residents or delivering campaign literature, meaning he wrestled with whether it was fair to ask them to go out and knock on doors.
Though Labour kept both of its Barnsley seats, the town's two MPs saw their majorities slashed along with others in former Labour heartlands while MPs in Don Valley, Wakefield and Rother Valley lost to the Conservatives.
Speaking after the result, Mr Jarvis said: “Too many people expressed anger at a Labour Party which doesn’t speak for them. We have to listen to those concerns and work to regain the trust where it has been lost."
In his LabourList article, the MP said: "I had countless conversations with people who expressed deep concern at a Labour Party that they thought no longer spoke for them. They felt we were out of step with the values of community, patriotism and respect for our armed forces that run deep throughout our town.
"The biggest obstacle faced on the doorstep was undoubtedly our party’s leadership. The disdain voters held was for a plethora of reasons, with virtually nobody believing Jeremy could be trusted to lead our country.
"The second most prominent reason for our collapse was our Brexit stance. We were perceived as a party of Remainers, intent on disrespecting the democratic decision made in 2016. Nearly everyone was exhausted by the delay, uncertainty and debate around a second referendum.
"Thirdly, our manifesto – while full of ambitious individual offers – was viewed by most as unrealistic. That is not to say the level and scale of change on offer is not deliverable but the groundwork preparing people for it was not laid. In the end, it cut through, but in a negative rather than positive way."