Lisa Nandy, tipped as a possible candidate to replace Jeremy Corbyn as leader, added that the phrase "Labour's not for us any more" kept coming up in conversations on the doorstep, as she tries to understand the party's heavy election defeat.
Ms Nandy also said a review set up by Labour Together was adopting the "wrong" approach, as it risked not listening to people such as an ex-miner she had spoken to in Ashfield, Nottinghamshire, where the party lost.
Former leader Ed Miliband is among those involved in the review, which will include interviewing all 59 MPs who lost their seats during the crumbling of Labour's so-called "red wall" of constituencies in the North, the Midlands and Wales.
Ms Nandy said she spent Monday knocking on doors in Ashfield, and found trust in Labour was the "major issue".
The Wigan MP told BBC Radio 4's Today programme: "There was a phrase that kept coming up yesterday that I've heard at home in Wigan as well, which is 'Labour's not for us any more'.
"There's been a lot of talk about the role of Jeremy Corbyn in this election campaign.
"But there was just a general sense that at the top of the Labour Party, that we don't speak for people like them any more, a sense we don't have skin in the game, that we're not rooted in those communities, and we're just not like them, and we don't come very often to just ask people what they think and to listen to what they've got to say.
"We often come in and tell them that we've got all the answers and we can fix it, and what people told us loud and clear in this election campaign is 'you're just not listening'."
Ms Nandy said crime was the "great unspoken issue" of the election campaign, with many communities facing problems with drugs and anti-social behaviour, while police and council resources have been cut by the Tories in government.
On the review involving Mr Miliband, she said she was among those who set up the Labour Together group.
Ms Nandy added: "I have to be honest though, I didn't know anything about this review until two days ago.
"And if the lesson is drawn from this election is, a review can be drawn up in a meeting room in Westminster without any reference to the two parts of the Labour movement - our councillor base and trade union base, that were probably the reason we didn't have a worse result, I just don't think that people are drawing the right lessons at all.
"We need to be out in places like Ashfield, listening to people like the ex-miner I met yesterday, not sitting in meeting rooms in Westminster trying to debate this out amongst ourselves with the help of a few think-tanks."
She added: "I just think the approach is wrong."