The dead men are all originally from Gambia, and some are Spanish nationals, who emergency services said were working when a 15ft (4.5m) concrete bay wall collapsed on them.
Blocks weighing about one-and-a-half tons each came down, along with tons of scrap metal being stored in an outside yard at the vast recycling site in a heavily industrialised part of Birmingham.
All five workers were pronounced dead at the scene following the incident at Hawkeswood Metal Recycling in the Nechells area of the city at about 8.40am.
The victims have been named locally as Saibo Saillah, aged 42, Ousman Jabbie, Mohammed Jangana, as well as Alimamo Jammeh and Bangaly Dukureh, who were both from Aston, in Birmingham.
All the men were married with young families; Mr Saillah had three-year-old twins, and Mr Jangana had a baby.
Friends of Mr Jammeh also revealed his wife and children were due to arrive in the UK on Sunday and had not yet been told of the tragedy.
Members of the community said they were hard workers on minimum wage who had been employed through a recruitment agency.
Prime Minister David Cameron has spoken of his shock at the tragedy, and said: “My thoughts are with the families of those involved.”
Prayers have been said at local mosques while the president of the city’s Gambian Association based in Ladywood said the community had been left “devastated” by the deaths.
Ansumana Barrow, 63, who works in Brierley Hill, West Midlands, said the association’s would be meeting to make sure the bereaved families - many with young children - get whatever help they need.
A sixth man injured in the accident is currently in hospital after suffering a leg injury, although it is not thought to be life-threatening.
Friends of the wounded man said he rang them from hospital with news of the deaths.
In a briefing at the sprawling site, which is bordered by a railway line, Detective Superintendent Mark Payne, of West Midlands Police, said: “We believe the victims are foreign nationals. We believe the men are from Africa.”
During a press conference at the site, Mr Payne said a joint Health and Safety Executive (HSE) and police investigation was now being carried out to establish the cause - warning that the process could take weeks.
He added one line of of inquiry would be to find out whether “any issues of negligence or malpractice” may have contributed to the collapse.
Mr Payne said: “We’re simply trying to recover the bodies of the men and do it in a way which will help understand exactly how that wall came to fall down.
“Clearly we are investigating together with the HSE, whether there are any issues of negligence or malpractice that have contributed to that wall falling down.”
Emergency services have accounted for all staff on site, and said there was no prospect of any survivors under the many tons of rubble.
It is hoped the bodies will be recovered overnight from the “extremely challenging” scene.
Mr Payne said: “It’s a wall constructed of blocks that weigh about 1.5 tons each - concrete blocks.
“That wall was supporting a body of scrap metal.
“It appears that that wall has collapsed on top of the men and then the scrap metal behind the wall has fallen on top.”
He added that the men had been working at the site which employs dozens of staff at the time they were killed.
Mr Payne said the incident went rapidly from being a rescue situation to a recovery operation, which was “likely to take the rest of the day”.
Manka Sawo, who was among about 30 members of the community gathered at the site’s entrance for news, said the surviving victim had rung to tell him what happened.
The news spread quickly through the tight-knit community, with friends of the dead men telling how the night before they had feasted to celebrate Eid al-Fitr.
He added: “I knew one of the men very closely; yesterday we celebrated Eid together. It’s very, very sad.”
Mohammed Kamarah, community leader, said he had visited the home of Mr Jangana family, and described how the man left behind a wife and baby.
Landlord Lang Dampha, 48, from Aston, said Mr Jabbie had been lodging with him a week after coming from Spain and had immediately got a job at the recycling centre, through an agency.
In an emotional tribute to his other friend and lodger Mr Jammeh, he said: “He is a perfect gentleman, down to earth, loves his work.
“He’s kind and generous, and always smiling.”
He said Mr Jammah was married with children, who before the accident had arranged to join him with plans to travel to the UK on Sunday.
Mr Dampha said: “I was working and I got a call, and they said ‘five of our guys have died’. I can’t believe it.”
Another man Kabiro Daboe, 40, from London, said Mr Jammeh had watched the Wales v Portugal game with him yesterday, before driving his friend to mosque.
Community leader Mr Barrow said Mr Jammeh had been at his house after he got back from prayers marking the end of Ramadan.
He found out about the deaths when one of Mr Jammeh’s relatives rang to tell him.
Mr Barrow said: “It is so, so sad. We are devastated.”
He added the focus would now turn to supporting the men’s families.
Mr Barrow went on: “We’re a hard-working community, and these men just came here to feed their families.
“It’s going to be very hard for those families now - that’s the problem.
“As a community now we will meet and see how best to gather the people, to help them.”
The industrial accident left rescue crews facing a “significant tonnage” of concrete and metal, which partially trapped the sixth victim before he managed to escape from the rubble.
A West Midlands Ambulance Service spokesman said it became apparent that on arrival nothing could be done to save the five other victims.
The injured man was later taken to hospital by air ambulance.
West Midlands Fire Service assistant chief fire officer Gary Taylor said: “Once the police have finished their initial examination of the scene, our absolute priority will be to ensure that the bodies of those who have lost their lives are recovered in the most timely and safe way possible, and with the utmost dignity and respect.”
A spokeswoman for the Health and Safety Executive said: “The HSE is aware and supporting the police in its investigations into the incident.”