The men had been working at the site of offshore company Claxton Engineering in Great Yarmouth, Norfolk.
Building work was being carried out by external contractors at the premises where the accident happened at around 2pm on Friday, the firm said.
The dead men were named locally as Dan Hazelton, 30 his brother Tom, 26, and Peter Johnson, 42, all from Stanton, near Bury St Edmunds, Suffolk, and Adam Taylor, 28, who lived close by.
Reports suggested a steel structure may have collapsed on the men as foundation work was carried out.
Organisers of two local football clubs said yesterday they had cancelled their matches in the wake of the tragedy.
Mike Powles, chairman of Walsham le Willows FC said: "Two of the lads used to play for us. It's a real community where everyone knows everyone else.
"We called the games off as a mark of respect and we understand Stanton have done the same."
The Rev David Messer of All Saints Church in Stanton, said: "It has hit many people in the village quite hard."
A joint inquiry has been launched by Norfolk and Suffolk Major Investigation Team with the Health and Safety Executive.
Further assessment of the site was carried out this weekend by police and health and safety investigators.
Det Chief Insp Neil Luckett from the major investigation team said: "This is a complex investigation and our inquiries – including assessments at the site – are likely to continue for several weeks."
Richard Howitt, Labour MEP for the East of England, said the tragedy was "a deadly example of why plans to lift health and safety protection should be halted".
He called for a freeze on Government plans to amend health and safety laws to make prosecutions more difficult.