The Trades Union Congress (TUC) said 1.2 million people were not getting any paid leave at all, while a million were not getting their minimum paid leave entitlement.
This is equivalent to one in 12 workers, the TUC said.
General Secretary Frances O'Grady said employees not taking holiday "puts them at risk of burnout".
She said: "The government must toughen up enforcement to stop bosses cheating staff out of their leave."
Almost all full-time workers are legally entitled to 28 days leave a year, and an employer can include bank holidays as part of statutory annual leave.
The TUC said workers were missing out on nearly £3 billion of paid leave annually, with employees being set unrealistic workloads and employers deliberately denying holiday requests and managing out people's leave.