Which? found more than three quarters of consumers in the two lowest household income groups rely on cash the most, while a quarter of them never use card payments.
Some four in five retirees are reliant on cash, using it at least two to three times a week, a survey by the watchdog found.
However 98 per cent of the general population used cash, with 73 per cent saying they used it frequently and just five per cent saying they rarely used it.
While just over half of people said they were not worried about moving towards a cashless society, more than two in five said they did have concerns.
Of those who were concerned, more than half were worried about the increased scope for scams with credit and debit cards and 46 per cent had concerns about the reliability of electronic payment systems.
The survey also found six in 10 of respondents had experienced a time when they could only pay by cash in the last three months, with 82 per cent of them saying this was because cash was the only method of payment accepted by the vendor.
Cash machine network Link, which has been at the centre of a row about the future of ATMs, announced earlier this year it was funding a review into how the shift from cash to digital payments was affecting people and small businesses.
The Link scheme is a national network connecting ATMs. A row previously broke out over the future funding of ATMs, sparking fears more people could struggle to access cash at a time when many bank branches are also closing.
Which? said the equivalent of 250 free-to-use cashpoints were closing every month despite Link’s Financial Inclusion Programme that promised to protect access to cash. It is calling for the Payment Systems Regulator (PSR) to review the programme and for the Government to place a statutory duty on the PSR to monitor and protect access to cash.
Which? Money spokesman Gareth Shaw said: “Widespread ATM and branch closures are threatening to leave behind the millions who still rely on cash and our research highlights lower income families and older generations will be among the worst hit. Clearly, Link’s commitment to protect access to cash in vulnerable communities is failing. It is critical the regulator now intervenes to ensure consumers are not at risk of financial exclusion.”