The part-nationalised bank, which had set aside £1.9bn to deal with the claims, could be asked by the Federal Housing Finance Agency (FHFA) to pay fines of more than £5bn over the sale of about $32bn of mortgage-backed debt in the United States, the report said.
An RBS spokeswoman declined to comment on the report. The bank agreed to pay a $99.5m fine in June to settle claims it misrepresented more than $2bn of mortgage-backed bonds during the housing bubble between 2005 and 2007.
The federal court in Connecticut is handling the new case which addresses a much larger portion of mortgage-backed securities.
RBS chief executive Ross McEwan said in October that the bank would not pay a dividend until it had strengthened its capital position and had more clarity over future misconduct charges.
Shares in RBS, which is 80 per cent owned by the Government, fell 1.2 per cent in trading.
“We believe that a $5bn settlement would be towards the top-end of market expectations for this particular issue, and slightly above the levels where other banks have settled in relation to the assets involved, but it’s not inconceivable” said RBC Capital Markets banks analyst Leigh Goodwin.
The report, citing unnamed sources familiar with the situation, said RBS’s £1.9bn provision was unlikely to be enough to cover the expected fines, which could be three times larger.
“The FHFA settlement is likely to be the largest of the outstanding litigation and conduct hits that RBS still faces, but there are many others, and they are likely to drag on for many years,” Mr Goodwin added.
“What is clear is that investors need to factor these costs into their earnings and dividend expectations.”
Analysts at Credit Suisse last year estimated European banks would take an $11bn hit from a raft of mortgage-related litigation costs in the United States.
In June the FHFA had totalled about $16.1bn of settlements, stemming from lawsuits it filed in 2011 to recoup losses on roughly $200bn of mortgage-backed securities purchased by Freddie Mac and Fannie Mae.