Members of the House of Lords Economic Affairs Committee urged senior figures at HMRC to take "aggressive action" to protect contractors who are trying to feed their families.
The Peers held a follow-up evidence session on the loan charge, with Jim Harra, Permanent Secretary and CEO of HMRC and Mary Aiston, the Director of Counter Avoidance.
The loan charge, announced by Government in 2016, was designed to tackle tax avoidance schemes where individuals receive income in the form of loans that are not repaid to avoid income tax.
Members of the committee were concerned about workers on modest incomes who were unwittingly being drawn into disguised remuneration schemes by umbrella companies.
During the session, Baroness Kramer,, said: “I'm also concerned about the employees because these abusive and rogue companies are undoubtedly putting those contractors at risk.
"I would like to know if you are going to tackle this by going after the umbrella companies or whether you intend to repeat the pattern of the loan charge and go after the individual contractors who have been caught up in these schemes, as probably one of the few opportunities they have to get work."
Mr Harra, said that, in terms of overall approach to umbrella companies, they do have tax obligations towards HMRC as the employer.
He added: “We do take steps to make sure that to, the extent an umbrella company poses risks to the tax system, that we tackle that.
“We look forward to the regulatory framework for that being strengthened, which the Government has said it will do, because that will help us in our task
“In the meantime, our approach to workers is to equip workers to understand whether they are being abused or misled and to enable them to contact us if they have got any concerns."
Mr Harra said HMRC had recently made a number of arrests in relation to mini-umbrella companies, which are often "fraudulent schemes".
Baroness Kramer added: “I’m slightly troubled by what I understand to be your fundamental approach, which is to say the individual, who may well be a cleaner or somebody working for £10 an hour in a test centre...should have the sophistication to be able to understand the complexity of all of these issues, which even HMRC struggles to grapple with, and if they get it wrong, they are there for the high jump.
“I was hoping to hear something much more aggressive in terms of providing protection for contractors who are finding themselves in this position because, thanks to the change in off-payroll working, there really is no other practical way in which they can actually put food on the table.”
Mr Harra said: “The vast majority of the yield that we have collected from tackling disguised remuneration has come from us tackling employers, that’s what the umbrella companies are..
“We have got a multi-pronged approach to this. Our approach to workers is to equip them, to make sure that they know what questions they should be asking, to protect themselves, when they engage with umbrella companies and also to give them routes to get advice and assistance from us if they feel they have been misled into something they shouldn’t be in and need to get out of."
He said HMRC had had hundreds of contacts from people asking for help to ensure they don’t get tangled up "in their tax affairs going wrong".
He added: "We are working very hard to make people alert and also to identify as quickly as we possibly can if anyone has got involved in a scheme that is going to cause them trouble.
“I cannot absolve taxpayers from the responsibility for keeping their tax affairs in order, but we do go after the employers as the first port of call if we possibly can.”
Lord Forsyth, the committee's chairman, said; “I’m getting the impression that what you’re saying is, ‘Look, all we’re interested in, is getting the money that’s due, and the fact that some people may have suffered hardship is down to them because they ought to have known better.
“Looking at some of the examples we have had: for example, of a social worker, who is told her job has disappeared on a Friday, but told she can come back as a contractor, and then years later, after she’s retired, gets an enormous bill from you.
“Isn’t there an issue of fairness here?..We just get the impression that the little people who get smashed by it are incidental and you are not taking any responsibility for the consequences of your own actions.”
Mr Harra rejected this claim, adding: “I absolutely agree with the area of focus...We recognise that the market has changed and that today the promoters really are targeting employment-based schemes and middle earners and increasingly moving down the earnings range and therefore those people are less equipped to get independent advice and understand what they need to do.. and also if they are presented with a big tax bill.. that is going to be a life-changing event for them and potentially distressing for them.
“We absolutely recognise that and we have been changing our strategy towards tax avoidance to reflect that."
He said HMRC was trying to identify where people get caught up in avoidance as early as it possibly can and "help them to get out of it".