Stocksbridge, Brighouse, Todmorden and Morley, all of which were in battleground seats at last year's General Election, were among the 16 Yorkshire towns invited to bid for up to £25m from the Towns Fund last year.
But a report by the National Audit Office that they were among the towns selected for the £3.6bn scheme despite getting a low score on the criteria used by the Ministry of Housing, Communities and Local Government (MHCLG) to decide which areas most needed the money.
Labour says there "are now serious concerns that ministers may have allocated funding for political gain" at the 2019 General Election, where the Conservatives won a swathe of Labour seats across Yorkshire.
And a Yorkshire MP representing a town that missed out on the funding said it showed that "the Tories have been playing political games to help their mates instead".
But the Government denies claims that nine out of 10 towns were ruled out of the competition for funding with no explanation. A spokesman said: "There were many factors to consider in the selection of towns and we are confident the process we took was comprehensive, robust and fair."
In total 101 towns across England were chosen to bid for up to £25m from the Towns Fund last September, with the amount of money ultimately awarded depending on the strength of investment plans drawn up locally.
The Government says the funding will help local leaders "transform their town’s economic growth prospects with a focus on improved transport, broadband connectivity, skills and culture".
The Yorkshire towns of Castleford, Dewsbury, Doncaster, Goldthorpe, Keighley, Rotherham, Scarborough and Stainforth were selected automatically after being graded as high priority by MHCLG.
Brighouse, Morley, Stocksbridge and Todmorden were chosen despite being rated as 'low-priority' and Goole, Shipley, Wakefield and Whitby made the list after being ranked as 'medium-priority'.
The NAO report shows 61 of the towns were chosen at the discretion of ministers led by Robert Jenrick, the housing and communities secretary. An analysis shows that all but one of them were either Conservative-held seats or Tory targets before the election.
In Morley, Conservative Andrea Jenkyns was defending a 2,104 majority from Labour, while in Stocksbridge, Tory Miriam Cates was looking to overturn a 1,322 Lib Dem majority. Brighouse and Todmorden both fall in the constituency of Calder Valley, where Conservative Craig Whittaker was defending a 609 majority from Labour.
The NAO report sets out why Ministers says each of the four was chosen despite meeting few of the MHCLG criteria.
Brighouse was said to have "recently suffered an economic shock when a manufacturer went into administration".
According to the report: "The company had a longstanding heritage in the local community and 313 local jobs were lost as a result of the company folding.
"The town scores in the bottom 28th percentile for productivity. The town and surrounding area have struggled to unlock its potential."
Morley in West Yorkshire was said to be "looking to invest in its transport hubs and is seen as an area with investment opportunities". The report adds: "The wider area around the town also suffers from low productivity and is at risk of potential economic ‘shocks’."
Stocksbridge in South Yorkshire is described as having an economy "dominated by the steel sector, which has experienced various periods of growth and decline over past decades".
The report says: "However, the current challenges facing the steel industry stand Stocksbridge out as a potentially vulnerable economy. Added to this, the town displays low household incomes and high levels of deprivation.
"The town has been identified by Sheffield City Region and City Council as a “priority” and the City Council has submitted a Future High Streets Fund bid in the past."
Todmorden in West Yorkshire was said to have "severe pockets of deprivation" and be "economically reliant on heavy industry, heavily reliant on the cotton spinning and weaving industry".
The report said: "Its industrial base is now much reduced and primarily operates as a commuter town for people working in surrounding cities including Manchester, Leeds and Bradford.
"There are regeneration opportunities for Todmorden which may help it build an economic base of its own. Affordable housing is a significant issue with limited land available for building."
Among the towns which failed to be selected despite scoring higher were Barnsley and Wombwell in South Yorkshire and Halifax and Knottingley in West Yorkshire.
Yvette Cooper, MP for Normanton, Pontefract and Castleford, which includes Knottingley, said: “Instead of investing in the towns that need it most, it turns out the Tories have been playing political games to help their mates instead.
"I’ve been calling for proper investment in our towns and for an end to austerity that has destroyed so many of our local services for years, and I’ve been to see several Tory Ministers to call on them to include Knottingley in the list, but we were rejected last year.
"Now this independent report from the National Audit Office shows that Knottingley was deliberately left out by Ministers even though it scored highest in the criteria for selection, and instead they gave the money to places which didn't have as high a need for investment to boost support for Tory MPs. Using public money for political games like this is appalling. We need a proper plan for investment in our towns, not just a plan for Tory MPs”.
The report stated: "Officials acknowledged that the scoring of each town was designed as a guide for ministers and was not the only way to assess eligibility.
"Officials recognised that some towns would be in similar situations, and a degree of qualitative judgement between picking towns with similar characteristics was inevitable.
"Ministers' selections resulted in towns being selected with lower scores than some other towns that were not selected.
"Officials concluded that the overall selection was acceptable because ministers had selected all 40 high-priority towns and provided a rationale for each of the towns selected from the medium- and low-priority groups."
Steve Reed MP, Shadow Communities Secretary, said: “Government ministers shrouded the town deals scheme in mystery but this report shines a light on how funding was allocated, with many deprived communities losing out.
“There are now serious concerns that ministers may have allocated funding for political gain at the 2019 election, something which breaks strict rules on impartiality.
“The Secretary of State must explain as a matter of urgency how ministers decided where to spend this money, why impartial officials were not involved, and why so many communities lost out. The public will not tolerate gerrymandering of public money for party political gain."
Chairwoman of the Commons Public Accounts Committee (PAC) Meg Hillier, a Labour MP, said: "This NAO report shows that some of the most deprived towns in England will be left behind once again.
"Nine out of 10 towns were ruled out with no explanation before they even reached the competition's starting line, while some relatively affluent towns are still in the running.
"Ministers relied on flimsy, cherry-picked evidence to choose the lucky towns. Those that lost out have not yet had the chance to make their case."
An MHCLG spokeswoman said: "It is completely untrue to suggest that nine out of 10 towns were ruled out with no explanation. This claim completely ignores the NAOs' detailed report, which shows that the Government put in place a robust process to identify towns for Town Deals.
"Furthermore, the report shows that the more affluent half of towns were ruled out, and the 40 most deprived towns were rightly favoured, with the remainder selected from a shortlist considering of a wide range of evidence.
"There were many factors to consider in the selection of towns and we are confident the process we took was comprehensive, robust and fair. We will deliver on our commitment to level up this country."