A boom in barn conversions means that rural towns and villages are missing out on vital infrastructure and local services such as roads, schools and GP surgeries, according to councils.
The Local Government Association (LGA) said the number of agricultural to residential conversions in England has increased by nearly 230 per cent - from 226 in 2015/16 to 743 in 2017/18.
It said that conversions, mostly in rural areas, have been taking place under permitted development rights, which allow developers to "bypass" the planning system.
Councils argued that they are unable to make sure homes are built to high standards with the necessary infrastructure in place.
The LGA, which represents councils in England and Wales, said it is concerned that rural areas in particular are not being provided with the local services or infrastructure, and in some places affordable housing, that would normally be required in any development going through the planning system.
Yorkshire had 36 barn conversions in the last financial year, each of which could be turned into up to three dwellings without planning permission.
These kinds of developments are increasing quickly - for example, barn conversions in North Yorkshire have more than doubled every year, increasing from two in 2015/16, to seven in 2016/17 to 19 in 2017/18.
Permitted development rights have been seen as one solution to the housing crisis allowing developers to create new homes quickly without red tape.
However, councils have much less control over where houses are built, which creates problems for planners.
David Renard, the LGA's planning spokesman, said: "We have concerns over the sharp rise in agricultural buildings being converted into homes without planning permission and the impact this is having on rural areas."
He continued: "Permitted development rules are denying councils and communities any control or oversight of this process.
"It is taking away the voice of local residents who need to be allowed to have their say over any developments in the streets and neighbourhoods where they live.
"Unless permitted development rules are scrapped, then communities face the risk of substandard housing without any of the vital supporting infrastructure and local services which residents need on a daily basis."
Other counties that also saw large increases in agricultural to residential conversions in the past year include Devon, Kent, Worcestershire, Herefordshire and Staffordshire.
Homes built under permitted development rights still need to comply with building regulations, which are administered by the local authority. Developers also need to take into consideration other planning issues, like ensuring the development meets listed building rules and has the right environmental permits.