Protesters who have gathered today (Saturday) in Full Sutton, in Yorkshire's East Riding, fear the plan by the Ministry of justice would harm the beauty and safety of their village.
There are also concerns the new prison could increase traffic and potentially drive up drug use in the area.
The village, which was recorded as having a population of 1,072 in the 2011 census, is already home to HMP FullSutton, a maximum security facility which in March 2018 was estimated to have a capacity of 558 male prisoners.
In July 2017, the MoJ was granted outline planning permission for a new 1,017-capacity prison next to the existing jail.
Since then, a fresh application has been made to increase the size of the proposed prison to 59,500sq metres in order to house 1,440 adult male inmates, sparking protests from residents.
Around 100 of them gathered in the nearby village of Stamford Bridge on Saturday, singing chants and holding up banners reading: "No to mega prison."
One protester, 70-year-old Colin Clarke, who has lived in the area for more 20 years, said of the proposed jail: "Structurally, it's going to spoil the village. It's going to make it a place that's less desirable for people to want to live.
"Besides all that, the traffic impact of having more than 500 staff working there, and visitors, is going to impact on the local road network.
"It's really a terrible situation, and that's the basis of objecting to it."
Mr Clarke has compiled a list of 16 objections to the proposal, writing at the bottom of the document: "Don't turn Full Sutton into a mega prison village."
Another concerned resident at Saturday's protest was Fiona Roberts, 61, who said: "There is a big problem with drug use in prisons, particularly in mega prisons like Wrexham, and the drugs have got to get into the prison somehow if they're going to be used.
"That will attract drug dealers, because the drugs can be lobbed over the fence, or flown in with drones."
In its planning statement to the East Riding of Yorkshire Council, the MoJ said investment in the proposed facility would "provide a significant boost to the local economy" and create 389 jobs for people "from the local area".
It adds: "The proposed new prison will provide greater opportunity for offenders from the region to be held closer to their home areas, which is an important factor in reducing re-offending rates.
"The decision to promote a new prison at Full Sutton is based on a national priority of promoting modernisation and efficiency in the operational prison estate."
A spokesman for the MoJ said of the protest: "We continue to engage with local stakeholders and the community regarding our plans for the new prison and have submitted a planning application to the East Riding of Yorkshire Council.
"This new prison is part of a significant investment across England and Wales to modernise prisons, it will boost the local economy and create jobs."