A public consultation has started on whether or not Pontefract Hospital's Friarwood Birth Centre will remain open.
Wakefield Clinical Commissioning Group (CCG), which is leading the consultation, has insisted that no final decision on the unit has been made, and it to hear from people who have used the service recently, or are likely to in the near future.
But at a health scrutiny meeting on Thursday, councillors expressed concern about the impact any closure would have on the local health service, with one member describing it as "insanity".
And the committee's deputy chair, Councillor Steve Tulley, said that if health bosses decide the maternity unit should be closed, elected members may ask the government to take over the process.
As the current health secretary, the final decision may rest with Matt Hancock.
Coun Tulley said: "As a committee, it's possible we will exercise our right to have this sent up to the Secretary of State.
"I know this is difficult for the CCG, but we don't want this to be to the detriment of people living in our part of the world, especially when we've got pressures across Wakefield.
"We've got to be mindful of the perception that we're losing services, and the impact that has."
Representatives of the CCG, who were at the meeting, were criticised by several councillors for the way the process has been conducted so far.
At the committee's last meeting before Christmas, MP Yvette Cooper raised several points in opposition to closing Friarwood, which members claimed had not been responded to.
Looking at papers which have been sent to NHS England outlining the case for closure, Coun Kevin Swift said: "That meeting on November 22 might as well never have happened.
"There is no evidence at all of any of the points that were raised having been taken into account, or the CCG's plans having altered."
But the CCG's director for corporate affairs, Ruth Unwin, stressed that the public's views would be taken into account and the fate for the unit was a long way from being determined.
She said: "It's important we make clear that this is just the start of the process. No final decision has been made.
"It's still all to play for.
"We want to hear from people who are likely to have a baby in the near future and those who've recently have one.
"That's what we're working through at the moment and that was the very strong advice we had from NHS England on how to go about this.
"After whatever decision is taken there will be an assurance process after that."