Leeds City Councillors discussed traffic and parking arrangements for the event at a Licensing Committee meeting in the Civic Hall on Tuesday.
Traditionally, taxis were able to pick up festival-goers from the same entrance buses used, known as the yellow gate, meaning they were able to access the site more easily.
But plans introduced last year mean taxis now have to access the site via the public entrance, known as the red gate, and face heavier traffic.
Drivers say this leads to delays of well over an hour, and have warned they may not attend as it simply wouldn't be worth their time.
Speaking after the meeting, Mike Utting, from the Leeds Hackney Carriage Association, said: "We have a living to earn, and we don't want to be stuck in a traffic jam for three hours.
"We are joining the queue with the public - they have the time to waste, but we don't. We are not happy about it at all."
The drivers say that between 2008 and 2016, taxis were able to use the yellow gate, which was designated for public transport. This was accessible via the A1.
But from 2017, taxis and private hire vehicles were made to access the site via the red gate, which was only accessible via the A64 York Road.
Javid Akhtar, another taxi driver, said: "We are supposed to be 'public transport' - why should we not be where the buses are dropping off?"
Andy Howard, from Streamline Telecabs, warned: "We may find that a lot of drivers just won't go up. Sunday is the day they come home, but there is no one encouraging us to go and take them back."
A letter was sent to the council by the association outlining these concerns. It stated that some drivers took up to an hour and a half to leave the festival grounds during the Saturday and Sunday of last year's event. It also warned journeys back into Leeds, which would normally cost £18-£20, ended up costing £40.
A report which went before councillors at the meeting this week said: "The main issues in 2017 were delays in the egress from the site of taxis/private hire vehicles on the Sunday evening due to poor communications between Festival Republic representatives and the trade.
"Contributory factors were also poor lighting and signage creating confusion leading to and in the pick-up areas."
Melvin Benn, managing director of organisers Festival Republic, told the meeting: "We have spent a lot of time talking with members of the community. We have built on last year's traffic plan - there is a change around the taxis so we get a drop off for parents and family members, and a pick up more or less in the same area as the taxis. We are very comfortable with the transport plan."
When asked about the letter from the Hackney Carriage Association, he told the meeting: "I am not going to comment on it - my position does not change one iota."
This year's festival will take place from August 24-26. Acts such as Kings of Leon, Fall Out Boy and Kendrick Lamar are expected to play in front of 70,000 people.