Both clubs entertained in a six-goal thriller and will now have to face each other again in a midweek replay to decide who will progress to the second round.
First-half goals from Ryan Jarvis and Josh Carson gave the visitors a 2-1 half-time lead, with Elliot Richards having equalised.
Substitute Matt Harrold levelled just before the hour mark and Preston loan striker Chris Beardsley put the home side ahead.
But a mistake from goalkeeper Steve Mildenhall cost the Pirates dear as substitute Wes Fletcher pounced four minutes from time to force a replay.
Jarvis put the visitors ahead when he poked home from close-range after 35 minutes.
But within two minutes Richards equalised for Rovers, pouncing from close-range.
However, it wasn’t long before Nigel Worthington’s team had regained their lead courtesy of a stunning strike from Carson, letting fly with a 30-yard thunderbolt into the top left corner.
In a first half they dominated, Jarvis had seen a shot charged down and Daniel Parslow rifled agonisingly wide.
As York went in search of a third goal after the break they were hit with a sucker-punch when substitute Harold struck having been on the pitch for less than a minute.
Harold then teed up Clarke, but his team-mate dragged a right-foot shot wide of the upright.
Bearnsley pounced to fire Rovers ahead for the first time, but Fletcher had the final say with his goal at the death.
Bristol Rovers: Mildenhall; Smith, Lockyer, Parkes, Brown; Santos, Clucas, Norburn (Harrold 58), Clarke, Richards; Beardsley (Brunt 84). Unused substitutes: Gough, Clarkson, Harrison, Keary, Southway.
York City: Ingham; Onyebanjo, Smith, Parslow, Fyfield; Carson, Platt, Whitehouse, Puri (Brobbel 54); Jarvis (Fletcher 74), Bowman (Cresswell 88). Unused substitutes: Kettings, Clay, Chambers, Allan.
Referee: T Kettle (Rutland).