The one pointer, kicked in the fifth minute of golden-point, completed an incredible comeback for the hosts who had trailed 26-8 with just eight minutes of normal time to play.
Three converted tries in a dominant spell sent the game to golden point and after a tense five minutes, Lilley kicked the one pointer from 40 metres to complete the comeback.
Head coach Ryan Carr was forced into naming a makeshift side, with Rovers' regular loan and dual-reg players made unavailable by their respective parent clubs.
Prop forward Jack Bussey was deployed at half-back and did a superb job, causing Bradford plenty of problems going forward.
Josh Walters moved from the pack into right centre while Josh Hardcastle was shifted to the wing.
Rovers did have the benefit of Leeds Rhinos' youngsters Jack Broadbent and Alex Sutcliffe on the left edge, however.
Featherstone started strongest and took an eight-point lead after Broadbent made it two tries in as many games for the visitors, finishing a slick passing move on the left.
Watson Boas, making his return from injury, added the conversion before kicking a penalty goal a few minutes later.
Bradford hit back immediately as they won the short kick off, which eventually led to Ethan Ryan bursting through a gap to plant down.
The Bulls soon levelled it as Elliot Minchella kicked a penalty goal from straight in front.
As the half wore to a close, Luke Cooper - playing in his 26th consecutive game - broke the line and almost nipped over for a try before being hauled down by two Bradford defenders.
The Bulls were penalised for offside from the play the ball and Boas made no mistake with the penalty from straight in front.
Featherstone extended their lead to six points on the stroke of half time as Bussey slipped the ball to Hardcastle who squeezed over for a try on the right.
Boas kicked two more points five minutes into the second half after Bradford were penalised 10 metres in front of their own posts.
Rovers produced a superb defensive effort in the first part of the second 40, holding the returning Ross Peltier up over the line as Bradford looked to reduce the deficit.
The away side struggled to find the attacking flow that they had enjoyed in the first half, resulting in most of the play taking place in the Featherstone half.
The visitors soon gained a grip on proceedings and after a few minutes of pressure on the Bulls' line, they extended their lead.
James Harrison took the ball on the last and crashed over underneath the uprights, Boas's conversion put 14 points between the sides.
Rovers extended their lead straight away as Josh Walters shrugged off the tackle and found Jack Render on his outside.
The full-back was brought down but was able to jump back up and stretch over to ground the ball.
Tries from Dalton Grant and Jordan Lilley reduced the deficit to a single try as Bradford produced a late onslaught on the visitors' line.
Bradford completed an almost-impossible comeback as David Foggin-Johnston dived over in the corner.
Minchella landed a towering touchline conversion to level the scores with 20 seconds to go.
After two missed drop-goal attempts from Rovers, Lilley kicked Bradford into the sixth round.
Bradford Bulls: Ryan, Grant, Minchella, Oakes, Foggin-Johnston, Lilley, Milnes, Bustin, Wildie, Crossley, Garside, Farrell, Hallas. Subs: Green, Webster, Flanagan, Peltier.
Featherstone Rovers: Render, Broadbent, Hardcastle Walters, Sutcliffe, Bussey, Boas, Ormondroyd, Maskill, Cooper, Day, Davies, Makatoa. Subs: Lockwood, King, Harrison, Beckett.
Referee: Matthew Rossleigh