Lee Frecklington’s scored twice in tewo second-half minutes to give Rotherham a seemingly unassailable 3-0 lead.
Kari Arnason set up Mark Bradley to open the scoring for the Millers.
Frecklington then scored with a long-range effort before pouncing on a loose ball two minutes later and sending the fans inside the New York Stadium into raptures.
But the League Two Millers were then given a real fright as Stevenage set up a grandstand finsih with two goals.
First, James Dunne’s long-range strike gave Stevenage hope before Filipe Morais scored after Marcus Haber’s shot was saved.
But Steve Evans’s side held on for a big win.