Semi Ajayi scored his sixth goal in five games to cancel out Kenny McLean's opener but Ben Godfrey's header claimed all three points for the visitors.
City now hold a four-point lead over Sheffield United after the Blades won at third-placed Leeds earlier in the day.
The result leaves the Millers third-bottom - a point adrift of Reading with just eight games left.
Norwich supporters packed out the away end at the New York Stadium and it made for a superb atmosphere.
Ben Wiles ballooned a shot over the bar early on for the Millers before Joe Newell warmed the gloves of City stopper Tim Krul after a clever pull-back from Billy Jones.
Michael Smith then nodded over the bar before skipper Richard Wood followed suit with a tame header just after.
Norwich had started slowly but began to assert themselves on the contest and Onel Hernandez drew a decent stop from Marek Rodak.
The visitors' growing dominance paid off right at the end of the half when Pukki lifted the ball into the path of McLean who cleverly tucked it beyond Rodak.
The hosts levelled proceedings seven minutes into the second half and it was through a familiar face.
Ajayi received the ball with his back to goal, spun his man and stroked the ball into the goal for his sixth strike in five outings.
The joy wasn't to last long though.
Pukki poked wide almost immediately for City before they restored their advantage from a set-piece.
McLean turned provider when his whipped corner picked out Ben Godfrey who planted a header beyond Rodak.
Ben Wiles blasted a shot wide before the classy Emiliano Buendia curled a shot just past a post for City.
Pukki then dashed half the length of the field but was kept at bay by an alert Rodak as the clock ticked away.
Jamal Lewis hit the target as City looked for a killer third goal before Buendia was denied after a blistering counter, but two goals was enough to ensure a hard-fought and gritty win for Daniel Farke's side.
Millers' chief Paul Warne said: "I thought the lads were excellent up against a formidable side in Norwich.
"With eight games to go, hopefully we can keep that level of performance up.
"I'm really proud to be their manger and I thought they were excellent.
"The disappointment for me was conceding from the set-piece for the second.
"But I asked the players to perform and they made me proud - I've got no complaints."
Rotherham: Rodak, Vyner, Wood (Yates 83), Ihiekwe, Jones, Ajayi, Forde (Taylor 62), Crooks (Towell 62), Wiles, Newell, Smith
Subs not used: Price, Palmer, Kayode
Norwich: Krul, Aarons, Zimmermann, Godfrey, Lewis, Trybull, Buendia (Vrancic 90), Hernandez (Klose 90), Stiepermann, Pukki (Rhodes 90)
Subs not used: McGovern, Hanley, Cantwell, Leitner