Individual mishaps saw his side lose from 1-0 up and end the match with 10 men.
Ryan Williams gave Millers the lead before Ellis Harrison inspired the fightback, scoring the first and setting up Liam Sercombe’s winner.
Frustrated striker Keiffer Moore was shown red late on for a stray elbow.
Warne said: “We need to go back to ugly defending. Obviously after a defeat I’m never happy.
“Tactically, the lads were set up perfectly to deal with what they had, but individual errors cost us.
“If we can eradicate that we have more than enough to pick up points in this league.”
Billy Bodin squandered a great chance when he headed over Dan Leadbitter’s cross.
Joe Newell caused the hosts problems from a corner as Adam Smith had to backtrack to tip over the goal-bound set-piece.
The Millers upped the pressure after the restart although Will Vauks could only fire wide after getting himself into a decent position.
However, former Barnsley player Williams got the opener after combining with fellow winger Newell.
The lead was short-lived as Harrison levelled just eight minutes later by beating Marek Rodak at the near post.
Harrison then turned provider as Sercombe sealed the points with a well-taken finish in the box.
Late on Moore was sent off for using an elbow on Lee Brown.
Bristol Rovers: Smith, Leadbitter, Lockyer, Sweeney, Brown, Partington (Lines 62), Clarke (Gaffney 62), Sercombe, Sinclair, Bodin, Harrison (Nichols 90). Unused substitutes: Slocombe, Broadbent, Telford, Broom.
Rotherham: Rodak, Cummings (Clarke-Harris 85), Ihiekwe, Ahayi, Mattock, Williams, Frecklington, Potters (Vaulks 23), Newall, Ball (yates 85), Moore. Unused substitutes: O’Donnell, Wood. Towell, Forde.
Referee: Brett Huxtable.