Argyle player of the year Carey won and converted the stoppage-time spot-kick as his side rallied from 1-0 down.
Joel Grant had earlier cancelled out Richie Towell's opener before Carey's winner, leaving Plymouth two points off sixth-placed Scunthorpe - who they visit on Tuesday - with two matches remaining.
With their play-off place already secured, Rotherham played with a freedom and were duly rewarded when Towell fired the Millers ahead in the 33rd minute.
Towell rifled in from the right hand side of the penalty area after being teed up by David Ball, who had an earlier effort ruled out for offside.
Grant went solo to level after 61 minutes, sweeping the ball past Millers goalkeeper Marek Rodak from 10 yards.
And Carey kept his cool from 12 yards at the death after he was fouled in the box by Will Vaulks.