The 26-year-old left the crisis-hit League One outfit earlier this summer due to pay issues after terminating his contract with the cash-strapped Shakers.
Thompson, a full Northern Ireland international, was the subject of reported interest from several League One and League One sides.
He trained with Bury's near-neighbours Bolton Wanderers in the close season and was also linked with Ipswich Town.
But he has now elected to head to South Yorkshire and signed a deal with the Millers until 2021.
The Harlow-born player is no stranger to Yorkshire after spending a loan spell at Bradford City in the 2017-18 season where he made 13 appearances in all competitions.
Thompson returned to Gigg Lane and re-established himself in the Shakers' line-up, helping the club win promotion last term - making 38 appearances and being a main-stay of the back four.
Thompson's performances had put him back in contention for a place in the Northern Ireland squad, who he has represented at several age-group levels.
The defender began his career in Watford's academy and made 16 senior appearances for the Hornets but spent much of his time at Vicarage Road out on loan with Brentford, Wycombe, Barnet and Southend.
A permanent move to Roots Hall followed in 2014 where he made 109 appearances before moving on to Bury in 2017.
The Millers are also close to signing two wingers to further boost their squad ahead of Monday's transfer deadline for League One and League Two clubs.