He has joined the Robins on a one-year contract with the East Yorkshire club also reportedly set to confirm a deal for Toronto Wolfpack half-back Ryan Brierley.
Ellis, 30, had been in his second spell at Wheldon Road after rejoining the club from Huddersfield Giants in 2018.
He still had a year left on his contract at Castleford but with the arrival of St Helens No7 Danny Richardson would have struggled for game-time.
Ellis missed most of the last campaign after undergoing knee surgery and did not play until August, making just six appearances.
But he is now hoping to kick on at KR where he has happy memories having spent a season on loan from Huddersfield in 2017.
He played a major role helping the Robins earn promotion back into Super League, forming a successful half-back combination with Jordan Abdull and scoring nine tries and kicking 157 goals in 30 appearances.
Obviously, Abdull has also rejoined the club after a quality campaign at London Broncos so they could reignite their partnership in 2020.
Rovers head coach Tony Smith said: “Jamie probably does not fit into the main brief of our 2020 recruitment criteria in terms of being ‘young and ambitious’ but upon meeting him and listening to his passion for the club, having already played with this team, it was great to hear.
“He loved his time here, still loves the club and had a very strong desire to express that. He convinced me that he is really ambitious and wants to do his best here.
“He understands that there are competition for places and that there are young and talented players in that position where he plays but he is willing to fight for that."
Smith added: “One thing we have said all along in terms of our recruitment is that we want competition for places and Jamie will add a different range of skill-sets to some of the other players.
“He adds a wealth of experience to our young squad in terms of playing for successful clubs and understanding what it takes to be successful.
“Jamie has good connections with players such as Adam Quinlan and Jordan Abdull already having played alongside them here, but also comes over with strong ties with Greg Minikin and Will Maher from Castleford and that is advantageous for us.
"Jamie does come to us with a minor knee ailment but he will be fully-fit within four to six weeks.”