The forward was approaching the end of his contract and a fresh start appealed to him.
The experience has given the one-club man a greater appreciation of what he has got at the Jungle.
"It came close, to be fair," Milner told The Yorkshire Post.
"Looking back now I've got a new coach, I'm pleased how things have worked out. I love this club and everything about it.
"I'm a local lad and I don't think the move would have been right for me or my family. I've got two young kids at home so spending time on the motorway wasn't the right thing for me.
"Being a local lad, it always means that little bit more when the team is doing well. Hopefully we can keep climbing the ladder and have a good end to the season."
Milner has made over 250 appearances for Castleford since his Super League debut in 2010, enjoying the best years of his career during Daryl Powell's time in charge.
The 30-year-old was capped three times by England in 2018 after catching the eye of Wayne Bennett.
While Milner is appreciative of the role Powell played in his career, he feels revitalised under Radford.
"It's different," said Milner. "I was under Powelly's leadership for eight years and it does take time to settle and adjust, but I think we're over that transitional period now.
"Radders is very approachable. He's very clued up as well. He's a smart man just like Powelly was.
"He likes to be in and around the lads and have a laugh and a joke, but on the field you know you've got to be on it all the time. Training sessions are intense.
"Having been under a coach for so long, to get that fresh start under a new coach has been very enjoyable. On a personal note, I'm really enjoying it under Radders at the minute."
It was a rocky start to the Radford era with the Tigers having to wait until round four for their first victory.
Castleford appeared to be on the right track thanks to a string of positive performances only to produce their worst display of the season at Hull KR in the Challenge Cup.
The players saw a different side of Radford after the game at Craven Park.
"The Challenge Cup game was a step backwards for us," said Milner.
"We'd made some massive strides and the cup competition was something that we really wanted to go for.
"Rightly so, we got the hairdryer treatment after the game. He certainly isn't afraid to voice his opinions when you're not doing too well.
"We've certainly bounced back from that disappointing performance pretty well getting the results that we have over Easter."
Three straight Super League wins have been just the tonic for the Tigers after crashing out of the cup, leaving Radford's men just outside the play-off positions.
Castleford's improved form has coincided with Milner's return to the side following his recovery from a troublesome back problem.
He is wearing 12 on his back this year but his versatility remains invaluable to the Tigers.
"It's another different role for me," said Milner on the switch to the second row.
"When I signed this new contract at Cas, I had a chat with Radders and he discussed playing me in the back row which is a position I'd never played.
"I spent a lot of time in pre-season adapting to that role but every week I'm finding myself playing different positions.
"I take credit that I can play a number of positions. I pride myself on understanding the game and different positions.
"Whether it's nine, 12 or 13, I know what my role is and it's giving my all for the shirt and the badge."
As Castleford prepare for Friday's trip to Catalans Dragons, Milner is eyeing a rare rugby league feat.
"The only position I've not played is full-back," he added with a smile.
"With the injury to Niall (Evalds), I keep putting my head into Radders saying, 'Have I got that one jersey this week?'
"Once I get the full-back position, I've completed the game and can retire pretty happy."