Having lost five of their last six games, including a Challenge Cup exit at Hull FC, they have dropped down to fifth ahead of tonight’s visit from Huddersfield Giants.
Sixth-placed Wakefield are level with them on points and improving Huddersfield will be, too, if they eke out a victory at Wheldon Road.
Hopes of challenging the top-two of St Helens and Warrington Wolves have certainly subsided and many critics feel Castleford do not have what it takes to reach a second Grand Final like they did for the first time in 2017.
England loose-forward Milner said: “I think a lot of people have probably written us off already.
“But we believe we can do something this year and we won’t stop until there’s nothing else we can do.
“We have had a break and come back in with a fresh mindset and fresh attitude and we’ve trained really well.
“Hopefully, we’ll kick-start our season now.
“There’s weeks when we have been down to our last 17 players; young kids have put their hand up and done a good job for us.
“But, hopefully, we’ll start getting some key players back and then start hitting some straps and start firing for the business end of the season.”
Milner, who made his England debut last autumn, has not hit the heights he knows he can yet.
The 27-year-old said: “I have been a bit inconsistent.
“I have been moved around, into nine with Macca (Paul McShane) being out a few weeks and I have probably been below my standard.
“But the break has been massive for me; I have enjoyed it and I am looking forward to sort of starting my season now and really getting into the back end of the year.”
Castleford – whose forward Mitch Clark has confirmed a move to Wigan Warriors in 2020 – had last weekend off due to the Challenge Cup.
Huddersfield had the same luxury, although they come in on the back of a 55-2 win over Hull FC at Magic Weekend – when Castleford were beaten by leaders St Helens.