Grimes, now viewed as one of the most consistent central midfielders at Championship level, admits that he found his time at Elland Road, where he made seven appearances in the 2016-17 season under ex-Leeds boss Garry Monk - who brought him to Swansea from Exeter City - hugely difficult.
Speaking about his journey in a recent Swansea match programme, Grimes, who joined Leeds when Swansea were in the top-flight, said: "Playing at Exeter was always upwards.
"I was progressing, and got my move. I needed to play more minutes than I was at the time to keep progressing. It plateaued a bit and then went on loan to Blackburn and got myself going again. I thought I was back on track and then went to Leeds.
"The mental state I was in that season (at Leeds), I don't wish on any footballer. I was training, not playing, going home and sleeping on the sofa because I was living on my own as I wasn't with my girlfriend at the time.
"I was just in a lull of training and not playing, running on the weekends and it impacted my mental health so much. I was stepping on the pitch, not feeling confident at all and I was thinking should I even be here?
"Should I be at a Premier League team? What am I going to do after this season because it has gone so badly. That, coupled with thousands of people outside of football telling me how bad I was, calling me all sorts of ridiculous things."
Despite his difficulties at Leeds, Grimes believes that it made him a stronger player in the long-run, mentally.
"To bounce back from it, without sounding cynical, I feel a lot of people need it to bounce back and build character, not that I wish people to go through that.
"It's so important, in this day and age, especially young players because they are subject to such a big scale of people watching and having their opinions.
"Obviously everybody is entitled to their opinions, but you just have to know what the right ones are to take in. Looking back now, it was a really good test for me if you like.
"At Leeds, because I wasn't playing, and when I was playing, I was playing poorly and then you get all the outside noise of you're not good enough. But because you aren't confident in yourself you started to believe that."