The star, 33, played Lady Edith in the TV drama, and is due to return to the upper-class character in the long-awaited Downton movie, which filmed in Yorkshire.
Julian Fellowes previously revealed how Wentworth Woodhouse in South Yorkshire inspired the film, which was shot at locations including Harewood House in Leeds with the help of Screen Yorkshire.
Speaking to Town & Country magazine, Carmichael said: "I was so close to not being a working actress. I had a hard time.
"It's such a painful experience, wanting to act and not doing it, and trying to figure out how to get in there."
Carmichael added that the drama, which was also a hit in the US, "has a long casting shadow, I've been in a lot of costume dramas..."
-> How the BBC's Gentleman Jack sparked global awe of Yorkshire rebel Anne Lister and Shibden HallBut the actress, who is in a relationship with co-star Michael C Fox - footman Andrew Parker in the ITV drama - said that working on Downton Abbey has "changed all of my life".
"When you are able to work, and love it, it is such a joy," she said.
"And I've fallen in love and made wonderful friends - how lucky am I?"
Recalling going up for the role, she said: "I went to audition, and because it was so top-secret, I didn't get the script before I arrived.
"I thought I'd probably have one line, but I had five scenes. I was reading them, going, 'My God, this is a proper part! Why are they seeing me for it?'...
"On my first day, I think Jim (Carter) saw I was like a rabbit in the headlights - I had to come in, speak, and serve myself some breakfast, which seemed utterly terrifying - and he said, 'Why
don't you come over here, lovey, and I'll help you?'"
Set in 1927, the film - out next month - sees the return of the principal cast, including Hugh Bonneville, Michelle Dockery, Dame Maggie Smith and Joanne Froggatt.
The storyline continues after the series left off in 1925 in the 2015 Christmas special.