The 42-year-old has been tipped as a possible presenter, alongside Chris Evans, who has signed up for the reincarnation of the motoring programme following the axing of Jeremy Clarkson.
O’Leary, who quit the X Factor after eight series, told Good Morning Britain: “I love my cars ... but I think ... he’s (Evans) such a main man.
“He’s a big deal for me, he’s one of the main reasons why I got into television in the first place. But I imagine he’ll want his own team on it.”
Asked whether he could join Evans on a new team, O’Leary replied: “I doubt it.
“If I was going to do it, I would want to do what Chris is doing - and he’s in charge of it.”
The presenter has also denied speculation that he could join this year’s Strictly Come Dancing, saying: “I’m a TV host. What I love doing is interviewing people and live television...
“I don’t want to be a contestant on Strictly and they’ve got two really good presenters (Claudia Winkleman and Tess Daly) so I can’t see it.”
He said of his decision to leave the X Factor: “Normally I’m doing a 13-hour day at bootcamp in July ... I did eight years. It was a wonderful, fantastic eight years. But it felt like the right time.
“I thought maybe it would be next year or the year after, but I just made the call quite quickly this year.
“You go through a couple of days of going ‘Have I done the right thing?’ and feeling quite sad and then your instinct tells you it’s the right time to go.”
Former Xtra Factor presenting duo Olly Murs and Caroline Flack are hosting this year’s X Factor.
Evans has called on Top Gear fans to send in short audition clips for a job on the new show, which will air in March or April.