The entrepreneur, who campaigned for Leave in the referendum, said it was quite outrageous that Brussels was demanding “billions and billions”.
Sir James also called for an end to corporation tax, insisting there were better ways to impose levies on companies.
He told BBC One’s The Andrew Marr Show: “Demanding billions and billions to leave is quite outrageous and demanding it before we negotiate anything is outrageous.
“So, I would walk away. I think that’s the only way to deal with them.”
Sir James said he believed 90 per cent of future growth would come from outside the EU.
Brexit Secretary David Davis said it would be “very expensive” for the UK if it sought to extend the negotiations by 12 months.
Asked why the UK would not consider pausing the Article 50 process to extend the time available, Mr Davis also said it would create extra uncertainty for businesses.
He said: “In order to pause it, well, not pause it, in order to extend it by a year, let alone pause it - I’m not sure you can pause it - even to extend it by a year takes unanimity of all the other 27.
“What do you think the price of that will be?
“When you require unanimity from 27 countries, I can tell you it’d be very expensive.”
The Howden and Haltemprice MP dismissed the suggestion that a no deal scenario is “more probable than it’s ever been before”, reiterating it is not an aim.
He also said the UK would be aware “no deal” is coming as he sought to play down concerns about the potential consequences.
Mr Davis said: “If we’re at this point with no deal, we’ll know it’s coming for a while and we’ll take measures to ensure that what you’re describing doesn’t happen, which is why I’ve talked about, at various times, a bare bones deal or a minimalist deal - we don’t want that either, frankly, but don’t assume we haven’t thought through the end contingencies of this.
“We have.There will not be a circumstance where aircraft won’t fly, there will not be a circumstance where we can’t exchange data with the European institutions, there will not be those sorts of failures that people are fearing.
“We’ll make sure that doesn’t happen.”
Last week it was reported that the Brexit negotiations have made “some progress” but more work is needed on specific issues.
The EU’s chief negotiator Michel Barnier said they still had to see “sufficient progress” on the arrangements for Britain’s withdrawal before they could move to the second phase of negotiations.
“We are not asking the UK for concessions, nor are we planning to make any concessions ourselves,” he told a joint news conference with Brexit Secretary David Davis last week.
Mr Barnier said the sixth round of talks had largely involved “deepening discussions, clarification and technical work”.
He said his “top priority” was to secure sufficient progress on the issues of citizens’ rights, the border with Ireland and Britain’s “divorce bill” by the time of the next EU summit in December.
“Only sufficient progress - that is to say sincere and real progress - on the three main key issues of these negotiations will enable the triggering of the of second phase of our negotiation,” he said.
Last week Mr Barnier indicated that he would need further clarification from the UK on its positions within two weeks to be able to recommend to European leaders that “sufficient progress” has been made at December’s summit.