Investment banker Jeffrey Blue says Mr Ashley promised to pay him £15 million if he used his expertise to increase Sports Direct's share price to £8 a share.
He says Mr Ashley paid only £1 million and he wants £14 million damages.
Mr Ashley denies the claim.
He told Mr Justice Leggatt on Wednesday: "It's nonsense."
Mr Justice Leggatt began analysing evidence at a High Court trial in London on Monday.
The judge has heard the dispute between Mr Blue and Mr Ashley relates to a conversation in a central London pub.
Mr Blue says Mr Ashley made a promise during a meeting in the Horse And Groom four years ago.
Mr Ashley told Mr Justice Leggatt, in a witness statement, how he met Mr Blue and three other finance specialists at the pub in January 2013.
"When we got to the pub we started drinking heavily at the bar and consumed a lot of alcohol during the evening," Mr Ashley told the judge.
"We must have had four or five rounds of drinks in the first hour.
"I can't remember the details of the conversations that we had in the pub as it was a heavy night of drinking."
He said he recalled talk about football.
"I can't remember the detail of these conversations but I do remember that we had a lot of drinks and a lot of banter," said Mr Ashley
"We were pulling each other's legs about what hypothetical value my shares would be worth 'on paper' at different share prices.
"It was a fun night, as it was intended to be, and everyone was on good form.
"We got on well, so much so, that we ended up going out after the pub to a bar in the West End where we stayed until the early morning.
"I had drunk so much that I can't remember which bar we went to.
"I don't remember Mr Blue joining us and I can't remember what time he left."
Mr Ashley added: "I find it incredible that Mr Blue is actually suggesting that I made a binding agreement for £15 million.
"If I did say to Mr Blue that I would pay him £15 million if he could increase (Sports Direct's) share price to £8, it would be obvious to everyone, including Mr Blue, that I wasn't being serious."
Mr Blue has told the judge that Mr Ashley is a "serious businessman".
He said the work ethic at Sports Direct was "like nothing else I have ever seen".
But he said Mr Ashley sometimes did business "in unorthodox ways and in unusual venues".
Mr Blue has told Mr Justice Leggatt how Mr Ashley vomited into a fireplace after a Sports Direct senior management meeting which was "effectively a pub lock-in".
He said Mr Ashley would nap under tables at "boring" meetings.
Mr Ashley told Mr Justice Leggatt that he was not "Obi-Wan Kenobi running the Death Star".
He referred to Mr Blue as "that liar" and he told a lawyer asking questions on behalf of Mr Blue that he did not stage business meetings in pubs as a norm.
Mr Ashley said the inference that Sports Direct had senior management meetings in a pub was "100% incorrect".
He said he occasionally made decisions in a pub.
"Definitely not as a norm," he said. "Otherwise I would have to live in a pub."
He added: "I take business decisions all day every day, from home, from the bath."
Mr Ashley went on: "If you are trying to portray the impression that I just merrily make massive decisions ... at the pub you are talking total crap."
Mr Ashley said what Mr Blue had called "senior management meetings" at a pub, he would describe as a "drink after work".
He said Shirebrook, Derbyshire, where Sports Direct is based, was a "very boring, lonely place".
Mr Ashley said "going for a drink" was "what we do after work".
He told the judge: "Serious, serious decisions are not done on drunken nights out."
Mr Ashley said he delegated decisions.
"Stop trying to paint me as this person who decides absolutely everything because it isn't true at Sports Direct," he said.
"Do you think I can have conversations with 1,000 people at a time?"
Mr Ashley said he paid Mr Blue £1 million as a bonus for work done.
He suggested that Mr Blue knew some evidence was "utter bullshit".
Mr Ashley told the judge that he was "really pissed off", "so annoyed" and "furious".
He said "nobody trusts me".