Giving evidence at the phone-hacking trial, the 46-year-old said the couple had an affair that started in 1998, but was not “continual” and there were long periods where they were just close friends.
The Old Bailey has previously heard that Brooks and Coulson had several “periods of intimacy” during their time working together.
Coulson followed Brooks as editor of the now-defunct Sunday tabloid between 2003 and 2007, and before that was appointed in 2000 as her deputy.
He told the court that the pair first got to know each other in 1996 when a friend of his had died, and then became professional colleagues in 1998 when Brooks was made deputy editor of The Sun.
Asked about their personal relationship, the former Downing Street spin doctor said: “There was an affair that started in 1998. It ended quite soon after but it did re-start, as the court has heard.
“What I want to say is that it was not by any means continual. There were very long periods - very long periods - where the relationship was what it should have been,” saying they were just close friends and colleagues.
“But I don’t want to minimise it or excuse it. It was wrong and it shouldn’t have happened and I take my full share of responsibility for the pain it has caused other people, not least my wife.”
Both Brooks and Coulson are accused of conspiring to hack phones and separate charges of conspiracy to commit misconduct in public office.
Coulson’s lawyer Timothy Langdale QC questioned him on how the affair affected their working relationship.
Coulson agreed he would exchange confidences with Brooks in a way he would not normally do with an editor.
But the married father of three denied the suggestion that the closeness of their relationship involved him “breaching professional standards or rules”.
Mr Langdale said: “In particular, it has been suggested as a result of the closeness of your relationship you would share sensitive or exclusive stories.”
Coulson said: “No, that did not happen - with the caveat unless on very particular occasions there was a pre-determined deal when there was a share between the two papers.”
After the affair, Coulson said remained friends with Brooks, even after his resignation from the NotW in 2007.
Earlier, Coulson told how he spent a weekend with David Cameron after he resigned as his media adviser amid controversy over what he knew about phone hacking while he was NotW editor.
But the 46-year-old said he had not spoken to the Prime Minister since that pre-arranged social occasion in spring 2011.
Coulson resigned as NotW editor in 2007, after the conviction of former royal editor Clive Goodman for hacking, and joined Mr Cameron’s team, first in opposition and then in Downing Street after the 2010 general election.
He resigned his post as Mr Cameron’s media adviser in January 2011 amid the controversy over phone hacking, and was arrested last that year, then charged in 2012.
Coulson today said he had had “sparing” contact with former boss Rupert Murdoch since his resignation.
And of his contact with Mr Cameron, he said: “My family and I spent a weekend with him in the spring after I left. I have not spoken to him since.”
He added that the invitation came before he left his Downing Street job.
Coulson described an atmosphere of intrigue and secrecy at the NotW when he joined the paper from the Sun as deputy editor under Brooks in 2000, saying it had become “destructive”.
Asked about managing editor Stuart Kuttner, who also denies conspiring to hack phones, Coulson described him as an “incredibly experienced professional journalist” and a “thoroughly decent man”.
He said although the pair worked together, they did not “keep tabs on each other”, and told the court: “He is a very experienced journalist, he did not need to have his hand held in any way or to be directed.”
Asked about the pressures to sell papers, Coulson said the main aim for an editor was to produce a successful newspaper, but also to produce it within budget.
But he said: “There’s some unspoken leeway and I think you could spend just a little bit more than the budget and argue your case.
“My experience is management expected you to spend every penny they gave you and perhaps a little bit more.”
Coulson said there was tension between editors who wanted larger papers with fewer adverts, and advertising departments who wanted smaller papers packed with “what they considered to be fascinating adverts”.
He admitted that an emphasis on “giveaways” of free DVDs or CDs was “depressing”, because readers came to expect them, with no “loyalty” to the newspaper.
Coulson was asked about 2005/06 budget proposals emailed to him by Kuttner. They included a 50% cut in hacker Glenn Mulcaire’s £105,000 annual contract, listed under his company Nine Consultancy for “special inquiries”.
Coulson said at the time he had heard of the company in relation to a budget document. But he denied knowing about private detective Mulcaire, saying: “I did not know the name Glenn Mulcaire until he was arrested with Clive Goodman.”
Asked if he ever inquired what Nine Consultancy did, he said: “I assumed the description special inquiries meant finding people, looking for people, possibly surveillance in the main and that Nine Consultancy was part of that process.”
He told the court: “It was not an area of the News of the World I was particularly interested in.
“For me, finding people in newspapers is an incredibly important part of the process. You have got to get there first. I always viewed that as a central function of special inquiries.”
Coulson said he could not remember arguing for or against the Nine Consultancy contract to be cut.
Asked about the figure paid to Mulcaire’s company, Coulson said: “I would not suggest that £100,000 is not a lot of money but in the context of a £32 million budget it’s not a massive sum.
“The NotW spent a lot of money, we had a lot of money to spend on stories and a lot of things.
“I don’t want to be dismissive of £105,000, but the reality is it wasn’t a lot of money in the business I was running. We paid double that, I think, to the astrologer.
“There were large sums of money being paid to people by the NotW.”
Coulson was questioned on a budget item, £22,000 for “flowers and Oddbins”.
There were laughs in court when Coulson said: “I think, it’s the flowers and alcohol”.
He would send bottles of champagne as “thank yous” to people for good work and at Christmas, he said.
He said a £750,000 sum described as “week 52” was used as a contingency if they had run out of money by the end of the year, or it could be dipped into earlier in the year.