Mr Williamson said that while working for a fireplace manufacturing firm in Scarborough in 2004 he engaged in a "flirtatious" relationship with a co-worker, confessing they shared a kiss a "couple of times".
The Conservative MP grew up in the town, and studied at Scarborough Sixth Form College and the University of Bradford.
He said admitting the infidelity to his wife Joanne had been one of the most difficult conversations of his life but said she had forgiven him.
Speaking to the Daily Mail, he said: "'My family means everything to me and I almost threw it away."
He added: "I had a good relationship with everyone I worked with, but with one person this started to develop into something more.
"We had to travel together and spent a lot of time in each other's company, the relationship became flirtatious and a couple of times we shared a kiss.
"It never went further than that, but this had a profound impact on us both and those close to us. It was a dreadful mistake and stopped as suddenly as it had started."
It is reported in the Daily Mail that Mr Williamson is understood to have revealed details to party officials before running for selection as an MP and has informed Downing Street. Press Association has contacted Downing Street for comment.
Mr Williamson took over as Defence Secretary in November after Sir Michael Fallon's resignation amid sleaze allegations.
The 41-year-old MP for South Staffordshire got his big break as parliamentary private secretary to David Cameron from 2013-16 and was selected by Theresa May as her parliamentary campaign manager for the Conservative leadership contest triggered by Mr Cameron's resignation following the Brexit referendum.
The former chief whip achieved the highly unusual distinction of being promoted directly into the Cabinet without having held a more junior ministerial job.