The Foreign Secretary, who grew up in the seaside town and went to Scarborough Sixth Form College, defied calls to quit over his trip to the Greek island, which he returned from as the Taliban captured Kabul in the middle of August.
Mr Williamson, a former defence secretary, was asked on Thursday if he would have gone on holiday amid concerns over the looming crisis in Afghanistan.
“As Education Secretary you tend not to have holidays,” he told LBC radio.
But asked if he had not had a holiday this summer, Mr Williamson said: “I got a couple of days in Scarborough to see my mum and dad, so that was very nice.
“Dom, at every stage, has been working tirelessly to make sure everything is done and the Prime Minister has absolute full confidence in him.”
There has been speculation that Mr Raab would be dropped at the next Cabinet reshuffle over his handling of the crisis, but No 10 has insisted Boris Johnson maintains confidence in the Foreign Secretary.