As our exclusive video and photo shows, the man, wearing a wetsuit, was in the sea in the town's South Bay as waves crashed in, whipped up by a strong north-easterly wind.
At one point, as he starts to walk back to the beach, he loses his footing and is carried along by the surge.
Scarborough RNLI is asking everyone to treat the sea with the utmost respect after a crew member saw the man trying to swim in the sea in what were stormy conditions.
The off-duty lifeboat crew member saw the man enter the sea near the lifeboat station just before noon.
“I witnessed the man very nearly get into serious problems,” the crew member said.
“The surge from this size of swell was incredibly strong and was washing him off his feet. It could easily have swept him out to sea in seconds.
“Luckily we were able to get the man’s attention and convince him to return to shore.
"He was very lucky not to get into trouble."
On the video a man can be seen shouting to the swimmer, warning him that he could have been swept against the wall.
A police officer and another person in a hi-viz jacket can also be seen nearby; the police officer spoke to the man afterwards.
The Met Office has issued yellow warnings of ice for the Yorkshire Coast until midnight and of snow until midnight on Wednesday.
The Environment Agency has issued flood warnings for along the Yorkshire Coast today as overtopping waves and spray were expected on high tides.
"Please stay away from seafront areas, large waves can be dangerous," said the Agency.
Entering the water on exposed beaches in a storm is extremely dangerous and should be avoided at all times, says the RNLI.
On its website, the RNLI advises: "If the water is rough, don’t go in."
If you see anyone injured or in distress on the coast, dial 999 and ask for the coastguard.