A passing boat with four men on a fishing trip spotted the overturned vessel and came to the rescue yesterday afternoon (Sunday).
One of the rescuers, Dale Price, from South Elmsall, told the Express the boat was already overturned by the time they reached it.
Four people on top of the sinking vessel were holding on for their lives and waving for help.
Mr Price, 38, said: "As soon as we got to the boat it was clear that it was going down.
"They ended up in the water but we were able to use a rope to get them out. A few minutes longer and they would have been goners.
"We had to keep them from drifting into more exposed rocks as it was low tide and get them onto our boat safe as possible.
"It's not often you see something like that but we didn't really think about it at the time we just had to act."
Mr Price, was on the fishing trip with his father, Leslie Price, and friends James Fox and Steve Bradbury, from Upton.
Mr Price added: "Everyone played their part in the rescue and we able to get them out of the water as the coastguard arrived."
Three of the men from the capsized boat were brought to safety by the passenger's boat while the fourth casualty was taken ashore by a Whitby coastguard.
Whitby RNLI's inshore lifeboat escorted the vessel to the lifeboat station where Whitby Coastguard Team and local paramedics were on hand to help.
One man was treated for hypothermia and all were checked for signs of cold water shock.
Geoff Hodgson who was helm of the inshore lifeboat at the time said: "The casualties were very lucky in this instance that they were reached quickly. The effects of cold water shock can be very serious and in some cases lead to drowning. We are all very relieved that this was not the case.
"Thank you to crew aboard a local fishing boat who reacted quickly and to all the emergency services who pulled together to deliver the appropriate first aid and treat the casualties.
"We would urge anyone who is out on the water, no matter how experienced you are to always wear a life jacket. It is also important to double check the tidal conditions before you set off on your trip. Things can change very quickly at sea if the conditions become manageable."