The 17-year-old youth is believed to have pulled himself clear of the water but returned to the sea to help one of his friends.
Emergency services received multiple calls about the incident from witnesses in the town which was busy with holidaymakers.
The teenager and two female friends were using an inflatable dinghy when they got into difficulty at Whitby's West Pier, a spokeswoman for RNLI Lifeguards said.
The boy, who has not been named but is believed to be from the Tyneside area, used a ladder on the pier wall to get to safety but had gone back to help one of his friends when he got caught in the rip current.
North Yorkshire Police said his friends were two girls believed to be aged 16, also from the Tyneside area.
They were also injured in the incident and were taken to hospital for treatment for hypothermia.
The drama unfolded just before 1pm when Humber Coastguard received multiple 999 calls reporting that three teenagers were being swept out to sea near the West Pier.
Whitby Beach lifeguards went into the water to recover the youths and the coastguard sent Whitby Coastguard rescue team, Whitby RNLI inshore lifeboat and a rescue helicopter from RAF Leconfield to the scene.
RNLI lifeguard Josh Jones reached the water's edge on a quad bike and entered the water, he reached one of the girls first and secured her with a rescue tube so she could float.
Megan Ellis, another lifeguard, took the girl to shore while Mr Jones went further out into the water to reach the boy, who was unconscious.
He kept him afloat until the inshore lifeboat crew reached them and recovered them both into the lifeboat.
The youngster was given emergency resuscitation in the lifeboat until a rescue helicopter arrived and transferred him to Scarborough Hospital.
He was pronounced dead a short time later.
The second girl is thought to have made it to the shore unaided, the RNLI Lifeguards spokeswoman said. Both she and the other girl were taken to the same hospital when the rescue helicopter returned to the beach and their injuries are not thought to be life threatening.
Both were conscious but were very cold and had swallowed a lot of salt water, prompting concerns they had secondary drowning where water enters the lungs. If untreated, drowning can occur several hours later as the water affects breathing.
Katie Gellatly, Humber Coastguard watch officer, warned of the dangers of swimming in the sea.
She said: "This incident reminds us all of the hidden dangers present when swimming at the beach, and we would urge all members of the public visiting the coast to use lifeguarded beaches where possible, and take note of any signs or flags that show which areas are safe to bathe in."
Hilary Jones, Scarborough Borough Council's strategic director, added: "We are shocked to hear of this tragic incident and our thoughts go out to the family of the young boy who lost his life and the other two teenagers involved."