The man, whose age has not been released, was on Castleberg Rock on Sunday when he fell just after 1pm.
Two visitors to the area, which is also popular with walkers, saw the accident and attempted CPR on him for around 15 minutes before paramedics arrived.
The Yorkshire Air Ambulance also attended but the man was pronounced dead at the scene and his body was removed from the crag by Cave Rescue Organisation volunteers.
Castleberg Rock was been popular with climbers since 2009, when a group of them worked with Settle Town Council, who manage the site, to open 22 routes while removing unstable rock and vegetation to create a limestone climbing and bouldering venue.
The tragedy is the second fatal outdoor activity accident to take place in North Yorkshire in less than week.
On Friday a kayaker died after getting into difficulties in the swollen River Ure at West Tanfield, near Ripon, following heavy rain. The man has not yet been identified and a second man who was with him managed to reach safety.