Parts of the stonework at the Roman castle were attacked over the weekend, according to North Yorkshire Police.
The castle, which has a history dating back 2,500 years, dominates the seaside town.
It features in some of the best known images of northern England and has loomed in the background of thousands of family holiday snaps.
A force spokesman said the incident happened between 7pm on Saturday and 9am on Sunday when “extensive damage” was caused to Roman stonework.
He said metal post caps were also removed and signposts were damaged.
English Heritage said site managers are also concerned offenders are risking their lives above 300ft high cliffs to get access to the fragile castle walls.
Pc Alasdair McNeill, from Scarborough police, said: “We need to find whoever is responsible for this completely mindless act of criminal damage.
“It’s very sad that people feel the need to destroy something which has stood for many hundreds of years, is a landmark for the town and contributes to the culture and economy of the area.
“The signposts can be repaired and replaced but the Roman ruins cannot. I urge anyone who has any information to contact the police as soon as possible.”
Peter Bleach, English Heritage’s site manager for the castle, said: “Scarborough Castle is often a target of anti-social behaviour and the latest damage to the castle is extremely disappointing.
“Deliberate acts of vandalism are illegal and we take this very seriously. People often climb the 300ft cliff walls to gain access during the night.
“This is very dangerous as the walls are 900 years old and are unstable. People risk a fatal fall whilst climbing them. Removing the fence and warning signs is also very irresponsible. Whoever did this placed themselves and the public at serious risk of death.”