Now a campaign has been launched to raise £300,000 to save the Flowergate Unitarian Chapel after the congregation dwindled to unsustainable levels.
Tucked away in an alleyway sandwiched between an art gallery and a cafe in the town centre, the chapel is a building surrounded by mystery - nobody knows exactly when it first began to be used for religious purposes, and it's got a cellar with its own secret passage.
Is Whitby really a rip-off? We investigateThe chapel's Friends group and Whitby Civic Society hope to turn the building into a community centre, and if funds are not raised they would be forced to sell it on the open market.
It's currently in a 'just about usable' condition, according to the Civic Society chairman John Field.
"The amount of work that's needed is hair-raising. It's still used for worship but not much, the congregation is tiny now - but it has huge historic significance for Whitby."
From 1888 until 1910, the chapel's minister was Francis Haydn Williams, a radical social reformer who led protests and demonstrations in the town and frequently ended up in court or imprisoned in York Castle.
He campaigned against the private enclosure of both the Whitby Abbey plain - the main public recreation area - and Malyan Spout, a scenic waterfall in Goathland.
Inside the Whitby pub where Captain Cook and Charles Dickens once drankThe chapel has medieval origins and was at first a three-storey house - it became a place of worship in the early 18th century, but nobody knows exactly when.
"It's been lost in the mists of time. The chapel has been neglected, and it needs to be updated for a variety of purposes.
"It's a very peculiar little building. There's an odd little cellar - nobody knows where it leads to, and the structure is medieval. There are a lot of mysteries, but it would be a great place for local groups to meet."
John also believes it could become popular with visitors.
This is how Whitby became the goth capital of Britain"The goths would love it, and the Friends would be happy for them to look around. It's a delightful place and very historic, although it's not a very accessible building and there is no parking nearby.
"We're looking for sponsors so we can apply to develop it, and it's a real opportunity for Whitby. If not then they would have no option but to sell it, which would be a great shame."