There are still a few hours to go before midnight, and already we have lost two of our party.
Sir Simon Marsden, the paranormal photographer whose book, This Sceptred Isle, on Britain's most haunted places, has sensibly retired to the Hilton Hotel opposite.
He's been doing the publicity rounds, and while happy to chat at length about his own beliefs in the otherworldly spirits, his latest research into French apparitions and the time an unidentifiable force threw him against a stone plinth, he draws the line at spending the night with a group of psychics in a building with no roof.
"I like being scared," says Simon, clearly not wanting to be labelled a coward. "I like horror films, I like being made to jump, but ghost hunts are not my thing.
"When I was thrown against that stone it was broad daylight, and I have to say if I was a ghost I don't think I'd be making an appearance if a group of complete strangers were wandering round my house, would you?"
Shortly after Simon disappears into the night, a reporter from a Middlesbrough newspaper also makes his excuses and leaves – something about a long journey home and an early start in the morning.
It's all slightly disconcerting, not least because I don't have particularly fond memories of Clifford's Tower.
The last time I was here, some 15 or so years ago, my mother had an attack of vertigo on the top of the tower. The situation was not helped by my observations that it would be ok if we fell as there was only a car park below. As we attempted to get back on terra firma, she became paralysed on the stone spiral staircase.
Frightening for my mother, chronically embarrassing for me.
Still, the team from Yorkshire Psychic Investigations, who have been called in by English Heritage, seem undaunted, and begin their preparations for the night ahead, which, worryingly, involve turning the tower's gift shop, where we are told to go if things get a little too spooky, into a safe space with the help of a few crystals to ward off the bad spirits.
As the sun sets, we head to the top of the tower, and it quickly becomes apparent that along with my mother I have also left my sixth sense at home.
While I'm preoccupied with looking at a couple becoming increasingly amorous on the top of the mound, Bridget Relton – the main psychic in the team – is suddenly overwhelmed by emotion and has to retire downstairs for a break.
When we return, Bridget, accompanied by team member and notetaker Danielle, who bears an uncanny resemblance to the secretary in Ghostbusters, is more composed and begins pacing round the top of the tower in search of its former residents.
During the 12th-century, the city's Jews sought refuge in the then wooden building after being attacked by a mob. Given the choice of either being baptised or killed, they went for a third option, setting light to the tower and committing mass suicide.
Given the site's bloody history, the first spirit Bridget encounters seems surprisingly laid back.
"I've got a child," she says. "Probably male, not unduly worried about being here. I think his first name is Edward. I keep thinking 1282. He's quite slim, muscular and tall for the time."
Now, he seems like the kind of otherworldly presence I'd like to meet, but as we move round, Bridget's contact with the well-proportioned spirit becomes increasingly fragmented.
"There's deep, deep sadness, heartache," she says. "Something to do with somebody being brought here. I keep getting metal, I'm seeing battles, lies, deceit and great pain. I feel sick, I just need to pace it out."
A few minutes of pacing later and we are heading back to the safe space for a cup of tea and a slice of cake, which in itself seems slightly unusual, but not quite as odd as Bridget's tale of her childhood experience of spirits.
"I was three years old and I had an imaginary friend who was a 6ft black guy with a jackal's head," she says without batting an eyelid. Its not a sentence I ever expect to hear again in my life, but, thankfully, Bridget breaks the silence.
"I know it sounds odd, but I always thought everyone was like me. I thought everyone felt things."
In fact, she only realised she was different when a couple of years ago she spent a night in a supposedly haunted house in Richmond to raise money for charity, and was outed by another psychic.
From there, Yorkshire Psychic Investigators was born and the group has since carried out a number of overnight stays, including Newcastle Keep, which they all describe as the most disturbing place they have been to. Leeds Grand Theatre, before it shut recently for redevelopment, was another place they visited.
With cups of tea finished, we head for the chapel – once used as the site office, where staff reported various odd goings-on – armed with voice-activating equipment and video camera.
In the pitch black, Bridget's other senses go into overdrive.
"I can smell wood, a really strong smell, can anyone else smell it?" she asks. I can't, but for a moment I think my psychic intuition may be picking up when I hear footsteps. Unfortunately, it turns out to be another of the investigation team making his way down from the top of the tower, unable to contain his own fear of heights.
"I can feel grounded energy," adds Bridget.
"There is someone here. I don't know who he is, but there is definitely a presence. I am being drawn to the corner of the room. There is definitely something missing from that wall. I don't know what it is, but it's not how it once was."
As we make our way once more to the gift shop – it has, after all, been at least an hour since we had our last cup of tea – the nice man from English Heritage reveals there used to be a portcullis on the wall of the chapel, and Bridget cheerfully confesses that not everything always goes to plan.
"We once set up the voice-activated equipment, but the microphone was too close to the recording device. We thought we'd got a result, but it turned out it was the ticking-over of the machine which set it off, but at least we admit it when we get it wrong."
It's approaching 1am as we begin the final walk round the castle's ground floor, where two of the team have already experienced an odd sensation standing in the chimney hearth.
Bridget goes to investigate and decides the feeling is not coming from the chimney, but is left-over energy from the previous wooden building and site of the massacre.
As we move to the entrance, Bridget tells us she is being drawn to look above and has a tight feeling in her chest.
"I think there were two people suspended from here at some point. It's just a very strange feeling. It's not pleasant."
On that note, I decide to make my own excuses to leave – I have to be in work the following morning, it's getting cold, it has started to rain and I have all but given up on seeing a ghost.
However, I hear on the grapevine that during the séance, the group allowed one ghost to pass to the other side, which in anyone's books sounds like a good night's work.
sarah.freeman@ypn.co.uk

This Sceptred Isle, published by English Heritage Publications, priced £30, can be ordered from the Yorkshire Post Bookshop. Call free on 0800 0153232. Post and packing costs £1.95.