Before lunch in a church-turned-café in the centre of York, I'm making notes when a man with a rucksack comes up. "Making notes, are you?" he asks. "You know the Angels window, do you? And the Noah's Ark window over there? And the reredos?"

I nod in a non-committal way, we have a bit of churchy chit-chat, and then he says: "I'm the world authority on woad, by the way. I'll give you a sticker." He delves in his wallet and hands one over and I stick it in my notebook. If ever you need to talk to someone about woad, Norman Wills of Long Sutton, near Spalding, is your man.

Church-visiting, it goes without saying, can attract its fair share of enthusiasts and eccentrics. And none was more eccentric, in a quiet sort of way, than Sir Stephen Glynne, whose 50-year passion for churches has brought me here in the first place. Glynne, a baronet whose Yorkshire Church Notes have recently been published, had, on the face of it, an uneventful life. Born in 1807, he lived at Hawarden Castle, near Chester. He became an MP, but never spoke in the Commons, perhaps aware that he would always be in the shadow of his brother-in-law, WE Gladstone.

Nor did he ever marry. Gossip that he might have been homosexual made his friends suggest finding a bride, but characteristically he decided that it would be too much effort. Self-effacing to a fault, he was a gentle soul who might have disappeared into worthy obscurity, had it not been for his obsession – visiting medieval churches.

Between 1825 and his death in 1874, he roamed the country for weeks at a time, "flying about", as he said, between churches. He clocked up an astonishing 5,500 of them in England and Wales, well over half the surviving total. He returned to some of them up to 40 years after his first visit, adding more notes to his first impressions on the same page of his notebook. The ink on the more recent entries tends to have faded less.

His usual plan was to take a train to the nearest town, then hire a horse or pony and trap to visit as many as 10 churches a day. Some of these were pioneering journeys. Most architectural historians of Glynne's day knew the churches in their immediate patches – Oxford, Cambridge, Kent – but, as Dr Lawrence Butler, editor of Yorkshire Church Notes, points out: "Glynne was going out and about in every county and occasionally coming across things which no-one had described before."

As well as writing on-the-spot descriptions, stronger on detail than interpretation, he did architectural sketches, sometimes inking them so heavily that they left a mirror image on the facing page of the notebook when he closed it too soon. He could have been the Nikolaus Pevsner of his day, but, modest to the end, he saw his expeditions as a purely private pleasure and never planned to publish his notes.

So it's good to have this painstaking new book, a fascinating historical gazetteer for modern "church-crawlers". It catches many buildings before Victorian restorers got their hands on them – and in about 20 cases before they were demolished.

And it has added appeal thanks to its charming illustrations – not Glynne's own ("careful, accurate, not particularly artistic" says Lawrence Butler), but 250 contemporary watercolours and drawings, many published for the first time. Glynne visited almost 400 of Yorkshire's 800 churches and didn't always like what he found.

In 1827, for instance, he climbed the steep steps up to St Mary's church in Whitby and found it "unsightly and deformed, from its irregularity of form and barbarous modern alterations". He similarly dismissed Leeds parish church ("does not contain much good work") and was outspoken to the point of petulance about St Peter's in Bradford (now the cathedral), whose interior, "though devoid of enrichment, would from its grand dimensions have a noble effect, if it were fitted with any attention to ecclesiastical propriety." St Mary's at Sand Hutton, near York, struck him as "a wretched little church... the interior looks something like the cabin of a ship".

And it wasn't just the churches. Sheffield in 1825 was a "large and dirty town" where "everything looks black and dingy"; Rotherham was "a gloomy and disagreeable town"; the road from Beverley to Hull passed through "flat, ugly country".

Bridlington had made a better impression, but his praise for the "truly magnificent" priory, then St Mary's church, is less interesting than his description of his journey there, before the seaside took over.
"The country between Driffield and Bridlington is for the most part bleak and bare. Beverley Minster is seen at 14 miles distant on the right, but the village and mansion of Burton Agnes embosomed in trees relieve the general dreariness of the country and shortly the sea comes into view, but the coast is particularly bare and exposed."

All told, Glynne comes across as the sort of antiquary lurking in MR James' ghost stories; a man who might push open a creaking church door one dark afternoon and find something ghastly in the choir stalls.
"He seems to have been a very refined and private sort of person," says Lawrence Butler, former senior lecturer in archaeology at Leeds and York universities. "He didn't seem to make friends very easily and didn't like the go-getting of politics. Other people with his background took up fox-hunting; he took up churches.

"He's interesting as an indicator of Victorian taste. Whether he would have done it differently if he had owned a camera, I don't know."
Dr Butler spent 10 years researching the Yorkshire notebooks, visiting every church mentioned (and settling on Patrington, out in Holderness, and Skelton, near York, as his own favourites). Glynne, he discovered, was supremely well-organised. "There's information in his diaries about where to get off and on the train and the distances between particular stations, to work out how much he could do in a day."

With that I mind, I retraced some of his steps in York, all of whose churches he visited. Some have gone completely – St Crux, St Lawrence, St Maurice, St Mary Bishophill Senior (not Junior), Christ Church. Others have noticeably altered: an extension to the tower at Holy Trinity, Micklegate, has gone (and a tourist from Hong Kong has summed things up in the visitors' book: "Old, excellent").

The exteriors of other churches have changed little, but their surroundings have changed enormously. St Mary Bishophill Junior (not Senior) still nestles in a quiet corner behind Micklegate, but is now hemmed-in by brick terraces and low modern flats. Some of its tilting gravestones are recognisable in the book's 1843 print.

There are changes of use. St John, Micklegate, has survived its many years as York Arts Centre and is now a bar with menu boards in Early English style. St Sampson, Church Street, has become an old people's centre and no longer occupies what the 18-year-old Glynne considered "a dirty and bad part of town". St Michael, Spurriergate, pictured in 1843 with locals posing picturesquely around a stagecoach, is now the Spurriergate Centre, with a thriving café that attracts young mums and world authorities on woad. Then there are changes of taste. The brick tower of St Martin, Micklegate, struck Glynne as "odious" and disfiguring; now it seems a novel change from stone.

Glynne admired the stained glass in All Saints, North Street, but didn't bother to describe the astonishing Pricke of Conscience window, which shows the last 15 days of the world with an all-horror cast of demons, skeletons and sea monsters, with people hiding in holes, stars falling from the skies and the world burning blood-red. Perhaps it was all a little lurid for the mild-mannered Sir Stephen Glynne.
"Sir Stephen Glynne?" says Fred William Luther, relief custodian at the box-pewed Holy Trinity, Goodramgate, now in the capable hands of the Churches Conservation Trust.

"He was a member of the Gladstone family, wasn't he? You often see him quoted."

How has he heard of him? "Oh, I've been a church freak since about four years old." And he goes into rhapsodies about reticulated tracery and cusped heads. What a treasurable world all this is.

The Yorkshire Church Notes of Sir Stephen Glynne (1825–1874) is published by York Archaeological Society in association with The Boydell Press (£30). To order from the Yorkshire Post Bookshop, call free on 0800 0153232 or go online at www.yorkshirepostbook
shop.co.uk. Postage and packing is £2.75