High Court judge Sir William Blackburne threw out her and her husband's claim against the Royal Borough of Kensington and the committee which refused the couple a key to the exclusive amenity surrounded by railings. They now face paying most of the costs of the case, estimated at about 173,000.
Sir William said Mrs Herrmann, 58, and her husband, Jeffrey, 64, moved from New York and bought 37 Ovington Square in 2008 after being assured by solicitors that they would enjoy a right to use the locked garden they could see from the front room of their new home.
"The Herrmanns attached and continue to attach considerable value to this as No 37 itself has no garden.
"Moreover it is not in dispute that, apart from the pleasure afforded by access to an amenity such as the garden, the existence of garden right adds considerably to the value of those properties which enjoy it."
The couple claimed that under the Kensington Improvement Act of 1851 they were entitled to a share of the garden because they lived in a house within the square area.
They also sued the Ovington Square Garden Committee because it had refused to make available a key to the garden.
The judge found that the Act governing those owners entitled to use the garden does not provide a definition of which houses are included in the square.
But he said he had "reached the clear conclusion" that only homes with a front or side facing into the square were covered by the Act.
"Since the front or side of No 37 does not face the Square in the sense intended, it follows that the Herrmanns, as occupiers of that property, do not enjoy the garden right that they claim."
He said it was irrelevant that it was possible to see the garden from their front window.
Sir William said if he had found for the couple, it would have caused uncertainty over the definition of a square and "potentially opened the door" to numerous other claims.
Mrs Herrmann, of successful hedge fund Paulson Europe, married her lawyer husband in 2000.