The tree - which could be nearly 400 years old if the legend is true - grows in a garden at the Charterhouse - one of Hull's oldest buildings.
It is said to be where the metaphysical poet and MP Andrew Marvell (1621 - 1678) played as a child when his father was Master of the Charterhouse.
The right-hand side of the tree - which is heavily laden with fruit - collapsed on Monday night or early Tuesday morning and it could now break up completely.
Tree experts who attended the Charterhouse - which has been on the same site for 700 years and is still an almshouse - today advised it may have to be felled.
The 39th Master, Canon Paul Greenwell said: "The initial view was that it probably has to come down - the first breath of wind it will snap off.
"Old people live here and we have to be compliant with health and safety.
"It is a shame because it has been here a very long time and it is very much part of Charterhouse life."
Historian Dr Robb Robinson said the only other mulberry tree in the city is the one standing outside William Wilberforce's house.
Mulberries have been grown in Britain since 1550.
In the 17th century King James 1 encouraged their planting in an attempt to produce silk-worms to make silk at home - but they did not grow well.
The rhyme Here We Go Round The Mulberry Bush is said to have originated from daily exercise around the tree which was at one time planted in prison yards.
Dr Robinson said: "It survived the English Civil War siege and the Blitz - hopefully the tree surgeon can give it a few more years.
"The tree is a living part of what is a very ancient and venerable institution - and it is very important to people who appreciate and understand the city's heritage."
Canon Greenwell said they would try and save what they could of the tree, but there was also plans to plant replacements.
The Charterhouse has a long and venerable history - it was founded as a Carthusian priory and almshouse in Hull a quarter of a mile from the town's walls in 1378 by Michael de la Pole, first Earl of Sussex.
It was closed as part of Henry VIII's Dissolution of the Monasteries in 1539, and some of the buildings were razed.
It was destroyed completely - together with the hospital buildings in 1643 prior to the second siege of Hull during the English Civil War.
A replacement was built in 1645, which was replaced again in 1780.
It now offers sheltered housing in 34 one- and two-bedroomed flats.