English Heritage has announced that it will not proceed with the project at Clifford's Tower in York.
The plan for a tourist building at the mound below the tower sparked public opposition and a High Court challenge after planning permission was granted.
The decision not to go ahead with it was made after separate plans were announced for a Â£30m city centre redevelopment scheme.
It also follows a new North of England Director, Andrea Selley, being appointed by English Heritage.
She said: "There are a number of things which influenced this decision. The momentum behind transforming the Castle Gateway area is genuinely exciting and it may open up opportunities for Clifford’s Tower that previously did not exist.
“We also became increasingly conscious that many people have a deep emotional attachment towards the mound."
Clifford's Tower is featured in countless images of the city and is last remaining part of York Castle, which was once at the centre of Royal power in the north of England.
The visitor centre plan included a gift shop and interpretation centre, along with new stairs up to the castle and improved tourist facilities inside the keep.
It received planning permission from City of York Council in October 2016.
A legal challenge was made by objectors to the scheme, led by independent city councillor Johnny Hayes, but their application for judicial review was rejected.