Elsie, 14, was attacked from behind and stabbed in the back and head as she walked through a railway tunnel just off a canal towpath in Wakefield in October 1965.
Detectives had been reinvestigating her murder since 2016 and were preparing to bring charges against convicted child killer and rapist Peter Pickering.
But the 80-year-old, known as the Beast of Wombwell, died last night after being taken ill in the secure psychiatric accommodation in Berkshire where he was being held.
His unexpected death has prompted the detective leading the Frost investigation to say publicly for the first that Pickering was believed to have murdered Elsie.
Elsie's brother, Colin Frost, said: "It's difficult to find the words to express how I feel. It's good that the police have put out his name and linked it to Elsie.
"The hard thing that's going to take some accepting is that he wasn't charged. It's been on the cards for quite a while. It's just painful really that that charge didn't come through.
"It would appear that we're still not going to get the justice we've been looking for, for Elsie. It feels like something has been taken away."
The case, one of Britain's longest running unsolved murder investigations, came back into the spotlight in 2015 when Elsie's family marked the 50th anniversary of her death.
Fresh appeals for information led to new intelligence and West Yorkshire Police's Homicide and Major Enquiry Team arrested Pickering in September 2016.
The pensioner was re-arrested and questioned for a second time in March 2017.
Mr Frost said that the family had been kept updated with the investigation's progress during weekly video conferences over the past 16 months or so.
"When Anne and I started out on this, we were just really looking for the case to be reinvestigated," he said.
"Everything we wanted at that time was achieved. However, as things have progressed over the last two years, our expectations have risen.
"Another difficult has been the fact that we've know what has been going on for so long and not been able to do anything."
Mr Frost said that after all the emotion and anxiety of recent years, the death of Pickering last night was a "bitter pill to swallow".
"Pickering was 80 years old and, believe me, we had questioned what would happen if he just dies," he said. "When the police rang me this morning, I just kind of lost it.
"I think there will be some angry, upset and annoyed people in Wakefield who have completed support us. It will be difficult for them too."
Thanking all those who had given support to the family, he said he hoped they would find some comfort in the police having named Pickering as their prime suspect.
"It's a big thing for us," he said. "There have been a lot of people still who have pointed the finger in the wrong direction. Hopefully they will see that [police comment] and they will understand we did have Elsie's murdered but we've not been able to go through the full process in court.
"We've remained happy with the way that the investigation has gone and supportive of West Yorkshire Police, but again we want to just thank everybody who has been constantly remembering Elsie for their support. We just didn't quite get there in the end."