He went missing in the park at around 11am and was suffering from dementia.
He was found in the lake after a member of the public raised the alarm.
West Yorkshire Police said:
"At 12:03pm today police received a call raising concerns for a 78-year-old man with dementia who had gone missing in Roundhay Park about an hour earlier.
"Officers were immediately dispatched to the area to search for the man.
"At 12:15pm a call was received from a member of the public reporting a man in the lake.
"He was recovered from the water by a PCSO who began emergency treatment. He was taken to hospital by ambulance but was pronounced dead a short time later.
"There are not believed to be any suspicious circumstances and the Coroner’s Office has been informed. A post-mortem is due to be carried out to establish the cause of his death."
Officers are looking to build up a picture of the man’s movements in the park prior to him being found.
He is described as white, about 5ft 8ins tall, and of slim build. He was wearing black jeans, a dark blue V-neck jumper, glasses and a blue baseball cap.
Anyone who saw him in the park during the time he was missing is asked to contact Leeds District CID quoting reference 13180407119.
Park visitors said the man was seen in the Lakeside Cafe before a woman accompanying him realised he was missing.
A police cordon remains at the scene, which is close to the cafe, although all paths have re-opened to the public.
A woman who did not wish to be named said:-
"I was sitting in the cafe having a coffee and a lady was with the man. She went up to order and I then saw her at the table looking for him. She spoke to some other people and then she went outside to make phone calls. The police turned up after that."
A member of staff at the Lakeside Cafe added:-
"The man and his family are regular customers. He's elderly. It's very sad"
In March 2017 a jogger found the body of a woman who had entered Waterloo Lake near the cafe.