The group of experts signed an open letter to the WHO and International Olympic Committee (IOC) asking for the Games to be halted or held in another location “in the name of public health”.
Professor Amir Attaran, one of the co-authors of the letter, told the Press Association that the Games risked becoming “the Olympics of brain damage” if they went ahead as planned this summer.
But the WHO told the BBC that suspending the Olympics or staging them elsewhere would “not significantly alter” the spread of the virus, which is linked to serious birth defects.
The letter signed by 152 global health experts states that the Zika virus has more serious medical consequences than previously known and that the emergency contains “many uncertainties”.
Some 500,000 foreign tourists are expected to attend the Games, which would lead to the virus being spread across the globe to areas it may not have reached if it was not for the Olympics, the letter warned.
The experts, many of whom have worked with the WHO, also voiced concerns over the relationship between the UN’s health agency and the IOC, which they said entered an official partnership in 2010.
Prof Attaran called the partnership “beyond the pale” and called into question the independence of the WHO.
“It is ignorant and arrogant for the WHO to march hand-in-hand with the IOC,” he said. “How can it be ethical to increase the risk of spreading the virus? Just because a fire has begun doesn’t mean you need to pour gasoline on it.”
The WHO declared the Zika epidemic to be a global emergency in February and in its latest assessment this week, said it “does not see an overall decline in the outbreak”.
Professor Attaran said he believed allowing the Olympics to go ahead in Rio would lead to the birth of more brain damaged children.
The majority of those infected with Zika will have no symptoms, but for others it can cause a mild illness with symptoms including a rash, fever and headache.
Serious complications that arise from infection are not common, but experts have said the virus can cause microcephaly, where babies are born with abnormally small heads due to the fact their brains have not developed properly.
Pregnant women have already been advised not to travel to Rio and the WHO have predicted the Zika risk in August would drop since it will be the south American winter and there should be fewer mosquitoes.
The experts’ letter dismisses this claim because many visitors to Rio may return to countries with a hotter climate.
The IOC has previously said it sees no need to cancel, delay or move the Rio Games because it had been “advised by the experts that the situation will improve over the next three months.”
WHO director-general Dr. Margaret Chan said earlier this month that the UN health agency is increasingly worried about Zika but stopped short of recommending the Olympics be moved or postponed.
In a statement, the WHO said: “Brazil is one of almost 60 countries and territories which to date report continuing transmission of Zika by mosquitoes.
“People continue to travel between these countries and territories for a variety of reasons. The best way to reduce risk of disease is to follow public health travel advice.”
No Olympic Games have been moved because of health concerns, but in 2003 Fifa moved the Women’s World Cup from China due to the respiratory virus Sars.
The opening ceremony of the Olympic Games is due to take place at Rio’s Maracana Stadium on August 5.
Here are some facts about the disease:
:: What is Zika?
Zika is a mosquito-borne virus which was first identified in rhesus monkeys in Uganda in 1947 and moved to humans in Uganda and the United Republic of Tanzania in 1952.
:: How is the virus transmitted?
It is spread by the Aedes mosquito, which usually bites during the day.
Zika can also be transmitted from person to person through sexual contact.
Other modes of transmission, such as blood transfusion and perinatal transmission, are currently being investigated.
:: What does Zika do?
The main concern from Zika is the threat to babies in the womb.
Zika is now known to be a cause of microcephaly, where babies are born with abnormally small heads due to the fact their brains have not developed properly.
Microcephaly can cause serious developmental problems and death in some cases.
:: What are the symptoms of Zika?
The majority of people show no sign of infection and for those who do display symptoms they are usually mild and last two to seven days.
Symptoms include fever, itching, conjunctivitis, red and sore eyes, headaches, rashes and joint pain.
A disorder of the nervous system known as Guillain-Barre syndrome has also been linked to the infection.
:: What can people do to protect themselves?
There is currently no vaccine against Zika. The best protection is to take steps to avoid being bitten by mosquitoes.
Official health advice says people should use insect repellents, cover up with long-sleeved clothing and keep windows and doors closed.
Still water and stagnant water such as in buckets and ponds also attracts mosquitoes to lay their eggs.
:: Which countries are affected?
Outbreaks of the disease have been recorded in Africa, the Americas, Asia and the Pacific.
So far 64 countries and territories have reported occurrences of the virus since 2007.
There is currently widespread transmission in much of Central and South America, including Brazil which has “increasing or widespread transmission”.
:: What is the current travel advice?
Across the UK, it is recommended that pregnant women should postpone non-essential travel to countries with an active and established Zika virus transmission until after pregnancy.
In addition, women should avoid becoming pregnant while travelling in an area with active Zika virus transmission, and for 28 days after their return.