Scientists demonstrated a link between the chemicals, sprayed on to crops to prevent damage by insect pests, and reduced sperm count and quality.
The study of 155 men found that those who ate the most fruit and vegetables with high pesticide residue levels had a 49 per cent lower sperm count than those who consumed the least.
They also had 32 per cent fewer sperm that was normally formed.
While the American nutritionist who led the research suggested switching to organic produce, other scientists said the findings should be treated with caution.
Leading British fertility expert Prof Allan Pacey, of Sheffield University, said: “The idea has been raised before, but to my knowledge this is the first paper that has investigated this question in a systematic way.
“That said, whilst the results are tantalising, they should be interpreted with caution as the study is not without its flaws and limitations.”
Among those in the “high” category were apples, pears, peppers, spinach and strawberries.
“Low” residue foods included peas, beans, grapefruit and onions.
Men with the highest intake of pesticide-heavy fruit and vegetables - at least 1.5 servings per day - had an average total sperm count of 86 million sperm per sample compared with 171 million for those whose consumption was lowest.
Low-to-moderate residue levels did not appear to affect semen quality, according to the results published in the journal Human Reproduction.
In fact, consumption of fruit and vegetables with low pesticide levels was significantly associated with having more sperm that was normally formed.
Study leader Jorge Chavarro, assistant professor of nutrition and epidemiology at the Harvard TH Chan School of Public Health in Boston, said: “These findings should not discourage the consumption of fruit and vegetables in general.
“In fact, we found that total intake of fruit and vegetables was completely unrelated to semen quality.
“This suggests that implementing strategies specifically targeted at avoiding pesticide residues, such as consuming organically-grown produce or avoiding produce known to have large amounts of residues, may be the way to go.”
Prof Pacey pointed out that the scientists did not measure the pesticide residues in the actual food the men ate but inferred the levels from other data.
It was also possible that some other aspect of diet or lifestyle might have been affecting men’s sperm quality.
He said it was also important men were not discouraged from eating fruit and vegetables.
He added: “There is also no evidence at present that switching to organic fruit and vegetables will improve semen quality, although it will obviously do no harm.
“But I hope that this paper will encourage other studies to take place in this area, so that we might be able to answer the question once and for all.”
Jackson Kirkman-Brown, of Birmingham Women’s Fertility Centre, said the research was part of a growing body of evidence that diet can affect male fertility and sperm quality but men should not worry unnecessarily.
“Men wishing to optimise their sperm quality should still eat a healthy balanced diet until more data is available,” he said.