Anti-doping authorities should concentrate on
tackling performance-enhancing drugs, and should consider taking "social" narcotics off their banned lists, he said.
Illegal drugs like cocaine and cannabis are on the list of substances proscribed by the World Anti-Doping Agency (Wada).
But Sheffield Central MP Mr Caborn told MPs on the House of Commons Science and Technology Committee that it should be for the police to deal with allegations of social drug use.
"We are not in the business of policing society. We are in the business of rooting out cheats in sport. That's what Wada's core function is about," he told the committee.
His comments were
condemned as "illogical
and damaging" by Conservatives, who urged Culture Secretary Tessa Jowell to disown them.
Sports stars were role models to many young people and needed to set a good example, said the Tories' Olympic spokesman, Hugh Robertson.
Mr Caborn told the meeting that Wada's code was based on three principles, performance enhancement, harm to the athlete and harm to the sport.
"I would give far more weight to the performance-enhancing of the those three," he said.
"And I would also look very seriously at the list, to take off what I believe are some of the social drugs."
Mr Robertson said: "Richard Caborn's comments are wrong and appear directly to contradict UK sport's very clear stance on drug taking by elite athletes.
"The performance of an elite athlete in any sport will be affected by taking recreational drugs and top sportsmen and women act as role models for many young people."
A spokesman for Mr Caborn later said that the sports minister was not calling for bans for illegal drug use to be dropped, but merely acknowledging that there was a grey area over substances like cannabis which had no known performance-enhancing effects.