The report – titled Global Player or Subsidy Junkie? Decision Time for the BBC –acknowledged the importance of the corporation as "the UK's strongest media asset".
But it said that scrapping the fee would better equip the BBC to operate as a major business on the world stage.
The document was compiled by David Graham, a former BBC producer who worked on Nationwide, The Money Programme and Panorama.
It said: "Continued commitment to subsidy via the licence fee will mean scaling back the BBC and diminishing its potential contribution."
The BBC has been undergoing a wide-ranging strategy review.
Last month, Culture Secretary Jeremy Hunt suggested that the licence fee could be cut, telling the Daily Telegraph he could "absolutely" see viewers paying less than the current 145.50.
Mr Graham predicted that while "the need for reform goes beyond the BBC... a serious debate on the BBC will start, in my view, in 2011 and 2012".
He argued that a voluntary BBC subscription would free the broadcaster from competitors' hostility.
Mr Graham said: "It would harness its huge reputation and high-calibre staff to new, ambitious objectives, and would encourage the BBC to lead the drive to create a bigger presence for itself and other UK players in global media."
He added: "Today, it is not unfair to describe the BBC as a subsidised entertainment firm with some non-commercial obligations.
"Some public service messages need to reach wide audiences and an entertainment medium can provide them. However, subsidising the entertainment itself is a different matter."
Mr Graham suggested that people support, at least in principle, public funding for "social priorities" like news and current affairs but see entertainment as "personal priorities".
He said the licence fee "discriminates" against "genuinely free" channels.
"People are paying for the right to watch free channels, supported by advertising, like ITV and Channel 4, because they cannot watch them without mandatory payments to a competitor, the BBC."
Mr Graham predicted the idea of a new basis for the licence fee would soon be mooted, to be based on broadband access rather than a given receiving device, as on-demand services grow in popularity.
He said: "There will be justified resistance to this, especially if the fee is, as now, proposed as the primary method of funding the BBC."
Mr Graham described the BBC as "the UK's strongest media asset" and a "global brand".
He put forward an alternative model whereby the BBC would allow licence fee payers to lapse or switch to voluntary subscription. Mr Graham, chief executive officer of media consultancy Attentional, argued that with a subscription comparable with HBO, the BBC could offer a comprehensive range of options.
n The BBC is holding talks with staff over concerns about new pension proposals.
Some employees are reported to be considering disrupting the coverage of key events such as the Pope's visit, in response to plans to change the pension scheme. Pope Benedict will make his first state visit to the UK in September.