BAT, which currently owns 42.2% of Reynolds, had offered to buy the remaining 57.8% of the company last month.
However, the merger had yet to be approved by Reynolds’ board of directors.
Reuters and Bloomberg are now reporting that the offer has been rejected.
Sources told Bloomberg that the two tobacco companies are still in negotiations and BAT is willing to increase its bid.
A spokesman for BAT declined to comment.
The deal was set to bring a raft of global brands under one roof, including BAT products like Dunhill, Lucky Strike, Rothmans, Kool, and Kent, and Reynolds’ brands like Newport, Camel, Pall Mall, Doral, Misty, and Capri slims.
BAT’s £38.3 billion offer valued the remaining 57.8% stake at 56.50 US dollars (£46) per share, representing a 20% premium against the closing price of Reynolds’ shares on October 20.
The offer comprised 27 billion US dollars (£22 billion) worth of BAT shares and 20 billion (£16.3 billion) US dollars in cash.
The companies’ combined earnings would be significant. BAT last year reported £5 billion in operating profit, while Reynolds reported net income of 3.3 billion US dollars (£2.7 billion).
BAT first invested in Reynolds through a deal which saw BAT’s US subsidiary Brown & Williamson merge with RJ Reynolds in 2004, giving BAT its original 42% stake in the US firm.
In 2014, BAT pumped 4.7 billion US dollars (£3.8 billion) into Reynolds in order to maintain its stake in the company following a mega-deal that saw Reynolds acquire sector peer Lorillard for 27.4 billion US dollars (£22.3 billion).
The proposed merger was expected to help BAT gain a further foothold in the US, and give the new company a significant presence in high-growth markets including South America, the Middle East and Africa.