Some 2.54 million new cars were registered in 2017 compared with 2.69 million the previous year, the Society of Motor Manufacturers and Traders (SMMT) said.
This marks the first fall in sales for six years.
December saw an even sharper decline of 14%, while demand for diesel models fell by 31% last month and 17% over the year.
SMMT chief executive Mike Hawes predicted the market's decline will continue in 2018, with registrations down 5-7%.
He described 2017 as a "very, very volatile year" with the fall in new car sales caused by "a drop in both business and consumer confidence".
Mr Hawes also revealed that uncertainty over transitional arrangements following the UK's withdrawal from the European Union in March 2019 has caused some manufacturers to delay investment decisions.
If there is no clarity by the end of March they will have to start implementing "contingency plans" which could harm funding for UK operations, he added.
Asked about the decline in diesels, Mr Hawes claimed "confusing anti-diesel messages have caused many to hesitate before buying a new low emission diesel car".
Rather than choosing alternatives, many people are "just sitting on their hands", he said.
In November's Budget, Chancellor Philip Hammond announced a tax hike on new diesel cars from April 1.
All diesels are expected to be subjected to a one-band increase in the first-year vehicle excise duty (VED) rate.
This came after the Government unveiled plans to ban the sale of all conventional diesel and petrol cars by 2040.
The SMMT said the amount of CO2 emitted by the average new car bought in 2017 is expected to have risen year-on-year for the first time in 20 years, reaching 121 grams per kilometre.
This is because the market share of diesel cars, which produce less CO2 than petrol cars as they are more fuel efficient, dropped from 48% in 2016 to around 42%.
The increase makes it harder for manufacturers to meet an EU target of 95 grams per kilometre by 2021.
Petrol registrations dropped 2.1% in December but rose 2.7% in 2017 as a whole.
Alternatively-fuelled vehicles reached a record market share of 4.7% last year.
New car sales to private buyers were down 6.8%, while fleet purchases decreased by 4.5% and business demand dropped by 7.8%.
Mr Hawes stressed that 2017 was still around the third best year for new car registrations in the past decade and followed record performances in 2015 and 2016.
"The market is still at historic highs," he said.
Jim Holder, editorial director of What Car? magazine, said turning the sharp decline at the end of 2017 into a loss of just 5-7% in 2018 will be a "real challenge" for the industry.
"That's going to take a massive swing in public perception," he told the Press Association.
"I would say there is a risk of double digit (decline) unless all the good news stories come true, and that includes turning some pretty entrenched views around on the economy and diesel."