Alex Oxlade-Chamberlain had given England the lead in the 70th minute after some poor Scotland defending, but Griffiths picked the perfect moment to break his international duck in his 13th cap.
The Celtic player curled 25-yard free-kicks into either corner of Joe Hart's net within three minutes, the second in the final minute of the 90.
However, Kane netted from close range three minutes into time added on to seal a 2-2 draw and maintain England's eight-year unbeaten run in qualifying matches as they stayed on course for a place in Russia.
Home skipper Scott Brown had set the tone for an unforgettable World Cup qualifier inside three minutes as he introduced himself to Dele Alli with a late challenge which earned a yellow card, and Scotland continued their aggressive start by forcing four early corners.
With a significant height disadvantage, they could not threaten directly from any and Joe Hart was fully behind Griffiths' dipping strike after one delivery ultimately broke to the forward outside the area.
Scotland were playing with a three-man central defence - all left-footed, including Celtic full-back Kieran Tierney - but wing-backs Ikechi Anya and Andrew Robertson were pegged back for much of the first half as Griffiths grew increasingly isolated.
Kane got two sights of goal when the visitors managed to penetrate. However, he blazed over after only partially bringing down Alli's lofted pass and then failed to make the most of Adam Lallana's cutback.
England looked more dangerous on the rare occasions they got their opponents turning and Kane had another half chance, with Craig Gordon stranded after coming out of box to head away a long ball.
Tierney headed his long-range chip off the line but Robertson's slip allowed Marcus Rashford a good chance. The Scotland goalkeeper recovered though to turn the shot wide with his foot.
Gordon made two more saves before half-time, pushing efforts from Lallana and surprise starter Jake Livermore off target.
Livermore hit the post five minutes into the half after his shot took a heavy deflection off Robertson and Scotland, with James McArthur on for the injured James Morrison, survived some nervous seconds before the ball was finally safe.
Griffiths set up a great chance when he held the ball up and picked out Robertson 18 yards out. The Hull player had time to take a touch but sliced over.
Griffiths soon set up Stuart Armstrong further out but his club-mate's chip was too high.
The home fans turned up the volume as Scotland pressed deep in the opposition half and Kyle Walker took his time with a throw-in as England regained their composure.
Both sides made changes in the 65th minute. England brought on Oxlade-Chamberlain for Rashford while Bournemouth winger Ryan Fraser made his Scotland debut.
Gordon soon made a decent stop from Kane's header but the Scotland goalkeeper failed to control Tierney's pass back and the visitors scored from the resulting throw-in after he scrambled the ball out under pressure.
Oxlade-Chamberlain had his back to goal outside the box but turned and rode Brown's challenge and got space to shoot from 15 yards after Armstrong ran into Tierney. The effort was straight at Gordon but the keeper could only flap at it and help it into the net.
Brown soon avoided a second yellow card for a late challenge on Alli and Lallana missed two chances to settle matters, heading over and having an effort saved.
Griffiths was well wide with a couple of efforts but he soon found his range in sensational fashion, with two free-kicks putting Scotland in the driving seat.
England were given a chance from closer in but Gordon looked to have averted the danger as he pushed Eric Dier's free-kick away. However, Scotland gave the ball away and it was soon returned into the box by substitute Raheem Sterling, whose ball found Kane unmarked to cushion a volley home.