Murray was heavy favourite to beat Anderson, having won five of their six previous meetings, but the big-serving South African played the match of his life to win 7-6 (7/5) 6-3 6-7 (2/7) 7-6 (7/0).
It means Anderson progresses to his first grand slam quarter-final, where he will now play Stan Wawrinka.
Murray led by a mini-break in the first-set tie-break but let it slip and the disappointment seemed to carry over into the second set, as he allowed Anderson to storm 5-1 clear.
The Scot fought back, breaking once, and almost again, but Anderson held his nerve to clinch a two-set lead.
Murray dug deep and sparked hopes of a comeback when he snatched the third set on a tie-break but the world number three left himself too much to do and Anderson closed out in the fourth.
“I felt like I had my opportunities there but didn’t manage to capitalise on them - when you’re playing against someone as good as him it’s tough,” Murray said.
“In the second set I felt like I was starting to put pressure on him there.
“When I had the break point at 5-3, I had a backhand pass that I really should have made.
“When you’re playing against players that are at that level, like him, you need to obviously make them think and then give them a chance to get nervous.”
Nerves could have played their part had Anderson been taken to a decider, particularly given the 29-year-old threw away a two-set lead to lose to Novak Djokovic at Wimbledon in July.
While the 6ft 8in ace-king moves into his first quarter-final, the defeat for Murray brings an end to an impressive run of 18 consecutive appearances in the last eight of major tournaments.
“That’s obviously something that is disappointing to lose,” Murray said.
“Obviously that’s many years of work that’s gone into building that sort of consistency. To lose that is tough.
“Also, to lose a match like that which was over four hours, it’s tough obviously after a couple of tough matches earlier in the tournament. It’s a hard one to lose.”
Elsewhere, Johanna Konta admitted her fairytale run at the tournament was like a “childhood dream come true” after she was knocked out by Petra Kvitova in the last 16.
Konta, ranked 97th in the world, defied all expectations to reach the fourth round at Flushing Meadows but Kvitova proved a step too far as the Czech sealed a 7-5 6-3 victory.
It is the first time Konta has gone past the second round at a grand slam and wins against ninth seed Garbine Muguruza and 18th seed Andrea Petkovic will see her climb into the world’s top 60 at the end of the tournament.
The 24-year-old was certainly not outclassed either under the lights of Arthur Ashe Stadium as Kvitova had to survive five break points in the first set en route to a hard-fought win.
“It’s obviously not the result I wanted,” said Konta.
“It would have been nice to have kept my run here going but I played against an incredibly tough player.
“She doesn’t give you much rhythm. She definitely doesn’t give you many chances to be able to take control in a point.
“So it was very difficult for me out there. But I had an amazing time on Ashe to be honest.
“My mum actually reminded me when I was speaking to her yesterday that when we were here back in like 2007 for juniors, I said, ‘This is like the most amazing stadium’.
“I completely forgot about that. I guess I had a little childhood dream come true, so that’s pretty special.”