The Terriers were rescued by Adam Clayton’s late penalty to cancel out Rob Hulse’s opening goal for Charlton.
But Southern’s controversial red card, following a challenge with Michael Morrison, changed the outcome of a game that Huddersfield were controlling.
It meant play-off chasing Town had to settle for a draw - but Grayson knows it could have been so much better.
“I’ve seen it on the DVD and it’s two committed players going in 50-50 on a wet pitch,” he said.
“He clearly got the ball first.
“But the referee made a quick decision. It’s changed the outcome of the game.”