Oliver’s controversial decision to show Huddersfield striker Steve Mounie a straight red card in the 32nd minute for his challenge on Yves Bissouma proved pivotal.
Huddersfield were leading 1-0 through Mathias Jorgensen’s first-minute header, but Brighton hit back after Mounie’s departure with goals from Shane Duffy and Florin Andone, who marked his full debut with a second-half winner.
Replays of Mounie’s challenge on Bissouma showed there was clearly no intent, but he did catch the Mali midfielder high on his shin.
“For me, the red card was not a red card. But he decided to give it,” said Wagner, who was angry other decisions had not gone his side’s way. “Maybe it was on the brink, but for me, it wasn’t.
“(Leon Balogun’s late) challenge on (Town defender) Erik Durm was a red card for sure, as well. I must have consistency in my decisions and from my point of view, this wasn’t the case.”
Wagner was also incensed his side were not awarded a first-half penalty when Alex Pritchard appeared to be wrestled to the ground in the first half.
“If you had seen the Pritchard incident, it was a clear penalty and a red card because it was a clear and obvious goal chance two yards from the goal. That is what we should discuss.
“When the big decisions go against you, football makes for no fun and no football team has a chance of being successful.”
The Terriers made a flying start in pursuit of their first back-to-back league wins since February, with Jorgensen heading the fastest Premier League goal of the season.
Brighton equalised in first-half stoppage time through Duffy’s towering header and Andone won it for the Seagulls with a near-post header from Solly March’s excellent 62nd-minute cross.
Brighton boss Chris Hughton said Oliver had been in an ideal position to make the decision, which was consistent with how the laws of the game are currently being interpreted.
“We experienced that two games ago with Dale Stephens and we are probably at a time now where a couple of years ago, it probably would not have been a sending off,” Hughton said.
“But the game has changed. We experienced it two weeks ago and somebody will experience it next week and the week after and the week after.
“The referee has to make the decision on the force of contact. There is no doubt he has caught him because Bissouma’s shin shows that.”
Hughton was delighted with his side’s response after they had been held 1-1 at home last week by Leicester, who played for over an hour with 10 men.
“I was really pleased with us against the 11 men and then going against 10,” he added.
“Last week, we have played against 10 men and we were far better against the 10 of Huddersfield than what we were last week.”