Police condemned the violence in Bolton, Lancashire, on Saturday which erupted between supporters of controversial right-wing group the English Defence League (EDL) and Unite Against Fascism (UAF).
Thousands of people took part in a rally in Victoria Square and at least two police officers and members of the public were injured.
At least 74 arrests were made, more than 55 from the UAF and nine from the EDL, police said.
Officers said UAF protesters were responsible for most of the trouble – and the organisation's joint secretary, Weyman Bennett, said he was arrested and charged with conspiracy to organise violent disorder.
He hit out at the policing, insisting he was grabbed by officers while protesting peacefully.
"I have been to more than 200 demos and never been arrested," he said.
"Officers came up to me as soon as I arrived and said they would arrest me. They are hostile to anti-racists and there needs to be an investigation. Police neutrality needs to be questioned."
Assistant Chief Constable Garry Shewan, from Greater Manchester Police, who led the policing operation, said his officers faced "a lot of hostility" and "unwarranted attacks", resulting in injuries.
He said the EDL had protested by flag-waving and "vitriolic name-calling" but he reserved his criticism for the UAF.
"We have seen groups of people, predominantly associated with the UAF, engaging in violent confrontation," the officer said.
"It is clear to me that a large number have attended with the sole intention of committing disorder and their actions have been wholly unacceptable."
A police helicopter flew overhead as hundreds of riot police and mounted officers battled to keep control of the rival groups, using batons and police dogs.
At the peak of the protest there were 2,000 EDL protesters in the square and around 1,500 from UAF, police said. Prior to the demonstration, both groups agreed to stay in two separate designated areas in front of the town hall, separated by barriers.
But they tried to smash through the barricades and many escaped from the pens, causing pockets of trouble away from the square.
Several demonstrators were hurt, including one man who was taken to hospital for treatment to a minor head injury.
Another man was taken to hospital for treatment to a minor ear injury, a 16-year-old girl was treated after suffering a panic attack, a 19-year-old man received treatment for an ongoing health problem and one was treated after collapsing.