Around 30 to 40 protestors, who had travelled from various areas of the country, had gathered close to the Minster before the scuffle broke out just after 3pm.
North Yorkshire Police, who were monitoring the protest, said they quickly brought the situation under control and arrested four of the protestors and one member of the public.
The rest of the protest group were escorted back to York railway station by police officers to prevent any further disturbances.
Superintendent Dave Hannan of North Yorkshire Police, said: “Everyone has a right to peaceful and lawful protest, however, the police will not tolerate racist, offensive or violent behaviour.
“The situation was soon brought under control with offenders arrested and the remaining protestors escorted out of the city.
“Detectives will continue to investigate the incident and will take action against any further offences they identify as a result of their enquiries.”
The arrested men include a 25-year-old man from Weymouth who was arrested to prevent a breach of the peace and for failing to supply his details.
A 25-year-old man from West Bromwich on suspicions of racially aggravated public order offence.
A 31-year-old man from Horley, Surrey on suspicion of using threatening words/behaviour. A 21-year-old man from Swansea on suspicious of a public order offence and a 27-year-old man from York on suspicion of using threatening words/behaviour.
Separately, far-right demonstrators will take part in a second march in Dover, following arrests as protesters took to the street on Saturday.
Two men were arrested in the town on suspicion of being involved in violence at a previous protest in January.
Saturday’s march, with counter-protests, passed peacefully following a large police presence in the town.
Before the demonstration began, police detained the men, a 33-year-old from Margate and a 35-year-old from Dover, on suspicion of violent disorder in connection with a protest on January 30 which erupted into violence.
A second march is set to take place later beginning at Dover Priory railway station.
Following the January demonstration, in which smoke bombs were let off and bricks thrown at police, officers seized more than 20 weapons, including a knuckle duster, and three people were arrested.