British Transport Police (BTP) announced the measure in a bid to "disrupt and deter criminal activity" on the rail network after the UK terror threat level rose to critical in the wake of the Manchester attack.
Armed officers have been patrolling on the London Underground network since December, but this is the first time they will travel on trains outside the capital.
BTP Chief Constable Paul Crowther said the deployment of armed officers will be focused on rail routes serving "big city locations" but will not be limited to those.
"The aim is for it to be unpredictable and widespread across the network, to create a deterrent and an immediate response to terrorists who may be thinking of (attacking) the transport network," he told the Press Association.
Among the first to be deployed were a group of four officers who boarded a Virgin Trains service at London Euston travelling to Birmingham New Street shortly after 2pm on Thursday.
Mr Crowther went on: "Since the devastating events in Manchester on Monday evening, our force has radically increased the presence of our officers nationwide.
"By having firearms officers on board trains we're ensuring that trains remain as safe as possible for passengers.
"Our patrols will be highly visible and passengers should feel comforted by their presence."
Mr Crowther urged passengers not to be alarmed by the presence of armed officers.
"These are normal officers," he said. "They are very approachable. They are there to engage with people, to talk to them about any concerns they've got. I'd really encourage people to approach them."
Since the Government activated Operation Temperer, BTP has benefited from additional firearms officers from the Ministry of Defence Police, who will remain at stations in London.
Mr Crowther insisted that BTP does not have any specific intelligence in relation to a threat to the rail network, but called for the public to remain vigilant.
He said: "In light of the dreadful events over these last few days, and the increase to the national threat level, nothing should be considered too trivial to report and any information - regardless of whether you feel it is significant or not - may be important to all of us.
"The cowardly acts of terrorists will never weaken our resolve."
Paul Plummer, chief executive of the Rail Delivery Group, representing train operators and Network Rail, said: "It is absolutely right that the British Transport Police takes all necessary steps to keep the travelling public and visitors to stations safe.
"Together with the police, everyone on the railway will be remaining vigilant."
Manchester Victoria station, which is attached to Manchester Arena where the blast happened, was closed to allow a forensic search of the area and has not yet reopened.