The 51-year-old, who is backing Labour's In for Britain campaign, told an audience in South London that the French are working tirelessly behind the scenes to keep undesirable people from coming to Britain.
After filming his Sky series Extreme World at the Calais 'Jungle' camp in 2015, he made a recent return to Eastenders' Albert Square for character Peggy's final farewell.
He said: "I witnessed first hand the situation going on in Calais. Undoubtedly some of the people there - some of them but not all of them - a minority of them, are undesirables.
"They have left their country for a reason and some of them are criminals and it's only the extra work done by French police and their border forces working in collaboration with ours, that scrutinizes and filters out a lot of the people that would actually be of harm to our country."
He said the border operation in Calais is currently underfunded for the amount of work they need to do.
"I've just come back from Iraq and Syria and there is a very clear route of people leaving certain countries who probably don't have the best interests of our country at heart," said Mr Kemp, who lived in Selby in the late 1980s when he appeared in soap Emmerdale Farm.
"It is only the good work and collaboration that goes on between the French Government and the British Government and other governments of Europe that is protecting our borders and it is done without us knowing."
As a long-term Labour supporter, Mr Kemp said he is giving his backing to Barnsley Central MP Dan Jarvis' campaign to remain in the EU.
He said he agreed with Mr Jarvis' view that Russia would be keen to see Britain leave Europe.
He said: "[Dan] made a very good point. Why is it the Australians, the Canadians and the United States want us to remain in the EU but Mr Putin wants us to leave?
"Surely that is a good enough indication as to how he would like to in someway diminish the security of our country."
Former paratrooper Mr Jarvis said he was making the "patriotic case" for Britain to stay in the EU at the June 23 referendum as he addressed the Poppy Club in East Mitcham on Wednesday May 25.
Britain has a bi-lateral agreement with France to allow the UK border force to operate in Calais.