President Uhuru Kenyatta named an opposition politician and retired army general, Joseph Nkaissery, to be the new interior minister, in charge of security.
Police Chief David Kimaiyo said he resigned for personal reasons.
Public pressure had been mounting for the two officials to be replaced following a string of extremist attacks.
The changes come after Islamic extremists from Somalia killed 36 quarry workers in northern Kenya early yesterday, targeting non-Muslims just like an attack 10 days ago on bus passengers.
The killings happened in Mandera County near the border with Somalia and the attackers escaped.
The group al Shabab, which has been battling for years to establish hard-line Islamic rule in Somalia, claimed responsibility for the killings.
A group of about 50 heavily armed people walked into the camp next to the quarry at 12.30am as the workers were sleeping and fired warning shots, said Peter Nderitu, who works at the quarry.
Mr Nderitu said when he heard the shooting he ran and hid in a trench from where he could hear his colleagues being asked to recite the Shahada, an Islamic creed declaring oneness with God. Then gunshots followed.
He only rose from his hiding place two hours later when he was sure there was no more movement, he said. The bodies of his colleagues were in two rows and nearly all had been shot in the back of the head, he said.
The gunmen singled out non-Muslims and killed them, said the police.
Al Shabab spokesman Sheikh Ali Mohamud Rage said the latest attack was a response to Kenya’s troop presence in Somalia and alleged atrocities committed by the Kenyan army.
Al Shabab claimed a recent air strike killed innocent people and destroyed their property. The Kenyan government said the air strike was in response to a November 22 al Shabab attack on bus passengers in Mandera County that left 28 people dead.
In that attack, the non-Muslims were also separated from other passengers and shot dead.
About 100 non-Muslims last week sought refuge at the army base in Mandera, demanding that the government evacuate them.
Mr Kenyatta’s chief of staff, Joseph Kinyua, attempted to persuade non-Muslim from leaving Mandera County, whose population is predominantly Kenyan Muslims of Somali origin.
Those who wanted to be evacuated argued that they could not stay because governor himself is not safe. Mandera governor Ali Roba survived an improvised explosive device attack on his vehicle on October 15.
The quarry attack came hours after a hotel in Wajir, also in northern Kenya, was hit by a grenade and gunfire, killing one person and wounding 13, said police.
Lebanese authorities detained a wife and suspected son of Abu Bakr al-Baghdadi,the leader of the “Islamic State” (IS) group several days ago, and she is being questioned, two senior officials have said. A military official said the woman and child were detained about 10 days ago while carrying fake ID cards.