Mr Javid warned that people who promoted extremist views were making it easier for terrorists to recruit British youngsters to their cause.
He said Muslim parents should question the views of Imams who refused to condemn terror atrocities like the massacre in Tunisia which left 30 Britons dead.
Mr Javid’s comments follow a call from David Cameron for British Muslims to do more to tackle those in their communities who ‘’quietly condone’’ IS’s extremist ideology.
The Business Secretary told BBC1’s Andrew Marr Show: “When I was growing up as a young Muslim in Britain, the extremist ideology that you see today just didn’t seem to be around, it didn’t seem to be an issue.
“Something has clearly changed over a number of years. I think the Prime Minister is right when he talks about if we are really going to combat extremism and terrorism then we have got to combat the ideology. It’s not about just military might.
“I do think there are too many people - let’s call them non-violent extremists - that feed this ideology. They may not agree with the terrorism ... but they might agree with the narrative.
“We have got to realise the damage that they are doing. They are, in that case, it’s like taking a young person to the door of the terrorist. Then you make the terrorist’s job of recruitment a lot easier because then they just have to beckon them in.
“I think all people, Muslims included - I guess especially Muslims - they have to talk to these people, let’s say the non-violent extremists, and say ‘what you are doing, spreading this ideology, you are hurting us, you are hurting yourselves ultimately, it must stop’.
“I think all people, but including British Muslims and others, have to do more to combat this poisonous ideology.”
He added: “If you are a Muslim parent and you send your children to a mosque, if the Imam in that mosque hasn’t condemned what happened in Tunisia, for example, you should be asking yourself, ‘why hasn’t he done that, what’s stopped him from doing that?’
“These are the kind of questions Muslims should be asking themselves because what’s happening is a peaceful, compassionate religion has been taken and twisted by this poisonous ideology and that cannot be allowed to stand.”
Mr Javid stressed that there was no barrier to being both a Muslim and a patriotic Briton.
“It is perfectly possible to be both and there are millions of Muslims who do that every single day.”