The previous maximum term of six months will be quadrupled under the plan to tackle the “cowards” who post abusive comments over the internet.
Mr Grayling said the plan was a signal of his determination to take a stand against what he described as a “baying cyber-mob”.
The move comes just days after the threats directed at Chloe Madeley after she stepped in to defend her mother Judy Finnigan’s controversial comments about footballer and convicted rapist Ched Evans.
The Justice Secretary said: “These internet trolls are cowards who are poisoning our national life.
“No-one would permit such venom in person, so there should be no place for it on social media.
“That is why we are determined to quadruple the current six-month sentence.”
Ms Madeley, a fitness instructor, was threatened with rape after intervening in the row over her mother’s comments about Evans. Finnigan had inflamed the debate about whether Evans should resume his footballing career by saying his crime was “non-violent” and did not cause “bodily harm” during a panel discussion on ITV’s Loose Women.
Speaking to a national newspaper Mr Grayling said: “As the terrible case of Chloe Madeley showed last week, people are being abused online in the most crude and degrading fashion.
“This is a law to combat cruelty - and marks our determination to take a stand against a baying cyber-mob.
“We must send out a clear message: if you troll you risk being behind bars for two years.”
Under the current law, people who subject their victims to sexually offensive, verbally abusive or threatening material on the internet can only be prosecuted in magistrates’ courts under the Malicious Communications Act, which carries a maximum prison sentence of six months.
However the new measures will allow magistrates to pass on more serious cases to the crown courts, where offenders could face a maximum of two years behind bars.
The proposed change will be made as an amendment to the Criminal Justice and Courts Bill which is currently going through Parliament.
The measures would also give the police more time to collect enough evidence to enable successful prosecutions to be brought.
Ms Madeley told the Mail on Sunday it was right for Mr Grayling to update the 10-year-old law.
“The current law obviously needs to be reviewed,” she said.
“It needs to be accepted that physical threats should not fall under the ‘freedom of speech’ umbrella.
“It should be seen as online terrorism and it should be illegal.”
With the rise of social media use, internet trolls are seen as a growing problem.
In June this year Jake Newsome, 21, was jailed for six weeks for leaving offensive remarks on Facebook about the death of Leeds teacher Ann Maguire just days after she had been stabbed.
Sentencing him after he admitted sending an offensive message at a previous hearing, District Judge David Kitson had said: “You’re entitled to express reasonable views, as are all of us. What none of us are entitled to do is abuse that freedom.”