Mark Stuart: Insurgent Ukip set to pile on the pain for Tories

But is all this effort by governing parties actually worth it? How “historic” are by-elections gains by parties? Is there not a tendency for by-election losses to be won back by the Government at the subsequent general election? And do by-election gains have any discernible political effect?

British political history since 1945 is littered with false dawns for small political parties. In March 1962, the Liberals sent shockwaves through the Conservative Government by winning the Orpington by-election. Harold Macmillan, the sitting Prime Minister feared in his diary that “we have been swept off our feet by a Liberal revival”. But that supposed revival petered out.

Similarly, the Glasgow Govan by-election of Margaret Ewing in November 1973 was supposed to presage a great upsurge in support for the Scottish National Party. In fact, Ewing only held the seat for 112 days before it was won back by Labour in the General Election of February 1974.

In short, by-election victories are like earthquakes. They produce major short-term shockwaves, and the occasional aftershock, but any political damage is soon rebuilt.

Only in the 1990s did the Liberal Democrats consistently pull off a string of spectacular by-election victories against John Major’s lacklustre government. For example, the Newbury by-election defeat in May 1993 is described by Major in his memoirs as “calamitous”. A second defeat occurred in July of that year, this time in previously rock-solid Christchurch.

The short-term political effect of these by-election losses, combined with nine rebellious Conservative MPs who lost the whip, was to deprive Major of a parliamentary majority.

Under Paddy Ashdown’s leadership, the Liberal Democrats engaged in the political equivalent of guerrilla warfare, saying one thing in a Northern by-election target and something completely different in a Southern constituency. Political principles were dispensed with. All that mattered was building up the number of seats. But a point was reached by the end of the 1990s when the by-election victories ground to a halt. The Liberal Democrats discovered they could not be all things to all men.

There was a limit to what became known as the “spinning plates” strategy. The Liberal Democrats were forced to choose between being an insurgent party and one that had pretensions to grow in size. And the moment, in 2010, when they tried to expand from being a small party of Opposition into a serious player with pretensions of sharing power, the Lib Dems flopped.

That same dilemma now faces Ukip. They are the new insurgents, replacing the Liberal Democrats as the party the voters choose to give the incumbent Government a kick in the teeth. But the irony is that the minute Ukip start having sustained success at by-elections, they will merely become yet another party belonging to the very Westminster establishment that they now seek to decry.

At present, Ukip can say what they like in Clacton in the South East, and something completely different in Heywood and Middleton in the North West, without worrying too much. But are they a right-wing libertarian party attracting disillusioned Conservative voters in the South or a traditional, left-leaning working class party tempting disgruntled Labour voters in the North? There is a limit to how long small parties like Ukip can face both ways.

Nevertheless, Ukip’s by-election victory in Clacton last month has had a political impact. It is shifting David Cameron further and further to the right, and forcing Labour belatedly to address the issue of immigration. The “Ukipification” of British politics is well underway. Moreover, Nigel Farage’s deliberate tactic of announcing the defections of Conservative MPs in a drip feed way maximises the agony for Cameron. Each Ukip by-election victory inflicts yet another wound on the Conservatives.

In that sense, by-election victories help to create political momentum. They are a clear sign that a party is virile and on the rise.

When David Cameron wakes up on Friday morning having lost another by-election to Ukip, he might look back longingly to the Crewe and Nantwich result in May 2008.

It was his party’s first by-election victory in 26 years, after years of decline. How long ago that victory must seem now.

Mark Stuart is a political academic from York who has written biographies of John Smith and Douglas Hurd.