FirEd?

Ed Miliband did what the professional speechwriters always say you should do. He presented an argument, rather than a laundry list. He did not dole out random policy nuggets, with a bit on foreign policy thrown in, in order to touch every base. Instead he made a case, arguing that the values cherished by British society are not reflected in the ethics that underpin our economy.

He even framed that idea in a pithy way, pitching "producers" against "predators", insisting that too often we reward the latter over the former. The speech deserved credit, too, for trying to rise to the moment of current crisis, advocating not mere tinkering at the edges but a recasting of our entire economic model. And yet I worry it will do precisely nothing to improve the prospects of either Miliband or Labour.

That's not because I want more detail on how his ideas will translate into action or how, exactly, governments will distinguish predators from producers. Nor is mine the familiar concern that no one listens to conference speeches any more. The anxiety I have is both more superficial and deeper than that – and it is a worry that goes beyond Labour, touching on the state of politics itself.

Put simply, my fear is that you can make all the speeches and policy statements you like – carefully devising a strategy on this and crafting a narrative on that – but what matters more are shallow considerations of looks, demeanour, speech patterns and biography. That, in short, it is personality, not policy, that counts.

How else to explain today's Comres poll, which had the Conservatives one point ahead of Labour. This despite rising unemployment, an enfeebled economy and a series of cock-ups and U-turns that should have the Tories gasping for air. Some of that can be explained by the Conservatives' success in persuading voters that they are stoically engaged in the hard work of clearing up a mess not of their making. But polls show that the Tories' numbers are boosted by the voters' high regard for David Cameron, while Labour's are dragged down by their lukewarm view of Ed Miliband. Comres found just 24% regard Miliband as a credible prime minister-in-waiting – compared to 57% who do not.

There was similarly depressing reading to be found in a voluminous survey commissioned, admittedly, by the former Tory party treasurer Lord Ashcroft, but whose reliability has not been doubted. He found a Cameron premium, with the PM more popular than his party, as well as a Miliband deficit, with more than one in three voters less favourable to Miliband than they are to Labour. The words focus groups used to describe Cameron were "determined", "competent" and "ruthless" – while the one volunteered for Miliband was "weird".

Was he deemed weird because of his stance on the 50p tax rate or on climate change? No. Those surveyed cited his fighting his brother for the leadership (which they called "creepy"), his failure to get married until recently, and his way of speaking ("geeky").

Politicians and those around them – including those of us who spill gallons of ink each week discussing the smallest policy shifts, trying to calculate their impact on the electorate's preferences – recoil from contemplating the implications of all this. It pains them – us – to think elections could be settled by matters so trivial. I remember long conversations with the Gordon Brown camp, in which they would speak of dividing lines and the like, confident that a clever strategic move on, say, the economy could transform his fortunes – when the unpalatable truth was that voters had simply taken a long, hard look at him and decided that they did not want him as their prime minister. The same fate dispatched William Hague, Iain Duncan Smith and Michael Howard.

Something similar is afoot in London, where Labour is currently 20 points ahead of the Tories – and where, despite that, Ken Livingstone is trailing far behind Boris Johnson. Why? According to YouGov's Peter Kellner: "It's an instinctive judgment, it's about character." Boris has out-Kenned Ken, stealing his predecessor's clothes as the cheeky, rebel politician. "Ken reinvented himself as a personality politician and he's now up against someone who does personality politics better than him," says Kellner.

It's not always been this way. Clement Attlee could be elected without having the obvious leadership look we'd identify today. Nor would Edward Heath, or even Harold Wilson, meet the modern definition of charisma. John Major was famously grey, yet he won in 1992 – but that was partly because he was up against Neil Kinnock, against whom many voters took a strong dislike. Ed Miliband's problem is that he is competing in Cameron against a man who has long looked the part, who even in a 10-second appearance on television has that mysterious, chemical quality that suggests a leader. Whatever that is, Miliband does not have it – yet.

Many will bristle at this kind of talk, insisting that such things should not matter, that, as Tony Benn always used to say, it's the issues that count. But Benn – no slouch as a personality politician himself – was always hopelessly idealistic on that score. Personalities do count. As I've written before, quoting an old teacher of mine: people don't believe in ideas, they believe in people who believe in ideas.

So this is the problem for Ed Miliband. He is a decent, clever man but he does not look the part. He looks too young; he looks more like the speechwriter than the speechgiver, an adviser to the leader rather than the leader. That could change; he might grow into the role over the next three-and-a-half years. One aide suggests that Miliband has challenged every other bit of conventional political wisdom – running against his brother, being unmarried, taking on Rupert Murdoch – and that maybe he will defy this one too.

"He's not an identikit politician," that aide admits, seeing strength in that fact. At this moment when everything is in flux and when the Labour leader is seeking to break a 30-year consensus on the economy, perhaps it is right to think that the old, admittedly shallow rules on what a prime minister must look like are ready to be broken too.

But it has unfortunate echoes of the Brownite refrain circa 2007, that people were ready for a non-celebrity leader for more sober times. It sounds principled, it sounds laudable. But right now the Tories are much safer than they should be, insulated by a leader with charisma and a touch of the X-factor. Labour desperately needs some of that too.