The Rational Optimist

Matt Ridley, lanky, diffident, sandy hair, spectacles, author of assorted elegant and successful works of popular science on the subjects of sex and evolution, human virtue, nature v nurture and – his big bestseller – genes, was non-executive chairman of said ignominious institution for the three years before it blew up, and had to be fixed by £16bn of taxpayer funded loans.

He must, therefore, have a few stories to tell. But he can't. Terms of employment, you understand. Thankfully, his new book is original, clever and will – you can just tell – prove controversial enough to compensate for this early setback. Climate change? We can beat it. Population explosion? Won't happen. Cancer, Aids? In retreat. Global food shortage? Already history. Economic depression? A mere hiccup. Peak oil? Really not an issue.

Ridley, you see, has the temerity to believe that life on earth can only get better, and the state of the earth itself with it – not a view, one imagines, guaranteed to win him instant and universal adulation. But we'll get to that later. In the meantime, he's sorry about the Northern Rock business. "I'm truly sorry," he says over coffee in the cafeteria at the Centre for Life, the award-winning life sciences centre in Newcastle he helped found.

"It was an awful time. A fantastically painful memory. There's not a week I don't think about it. But the problem is that every time I try to work out how I can go on the record about it, I end up having to go into detail. And that's what I'm not allowed to do."

It seemed, presumably, a sensible thing to do at the time. Born to a noble Northumberland family (Ridley's father, Matthew, is the fourth Viscount Ridley; his uncle was Nicholas, the late member of Margaret Thatcher's cabinet; the family pile, Blagdon Hall, is a Grade I-listed mansion in 8,500 acres) whose fortunes were assured by "a spectacularly successful 18th-century coal merchant", Matt was the third generation of the family to sit on Northern Rock's board.

"I was on the board of quite a few companies in the north-east by that time," he says. "Of course I was unqualified, but the mantra at the time was that banks needed people with independent experience. Those with chairmen who had more relevant experience got into just as much trouble." The main problem was: "Like everyone living in a bubble, you don't know you are."

So Ridley came in for a fair amount of stick for his part in the Northern Rock debacle, not least before a parliamentary committee, where he was castigated for not "recognising the risks of the bank's strategy" and "harming the reputation of the British banking industry".

Other critics, notably the environmentalist and Guardian writer George Monbiot, were harsher, arguing that Ridley's writings promoted a "geneticist's version of the market", amounting to a call for precisely the kind of libertarian, anti-regulation, non-interventionist approach that, applied to banks, had precipitated the global financial crisis. He wanted less regulation, his critics said – and look at the result.

Ridley says that's not the case. It's true that he once wrote, in a 2006 article, that "the more we limit the growth of government, the better off we will all be". But, he says, "I never 'lobbied' for deregulation, let alone in financial services. Writing articles criticising bad regulation in non-financial areas, and criticising excessive regulation – also in non-financial areas – as part of a wider critique of government fallibility is the closest I've come. But I don't call that lobbying."

The whole foul experience, though, undoubtedly helped inform his thinking. Which is what, exactly? How does he justify such immoderate optimism in the future of the species, and – even more immoderate – its habitat? Ridley is, at root, a naturalist. It was the "happy distraction" of bird-watching at Eton that first got him interested in science, and he eventually left Oxford with a DPhil in zoology: "On polygamy in birds, with particular reference to the pheasant."

He joined the staff of the Economist in 1983, and was the magazine's science editor from 1984-87. He then spent three years in the US as Washington correspondent, then returned to Britain as American editor for a couple more. Then he left. "I wanted, gradually, to take over the running of the house from my father," he says. "I wanted to start a family, and to do that in the north-east. My wife [neuroscientist Anya Hurlbert] had got a job at Newcastle University. I opted for a freelance writing career. I was lucky enough to have the means to do it."

His first book (bar a "rather embarrassing five-week job on the election of George Bush") was The Red Queen, about the evolution of sexual reproduction. The Origins of Virtue, published in 1996, traced the evolution of society in genes, animals and humans. The big breakthrough came three years later with Genome: The Autobiography of a Species in 23 Chapters.

Appearing just before the mapping of the human genome in 2000, Ridley's book, variously praised as absorbing, lucid, perceptive, exhilarating, stylish, witty and brilliant, has since sold more than 500,000 copies and been translated into 25 languages. He is equally proud, though, of a later book that made a rather smaller splash, Nature via Nurture.

And so to the new one. The Rational Optimist: How Prosperity Evolves is out this month. It is, essentially, about progress: how, of all the species on earth, only humans have managed so radically and completely to change the way they live. Animals, even the most intelligent ones, have not thus far known "economic growth" or "rising living standards" or "technological revolutions" (or, indeed, "credit crunches"). Why?

"I'd written four books about how similar humans are to animals," Ridley says. "This one's about how we're different. We think we're free agents, but human nature itself is a constant. If you go back 30,000 years or so, you'd find you were on much the same wavelength as the man who painted the images in the Chauvet cave in the south of France. You'd know what he was about, his nature; you'd recognise him as human in every psychological way."

So the question that intrigued Ridley was this: "If people are all the same underneath, how has society changed so fast and so radically? Life now is completely different to how it was 32,000 years ago. It's changed like that of no other species has. What's made that difference? Clearly our genes haven't changed; this process has happened far too fast for genetic change. My answer, bringing together my evolutionary knowledge and a lot of economic reading, is this: sex is to biology as exchange is to culture."

That probably requires a little unpicking.

Ridley's argument is that just as biological evolution depends on sex (because sex is what brings together the genes of different individuals, allowing natural selection to come up eventually with, say, an eye), cultural evolution depends on ideas meeting and – Ridley's term – mating, allowing mankind to come up with, say, chocolate ice cream. Or a jet engine. Or even The X Factor.

And what allows those ideas to meet and mate is our facility for exchanging. Except, of course, it's not just ideas that humans exchange, but things. And as soon as human beings, sometime before 100,000 years ago, began exchanging ideas and things, "culture became cumulative, and human economic 'progress' began," says Ridley, setting his coffee aside, suddenly animated.

"It just dawned on me how unusual exchange is in other animal species – not teaching or learning, as many animals do, but genuinely exchanging. Then you start reading, and you realise Adam Smith had seen this: 'Nobody ever saw a dog make a fair and deliberate exchange of one bone for another with another dog.' And you realise, maybe, you could be on to something."

What Ridley was on to was a bold and immensely seductive theory, extensively and elegantly demonstrated over 350-odd pages, with examples from the Stone Age to the internet and Tasmania to Ming-era China. Through "exchanging", he argues, humans discovered what he calls "the specialisation of efforts and talents for individual gain". Specialisation, in turn, encouraged innovation. And innovation saved time, which, in Ridley's thesis, equals prosperity – and rising living standards.

The world, he thankfully volunteers at this juncture, is far from perfect. Millions still live in misery. But the majority are infinitely better off than they were 200 or even 50 years ago: since 1800, average life expectancy has more than doubled, and real income multiplied more than nine times. Even since 1955, people earn lots more, eat better, lose fewer children, are better educated, and own more useful things such as telephones, toilets and fridges.

They are also far less likely to die as a result of murder, accidents, natural disasters and from any one of a whole range of diseases. And that's not just "on average": even including impoverished Africa, Ridley says, it's hard to find any region in the world where longevity, health and wealth haven't risen overall since the 1950s; even the UN reckons that globally, poverty has fallen more in the last 50 years than in the previous 500.

"The fact is," he says, "that the more we specialised as producers, and diversified as consumers – the more we exchanged – the better off we have become. And the better off we will be in the future, because this is a process that doesn't stop. It just trundles along, producing higher living standards, in an inexorable way.

"And if 6.7 billion people continue to keep specialising and exchanging and innovating, there's no reason at all why we can't overcome whatever problems face us – population explosions, food shortages, disease, poverty, terrorism, climate change, you name it. In fact I think it's quite probable that in 100 years' time both we and the planet will be better off than we are now."

Well, it's certainly a tantalising idea. Just a shame no one else seems to believe it.

But that, suggests Ridley, is simply because we are pessimists, on a grand scale. People have been predicting the worst for at least the last couple of centuries – and so far they've always been proved wrong. Growing poverty, worldwide famine, population explosion, imminent pandemics, disastrous resource shortages, thinning ozone layers, Y2K bugs, acid rains: none happened as predicted.

"For reasons I don't fully understand," concedes Ridley, "it just sounds smarter to shake your head when everyone else is clapping. And catastrophe prediction sells: look at any bookshop." Added to this, humans have a selective memory: we like to believe things were better in the past.

Of course, they weren't. "For the vast majority of people," says Ridley, "they were a lot more miserable – lives were harder, poorer, shorter, sicker. We look at Jane Austen on the television and think: the early 19th century looks quite nice. But most of us wouldn't have come anywhere near the ballroom. And even if we had, the body odour would have been terrible."

The underlying point about doom-mongering is that it's all based on extrapolation: if we carry on like this, we'll run out of oil, or paper, or food, or water (or temperatures will rise by 6C) within 20, or 50, or 100 years. Rubbish, Ridley says, briskly: "Of course it's right to say, 'We can't carry on like this.' We can't. But the point is, we don't: we find ways to replace things, to be more frugal, more productive. We shouldn't be saying, 'We can't go on as we are,' but: 'How can we best encourage change?'"

And the answer to that – assuming you buy Ridley's line – is obviously through exchange. So that's it, then? Our salvation lies in free markets? "Ah," he says, taking stock for a moment. "I'm not talking about all markets. I'm talking, really, about commerce." This book – and, clearly, that unfortunate stint at Northern Rock – have taught him an important lesson: "Markets in goods and services are very different from markets in capital and assets."

The latter are "prone to bubbles that pop; the former are not. In fact it's hard to design the latter so they work at all, which is why they need good regulation. I've never argued against that. The former, on the other hand, work so well in delivering efficiency and innovation that it would actually be quite hard to design them so they'd fail."

The key difference, he says, lies in the fact that commerce is "a two-sided thing. I think if you put people in front of some huge temptation where it's possible to grab as much as they can for themselves, almost everyone will. The beauty of commerce is that it mutes that. The chap behind the counter in the corner shop has no interest in short-changing you, because he wants you to come back." Markets in assets are not nearly so sensible.

And these markets in goods and services need less, not more regulation? He reflects again. "There is," he confesses, "an element of the libertarian in this, but I think socially as well as economically libertarian. The conclusion I've reached is that prosperity, enrichment, is a bottom-up process: it relies on people being able to get on and do things. Governments see it as top-down. And they're cautious, they see the bad things that may come of a new technology rather than the good things. A balance needs to be struck."

Not, he hastens to add, that he's opposed to government altogether: "I'm not a great fan of it after the last 1,000 years, but I'm not an extremist who wants rid of it. What we don't want is for the chiefs, priests and thieves to kill the goose that lays the golden egg – which they've often been guilty of in the past.

"And I absolutely see that the process of enrichment produces inequalities, and that a government has a responsibility to redistribute. My objection to redistribution in this country is not redistribution in itself, it's that in the process so much gets trousered by bureaucrats." (In keeping with this view, he says, he's giving away half the post-tax proceeds from his book to agricultural charities in Africa.)

Hence, he says, standing up and loping off to meet the photographer, the Rational Optimist: not because he was born that way ("I'm as low as anyone else at three in the morning"), but because he's looked at the biological, economic and cultural evidence and convinced himself that we can be optimistic about our future.

There are plenty, doubtless, who will think him mad, or bad, or possibly both. It would be kind of a relief, though, wouldn't it, if he was right?