A review of Tetlock’s ‘Superforecasting’ (2015)

Spectator Review, October 2015

Forecasts have been fundamental to mankind’s journey from a small tribe on the African savannah to a species that can sling objects across the solar system with extreme precision. In physics, we developed models that are extremely accurate across vastly different scales from the sub-atomic to the visible universe. In politics we bumbled along making the same sort of errors repeatedly.

Until the 20th century, medicine was more like politics than physics. Its forecasts were often bogus and its record grim. In the 1920s, statisticians invaded medicine and devised randomised controlled trials. Doctors, hating the challenge to their prestige, resisted but lost. Evidence-based medicine became routine and saved millions of lives. A similar battle has begun in politics. The result could be more dramatic.

In 1984, Philip Tetlock, a political scientist, did something new – he considered how to assess the accuracy of political forecasts in a scientific way. In politics, it is usually impossible to make progress because forecasts are so vague as to be useless. People don’t do what is normal in physics – use precise measurements – so nobody can make a scientific judgement in the future about whether, say, George Osborne or Ed Balls is ‘right’.

Tetlock established a precise measurement system to track political forecasts made by experts to gauge their accuracy. After twenty years he published the results. The average expert was no more accurate than the proverbial dart-throwing chimp on many questions. Few could beat simple rules like ‘always predict no change’.

Tetlock also found that a small fraction did significantly better than average. Why? The worst forecasters were those with great self-confidence who stuck to their big ideas (‘hedgehogs’). They were often worse than the dart-throwing chimp. The most successful were those who were cautious, humble, numerate, actively open-minded, looked at many points of view, and updated their predictions (‘foxes’). TV programmes recruit hedgehogs so the more likely an expert was to appear on TV, the less accurate he was. Tetlock dug further: how much could training improve performance?

In the aftermath of disastrous intelligence forecasts about Iraq’s WMD, an obscure American intelligence agency explored Tetlock’s ideas. They created an online tournament in which thousands of volunteers would make many predictions. They framed specific questions with specific timescales, required forecasts using numerical probability scales, and created a robust statistical scoring system. Tetlock created a team – the Good Judgement Project (GJP) – to compete in the tournament.

The results? GJP beat the official control group by 60% in year 1 and by 78% in year 2. GJP beat all competitors so easily the tournament was shut down early.

How did they do it? GJP recruited a team of hundreds, aggregated the forecasts, gave extra weight to the most successful, and applied a simple statistical rule. A few hundred ordinary people and simple maths outperformed a bureaucracy costing tens of billions.

Tetlock also found ‘superforecasters’. These individuals outperformed others by 60% and also, despite a lack of subject-specific knowledge, comfortably beat the average of professional intelligence analysts using classified data (the size of the difference is secret but was significant).

Superforecasters explores the nature of these unusual individuals. Crucially, Tetlock has shown that training programmes can yield big improvements. Even a mere sixty minute tutorial on some basics of statistics improves performance by 10%. The cost:benefit ratio of training forecasting is huge.

It would be natural to assume that this work must be the focus of intense thought and funding in Whitehall. Wrong. Whitehall has ignored this entire research programme. Whitehall experiences repeated predictable failure while simultaneously seeing no alternative to their antiquated methods, like 1950s doctors resisting randomised control trials that threaten prestige.

This may change. Early adopters could use Tetlock’s techniques to improve performance. Success sparks mimicry. Everybody reading this could do one simple thing: ask their MP whether they have done Tetlock’s training programme. A website could track candidates’ answers before the next election. News programmes could require quantifiable predictions from their pundits and record their accuracy.

We now expect that every medicine is tested before it is used. We ought to expect that everybody who aspires to high office is trained to understand why they are so likely to make mistakes forecasting complex events. The cost is tiny. The potential benefits run to trillions of pounds and millions of lives. Politics is harder than physics but Tetlock has shown that it doesn’t have to be like astrology.

Superforecasting: the art and science of prediction, by Philip Tetlock (Random House, 352 pages)

Ps. When I wrote this (August/September 2015) I was assembling the team to fight the referendum. One of the things I did was hire people with very high quantitative skills, as I describe in this blog HERE.

7 thoughts on “A review of Tetlock’s ‘Superforecasting’ (2015)

  1. Pingback: On the referendum #21: Branching histories of the 2016 referendum and ‘the frogs before the storm’ – Dominic Cummings's Blog

  2. Pingback: On the referendum #21: Branching histories of the 2016 referendum and ‘the frogs before the storm’ – Dominic Cummings's Blog

  3. I am late to this but came across it reading about Tetlock elsewhere…

    An area of HMG policy where this seems to apply particularly well is climate change. The climate could not be more complex or multi-disciplinary, so Tetlock’s findings tally precisely, in that the climate has not done what experts told us for decades that it would. But it’s easier to be a hedgehog and pay great heed to the idea of ‘climate change.’ Hence we are throwing phenomenal amounts of money around on what looks increasingly to be a non-issue.

    This links which what this blog has outlined elsewhere, especially in the Brexit post, where evidence and rationality is put to one side in favour of moral grandstanding.


  4. Pingback: #29 On the referendum & #4c on Expertise: On the ARPA/PARC ‘Dream Machine’, science funding, high performance, and UK national strategy – Dominic Cummings's Blog

  5. I am intrigued by the parodies in so much of your work.

    I would urge you to note the effects of social media use with cognitive dissonance and the effective rise in preventable infectious diseases through anti-vaxxers spreading false information and pseudo-science. This is a middle-class epidemic, rather than the assumed ‘hippy’ base, where a little knowledge and wealth to spend on unregulated holistic therapies has opened Pandora’s box.

    Much like Brexit, computer algorithms can speed up and target the worst of human nature, as human nature is in itself, more complex and changeable than computers. Even with AI and predictive evolutionary learning, a single human can upset the applecart. Although admittedly this would be temporary before the AI caught up and recalibrated, the idea that an omnipotent computer-based algorithm could generate the future of civilisation has been dramatised in the show Travellers (Netflix). The Travellers in the show had the benefit of time travel to ‘re-do’ the errors within the system based on human fallacy.

    I am concerned that you note that systems are rotten – agreed – yet feel another system is the answer. Not merely that but a system that isn’t answerable to anyone (or will the AI have programmes for compassion and empathy for the ‘human condition’?) and those that control it will have a vested interest to manipulate outcome on a grander scale than anything current governments can conceive. Granted many ideas and inventions are overlooked because of short-sighted fat cats/middle manager types looking to uphold the status quo and as a business venture with tight controls in various sectors. I am particularly interested in public health and the field of epidemiology. The ability to cross-reference products or environments that may have negative impacts on mental and physical health on a scale unprecedented to date is fascinating. Would the conglomerates controlling the AI want this to happen, I wonder? Algorithms that can look into biopsychosocial aspects of health and predict illnesses within families and offer guidance, for example. Even though such enterprises can be manipulated for insurance billionaires to take over, weighing up the potential health benefits makes that attractive (while we have the NHS, it is imperative to note). An AI that can record someone who has a fatal diagnosis’ wish for euthanasia and effect this on behalf of the patient once the disease reaches it’s climax (looking forward to a time when a change in the law allows for this, such is the strain on the NHS). Such ideas can eliminate the risk practitioners take on ethically and be truly ‘patient-led’. Ethics is largely the reasoning behind so much of what is troublesome to the NHS, yet it is also the most basic part of being human; being humane. If AI can format to similar frameworks for biomedical ethics, you reboot the old system, as you have said.

    However, the application of this in a larger, over-arching manifestation – such as the Brexit vote – has meant this thrust of your argument has been lost. Instead the public and politicians will more likely be fearfully instigating an AI ‘arms race’ for want of a better phrase, to ensure their place in power is unchallenged in forthcoming elections. Frankly, you’ve given them the keys to the castle. Now we look forward to a future where the richest will win, as they can pay for the AI they require and do as little in the public interest as is possible. We can already see where the spending has been going since Brexit (which has now cost us more in FSTE damage alone than we have paid into it in 45 years) and it certainly isn’t on “the people” or education.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s