Publications
Partners
Superforecasting: The Art and Science of Prediction, by Philip E Tetlock and Dan Gardner
Author Professor Philip Tetlock has conducted the most comprehensive, scientific assessment of expert judgement to date by studying the quality of experts’ advice from 1984 to 2004.
Every day we hear the opinions and forecasts of experts, but we rarely ask how good the forecasts actually are. “Old forecasts are like old news,” Tetlock explains, “Soon forgotten - and pundits are almost never asked to reconcile what they said with what actually happened.”
His twenty-year study showed that accuracy declines the further out experts tried to forecast, becoming no more accurate than a chimpanzee throwing a dart.
We are all forecasters. When we consider a career change, starting a business, marrying, buying a home or an investment, or retiring, it is always based - even if tacitly - on how we expect the future to unfold. These are forecasts. And of course, when significant events occur, whether pertaining to our health or the markets, we turn to the experts for advice on the tumour, or the impact of a GDP announcement on future investment.
In 2011 Tetlock launched a geopolitical and geo-economic forecasting tournament. It covered about 500 questions of international importance, such as which country will withdraw from the Euopean Union by a certain date, and the likelihood of a clash in the East China Sea claiming more than 10 lives. Over 2 000 amateur forecasters participated, some with astonishingly accurate results whom Tetlock called the “Superforecasters”.
“Explaining why they’re so good, and how others can learn to do what they do, is my goal in this book,” Tetlock explains. He believes that it is possible to see into the future, in some situations to some extent, and “that any intelligent, open-minded, and hardworking person can cultivate the requisite skills”. One need only consider how many people rely on us to provide leadership, to see the need for this skill.
In 1972 an American meteorologist, Edward Lorenz, discovered accidentally that tiny data entry variations in computer simulations of weather patterns (replacing 0.506127 with 0.506) could produce dramatically different long-term forecasts.
In life and politics, as in the impossibly complex network of atmospheric actions and reactions, a minute difference of 0.000127 could produce material differences. It may be the difference between a tornado or a storm, or the difference between the Arab Spring or the continuation of decades-long dictatorships.
“In one of history’s great ironies, scientists today know vastly more than their colleagues a century ago, and possess vastly more data-crunching power, but they are much less confident in the prospects for perfect predictability,” notes Tetlock. Despite this limit on our ability to forecast, it would be a mistake to undervalue forecasting.
The description of what Superforecasters do, is summarised in a set of “commandments” for aspiring forecasters. A key point is what Superforecasters do, not what they are. “Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs. These habits of thought can be learned and cultivated by any intelligent, thoughtful, determined person,” Tetlock concluded.
How it works
Consider this technique: break a seemingly intractable problem into tractable sub-problems. In the pre-internet era, Enrico Fermi (a central figure in the invention of the atomic bomb) would pose this problem to his students. “How many piano tuners are there in Chicago?”
This question can be broken down by asking: what information would enable answering this question? The question could be answered if you knew the answer to four facts: the number of pianos in Chicago; how often pianos are tuned; how long it takes to tune a piano; and how many hours a year the average tuner works.
Based on the number of people in Chicago and estimating how many could afford a piano, and would want one (guess = 1 in 100), that would be about 50 000 pianos. If a piano takes two hours to tune, and each needs tuning once a year, that is 100 000 hours of work. Divide that by the 40 hours a week for the 50 weeks a piano tuner can work, with 20% of that time taken by getting to the job, and the piano tuner works 1 600 hours a year.
So there should be about 63 tuners in Chicago. (The Yellow Pages have 83 listings for piano tuners, but there are many duplicates as some have more than one phone number.)
The above was based on guesswork, but is surprisingly close to the probable number. A similar, but more complex process can be applied to geopolitical or geosocial problems. What this approach defines is the “outside view” of the problem, avoiding the psychological problem of ‘anchoring’. When a number is given prior to a question, our minds anchor about that mark.
Will there be terrorist attacks in Central Europe in the first three months of the year? From Wikipedia’s list of attacks in these countries over the past five years, it is possible to calculate the base rate of 1.2 per year. Having established the “outside view”, the other factors, the “inside view” can be considered. There have been increasing threats from a more prominent ISIS, a heightening of security since Charlie Hebdo, and so on.
Another significant finding from Tetlock’s tournament was the correlation between the number of refinements to the forecasts of the Superforecasters, and those of the other forecasters. Superforecasters delivered many more refinements, and were clearly more open to new information.
Superforecasters were more like the “foxes” than the “hedgehogs” of Isaiah Berlin’s essay. Hedgehogs hold an overarching position such as socialism or capitalism, and make all information conform to this view. Foxes, on the other hand, are more responsive to information for itself, without the need to fit it into a big idea.
All people default to System 1-type thinking, the rapid intuitive knowing (what is 5x10?) and only use System 2-type thinking through deliberate choice (what is 967.48 x 194.5?).
If nothing else, reading the book on how Superforecasters operate will incline you to give pause when you think about the future. At best, the many techniques it describes will help you hone a most important leadership skill.
Readability: Light ---+- Serious
Insights: High +---- Low
Practical: High --+-- Low
* Ian Mann of Gateways consults internationally on leadership and strategy and is the author of Strategy that Works. Views expressed are his own.
