Nate Silver, at 34 years old, has the pale, rather awkward countenance of a man who has spent much of his time alone. The statistician was much celebrated last November after he correctly predicted the outcome of the US presidential election in each of the 50 states.
We meet during a publicity tour for his book The Signal and the Noise , on the art and science of prediction. It has been described as "the most momentous book of the decade", and Silver – much sought on the lecture circuit – has been labelled "our age's Brunel" and "the Galileo of number crunchers".
Silver’s view of predictions draws on the work of the 18th-century English clergyman Thomas Bayes, who disagreed with the philosopher David Hume’s contention that predictions are irrational because nothing can be predicted with certainty. Predictions are rational, Bayes argued, if they are dealt with as a matter of probability; as the sun has always risen in the morning, we are justified in believing that it will rise tomorrow, even if one day it will not.
The world is not doing so well with its predictions currently, Silver argues, and the torrent of information now available in the computer-driven world of big data does not necessarily mean matters will improve. Economists predicted just two of the world’s 60 modern-day recessions a year ahead of time, he says, and for the most part they have failed to recognise the existence of a recession even after it has already begun.
First, however, people must recognise, if not abandon, their own biases, as too often, he contends, they want the facts to fit their views rather than being prepared to have their views shaped by the facts that surround them.
Some companies realise this more quickly than others, particularly those in technology, such as Google, which is able to test dozens of innovations each day but "pull the plug on them quite quickly if they aren't working".
“Technology businesses have been especially receptive to the message of the book. They are performing experiments on their data all the time; they know that a lot of ideas that look really good on paper don’t really take well with real customers,” says Silver.
"Equally, some ideas that they think are mediocre really explode with customers. I am not sure that you could have pinpointed Facebook ahead of time and said that it was such a unique business idea, since it was similar to what MySpace had done. But for some reason, by engineering the product a little differently, it became the biggest social-media brand in the world."
Silver continues, “In a big-data environment, which not all companies are [in] – of trial and error and gradually converging towards a better outcome, and being process-oriented rather than results-oriented – you have to have a good process.
“It has to be a lifelong chore to keep trying to improve your decisions: the minute you make one innovation it becomes the baseline from that point forward. Some businesses get that, and some don’t as much.
“The practitioners, the quants and data geeks get this, rather than the executives. In part, being an executive is all about leadership, supposedly. You are supposed to have a big idea and push it forward and be brave.”
The Signal and the Noise argues that such macho leadership is flawed. "You more want foxes, versus hedgehogs," says Silver. "Foxes are the people who scavenge around for different information and collect it from lots of sources and try out different things and hedge their bets a lot more."
The reception to his message varies. “People are always polite and receptive, but, you know, executives read a lot of books and go to a lot of conferences where they hear something, and maybe they’ll draw something from it.”
Companies of all types are drawing ever more information about their customers, but, says Silver, big data can give business new ways to get old things wrong. “Human nature is not going to change all that much, and that is usually the ultimate constraint on technology. We already have more data than we know what to do with. Doubling or tripling the amount won’t necessarily help things across the board. In some contexts, yes, it will, but in other cases it won’t. It will leave people confused.
“Inevitably, there will be disappointment when people realise that there aren’t any simple solutions here, and if you don’t know what you are doing then big data will find random correlations that don’t have any structural, causal link, and will only get you into trouble,” he says.
People’s desire for privacy will play a role. “I’m at the tail end of Generation X. People of my generation seem much less concerned with privacy than I do. They seem much more willing to share all sorts of components about their lives which maybe they shouldn’t share.
“I am a relatively private person myself, but you could argue that privacy in western society now is high by global standards. Before, you had real communities that you shared a lot with, so the concept of privacy in the western sense is itself the aberration.
“Before, you’d sleep in the same quarters as a lot of other people, so you didn’t have as much privacy. Everyone knew your business, but you knew their business, too. It was equitable in that way.
“Now, if Facebook or Google know your business, that could be a little frightening.”
