Book Review: The Signal and the Noise

Author: Nate Silver Publisher: Penguin Books Reviewed by: Stephen Keim

I first became aware of Nate Silver in the lead up to the first Obama election in 2008. Denise, who is an assiduous reader of news on the internet, found the election site FiveThirtyEight before me and it was through her that I became an avid fan.   Silver came to politics through sport. He had developed PECOTA, a system for analysing and predicting the career development of Major League baseball players. He managed the publication of articles based on predictions by the scheme for Baseball Prospectus from 2003 to 2009.

Silver is a statistician and an analyst. That places him squarely at the geek end of the human continuum. FiveThirtyEight was based on a system of prediction or model combining the use of polling data with available demographic data to produce ongoing predictions about the likely result of each aspect of the 2008 elections.1

The fact that the model, and thereby Silver, correctly predicted the result in 49 of the 50 states provided a great boost to FiveThirtyEight  and to Silver’s reputation. The fairly hand to mouth excitement of a small operation changed a little as FiveThirtyEight was first purchased by the New York Times and, two years later, by ESPN. Silver remains editor in chief of the current operation.

Fans, whether they be obsessed by baseball, politics or something else, do love predictions. They particularly love them when they are predicting that your team or party is going to win.

But a publication based around prediction needs something else. It needs stories. It needs points of interest. It needs to give perspectives to the news of the day. It is in that area that Nate Silver is no ordinary geek. He has the ability to find ways to analyse the events of the day that generates both interest and new understanding. He can grab data from different institutions, relating to different aspects of our society, and explain, all over again, from a different perspective how the world works; why it works that way; and what it means for us.

His explanations of complicated data are clear and easy to follow. His insights are generally clever and unexpected. And he always manages to retain the excitement of the fan in his sharing of this new view of the world.

I remember, in 2008, we had come to Convention time. First, one party, then the other, obtained the traditional bounce in the polls. The rest of the world predicted a new winner after each bounce.

But Silver turned his mind and his pen to explaining the reasons for a bounce of this kind. I found out how most polls were conducted. They are robo polls. A machine makes the phone calls; asks the questions; and records the result. Most people do not like answering polling questions and so the machine may make a hundred phone calls in order to get one or two responses.

So Silver suggested that the effect of good news for a party (such as at convention time) may not be to change actual support for a party but merely to make existing supporters just a little more likely to respond to the robo phone calls from the polling companies. A couple of percentage points up or down in willingness to respond may then turn up in the polling results, when published, as increased numbers intending to vote for that party.

Ingenious, I thought. 538 had changed my world view that tiny bit with that little extra understanding of how the world works.

The Signal and the Noise, first published in 2012, may just be, then, what you do when your geeky back room analysis skills make you both comfortably wealthy and a huge celebrity. You take time off to write a book drawing on those skills.

The Signal and the Noise is about prediction and the difficulties of prediction. Prediction depends on data. More and more data is being collected and made available for those statisticians who are interested in making predictions. Having data about how the world has behaved in the past, however, does not mean that prediction as to what will happen in the future is either possible or easy.

The title of the book derives from the crux of the problem. In dealing with the data, the analyst looks for patterns. Finding patterns does not mean that your predictions will be successful. The analyst may be over reading the data. The patterns may have no relationship to how the same events will unfold in the future. There may be no signal. The analyst may just be reading the noise.

Ten years of data may reveal that prices went up for three years and then levelled out before dropping. The analyst may be tempted to predict a similar pattern to occur over the next ten years. That pattern may, however, just be noise in the data. A longer term view may suggest that prices are at an all time high and are likely soon to fall back, consistently, in a pattern that may persist for decades.

Knowing when you have truly found the signal and avoiding being taken in by the noise is the challenge of all prediction. It is an important science, of course. But it is also an art. And The Signal and the Noise guides us on a journey through different aspects of the world and looks at the way the art and science of prediction fares across many areas of human and non-human activity.

In The Signal and the Noise, Silver introduces the reader to a number of different disciplines. He has, of course, had to master the lore of each of those disciplines and access and master both historical and up to date data on each one of those disciplines. He mixes the dry art of research with the live action of interviews with experts in each field.

The surprising message from the book is that some phenomena are more predictable than others. Earthquakes are predictable in a long term, unhelpful way. The experts can, comfortably and confidently, predict that an earthquake of a certain severity will hit a particular vulnerable area with a certain frequency, the more severe earthquakes occurring less frequently than the less severe.

What seismologists have been desperate to unravel, however, is the the pattern in the plethora of small tremors that will presage the coming of a rare but major earthquake. But the data, so far, is all noise. The coming of the major earthquake remains unpredictable other than, say, it will occur, on average, in that place, every 50 or 100 or 250 years, depending on the severity of the quake that one is looking at.

Predicting weather, on the other hand, is getting better and better. It is all about applying more and more computing power to the old models and predictions are able to be made more accurately, further into the future.

But Silver also reveals the culture and business of forecasting. At least until the recent boom of the BOM, in Australia, the Bureau of Meteorology website, people have got their weather information predominantly from the mass media. But the TV stations and the newspapers cheat. They weight their forecasts towards wet weather on the simple basis that “our consumers won’t hate us if we predict rain and snow but it turns out to be a nice day. They will, however, hate us if we fail to warn us of bad weather”.

So, for many decades, the watchers and readers of mass media have been getting a jaundiced view of an increasingly accurate and reliable set of forecasts.

Public health, like seismology, is another area that is difficult to predict. Governments, health departments and corporate providers of health can spend billions of dollars according to their experts’ predictions of a returning or newly emerging epidemic. Unlike the sea of tremors in which seismologists try to find a signal, the health researcher may have only limited data collected at the early stages of the outbreak of the disease. As well as paucity, the data may suffer from being based on a selective group of people who have contracted the disease for reasons which may not emerge until well after the health billions have spent in accord with the unfulfilled predictions.

The Signal and the Noise directs itself to prediction in the game of poker. Silver had made a reasonable living out of playing internet Texas Hold ‘em poker for 18 months when the early internet boom for the game was still unfolding. He unmasks the myths about being able to predict a bluff by the sweat emerging on an opponent’s forehead or the changing colours of the iris.

Rather, one sizes up an opponent by the opponent’s past behaviour across a number of hands as somebody who is cautious or more inclined to take risks. One can also draw conclusions about the likely shape and strength of an opponent’s hand by relating ongoing bets in the hand to the possible combinations that may be available to the opponent.

Silver reveals ways in which cards that offered little in the early stages of a Hold ‘em hand can, unexpectedly, produce a straight or a flush, later on, also producing a rapid change of tactics on the part of the holder of that hand.

The use of computers to analyse and play chess and the amount of computing power which has allowed computers to compete with and beat the best players is the subject of a fascinating chapter.

And Silver reveals how a misconception about his machine opponent led to a shattering of Gary Kasparov’s confidence in his famous lost match against Deep Blue. Towards the end of the first game, which he won easily, Kasparov observed an unexpected move by the computer. He concluded, after much thought, that, even in a lost position, the computer was calculating and seeing opportunities at a much deeper level of analysis than any human could achieve.

Thus, Kasparov went into the later games, expecting and fearing much more than the computer could actually deliver. However, Silver’s interview with a Deep Blue, long after the event, revealed that the unexpected move had been the result of a programming glitch. Rather than dastardly calculated, the move had been completely random. Kasparov, by making a God out of the Machine, had destroyed himself and his chances of winning the match.

Perhaps, the most valuable and important part of The Signal and the Noise is Silver’s introduction to us of Thomas Bayes and his probabilistic view of the world through the simple formula he borrowed from French mathematician, Pierre-Simon Laplace.

The Bayes’ Theorem is directed at calculating the probability of something being true in the light of a particular piece of evidence. Silver uses the example of the chances that your husband is cheating on you if you find an unexplained pair of panties in a drawer at home.

The formula and the resultant method use three calculated or estimated Bayesian probabilities. In the example, they are the probability of the underwear appearing because he is cheating on you (the positive hypothesis); the probability that the underwear got there without him cheating on you (the negative hypothesis); and, third, the probability that he is cheating on you if there were no panties evidence (this called the Bayesian prior probability).

In the example, The Signal and the Noise came up with relatively low prediction of a 29% probability that the husband in question was cheating which may be of comfort for anyone who has discovered unexpected panties in recent weeks.

Without knowing the precise formula, one can still gain insights2  into the world of statisticians and analysts.

We can see this from another example. It is one week from a by-election. The governing party leads by 4 percentage points in an opinion poll. What are the chances that the government candidate will win.

The positive hypothesis directs one’s search for data. How often does a candidate leading by four percentage points, one week out from the election, win? Is that affected by the nature of the election? Are by-elections more unpredictable? Is the nature of the seat important?  A normally safe seat for the governing party may be more likely to stay the course even in difficult times.

Each of these questions guides the analyst in search of collected and available data which might shine a light on the positive hypothesis.

The negative hypothesis gives another set of relevant questions and indications of relevant data. How often are opinion polls a week out in error?

And the Bayesian prior leads in more directions. What would be the probability if we did not have this opinion poll? What was the calculated probability based on last week’s polls? Have any political events happened in recent times which would suggest the candidate would win? Or lose?

So the analyst is searching all the data banks she can find which might feed into the hypotheses. But she also has an eye for the real world which might cast doubt on the latest datum from the latest opinion poll.

And one can see how the approach works. If all the previous polls have been predicting a government loss, one will put less faith in this one off poll which may well be an outlier based on poor sampling techniques.  If a common sense reading of the news headlines, suggests that the government did very well in the last week, the prior hypothesis might be adjusted in the government’s favour and more faith may be put in the poll which indicates that opinions have changed.

The second insight into the analyst’s world is that what the formula spits out is a probability. The formula will not predict that the government candidate will win. It may say that the government candidate has a 65% chance of winning.

If you live long enough with the Bayes’ theorem, your view of the future will become one of probabilities governing every future event rather than absolute predictions that a particular event will or will not happen. You may be surprised but you will never be absolutely surprised.

I am reminded but for a moment of George Cockroft’s3 cult novel of the seventies, The Dice Man, about a bored psychiatrist who allows his life to be controlled, including in the direction of sexual assault and murder, by the cast of two dice.

The Signal and the Noise is nowhere near as subversive as The Dice Man and will not lead you on the path of murder or other serious crimes. You will, however, find it full of interest and there is a 37% probability that it will change your life. 

It will certainly give you greater understanding of how the world operates. You may expect more and more of the BOM and less of seismologists.

Stephen Keim SC

Footnotes

1. 538 is taken from the number of members of the Electoral College chosen by the voters in a Presidential election. The Electoral College is the body formally charged with electing the President.
2. The actual formula is set out at page 245 of my edition
3. Published under the pen name of its protagonist, Luke Rhinehart