Nate Silver — who founded the FiveThirtyEight blog, and has been analysing polling data better than anyone else in his field for the past several years — has finally written a book, and it’s an achievement.
The Signal and the Noise is great for a number of different reasons. Silver aimed much higher than a Chicken Soup for the Statistician’s Soul.
The book, much like the blog, trusts the reader and doesn’t try to preach or condescend — it’s easily the most clear discussion of statistical forecasting for the average person available.
You can buy the book from Amazon on the 27th, but for now, here’s the twelve coolest things we learned from Nate Silver:
Earthquakes are unpredictable, and probably will never be predictable, and anyone who says otherwise is a charlatan.
A big discussion in the book is about the limitations of modelling and simulation in the aid of forecasting. For a guy who makes his living being one of the most prominent users of big data, the Signal and the Noise has a lot of cautionary tales. One comes in the form of earthquake prediction which — despite decades of research — is usually only accomplished by accident. While many, many people have tried to find a legitimate way to predict earthquake swarms, nobody actually has.
The ratings agencies got the housing bubble wrong because they were behaving like first-time drunk drivers
It’s called an Out of Sample error, and here’s how he describes it. Someone who has never driven drunk needs to decide whether to call a cab home.
Out of a sample of 20,000 car trips, you’d gotten into just two minor accidents, and gotten to your destination safely the other 19,998 times. Those seem like pretty favourable odds. […] The problem of course, is that of those 20,000 car trips, none occurred when you were anywhere near this drunk. Your sample for drunk driving is not 20,000 trips but zero, and you have no way to use your past experience to forecast your accident risk.
Silver then points out that this is the exact type of forecast error made by Ratings agencies with regards to housing, and the fact that they based their mortgage default correlation models on a set of data where housing never decreased in value.
Television pundits are horrible at their jobs
Silver evaluated nearly 1,000 predictions made on The McLaughlin Group — a show where people paid to discuss politics as a living made predictions — and of the testable predictions, the panel did about as well as a coin flip. The panel’s predictions were 46% true and 47% false. “They displayed about as much political acumen as a barbershop quartet”
Silver’s own contribution to Moneyball still isn’t as good as the scouts
Silver first came on to the forecasting scene with his groundbreaking PECOTA statistic developed for Baseball Prospectus. He went back to see how his model — which sought to identify undervalued minor league players based on their career similarities to former Major League ball players — did compared to the scouts. The answer? While it did perform really well, the professional baseball scouts still did better.
Baseball America’s Top 100 prospects, determined by scouts, generated 630 wins for their major league team, while top 100 prospects identified by Silver’s PECOTA model generated only 546 wins in the same span of time.
The National Weather Service makes one of the best models in the world
The weather — “the epitome of a dynamic system” — has been increasingly predictable due to the National Weather Service’s use of modelling on supercomputers. They’ve halved the average error in a temperature forecast since 1970, and they’ve been able to cut down the average error in the location of hurricane landfall from a 350 mile radius to a 100 mile radius in a mere 25 years.
Their calibration is as near to perfect as can be expected when it comes to forecast probability: “When they say there is a 20 per cent chance of rain, it really does rain 20 per cent of the time.”
Your local meteorologist is horrible at his job, though
The Weather Channel isn’t all that great — Silver notes that when they say there is a 20% chance of rain, it’s only rained about five per cent of the time. It’s because “people notice one type of mistake — the failure to predict rain — more than another kind, false alarms,” and the Weather Channel would rather err on the side of not ruining picnics.
But local TV meteorologists take this to an extreme. They’re much more likely to overstate the probability of rain — in fact, “when a Kansas City meteorologist said there was a 100 per cent chance of rain, it failed to rain one third of the time. “
Statisticians who study disease have some of the hardest jobs in the world
Silver devotes a whole chapter to one of the most difficult but crucial types of modelling, disease control.
One of his main points in the chapter is that while simple models can aid in a researcher ‘s ability to solve a problem, Occam’s razor — the idea that the simplest solution is usually the right one all else being equal — doesn’t always hold.
The issue with disease control? While weather happens every day and the NWS has largely figured out what makes weather happen, disease shows up irregularly, and it’s only after the fact that researchers can usually figure out the statistics why.
One of the most interesting examples was where a team of researchers realised that the deadly spread of MRSA in inner city Chicago was due to poor hygiene at the Cook County Jail, and that outflows from the penitentiary consistently led the the introduction of the deadly infection.
When Deep Blue beat Garry Kasparov, it was because of a bug
One of the best chapters in the book is Silver describing a fast paced, high stakes chess game with the infectious enthusiasm of a Monday Night Football commentator.
He looks back at the famous chess match between Garry Kasparov, then the best chess player in the world, and Deep Blue, the IBM chess computer. Kasparov was savvy, and made a number of early moves designed the throw the computer off. But even when he won the first game, something the computer did scared the life out of him.
Deep Blue, in the end game, moved a rook in a seemingly irrational move. Kasparov, who understood the plausible trajectory of the game around five moves ahead, would interpret this as the computer seeing a dozen or more moves ahead, startling him deeply. He changed his play to adapt to this possibility, and would go on to forfeit winnable games because of it.
Here’s the most interesting part though. The move of the rook, according to the IBM programmers, was because of a bug in the code. The programmers thought they had removed all of the code that said “if Deep Blue cannot win, make a random move” but failed to, and the move of the rook was nothing more than a goof.
The economics of online poker meant that the worst players provided nearly all of the winnings
Silver rode the online poker boom, and made a significant amount of money in the process. One of the most interesting parts of the chapter on poker is when he describes the economics of the poker bubble.
Essentially, the worst players in the game (“the fish”) were providing almost all of the winnings, and even mediocre poker player were making money hand over fist because the fish continued to fuel the online poker economy. According to Silver, in a hand of 10 players, the estimated loss per hundred hands for the worst player was four hundred dollars. Everyone from the sixth-best player to the best player either broke even or made more than $100.
The poker bubble burst when these amateurs who were providing the income for the top 60% all left in droves as the outgoing 2006 Republican congress passed the Unlawful Internet Gambling Entertainment Act, which “scared the hell out of many online poker players.”
Silver’s interview with Donald Rumsfeld about Pearl Harbor and 9/11 is awesome
The last chapter in the book has to do with terrorism, and he and Donald Rumsfeld had a long chat about the topic, contextualized with the idea of Pearl Harbor, a personal interest of the former Bush Secretary of defence:
In Pearl Harbor, what they prepared for was things that really didn’t happen […] They prepared for sabotage because they had so many Japanese descendants living in Hawaii. And so they stuck all the aeroplanes close together, so they could be protected. So of course the bombers came and they were enormously vulnerable, and they were destroyed.
The whole section is a must read. On 9/11, Rumsfeld said this:
You can reasonably predict the behaviour if people would prefer not to die. But if people are just as happy dying, or feel that it’s a privilege or that it achieves their goals, then they’re going to behave in a very different way
9/11 wasn’t an outlier
Terrorism is like disease control, says Silver, in which there isn’t a long, prolific and consistent database to work from. According to one analyst he interviewed, “Detecting a terror plot is much more difficult than finding a needle in a haystack, he said, and more analogous to finding one particular needle in a pile full of needle parts.”
In fact, one expert Silver talked to suggested that terror prediction is closer to earthquake prediction than anything else. If the Oklahoma city and Lockerbie bombings were a 7 on the Richter scale, the World Trade centre attacks were an 8.
When viewed on a logarithmic scale with all the other terrorist attacks in NATO nations since the late seventies – with the Y axis as frequency, and the X axis as fatality – the attacks form a straight line, and 9/11 was well within the realm of reality and the furthest thing from an outlier. The implication? “A September 11-scale attack would occur about once every 80 years in a NATO country, or roughly once in our lifetimes.”
You can buy The Signal and the Noise starting September 27.
Business Insider Emails & Alerts
Site highlights each day to your inbox.