Monday, October 29, 2012

In Defense of Nate Silver

It's Nate Silver's job to analyze the news—so it must have come as quite a shock to him today to find himself become the news. While criticism of Silver has been out there for a long time, its most recent form has cut straight at the heart of Silver's analysis and represents the same type of anti-intellectual fear that has followed trailblazers like him around for centuries. In a surprisingly acidic POLITICO article, Dylan Byers makes Joe Scarborough's case against Silver and his data-driven polling analyses:
"So should Mitt Romney win on Nov. 6, it's difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance of winning (way back on June 2) and — one week from the election — gives him a one-in-four chance, even as the polls have him almost neck-and-neck with the incumbent."
Critiques of this ilk betray an inability to even speak intelligently on the subject of statistics, let alone a leg to stand on when presenting a counterargument to the findings of Silver's trademark Electoral College–predicting model. As I write this, Silver and his model give Barack Obama a 74.6% chance of victory on November 6. That number is very prominently labeled on Silver's website as "chance of winning." There's not much ambiguity in that. It should be obvious to anyone looking at that figure that what that means it that, in the judgment of the model, Mitt Romney has a 25.4% chance of winning the presidency.

But Byers, in the passage quoted above, clearly misses that point. Nowhere does it say that those 74.6%-to-25.4% figures are a prediction that Obama will win or that Romney will lose. It is an attempt to take a snapshot of the data and figure out odds. As Silver told Byers in the POLITICO article, there is still a significant chance that Romney wins—indeed, specifically, a one-in-four chance. If Romney wins, the model was not necessarily wrong. Indeed, every fourth time the model was run (Silver runs 10,001 simulations per day), Romney did win—and it's not a contradiction to say so while still handicapping Obama as the favorite.

Scarborough and, apparently, Byers seem to have a problem with this, but they don't seem to understand that this is the scientifically responsible way of doing this sort of thing. There is an academic discipline known as statistics, and they've been doing this a whole lot longer than any of us. Silver and others trained in this fickle art adhere to time-tested tactics such as the scientific method, gathering as-large-as-possible sample sizes, and acknowledging and even embracing the possibility of error.

In a world of post-debate insta-polls and Senate race rankings that are either Lean Democrat or Lean Republican, we as a society place a huge emphasis on "calling" states, elections, World Series, you name it. Audiences want instant gratification, and pundits give it to them with iron-clad predictions that they finalize and stick to come hell or high water.

What makes Nate Silver so unique—and so valuable—is that he resists that entirely (and yet still manages to be popular; imagine that!), favoring instead a scientifically responsible spectrum. The core tenet of this method lies in the difference between a 49% chance of an Obama win and a 51% chance of an Obama win. For most pundits, those are opposite predictions. On a spectrum, they're virtually identical. Given that it only takes a two-percentage-point swing to make up that difference, that's the right way to think about it.

Likewise, a spectrum always leaves room for some doubt. Even very safe predictions have a small chance of not happening, and a probability spectrum is honest about that fact, setting 99% or 99.9% odds for a very likely event. In other words, a good scientist always leaves room for the possibility that anything from extreme X to extreme Y will occur; the trick of creating a utile spectrum is knowing where to fix the "tipping point" between "lean X" and "lean Y," not picking one or the other. The beauty of a good probability spectrum is that it allows for every possibility. That's because the chance always exists, however small, that something extremely unlikely (e.g., a Romney landslide) will happen. In that sense, spectra like Silver's model will always be accurate.

And maybe that's the problem; skeptics see Silver's model's tolerant spectrum as wishy-washy—an attempt to take credit for being accurate no matter what the outcome. Science has one word for these people: "Tough." We have no choice but to accept this little ambiguity in our lives, because we have no way of ever being certain about anything. I understand that that is unsettling for many people, but that's what being a scientist—or even just being intellectually curious—is all about.

It also doesn't help matters that the exact figures and contours of a probability spectrum are impossible to prove. No one can ever say for sure that, on October 29, Barack Obama had a 74.6% chance of winning the race, even if Romney does win in that landslide. All we will know is the binary outcome: did Obama win or not, and by how much. It takes a much broader body of work to "prove" (to the extent anything can be proved) that those odds were correct—a body of work that, sadly, we'll never have. (You'd need the 2012 election to duplicate itself in future elections exactly the same way through October 29 a few hundred times, then see who won in each of those cases. In laboratories, these types of experiments are possible. Not in political science, where this is only the 57th presidential election in American history.) The best Silver—or anyone mortal in the whole wide world—can do is make an educated guess based on the data we do have. You may criticize which data—which polls or which economic variables—get plugged into Silver's model; you may not ignore science or the discipline of statistics.

Yet people do. People rely on their "gut" more than on the data in many more fields than just politics, and Silver has been dealing with them his whole life. As an early employee of Baseball Prospectus, Silver invented the PECOTA system and was an early figure in baseball's sabermetrics. He and co-Moneyball-ers tried to bring a rational, data-driven approach to predicting baseball the same way he has done in politics—and met with the same uninformed ridicule.

Baseball is full of the same "anti-statheads" that have come out of the woodwork in politics recently. You know them as the people who think of pitchers' wins as still a valuable statistic. They're the ones that denigrate WAR by saying that a better measurement of skill is actually how many wins you generate above a replacement level. They believe in momentum in baseball, in "clutch" hitting, and in the idea of lineup protection just because their experiences have led them to.

(Note: I'm painting with an extremely broad brush. In fact, I would like to see more rigorous statistical study on each of those last three. And you can indeed have a reasonable argument with other baseball experts or fans about those things—as long as the argument is empirical and grounded in data and facts, not "general impressions.")

The anti-Silverites in politics we see today are the descendants of the meanest versions of that baseball old guard: the old-timey scout who believes stats and innovation have nothing to offer him; the longtime columnist who bullies and mocks statisticians as "eggheads" or "binder boys." These people are as closed-minded as Nate's probability spectrum strives to be open-minded. The best analysis, and the best predictions, will inevitably come from viewing all available data and considering them holistically. As HardballTalk head blogger Craig Calcaterra says, quite astutely I think, if you worked in any field other than baseball and stubbornly ignored new information and new technology in your job, you'd be fired. Any field other than baseball or politics, I guess.

Maybe it's my wishful thinking, but it seems to me that those people in baseball are, fortunately, becoming more and more marginalized. Unfortunately, though, that's what makes those critics in politics much more dangerous—they are actually "important" pundits who are taken seriously. Indeed, in baseball, people can be as ignorant as they like, but the only real damage they're doing is taking up column inches and maybe, just maybe, encouraging a stupid trade to go down. In the powerful field of politics, ignorance can have a real effect on policies or the next leaders of the United States. They're playing with fire.

That's all the more reason to make sure Silver's voice of reason isn't drowned out. Unfortunately, Nate has caught onto the fact that many people in politics are jerks, and it may hasten his "retirement" from the field of political forecasting. (This most recent incident can't have helped.) But with Nate gone, unlike in baseball, the statistics-ignorant crowd will have won out, and political observers and the viewing public will go on thinking that the tools of ignorance are an acceptable forecasting model for elections.

I happen to roughly agree with Nate's prediction on the outcome of the presidential race, but losing his crystal ball is not even close to the reason his departure would sting. Rather, it's the loss of the reasonable, data-driven approach that he represents and has brought to the fields of baseball, politics, and others. Silver stresses that prediction is an imperfect science that he's just trying to make sense of, not solve. He is a student of the science of prediction, not a prognosticator per se. As he likes to say, it's not about which predictions are right, but rather which are less wrong. Instead of trying to eliminate error like those pundits and their iron-clad predictions, he truly does embrace it and see its value in helping improve subsequent forecasts. He's enhancing the study of predictions and helping us understand how to make them better—a goal that's bigger than elections, in some cases even helping to save lives.

Anyone who has given Silver's New York Times blog a more than cursory read recognizes all this, because Nate goes to pains to point it out (undoubtedly stung by ignorant would-be statisticians before). This isn't weakness, or being "unmanly," as it was distastefully put this week. It's realism and nuance, two traits that are essential for level-headed people in any field—only when it comes to predictors, they're basic qualifications.

Tuesday, October 23, 2012

Introducing the Baseballot Gubernatorial Rankings

With only two weeks left to go before Election Day, I considered the other day the relative lack of attention that governors' races are getting this cycle. This is hardly a new phenomenon for the relatively few gubernatorial elections that take place in sync with the presidency, but it is an unfortunate one. Governors, along with state legislatures (whose electoral prospects are fascinating but, alas, I will not have time to address before November 6), affect people's day-to-day lives in the states in question arguably more than the Senate and president will. Accordingly, I wanted to devote some more space on my little corner of the internet to assessing the 11 gubernatorial races of 2012.

At the top of the page, you'll see a new tab, "Gubernatorial Rankings," added to the menu. Click through to see my race rankings for the 11 campaigns in the same chart form as my Senate rankings. The scale is also the same—Solid/Likely/Leans for each party, explained in detail here.

Because of the relatively few number of gubernatorial races, however, I figured I could devote a little bit of time to a qualitative analysis of each one as well. In alphabetical order:

Delaware (Solid Democrat)
Sometimes, less is more. Democratic Governor Jack Markell is a popular governor in a blue state in a presidential year. His Republican opponent is not a big name. Easy hold.

Indiana (Likely Republican)
This could just as easily be Solid Republican, since there's no polling evidence that Democrat John Gregg is competitive against Republican Congressman Mike Pence. However, Gregg is a solid candidate who has been able to get active on the airwaves and has achieved some fame for his crazy moustache. With the competitive Senate race in Indiana, this at least could be closer than the GOP blowout it otherwise would be.

Missouri (Likely Democrat)
Governor Jay Nixon has proven to be a very strong candidate for the Democrats, winning over solid crossover support despite a rocky beginning to the cycle. Furthermore, whether hurt by association with Todd Akin or just falling flat of his own accord, Republican Dave Spence—oddly, in my opinion—hasn't been able to tap into Missouri's growing GOP base of support.

Montana (Tossup)
Sparse polling hasn't given us many clues about who's favored in this race. Montana is a reliably red state in the presidential race, but a close race for Senate and its love for outgoing Democratic Governor Brian Schweitzer has made it swingier this cycle. Schweitzer afterglow would probably be responsible if Republican Rick Hill loses this one to Democratic Attorney General Steve Bullock.

New Hampshire (Tossup)
A true swing state on every level (president, Senate, both House seats, and governor in 2012), New Hampshire has a popular Democratic governor who is retiring. I would have thought that that fact, plus a weak (in my opinion) Republican candidate in Ovide Lamontagne (he was Kelly Ayotte's Tea Party nemesis in 2010), would create an advantage for Democrat Maggie Hassan, but polling so far has been tight—with the two candidates seemingly playing hot potato with the lead with every poll that's released.

North Carolina (Likely Republican)
North Carolina may end up the only state where the corner office changes party control in 2010. Republican Pat McCrory is well positioned to win thanks to an unpopular Democratic incumbent, up by double digits in most polls. Ironically, though, Democrat Walter Dalton's status as lieutenant governor—meaning he's won statewide before—is the only thing preventing me from ranking this as Solid Republican. Wild card: will Romney's withdrawal from North Carolina hurt McCrory?

North Dakota (Solid Republican)
Call it Delaware in reverse: Republican Governor Jack Dalrymple is very popular, and an unknown Democrat is just not going to overcome him and Mitt Romney.

Utah (Solid Republican)
One of the reddest states in the country electing its governors in presidential years? No wonder Utah hasn't elected a Democratic governor since 1980. Republican Governor Gary Herbert is safe.

Vermont (Solid Democrat)
Vermont, along with New Hampshire, elects its governors every two years. While Democratic Governor Peter Shumlin won office in a squeaker in 2010, this year will be much friendlier to Democrats. That's been borne out by polling showing Shumlin clearly in the lead over Republican Randy Brock.

Washington (Leans Democrat)
This is the heavyweight battle of 2012—Democratic Congressman Jay Inslee in a blue state against moderate Republican and sitting Attorney General Rob McKenna. Washington hasn't elected a Republican governor since 1980, but if anyone is going to do it, it'll be the extremely strong and likeable McKenna. He led in pollsfor much of the spring and summer, but as voters have been reminded of the presidential race and their preference for national Democrats, Inslee has taken a slight lead. While most of his leads remain in the margin of error, various pollsters have all been consistent about saying Inslee is the one with the edge lately, so I will too.

West Virginia (Likely Democrat)
This seems like it should be safe for Democrats, as current Governor Earl Ray Tomblin was able to win a special election in 2011 when the national mood was still very anti-Democrat. However, there has been no polling in the race, leaving it a bit of a dark horse. West Virginia has identified with the Democratic Party for decades, remaining faithful to it on the state level even while voting consistently for Republican presidential candidates. Tomblin is the right kind of pro-coal Democrat whom Romney voters won't hesitate to support, but Republican Bill Maloney should have to only subtly shift Tomblin's public image to make him look much more like one of those hated "Washington liberals." If that happens, you could see a 20% shift of voters from Tomblin to Maloney en masse. Right now I see Tomblin winning this race by about 10 points, but the tables could suddenly turn. One sure prediction: in this presidential year, it won't be a close one; the most Democratic Romney-compatible candidate should be able to dominate.