What I Didn't Expect in Politics in 2016

This year was a bad one for people in the political prognostication business. I'm not afraid to say that I failed as spectacularly as anyone: throughout the Republican primary, I clung to the orthodoxy that "the party decides" and that the GOP would never let Donald Trump become its nominee. Throughout the general election, I dismissed Trump's chances of winning over a country with sexism, racism, and scare tactics. Now, we're 41 days away from him taking the oath of office.

I believe that any responsible pundit must reflect on such a failure before reengaging in the profession. We—or at least I—make our predictions based on past information. The election of Donald Trump represented a new piece of information that we must add to our calculus going forward. But we also must avoid overreaction. Throwing our hands up and declaring elections unpredictable is an overreaction. Saying that polling and other hard data are useless is an overreaction.

The truth is that Hillary Clinton did win the popular vote by about two percentage points, as predicted by the national polls. Under different rules of the game, that would be enough to hand her the presidency, and none of this Democratic and media handwringing would be happening. That's right—not a single person could have voted differently and everyone who currently looks so dumb would actually have been totally correct. Not a single person could have voted differently and the prevailing narrative would have been the implosion of the Republican Party instead of the meltdown of the Democratic. But because the votes were distributed just right, and because we use the Electoral College, not the popular vote, we got the outcome we did.

To me, the lesson of 2016 was randomness. Sometimes, data have errors. Sometimes, you have bad luck. Sometimes, unlikely things happen. In looking back at my predictions, I have a hard time seeing what I should have done differently. To predict a Trump win (in the general, at least) would have required a series of assumptions that, at the time, would not have been well grounded in fact or theory. Given the information I had before the election, I actually still think I made a sensible call in expecting Clinton to win. But my mistake was being so sure of it. In an effort to draw attention and project confidence, I treated an event that was likely to happen as one that was certain. I didn't fully appreciate that we live in an uncertain world. I didn't properly allow for the possibility of a polling error or something else weird happening. I didn't stop to think that a 90% chance still comes up empty 10% of the time.

On May 21, 2010, with two outs in the third inning, the Los Angeles Angels intentionally walked St. Louis Cardinals outfielder Skip Schumaker to load the bases for pitcher Brad Penny, a career .157 hitter. Six out of seven times, Penny makes the final out of the inning, preserving the 4–4 tie. On this occasion, though, Penny promptly hit a grand slam for what proved to be the winning hit. Was it the wrong decision by the Angels? I would argue that it was the right call; it just didn't work out this time. Sometimes, the right strategy can still lead to a bad result (and vice versa). It doesn't automatically discredit the strategy, as long as the strategy has been proven sound over a larger sample size. Data and political science remain the most accurate tools for predicting elections over the long term; one bad year doesn't change that.

Going forward, I will continue to use factors like polls and demographic trends in predicting elections. But I will change my attitude to be more humble about my own fallibility. Never again will I declare anything as a "slam dunk" or a "sure bet"; the fairest analysis acknowledges that anything can happen while still pointing toward the overall likelihood of one outcome. To those who heeded my political advice in 2016, I apologize for being overconfident. In the future, I promise that I will retain a more open mind and always be aware of my own limitations as a soothsayer.

One way to do that is to move away from the language of "calling" states and toward a probabilistic election forecast. The Cook Political Report uses such a system through its Solid-Likely-Leans scale of handicapping races. Every year, I use the same scale to make my annual constitutional-office election predictions, with much more favorable results than my failed presidential prognosis. Indeed, while I wasn't perfect in calling downballot races this year either, my more nuanced assessments of each campaign stand up better to post factum scrutiny.

In 2016, I issued Cook-esque race ratings for 52 downballot races: lieutenant governors, attorneys general, secretaries of state, treasurers, auditors, superintendents of public instruction, commissioners of insurance, commissioners of agriculture, a comptroller, a commissioner of labor, and a commissioner of public lands. In the 40 races where I gave one party the edge, that party prevailed 37 times. Specifically:

• Democrats won all 11 races I rated as Solid Democratic, including three that lacked a Republican opponent.
• Democrats won five of the six races I rated as Likely Democratic.
• Democrats won five of the seven races I rated as Leans Democratic.
• Republicans won 10 of the 12 races I rated as Tossup.
• Republicans won all five races I rated as Leans Republican.
• Republicans won all three races I rated as Likely Republican.
• Republicans won all eight races I rated as Solid Republican, including two that lacked a Democratic opponent.

Here is the Democratic margin of victory or defeat in each of the elections (save Washington treasurer, which was a Republican-on-Republican contest). Note that these numbers are unofficial election results from the Associated Press.

As in 2014, I underestimated Republicans across the board, but not egregiously so. Democrats' average 3.2-point margin of victory in Leans Democratic races, for example, is right where you want it to be, and it isn't a shock that one or two (OK, two) of the elections I rated thusly actually flipped the other way. However, the same can't be said of the races I dubbed Leans Republican. The GOP won them all, by at least 7.1 points (an average of 15.5). My Tossup races behaved more like Leans Republican should: the GOP won them by an average of 6.1 points, losing only two of the 12. That's what happens in a wave election, though: close races all tend to break the same way.

I can console myself with the fact that my biggest errors were isolated to just a few states. Misread the political climate in just one state, and you can blow several races, even as you accurately predict the rest of the nation. For me in 2016, that state was North Carolina. The Tarheel State was responsible for my biggest miss: declaring Democrat Wayne Goodwin and Republican Mike Causey's battle for state insurance commissioner a Likely Democratic contest. Instead, Causey defeated the two-term incumbent by 0.9 points. Almost as embarrassingly, Democratic Auditor Beth Wood—whom I deemed Likely to beat Republican Charles Stuber—prevailed by only 0.1 points in initial returns. (The race isn't even officially called yet, as Stuber has requested a recount.) North Carolina also played havoc with my projections by almost ousting Democratic Secretary of State Elaine Marshall (Solid Democratic) and rejecting Superintendent June Atkinson (Leans Democratic).

Oregon and Missouri were two other states that I misjudged. Democrat Tobias Read won the Oregon auditor's race by only 1.9 points, even though I had dismissed the contest as Solid Democratic. Meanwhile, my third and final incorrect "call" came in the Oregon secretary of state election, where Republican Dennis Richardson defeated Democrat Brad Avakian in a Leans Democratic competition. Farther inland, Missouri singlehandedly goosed the average Republican margin of victory in my Tossup and Leans Republican races when it swung much farther to the right than anyone was expecting. Its Democratic candidate for lieutenant governor lost a supposed Tossup race by double digits, and, while I expected its attorney general and secretary of state races to Lean Republican, I didn't anticipate them doing so by 17.4 and 19.5 points, respectively.

Elsewhere, I called most races right on the money. I'm especially proud of my performance in West Virginia, where I correctly foresaw a close race for secretary of state (presumed favorite Democrat Natalie Tennant was stunned by a 1.8-point loss) and GOP wins for agriculture commissioner and auditor, despite the state's ancestral Democratic tone. Clearly, handicapping downballot races is my calling more than presidential forecasting is—and that's OK by me. I'm looking forward to bringing these understudied races even more into the light as we turn the page to 2017 and the busy midterm year of 2018, when the biggest batch—over 140—of constitutional offices are on the ballot.