The election-night coverage of the 2016 vote began with images of the Clinton campaign team gathering in the Javits Center—under the symbolic glass ceiling. Hillary Clinton supporters were enthusiastic, upbeat, and expectant. The polls gave them every reason to expect that America was electing her as the country’s first female president. There’s just this one little detail, however: the polls were wrong, and the party never happened.
Not since 1948, with the non-election of Thomas Dewey, have the polls been so wrong. Just as in 1948, however, the problem does not rest in the field of statistics, but rather in the field of research methodology.
RealClear Politics identified 16 battleground states and provided data for the latest polls in each of those states. They had combined those polls and posted an overall average for each state. We all understand that these polls include a margin of error that is mathematically linked to the sample size. All else being equal, larger samples have less error; smaller samples have more error. Although RealClear does not provide an overall margin of error for their average of polls, that margin can be easily calculated by taking into account the size of the various samples (and making one or two reasonable assumptions).
Based on these numbers, the RealClear difference between Trump and Clinton was indeed within the margin of error for seven of the 16 battleground states, including Arizona, Colorado, Florida, Georgia, Maine, New Hampshire, and Virginia. In one state, the polls underestimated Clinton’s performance. The RealClear summary hinted at a small Trump lead in Nevada, a state that Clinton would win. The other eight errors, however, were all cases in which Trump’s performance was underestimated, including Iowa, Michigan, Minnesota, Missouri, Ohio, Pennsylvania, North Carolina, and Wisconsin. Some of these states had been correctly predicted to go to Trump (like Missouri and Ohio), but the differences were larger than anticipated. Some of these states eventually went to Clinton (like Minnesota), but the margins were much smaller than expected. Ultimately, however, the unexpected differences in states like Pennsylvania and Michigan (which, for the record, still has not been called) moved the celebration from the Javits Center to Trump Tower.
Republican political consultant Mike Murphy suggested, “Tonight, data died.” He pointed to a very real failure of election forecasting, but his quote suggests a problem with statistics. In fact, however, the problem was in the failure to obtain representative random samples.
The error in 1948 was because many voters did not have telephones. A random sample restricted to homes with telephones might totally misrepresent the political tendencies of homes without telephones. The homes without phones generally had fewer economic resources and different voting priorities. The failure to use random representative polls of all potential voters resulted in the faulty prediction.
Although some will suggest that the polls were intentionally rigged, the fact that the error in Nevada was in the opposite direction suggests otherwise. Pollsters were trying to get an accurate read. In fact, the polling has been extremely accurate in recent elections. Based on the polling data in 2004, for example, I successfully predicted the winner in every single state.
What has changed? In 2016 we use our telephones differently than we did historically. In the past, when my phone rang, I answered it. I answered when my hands were wet, as I washed the dishes, or were dirty with grease, as I attempted to change my oil. After years of robo-calls, however, I no longer answer my phone until and unless I decide whether or not I want to talk to the caller. I check the caller ID, and I practice personal control over how I use my phone.
As a consequence, it is progressively more difficult for pollsters to complete a representative random sample of voter behavior. Our polls are more likely to predict how people who do not use caller ID will behave. It is possible that people who seek more control over their telephones might tend to be the same sort of people who seek more control over their government and other areas of their private lives.
The 2016 poll numbers might have been rigged, but not through any intentional work by the media. Rather, they were unintentionally rigged by the realistic hurdles faced by behavioral scientists, as they seek to explain and understand human behavior.
Even if we are able to fix the research methodology, it is important to remember that statistics always include a margin of error. The data only provide insights into the world. Like the weather forecast, they are not intended to provide any guarantees. Stephen King’s comment about statistical analysis is right on target: in his novel, “11/22/63,” King writes: “There’s always a window of uncertainty.”
Dr. Gary L. Welton is assistant dean for institutional assessment, professor of psychology at Grove City College, and a contributor to The Center for Vision & Values. He is a recipient of a major research grant from the Templeton Foundation to investigate positive youth development.
