A Pollster’s Life is Never Easy

I enjoyed our discussion of the differences between serious, non-partisan public opinion polls and their imitators. But let’s take a moment for scientific humility and dwell momentarily on two of the most famous polling disasters in American political history. They occurred 62 years apart, and involve two of the most interesting names in American politics – Franklin Roosevelt and Jesse Ventura.

During the 1936 election season, the Literary Digest conducted a random telephone survey in an attempt to predict the election outcome. The results indicated a solid victory for Republican Alf Landon. Of course, Democrat Franklin D. Roosevelt won in a landslide. What went wrong? The Literary Digest’s pollsters forgot that during the depression only the rich owned telephones! So although they called random numbers, the person at the other end was always wealthy and overwhelmingly likely to be a Republican. Roosevelt’s Democratic supporters tended to be poorer, and did not own telephones. But they turned out to vote – Roosevelt became President, and the Literary Digest was bought by Time Magazine in 1938.

Time passed, elections came and went, and researchers improved the science of public opinion polling…

Then, in the 1998 Minnesota gubernatorial election, the Minneapolis Star Tribune made a surprisingly similar error. Just six weeks before the election, the Star Tribune’s survey predicted a victory for Democrat Hubert Humphrey III, with Ventura of the Reform Party a distant third. Of course, Ventura won the three-way race, and became governor of Minnesota. It took exit polling to figure out what went wrong. Exit polls are conducted by researchers who interview people exiting voting facilities. The Star Tribune had surveyed only “likely voters” – those who, based on their demographics, were likely to vote in the election. If only these ikely voters had voted, the Star Tribune would have correctly predicted the outcome, but Jesse Ventura’s candidacy had inspired large numbers of unlikely voters to participate in the election. They turned out to vote, and won the election for Ventura. The Star Tribune’s attempt at greater accuracy actually caused an unexpected error in its survey.

And just to make things more difficult for pollsters…

The ubiquity of caller ID, voicemail, do-not-call lists, and cellphones has made life even more difficult for pollsters. These problems were highlighted in a recent article in the St. Louis Post-Dispatch.

• Pollsters have to call more people than ever in order get enough responses, which makes polling more time consuming and expensive. According to the Post-Dispatch article and other sources, the response rate on telephone surveys has dropped from a recent average of 40% to as low as 20%. About the same percentage of people are hanging upon pollsters, but far more are simply screening their calls with called ID or voicemail. This problem is especially acute among younger voters.
• Because cellphone area codes do not necessarily reflect the physical location of the phone, pollsters trying to sample specific geographic regions are having a harder time than ever pinning down their sample.
• Cellphone numbers are not usually as publicly accessible as landlines, so pollsters are less likely to have a good list of numbers to dial at random.
• Higher income individuals tend to have several telephones, including perhaps three landlines (home, office, and home office) and at least one cell. Middle and lower income persons often have just one or two telephones, and 18-34 year olds of any economic group are increasingly likely to have only a cellphone. This increases the likelihood that a pollster will reach a wealthier, older potential respondent, which may skew the results of random telephone surveys.

But pollsters soldier on! Even now, social scientists are developing new research methods, including email and text message surveys, in order to bring us the most accurate public opinion polls possible. So if you get a call from a real pollster, give him a break – you may be the only person he talks to all day.