Lies, Damn Lies, and Statistics

23Jul09

Statistics are an interesting beast.  You can make statistics say whatever you want, provided you ask the right questions (or ignore the “wrong” answers.  This came to mind today when I received a mail advertisement for an auto insurance company.  The outside proclaimed that “people who switched saved an average of $484”.  This is a sales pitch I have seen from many insurance companies.  It seems impressive on the surface–who wouldn’t want to save hundreds of dollars on anything?  But the skeptic in my head screams out questions: How long did it take for them to save that much? Was it one year or five years?  How many people didn’t switch because you cost more? I mean, if 10,000 people save an average of $484 per year, great, but what if there’s another 50,000 people who didn’t switch because they would have paid an EXTRA $500 per year.  Then suddenly you don’t look so great.  Obviously, they limited their selection sample to the people who actually switched because it gives them a better looking statistic.

A while back I looked at a commentary discussing unfairness in the tax system.  Ari Fleischer was asserting that the wealthy already pay more than their fair share of taxes.  He would switch between talking about percentage of population and percentage of money in order to make his point more dramatically.  He never lied: every thing he said was true according to the CBO reports he referenced.  But he did change the frame of reference depending on the point he was making.

Polling statistics can be manipulated based on the questions asked.  A CBS/New York Times poll conducted in June showed that 57% of Americans were willing to pay higher taxes for guaranteed health insurance for all Americans.  But when asked if they were willing to pay $500 per year in higher taxes for guaranteed health insurance for all Americans, the approval rate dropped to 43%.  (Go to page 4 of the CBS summary here.)

You can also manipulate statistics in polls by who and where you ask questions and how you conduct the poll.  If I go around my neighborhood in Saint Paul and ask people if they support President Obama, I’m very likely to get a very strong positive response (Saint Paul voted in favor of Obama 75% to McCain’s 22%).  I move the same survey to the city of Elk River, MN and the results will likely shift dramatically (42% Obama to 56% McCain).  If the survey is on Look True North I am likely to get a different result than if it is on MN Progressive Project since they each appeal to different audiences.  Or if the poll doesn’t prevent multiple votes the results will be skewed depending upon how passionate and/or bored people are.  Political polls can also depend on the simple designation of “registered voters” and “likely voters”.  I wish I could find polls from back then, but I seem to recall around 10 years ago when a certain colorful person was elected Governor here.  If my memory serves me right, Ventura was trailing in polls as little as a couple days bafore the election.  He won in part because he was an attractive candidate to a younger crowd that tended to vote less often (“unlikely voters”) but turned out in greater numbers than usual.

Polls and statistics can give you useful information, but only if you understand the framework.  Always think about what was asked and what wasn’t.

Advertisements


No Responses Yet to “Lies, Damn Lies, and Statistics”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: