Humans, it seems, have an innate tendency to overgeneralize from small samples. How many times have you been caught in an argument where the only proof offered is anecdotal? Perhaps your co-worker saw this bratty kid make a mess in the grocery store while the parents appeared to do nothing. “They just let that child pull things off the shelves and create havoc! My parents would never have allowed that. Parents are so permissive now.” Hmm. Is it true that most parents commonly allow young children to cause trouble in public? It would be a mistake to assume so based on the evidence presented, but a lot of us would go with it anyway. Your co-worker did.
Our propensity to confuse the “now” with “what always is,” as if the immediate world before our eyes consistently represents the entire universe, leads us to bad conclusions and bad decisions. We don’t bother asking questions and verifying validity. So we make mistakes and allow ourselves to be easily manipulated.
Political polling is a good example. It’s actually really hard to design and conduct a good poll. Matthew Mendelsohn and Jason Brent, in their article “Understanding Polling Methodology,” say:
Public opinion cannot be understood by using only a single question asked at a single moment. It is necessary to measure public opinion along several different dimensions, to review results based on a variety of different wordings, and to verify findings on the basis of repetition. Any one result is filled with potential error and represents one possible estimation of the state of public opinion.
This makes sense. But it’s amazing how often we forget.
We see a headline screaming out about the state of affairs and we dive right in, instant believers, without pausing to question the validity of the methodology. How many people did they sample? How did they select them? Most polling aims for random sampling, but there is pre-selection at work immediately, depending on the medium the pollsters use to reach people.
Truly random samples of people are hard to come by. In order to poll people, you have to be able to reach them. The more complicated this is, the more expensive the poll becomes, which acts as a deterrent to thoroughness. The internet can offer high accessibility for a relatively low cost, but it’s a lot harder to verify the integrity of the demographics. And if you go the telephone route, as a lot of polling does, are you already distorting the true randomness of your sample size? Are the people who answer “unknown” numbers already different from those who ignore them?
Polls are meant to generalize larger patterns of behavior based on small samples. You need to put a lot of effort in to make sure that sample is truly representative of the population you are trying to generalize about. Otherwise, erroneous information is presented as truth.
Why does this matter?
It matters because generalization is a widespread human bias, which means a lot of our understanding of the world actually is based on extrapolations made from relatively small sample sizes. Consequently, our individual behavior is shaped by potentially incomplete or inadequate facts that we use to make the decisions that are meant to lead us to success. This bias also shapes a fair degree of public policy and government legislation. We don’t want people who make decisions that affect millions to be dependent on captivating bullshit. (A further concern is that once you are invested, other biases kick in).
Some really smart people are perpetual victims of the problem.
Joseph Henrich, Steven J. Heine, and Ara Norenzayan wrote an article called “The weirdest people in the world?” It’s about how many scientific psychology studies use college students who are predominantly Western, Educated, Industrialized, Rich, and Democratic (WEIRD), and then draw conclusions about the entire human race from these outliers. They reviewed scientific literature from domains such as “visual perception, fairness, cooperation, spatial reasoning, categorization and inferential induction, moral reasoning, and the heritability of IQ. The findings suggest that members of WEIRD societies, including young children, are among the least representative populations one could find for generalizing about humans.”
Uh-oh. This is a double whammy. “It’s not merely that researchers frequently make generalizations from a narrow subpopulation. The concern is that this particular subpopulation is highly unrepresentative of the species.”
This is why it can be dangerous to make major life decisions based on small samples, like anecdotes or a one-off experience. The small sample may be an outlier in the greater range of possibilities. You could be correcting for a problem that doesn’t exist or investing in an opportunity that isn’t there.
This tendency of mistaken extrapolation from small samples can have profound consequences.
Are you a fan of the San Francisco 49ers? They exist, in part, because of our tendency to over-generalize. In the 19th century in Western America and Canada, a few findings of gold along some creek beds led to a massive rush as entire populations flocked to these regions in the hope of getting rich. San Francisco grew from 200 residents in 1846 to about 36,000 only six years later. The gold rush provided enormous impetus toward California becoming a state, and the corresponding infrastructure developments touched off momentum that long outlasted the mining of gold.
But for most of the actual rushers, those hoping for gold based on the anecdotes that floated east, there wasn’t much to show for their decision to head west. The Canadian Encyclopedia states, “If the nearly 29 million (figure unadjusted) in gold that was recovered during the heady years of 1897 to 1899 [in the Klondike] was divided equally among all those who participated in the gold rush, the amount would fall far short of the total they had invested in time and money.”
How did this happen? Because those miners took anecdotes as being representative of a broader reality. Quite literally, they learned mining from rumor, and didn’t develop any real knowledge. Most people fought for claims along the creeks, where easy gold had been discovered, while rejecting the bench claims on the hillsides above, which often had just as much gold.
You may be thinking that these men must have been desperate if they packed themselves up, heading into unknown territory, facing multiple dangers along the way, to chase a dream of easy money. But most of us aren’t that different. How many times have you invested in a “hot stock” on a tip from one person, only to have the company go under within a year? Ultimately, the smaller the sample size, the greater role the factors of chance play in determining an outcome.
If you want to limit the capriciousness of chance in your quest for success, increase your sample size when making decisions. You need enough information to be able to plot the range of possibilities, identify the outliers, and define the average.
So next time you hear the words “the polls say,” “studies show,” or “you should buy this,” ask questions before you take action. Think about the population that is actually being represented before you start modifying your understanding. Accept the limits of small sample sizes from large populations. And don’t give power to anecdotes.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.