The Top Five Most Annoying Statistical Fallacies

Mathematical statistics and probability is hard. It often involves what, at a first glance, involves complicated calculations and the sheer volume of data coming out of some studies can often be hard to interpret, even if you know all of the mathematics behind it. Although it is important to understand the math, it is equally important (or perhaps even more important) to understand what the results mean and don’t mean. It is easy to get dazzled by fancy mathematics or over-interpret results to mean something they really do not. Therefore, a basic understanding of statistical fallacies should be a part of every scientific skeptics toolbox or baloney detection kit.

Here is a list of the most common statistical fallacies, what they are and how to combat them.

1. Confusing correlation with causation

A correlation is when two variables vary together, whereas causation occurs when one factor causes the other. It may be tempting to think that the former implies the latter, but that is hardly ever the case. For instance, ice cream sales may increase in the summer and decrease in the winter. The same may be true for drowning accidents. Does this mean we can draw the conclusion that drowning accidents causing ice cream sales? Does this mean that people have become so selfish and morally vile that they prefer to buy ice cream and watching people drown than trying to save them!? Fortunately, not really. Just because two variables vary together does not mean that one caused the other. It might be that the other caused the first, that they both cause each other or that a third factor causes both. In the case of ice cream sales and drowning accidents, a third factor that probably explain the correlation is season. In the summer, more people eat ice cream and go bathing, but fewer to these things in the winter. Confusing correlation with causation is widespread in many areas of pseudoscience, such as the anti-vaccination movement; one of their claims is that as the number of vaccines given have increased, so has the rates of cancer. This shows that the two factors correlate, not that vaccines caused cancer (in fact, the vaccine against HPV and Hepatitis B can prevent cancers) is a correlation, not a causation. A more likely factor is better healthcare as a third factor; better healthcare has meant more vaccines, but also increased lifespan, which is associated with an increase in the risk of cancer.

2. Post hoc

Post hoc and denotes the fallacy of thinking that A causes B just because B follows A in time. This fallacy, like the fallacy of confusing correlation with causation, is understandable from an evolutionary perspective. Those that where too skeptical of attributing an upset stomach to poisonous berries where less likely to reproduce. However, this kind of instinct-based reasoning can no longer be thought of as justified in our modern society. A clear example of this fallacy is thinking that because dawn occurs after the rooster crows, therefore the rooster caused dawn to occur. This fallacy is pretty much a core feature of anti-vaccination rhetoric.

3. Thinking that the average says anything about the spread

This fallacy can often by found in racist or anti-immigration rhetoric. It is the fallacy of claiming that the average (say, intelligence or prevalence of crime) is different a particular group A compared with another group B, therefore, it is reasonable to treat individuals in group A as if they, say, had lower intelligence or higher prevalence of crime. This is an erroneous argument, because the average says nothing about the spread, that is, how different individuals in the group are distributed. When it comes to IQ scores, a group A in which individuals all have precisely 120 IQ will have a higher average IQ than a group B in which half of the individuals have 95 IQ and the other half 140 IQ (0.5*95+0.5*140 = 117.5), yet clearly half of the individuals in group B have a much higher IQ than individuals in group A, despite the fact that group B containing individuals with a lower IQ than individuals in group A. This could feed into a stereotype of thinking that individuals in group B have less IQ than those in A. These groups and figures are just made up for the purpose of a simple explanation and does not refer to any actually existing group.

Follow Debunking Denialism on Facebook or Twitter for new updates.

4. Confusing a priori probability of a specific event with a fortiori probability of a specific class of events

This sounds really difficult, but it is the most used statistical fallacy by creationists. It revolves around asserting that this or that biological structure has a very low probability of arising, because, say, the mutations needed are so improbable to occur in the right sequence, so therefore, evolution cannot have produced them. This is the a priori probability. This may seem like a plausible argument to laypeople, but imagine the following analogy. Have you ever played bridge? The number of possible bridge hands is 635013559600, so the a priori probability for a specific bridge hand is 1/635013559600. Imagine the silliness in getting a particular bridge hand, then exclaim that you could not possibly have gotten that particular hand that you just got since the probability is astronomically low. This is exactly the same error as these kinds of creationists are doing. In the same way that the question “how likely is the given hand I just got?” does not make sense in this context, but instead the relevant question is “how likely is that I get any bridge hand” he question should not be “what is the a priori probability that this sequence of mutations occurred”, but rather “what is the probability of any mutation giving a viable organism?”

5. Confusing statistical significance with clinical significance

The term “statistical significance” sounds complicated, but if a result is statistically significant, it just means that there is a low probability that the results (or something more extreme) would have occured if the null hypothesis was true. However, this says nothing about the probability of the null hypothesis given the evidence. It also does not mean that the results are “significant” in a clinical or scientific context. That is, just because the difference between, say, a placebo group and a group given a treatment is statistically significant, does not mean that the treatment is highly effective. It is entirely possible that the difference is real (in the sense of not just being caused by chance), but that the difference is not really that large to merit thinking of the treatment as highly successful in a clinical setting.

There are plenty of other statistical fallacies and ways to misuse and abuse statistics, such as data dredging, unrepresentative sample, too strong conclusions about the population from a sample and pseudoreplication. Perhaps future blog posts will treat some of these in more detail.