Debunking Denialism

Defending science against the forces of irrationality.

The Plural of Anecdote is not Scientific Evidence

9 Comments Posted by on August 2, 2012

research

Note: I am in no way saying that racism do not exist in the skeptical community. It does. It is probably as prevalent in the overall population. My general point is that we need scientific evidence, not anecdotes, to accurately quantify the size of the problem and that such a quantification helps us find appropriate solutions to the problem in ways that anecdotes do not (clarification added 20120802 22:36 GMT +1 DST).

Note: Yes, anecdotes and testimonials could be extremely useful in convincing people in practice, but that is not the same as being useful for rational refutation (clarification added 20120803 01:29 GTM + 1 DST).

Let’s say that a person, which we can call X, comes up to you and tells you that X has a successful treatment for a given chronic disease. Or maybe X tells you that the scientific community is oppressing or discriminating proponents of this treatment.

A successful treatment for a chronic disease is great news of course, if it actually works. Also, discriminating is clearly morally unacceptable. Or is this another quack “treatment” that is exploiting chronic disease patients? Is X just playing the martyr card?

As a scientific skeptic, however, the first thought that goes through your mind probably is something along the lines of “That’s interesting! I wonder if there is any scientific evidence exists for that?”. So you ask X about the evidence, and X responds “There is a lot of evidence!” and shows you a couple of testimonials where people say that the treatment has cured them of their chronic condition. X uses other testimonials as examples of discrimination from the scientific community.

“But wait a minute…”, you interject. “That is not scientific evidence, those are merely testimonials!” Something is obviously wrong here. “Why are you dismissing the experience of these people? They have gone through a great ordeal of having a chronic disease and on top of that, been discriminated against by the scientific community!”, replies X in a high-pitched voice, signaling agitation and emotional arousal.

Clearly, you have to tread slowly here or this might get ugly. You take a deep breath before responding that “I am not trying to dismiss their experience, I just want to know about the evidence. When you present testimonials, there are two problems. First, there is no independent support for what is being said. How do we know they are accurate? Should we just trust people’s beliefs on issues as serious as discrimination and medical treatment? Second, there is no way of telling, just from the testimonials themselves, if they are representative. Do these successes represent 99% success of treatment, 50% or 0.00001%? How do we know that it is not just spontaneous remission, or the placebo effect? The plural of anecdote is not scientific evidence. It is just a bunch of anecdotes with questionable scientific relevance.” You exhale, thinking that you nailed the situation and that X will understand your reasoning. You could not be more wrong.

X responds predictably that “So you support discrimination and would refuse giving successful treatments to these chronic disease suffers!?”. You sigh. “No, that is not what I am saying at all because….” You do not get a chance to finish, because X interrupts you with “They can’t win, can they!? If they don’t speak out about our treatment and the discrimination, they have to suffer in silence, but if they do speak out, they get your repugnant skepticism and hypertechnical demands for evidence!” You have finally realized that conversation is impossible. You give it one last try. “I just want to know what evidence…” you explain slowly, but are yet again interrupted: “Evidence!? What kind of evidence do you need then!? What exactly do we need to do to proceed and convince you die-hard skeptics!?” Bingo! Now we are becoming productive.

You take another breath and launch the winning response: “I want well-controlled scientific studies published in reputable, peer-review journals. Only then can we find out if the treatment actually works and the extent of discrimination of proponents of your treatment.” That ought to do it. Yet you are wrong again. X replies that “but the treatment and presence of discrimination cannot be tested because of special feature A, B and C with the treatment.” Predictable response by X, but now the tables have turned. A, B and C are arbitrary, irrelevant or can be included in the study. Your final response becomes “if scientific studies cannot show that the treatment work or that discrimination is extensive, then you have no justification for claiming that it does. End of story. Less anecdotes, more science. If your treatment actually works, then present the evidence or stop screwing chronically ill people out of their money.” before you walk away. Not all discussions that can be had are worth having.

Skepticism ,

9 Responses to The Plural of Anecdote is not Scientific Evidence

  1. Pingback: Scientific Studies of Racism? | Greta Christina's Blog

  2. Maude LL August 2, 2012 at 20:42

    Your analogy is deeply flawed. First, with regards to racism, the equivalent of bunk cancer treatment would be the person saying that racism hasn’t been “proven”. Just shut your eyes and it ceases to exist. DeepakChopra-esque! Anyway, there’s a reason social sciences stopped trying to be physics about a hundred years ago: it was very unscientific (i.e. it didn’t work, results were in fact catastrophic). The scientific method in social sciences is different, social sciences must use more than one method to gather information. Collecting data alone does not work. In this instance, the only way we could really quantify racism perfectly is if we had a way to distinguish racism from other motivations in the brain of the person dismissing another person of a different ethnic background. This is why peer reviewed papers from social scientists often consists of a “plurality of anecdotes”.
    Here’s an example. As with everybody else, I’ve been in many situations were I was dismissed. If you take the aggregate of the times I was dismissed, it appears to be higher than the male counterpart of similar background. You will find individual males who have been dismissed more often than I. So far, we don’t have much information. However, if we get a large random sample of women, take account of their experiences with as much objectivity as possible (i.e. some women do choose to work part-time to take care of their family, therefore cannot reasonably become CEO, so other factors need to be accounted for), we note a trend. The average white middle class man will have less obstacles than the average white middle class woman. Now that’s when you can turn this into relevant information, not before. Very rarely can we know for a fact that a particular action was directly due to racism. However, we can see an aggregate of anecdotal evidence that underline a specific dynamic favoring a certain group.
    I used the example of sexism here because I am a white woman, but I believe it applies to race as well. I’ve had frustrating experiences with people calling me a racist. For example, I worked as a server and always emptied the tip jar when I started my shift. One time, I did just that and a customer got angry because he thought I was hiding the money because he is a Native American. If that information got into a study, that particular instance would have been wrong. This is where the researchers have to be careful, there is a confirmation bias on the part of the oppressed group. The group has identified systematic racism therefore attribute to racism actions that due to other factors.
    I’m not sure if you are trying to imply that people raising the problem of racism in the atheist community are like the Burzynski clinic or something… I don’t want to put words in your mouth; are you saying that racism could be non-existent in our community? Because that would be baffling. (and would require you to learn beyond an overview of an “introduction to methodology in social sciences” class)

    Reply
    • Emil Karlsson August 2, 2012 at 21:34

      1. You are making a straw man argument. The analogy was not between the treatment for chronic disease and racism, but between discrimination and racism.

      2. Actually, social sciences have increasingly been using similar methodologies to natural sciences, such as statistical analysis of quantitative data. It works and it is very common.

      3. To quantify racism we do not need to separate it neurologically from other motivations, just analyze behavior. Some behaviors are considered racist, even if the person intended something else. In other words, we have a different view on how to operationalize racism.

      4. Furthermore, the quantification does not need to be perfect. In fact, no quantitative study, in natural or social sciences, needs to be or claim to be perfect.

      5. You are confusing scientific evidence with anecdotal evidence. Filling out questionnaires may seem like they are providing anecdotal evidence, but this becomes scientific evidence after careful and appropriate statistical analysis by a scientist. That is because it provides, if the study is good, a defensible quantitative estimation on prevalence, rather than an over-reliance on anecdotes.

      6. I am not saying that racism is non-existent. It exists, and it is probably prevalent. The general point is that we need to use scientific studies, rather than anecdotes, to get a good idea about the size of the problem and that it is such considerations that help us find solutions in ways that anecdotes do not.

      I will add an explanatory note to the post for clarification.

      Reply
    • Parsa August 3, 2012 at 01:43

      “Anyway, there’s a reason social sciences stopped trying to be physics about a hundred years ago”

      Do you mean, say, analytical (in much the same sense that Anglo-American philosophy is called “analytical”), facts-not-values-oriented, mathematical / computational, etc.? Ah no, that’s an unwarranted generalization: agent-based computational economics and sociology are two examples of research programs in that vein that are undoubtedly alive and kicking. There is even a modern effort to bridge physics and social sciences in a less naïve way than was attempted in the 19th and early 20th centuries, not surprisingly called econo- and sociophysics.

      You should know: there’s a great wide world outside of “[name of protected group] Studies 101″.

      Reply
  3. Mark Erickson August 3, 2012 at 05:41

    I just gave you my personal support on Greta’s blog post. Not taking sides on the argument, but saying you’re a good guy and to give your blog a chance. I did a bit of analysis on you there, too. I apologize if it offends, let me know if it does. Just trying to help.

    Reply
    • Emil Karlsson August 3, 2012 at 18:02

      Thanks for your input. I spent about five hours commenting on the two blog posts and I feel that I have gotten through to Greta with the substantive parts of my position and I interpret we are in broad agreement, although we may differ on some specific issues.

      In that sense, I feel done with this particular issue.

      Reply
  4. Iamcuriousblue August 3, 2012 at 20:07

    It looks like your arguments over at Greta’s blog are being dealt with by the commentators by their usual “rational” method, namely, piling-on. As with many of the “freethought” blogs, things like nuance and disagreement over specifics quickly go out the window, and things get reduced to either towing the line or being The Enemy. Hang in there – your points are stronger than the dismissive shoutdowns they’re receiving would indicate.

    Reply

  5. Pingback: The Triumph of Modern Quantitative Science « Debunking Denialism

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out / Change )

Twitter picture

You are commenting using your Twitter account. ( Log Out / Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out / Change )

Cancel

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 57 other followers

%d bloggers like this: