A cultural anthropologist named Will posted an attempt at rebutting the general argument I laid out in The Plural of Anecdote is not Scientific Evidence (and various comments at FtB) over at the Skepchick group blog. Unfortunately, Will cannot correctly characterize a single of the arguments that have been advanced and misunderstands vital elements of quantitative methods in science. He also makes a number of other fallacies and errors that is worth looking into. His original post is called A Cult of Quantity. Let’s rip into it.
1. Background and demonstration that Will does not understand the situation
Let me outline the position as it actually were delivered and then we can compare what Will thinks the issue is.
My argument was that since the question primarily (but not exclusively) was about how racists (falsely) believe that the prevalence of racism was so low that it was negligible, we should rationally refute this claim by providing quantitative scientific studies showing that the problem is large instead of providing anecdotes or testimonials. This is because they have no independent evidential support and they may not be representative. Therefore, as a rational strategy aimed at refuting their position, it makes more sense to refer to the science.
That’s it. I thought it was a very clear description of what I meant and that it would be almost universally accepted among scientific skeptics. Apparently, I was wrong. Here is how Will mangled it:
One of the themes that I’ve seen within the skeptical community over time (and, more specifically, in recent discussions about race […] and gender […]) is the hyper-skepticism about social issues and a hyper-rationality when seeking to understand social issues. It often comes in the form of JAQing, typically from people who “just want data.” What they often (but not always) mean by this is that they want specific, quantitative, peer reviewed, top-tier journal published research on whatever specific topic is being discussed, and if that type of research doesn’t exist, then the problem is moot.
There is nothing hyperskeptical or hyperrational about asking for quantitative data when evaluating the size and scope of a problem, or pointing out that anecdotes tells you nothing about it. I also was not JAQing off. I never claimed that I was just asking questions, I did not spout accusations or hide behind them and I did not avoid explaining my position. I did not even request the data, but rather recommend a certain approach when arguing against racists who hold the particular flawed position that the problem is negligible.
There was no specific request for top-tier journals, but simple about quantitative research for answering a quantitative research question. Furthermore, I never said that the problem would be moot if these did not exist. On the contrary, I strongly hold that this research does exist, and that shows that racism is a large problem, so the problem is by definition not moot.
I fully admit that I should have made an extra clarification in my initial post over what I was not arguing. My name was not mentioned by Will and my blog was not linked, so how can I know it is directed at my arguments? Simple. The arguments that Will attempts to criticize closely mirror, apart from the substantial straw men distortions, the arguments that I put forward. Despite having a torrent of straw men arguments directed at me, I am not going to play the martyr card. Instead, I am going to explain precisely where and how Will is wrong.
2. The cult accusation
Cambridge Advanced Learner’s Dictionary defines “cult” as (1) “a religious group, often living together, whose beliefs are considered extreme or strange by many people” and (2) “a particular system of religious belief. Will claims that emphasizing quantitative data over anecdotes and testimonials qualifies as a “cult of quantity”, yet he provides no reason for why this should be considered a cult and he does not provide any scientific evidence for his assertion. In fact, most of modern science (increasingly including social sciences) are becoming more quantitative, so the cult accusation can be interpreted as an attack on quantitative science as a whole.
The cult accusation can also be considered to be a guilty by association fallacy. Religious cults are viewed with scientific and moral skepticism and so associating things like modern quantitative science with it appears to cast a shadow of scientific and moral dubiousness over it.
Finally, calling modern quantitative science a cult is a non sequitur. Even if it was a cult, it would not necessarily be methodologically flawed.
3. The plural of anecdotes is…?
Here is how Will outlines the argument:
Argument: “The plural of anecdote is not evidence/data.”
Response: That depends on the question being asked. For example, if your question is “how many people experience racism/sexism?”, then anecdotal data are not helpful because you’re seeking a quantity.
However, if your question is “what are some of the ways that people have experienced racism/sexism?” then anecdotal data is absolutely helpful. The answer to this question could certainly be compiled to put a percentage on the ways, but the question itself isn’t focused on quantity but quality.
Here Will makes a straw man fallacy. He asserts that the argument is that the plural of anecdote is not evidence/data, when in fact, the argument was that the plural of anecdote is not scientific evidence. It should be understood that the context makes it clear that what is being referred to is that the fact that many anecdotes exists does not automatically qualify as scientific evidence. Anecdotes, in the form of rigorous case studies can very well qualify as useful data after critical evaluation by the scientists performing the case study. But only if this criteria is fulfilled. In other words, the idea with the arguments is that testimonials, at best, tells you that something exists. I say at best because there is usually no independent evidence for the content of testimonials, such as those promoting alternative medicine “treatments”. Even if we suppose that testimonials can be shown to have independent evidential support, that alone does not tell you if these are representative of the population. This is also a clear limitation in any qualitative study. A cultural anthropologist like Will should known this, and he probably does already.
There is also a incorrect characterization of the conversation. The two approaches discussed in the comment section of the post on racism that he links to was not “what are some of the ways that people have experienced racism/sexism?” versus “how many people experience racism/sexism?”, but rather “does examples of racism exist in the skeptical community?” versus “how big is the problem of racism in the skeptical community”. The racists primarily asserted that racism existed, but that it was so uncommon that it was a trivial problem that could be neglected. Providing testimonials in this situation does not rationally refute the racist position and it is unhelpful because it can give the appearance that no evidence exists for the problem (since it was not presented upon request). Clearly, the best way to refute this statement is to show that racism is indeed quite prevalent, and therefore a huge problem.
Yes, asking questions such as “what are some of the ways that people have experienced racism/sexism?” is very valuable, but it was not the question primarily being discussed in those comments and tweets. The racists in question believed that the problem of racism was negligible, and the best way to rationally refute that is to show and discuss quantitative data, not present testimonials.
4. Relationship between size of problem and credible solutions
Here is how Will characterize my argument:
Argument: “We cannot advance a solution unless we know the size and scope of a problem.”
Response: The only thing we learn from quantitative data on the scope of a problem is how widespread the problem is. It tells us very little (if anything) about why the problem is occurring. Qualitative research, on the other hand, is much more useful in helping us to understand why certain behaviors occur, because it utilizes much more focused data collection methodologies. What I mean by “focused” here is that qualitative research seeks a narrow depth whereas quantitative research seeks a shallow breadth.
What this means is that multiple qualitative studies are important for understanding the underlying causes of problems more broadly. In other words, we need more than one qualitative study to be able to generalize. Conversely, we can start with a quantitative study and then move to qualitative data collection to give some depth to the generalized statistical data.
This is, sadly, yet another straw man. The argument was not that solution crafting is impossible if you do not know anything about the size and scope of the problem, but that credible and evidence-based solutions need to be informed by the size and scope of the problem. If we, for instance, underestimate the size of the problem, our solutions will be less than optimal. If you know, with reasonable confidence, that water will fall from the clouds tomorrow, you could go out and get an umbrella. But what if it is a flood? Clearly, the umbrella will not be of much use. When trying to mitigate flood damage, do we care about why it rains? About the intimate details of meteorology? Probably isn’t our top priority. The initial causes of a problem are not necessarily equivalent to the processes that maintain the problem. Understanding why a problem occurs is good, but it is not necessary or sufficient for solution crafting.
5. Will’s bait-and-switch trick: anecdotes vs. qualitative studies
In the above quote, Will makes an interesting bait-and-switch. Previously, we were talking about anecdotes and testimonials. Then Will shifted to qualitative studies and pretended that the argument against anecdotes was being made against qualitative studies, when in fact, no such argument was being made.
The reason for the lack of this argument is simply because qualitative studies are not anecdotes. Qualitative studies with a solid methodology actually transforms anecdotes to reliable data when the scientists are critically analyzing, interpreting and publishing their results. Therefore, Will is erecting a straw man when he claims that the argument against anecdotes is an attack on qualitative studies. It is not.
Of course we need both qualitative and quantitative studies in social sciences. But what we do not need is testimonials and anecdotes, devoid of a critical assessment of independent evidence and representativeness or used regardless of the claim that is actually being criticized.
6. The problem with anecdotes, take 10256
Argument: “Humans are flawed and thus anecdotes and testimonials are not completely accurate; therefore, anecdotes are invalid and tell us nothing.”
Response: This confuses accuracy with reliability. Anecdotes/testimonials can be quite reliable and informative, even if they are not completely accurate. They can paint a personalized and specific picture of a problem that is otherwise overlooked in quantitative data. This can help to humanize social issues (e.g., look at how much American culture has changed with regards to gay and lesbian people over the last ten years—it’s due mostly to the visibility of queer people sharing their lived experiences, not because some statistical dataset was pushed into the public sphere).
Clearly another straw man. The argument was not that because anecdotes and testimonials are not completely accurate, they must be invalid and completely uninformative. The argument was that the major problems with testimonials and anecdotes is that (1) there is no independent evidence for accuracy and (2) they do not tell us how representative the descriptions are (i.e. to what extent they can be generalized to the population) and that these two factors need to be taken into account. Nowhere did I make the claim that anecdotes are not completely accurate. The issue of accuracy and the issue of independent evidence of accuracy are separate things. Anecdotes can very well be completely accurate, but that does not mean we necessarily have a clear-cut way of telling if the evidence supports that conclusion or not (and therefore if we should accept the anecdote or not). Do not confuse ontology with epistemology.
Because of these two problems, anecdotes are invalid in the specific sense that they do not tell us anything about the size and scope of the problem. They are invalid replies to the horribly flawed assertion that the prevalence of racism is so low as to be negligible. Anecdotes are of course not completely uninformative or useless in a global sense. They can, if critically analyzed and interpreted, be used in case studies and case studies can give us a rich tapestry of descriptions, suggest hypothesis for testing and has the ability to study exceedingly rare phenomena in detail. One such classic example is case studies of feral children.
7. Uninformed attacks of quantitative studies
Accuracy and reliability are also issues in quantitative, statistical data (obviously, since statistical data is always accompanied with margins of error). But we don’t dismiss a quantitative study because it had a 95% accuracy rate. That would be pretty damned reliable, despite not being completely accurate.
Will is of course comparing apples to oranges. The fact that well-defined margins of error appear in quantitative studies is a strong argument in their favor, while the the lack of independent corroboration and lack of knowledge of representativeness of an anecdote is a strong argument against it. Both qualitative and quantitative studies will have limitations and inaccuracies, we have to understand that there is a huge difference between the level of uncertainty between quantitative results with well-defined error bars and the average quantitative study (present one with 95% accuracy!) and that the latter generally do not, and indeed usually cannot in practice, even attach a margin of error to their raw data to begin with. You can discuss alternative hypothesis and interpretations, but that is a different matter.
8. On Objectivity
It seems to me that this comes from an over emphasis on objectivity such that many skeptics feel that they can escape subjectivity. But this is not possible. We are temporally and materially situated beings living in the world. By necessity, we cannot be completely objective […]
This is another fallacy, this time argumentum ad hominem circumstantial. Will is attacking a position by explaining the processes he thinks made his opponents come to that conclusion. This is of course irrelevant. The validity of a claim rests on the merits of the claim itself, not by what path a person took to reach that conclusion. Furthermore, the very nature of quantitative studies includes an appreciation of subjectivity in the form of error bars. Moreover, we already discussed that it is really a spectrum, and that most quantitative science with a solid methodology is more objective than the average qualitative studies in some aspects because the subjective opinions in qualitative study sample may be poorly understood and analyzed because of subjective misrepresentations by the researcher. In quantitative studies, on the other hand, there is operationalization of variables (and therefore an unambiguous way to measure them) and replicates. Quantitative studies are not perfect and have their own set of limitations, but there is a difference in degree of subjectiveness.
When racists falsely claim that the problem of racism is negligible because the prevalence is very low, the most rational rebuttal is to point out the scientific studies that show that the prevalence is not negligible. Anecdotes or testimonials are invalid rebuttals and counterproductive, since it may be seen as a way to avoid responding to the assertion. Anecdotes and testimonials are also usually without independent evidential support and their representativeness (and therefore the degree to which they can be generalized) is unknown. Evidence-based estimation of the size and scope of a problem informs reasonable solutions. An umbrella might work against rainfall, but is hardly effective against a flood. In that sense, reasonable solutions need to take into account the size and scope of a problem. Quantitative studies are not just “collections of anecdotes”, so the bait-and-switch does not work successfully. They are critically investigated, analyzed, interpreted and published by scientists. This transforms the data into evidence. Quantitative studies have error bars and uncertainty, but we have to appreciate that we are dealing with a spectrum. Qualitative studies rarely, if ever, attach error bars to their raw data (how could they?). Because qualitative data involve additional layers of subjectivity and subjective interpretation they can generally be viewed as more subjective than quantitative studies (even though the distribution probably overlaps). Will does make some good points such as being a scientific skeptic does not imply that you are objective and so on, but those are outweighed by the intellectually unimpressive reasoning throughout the rest of the post.