How to Critically Read and Evaluate a Scientific Paper

There are many pseudoscientific groups out there, from creationists and 9/11 truthers to opponents of vaccination and psychiatry that attempt to prop up their beliefs by referencing the scientific literature. However, they often do this in deceptive ways, either misrepresenting good science or taking it out of context or referencing to bad science, published in journals with very low credibility and often not even peer-review. If you are ever faced with someone who is clearly presenting pseudoscience referencing the scientific literature, there is one overarching rule to keep in mind: read the article. I cannot possibly stress this enough, read the article, or at least read a credible summary of it or an already written debunking of the argument using that reference from a reliable source. The goal of this post is to serve as a short introduction to how to read and critically evaluate a scientific paper.

How do you gain access to the paper in question? Easy. Buy it online, or go get a library card from a university library. They often give you the ability to access the papers remotely. Sometimes you can find the paper in full online, if it was published in an open access journals. Key questions we should ask ourselves are: does the article support the claim that the person referencing does? Is the article published in a peer-review journal? What is the impact factor of the journal? Is the method flawed? Is the conclusion in proportion to the results? Have there been independent replication? We will look at these questions, and more, below. But first, let us take a look at what a scientific paper looks like. I have been intentionally vague so far in what I mean by a scientific paper. There are actually two main categories of scientific papers (there are more): research article and review article. Let us look at review articles to begin with.

1. Review Articles

A review article is a sort of summary of the research already been done on an area. The quality of a review article is often linked to the sample of research it summarizes. Generally speaking, the more well-carried out research it uses, the better. The example of a review article I will use is Vaccines and Autism: A Tale of Shifting Hypotheses by Gerber and Offit.

The key way to handle the situation when a pseudoscientists reference a review articles is checking if the article actually says what the person that is referencing is claiming it says. There are two main ways to do this: (1) read the conclusion of the review article and see if it fits or (2) read the quote given by the person in context, to see what the author(s) really meant. If the person misrepresented the article, quote the relevant parts of the conclusion back to him or her. If the person is quoting the author(s) out of context, quote back the context. In both these cases, explain how it is misrepresented or out of context quote as a lot of readers or listeners may skip the academic quote and want a fast summary of the problem.

It may be cases where none of this is true: the quote is in context and it is not a faulty of the review article. If so, you can go further and check the research being summarized. Review articles usually references the research being discussed in close proximity to the quote. Research articles is where we now turn our attention to.

2. Research Articles

A research article is usually an article reporting original research. There are many different methods that can be used depending on the area of science and the specific question being investigated, such as case study, experiment, correlational study, retrospective cohort etc. with specific benefits and drawbacks.

A fairly standard research article contains an abstract, an introduction, a method section, results and lastly a discussion. The abstract is a summary of all the parts of the article, giving a few lines or sentences to each other part. It is used to quickly get an overview of what the article is about and establish if it is relevant for what the person is after. The introduction gives a background to the topic, often starting with generalities and leading up to the purpose of the study. The method section explicitly describes how the study was done in details. This is done so that other scientists can replicate the study or evaluate the method for suitability. The results section gives the results of the study, usually in a diagram or table, along with texts describing these. A discussion looks at the conclusions that can be drawn from the results, possible sources of error, and where to go from here. An example of a research article is A Population-Based Study of Measles, Mumps, and Rubella Vaccination and Autism.

There are to main angles that one can pick to criticize a research article: either point out flaws in the method or an overstated discussion. Let us look at these two in turn.

One major issue in experiments is that correlation does not imply causation. In other words, just because two factors or variables (say, A and B) co-vary, does not mean that A caused B. It may be the case that B caused A (reverse causation), that both A causes B and B causes A (bidirectional causality) or that a third factor (say, C) causes both A and B (third-variable problem; common-cause variable). A classical example of the problem with correlation and causation is ice cream sale and drowning accidents. These follow each other pretty closely during the year. In the winter, the ice cream sale is low and drowning accidents are also low. In the summer, both are higher. Does this mean that drowning accidents cause an increase in ice cream sales? Most reasonable people quickly see that there is a third factor causing both, namely season. When it is summer, people go swimming and eat ice cream. Analogously, the problem with the method section can potentially be that it did not control for an extraneous variable.

An overstated discussion is a discussion that is not motivated by the result of the study. This depends largely on how the discussion is formulated and there are different standard skeletal phrases depending on how strong the results where. As we saw in the summary of Harriett Hall’s discussion of Kirsch and the effect size of antidepressants, a problematic standard for, say, clinical significance, could be used, which influences the nature of the conclusion, sometimes in radical ways. It is also important to separate statistical and practical significance. Even if the statistics and controls are solid, maybe the difference between two groups are too low to be practically interesting.

3. Authors and Journals

There are many other ways to critically examine an article, but these are not deductive arguments, but rather indications. These cannot be used to refute the content on a paper, per see, but they may be useful. Is the author an expert on the topic? Is he an respectable scientists or just a crank? What journal is the paper published in? There is a world of difference between a large, respected journal that uses peer-review from a journal that no one has heard of lacking peer-review. There are many different ways to check this, such as impact factor (how important articles have been) and citation index (relates to the number of citations of articles). Has there been independent replication?

4. Summary

There are many things to keep in mind when critically investigating a scientific paper. Who wrote it? What journal has it published in? Does the journal apply peer-review? Have the person citing it incorrectly characterized the study? Taken quotes out of context? Does the method of the paper have serious flaws? Does the results support the conclusion? Has the results been independently replicated?

Follow Debunking Denialism on Facebook or Twitter for new updates.