"Scientists have discovered": reading between the lines
How to read scientific news
Egor Voronin based on the materials of Science-Based Medicine (Harriet Hall, Evaluating Treatment Claims: A Primer)
and U.S. News Health (Dr. Michael F. Roizen, Media's Responsibility: Understand Science and Don't Misunderstand).
A short list of questions worth asking when you read again about what "British scientists have discovered". With an emphasis on medicine and in order of approximate importance.
1. On which organism was the study conducted? Studies on cell cultures, mice, monkeys can show excellent results and be very informative, but they are very rarely repeated in humans without additional research (most are not repeated at all).
2. What exactly was measured, and what is extrapolation? If a drug kills cancer cells, it means that "the drug kills cancer cells", it does NOT mean that "the drug cures cancer". If the first stage of clinical trials of the vaccine was successful, it means that "the vaccine is safe", but it does not say anything about whether it causes the desired immune response, and even more so that "the vaccine is effective".
3. What is the effect size? Often the news reports only that "people who took X experienced Y significantly more often," but this "significantly" may refer to the statistical reliability of the result, and not to its size. For example, it can be very reliably shown that a diet reduces the risk of cancer, but if the risk reduction is only 1%, then it may not have any practical value.
4. Can I trust my reputation? Where was the study published? On Facebook? On the scientist's personal page? In a reputable scientific journal? Who did the research and who is commenting on it? If both are scientists regularly published in scientific journals, then they have more confidence than if the researcher is known mainly as the author of books about diet, and famous actresses comment on his results, swearing that the diet works.
5. If this is a human study, what was its size? Is this an isolated case, a group of 10 people, out of 100, out of 1000? As a rule, the larger the size, the greater the reliability of the result. The flip side of the coin is that the larger the sample, the smaller the difference can be detected, and then see point 3.
6. What is the design of the study? Are there proper controls? Double-blind studies can usually be trusted more, especially when it comes to symptoms, where the assessment of effectiveness is subjective. Often, in order to answer this question, you need to be an expert in this topic, but sometimes it can be clear to a person from the street.
7. Have similar studies been done before and what did they find? The results obtained in several studies and by different groups can be trusted more than a single study. The same results obtained by different methods can be trusted more than a simple repetition. Unfortunately, not being an expert in the field, it is difficult to answer this question, but if the news tells about previous studies, then this is a good sign.
8. Are there explanations of the mechanism of the observed phenomenon based on modern knowledge of biology and chemistry? This is not a mandatory requirement, but unexplained results should be treated with great skepticism (yes, homeopathy, it's about you).
9. What is the general tone of the message? Are weaknesses or directions for future research indicated? If it is claimed that something is 100% working and there are no nuances, then either the journalists have misrepresented, or it's bullshit. If it is claimed that something cures various unrelated diseases (cancer and Ebola), then this is 100% bullshit. If there are restrained comments in the news (even if generally laudatory) from scientists, then there is more trust.
Portal "Eternal youth" http://vechnayamolodost.ru17.07.2014