Natural selection of "bad science"

Low-quality research is getting more and more money

marks, Geektimes

In 1962, Jacob Kang, a psychologist from New York University, analyzed about 70 articles published in the Journal of Abnormal and Social Psychology and discovered an interesting fact. Only a small number of scientists acknowledged the failures of their research in their works. For these materials, he calculated their "statistical power". The term "statistical power" means the probability of rejecting the null hypothesis if it is factually incorrect.

According to statistics, the confirmation of the result expected by the researcher is manifested in the course of only 20% of the experiments performed. As it turned out, in almost all the works studied by Cohen, the authors indicated a positive expected result of the research. It turns out that the authors simply do not report failures. Moreover, some authors distort the results of their research, indicating a positive effect even when there is none.

The amount of power when testing a statistical hypothesis depends on the following factors:

• the magnitude of the significance level, denoted by the Greek letter α (alpha), on the basis of which a decision is made to reject or accept an alternative hypothesis;
• the magnitude of the effect (i.e. the difference between the compared averages);
• the sample size required to confirm the statistical hypothesis.

More than half a century has passed since the publication of Jacob Cohen's work, but the authors of scientific research still talk about their successes, hiding defeats. This is proved by the results of another paper published recently in The Royal Society Open Science (The natural selection of bad science). The authors of this work are Paul Smaldino from the University of California and Richard Mcelres from the Max Planck Society Institute for Evolutionary Anthropology. According to the researchers, modern articles have not become better. At least, articles that relate to psychology, neurology and medical science.

Having studied several dozen articles published in the period from 1960 to 2011, scientists have determined that the average statistical power in this case is 24%. This is only slightly higher than the parameter that was calculated by Cohen. And this is despite the fact that in recent years the methods of scientific research have become more accurate, and more and more books and articles describing the principles and methods of scientific work are being published for researchers.

Average statistical power of publications,
published in scientific journals from 1960 to 2011 .

Having received such a result, scientists thought about what could change the current state of things so that the authors of scientific papers become more conscientious. To do this, Mcelres and Smaldino created a computer evolutionary model. Within the framework of this model, about 100 virtual laboratories competed for the right to receive remuneration. It was paid if the laboratory team received a really significant result within the framework of the study. To determine the amount of remuneration, scientists used such an indicator as the volume of publications.

As it turned out, some laboratories worked more efficiently than others, showing more results. At the same time, these laboratories often gave out the expected for the real. In this case, the results were verified worse, and the results were interpreted as positive. If the results of the work were verified more carefully, then fewer works were published.

In each simulation cycle, all simulated laboratories performed experiments and published the results. After that, the scientists removed the oldest laboratory from a number of randomly selected ones. And laboratories from another random list (the sampling criterion was the maximum number of rewards received) allowed them to create their own department, which was actively engaged in the publication of scientific materials. Preliminary results of the analysis of the computer model demonstrated that the laboratories that published the most papers devoted only a small fraction of time to checking the results and became the most authoritative, spreading their research methods in the scientific community.

But there was something else. As it turned out, the repetition of the results of the work of one laboratory by the team of another leads to an improvement in the reputation of the first laboratory. But failure to repeat the results of any experiment leads to problems and a decrease in the reputation of the laboratory that conducted such an experiment first. In this case, a filter is triggered that prevents the appearance of fake studies with modified research results in the scientific community.

The stronger the punishment for those who published unverified results, the more powerful the filter of low-quality studies turned out to be. With a maximum penalty of 100 points for laboratories with fake data, the number of publications with real results sharply increased. In addition, the number of repeated experiments conducted by other laboratories with the intention of repeating the results obtained by someone was also growing.

Let me remind you that all of the above is a situation modeled on a PC. The authors of the study draw the following conclusion: as before, now scientific organizations that publish more papers than others are considered the most authoritative. Unfortunately, the filter of low-quality publications, which worked in the virtual world, does not work too well in the real world. The fact is that research institutes and individual researchers do not check each other's results too often. If such checks with the intention to repeat the result obtained by the partner were carried out more often, then there would be significantly fewer "fake results" in the world of science.

The authors of the study believe that the computer model showed the possibility of changing the current state of things. If foundations and scientific organizations did not give money to those scientists and laboratories who published unverified results of their research, passing them off as a positive result, then there would quickly be fewer cheaters. But it is quite difficult to implement such a model in the real world. "Easier said than done," Smaldino says.

So for now, those organizations that publish a lot of articles are in the black. But organizations that carefully verify their results are published less often.

Portal "Eternal youth" http://vechnayamolodost.ru  27.09.2016