Everything is useful that got into your mouth

Don't believe what they write about nutrition

Christie Aschwanden, FiveThirtyEight: You Can’t Trust What You Read About Nutrition

Translated by Vyacheslav Golovanov, GeekTimes

For links, see the original translation.

At the end of each year, millions of people vow to change their eating habits. Usually, at the same time, people divide food into moralistic categories: good/ bad, healthy/harmful, nutritious/ tasty, weight loss/fattening – but opinions differ about which food belongs to which of these categories.

The US Nutrition Recommendations Committee recently issued a new set of recommendations defining a healthy diet as a diet that focuses on vegetables, fruits, unprocessed grains, low-fat foods, seafood, legumes and nuts, and reduces the content of red and processed meat, refined grains and sugar-containing foods and beverages in the diet. These recommendations immediately caused a storm of controversy. An editorial in the medical journal BMJ concluded that there was a lack of rigorous evidence; this statement was actively disputed by the members of the committee.

Some cardiologists recommend a Mediterranean diet rich in olive oil, the American Diabetes Association supports both low-carb and low-fat diets, and the Committee of Physicians for Responsible Medicine promotes vegetarianism. Ask a fit crossfit fan, and he will recommend you a "paleo diet" based on what our ancestors supposedly ate in the Paleolithic. My colleague Walt Hickey advocates a ketogenic diet.

Who is right? It's hard to say. On the issue of nutrition, everyone has their own opinion. But no one has solid evidence. The problems start because of a lack of agreement about what makes a healthy diet. Weight reduction? Muscle building? Maintaining bone strength? Preventing heart attacks, cancer, dementia? Whatever you are worried about, we will have no shortage of special diets designed to help you. It is ridiculously easy to link eating habits and food addiction with health factors – as you will soon see from a small experiment conducted by our editorial staff.

Our intrusion into the field of nutrition science has demonstrated the depressing state of the articles written about the impact of food on health. As proof, we will take you behind the scenes and show you how these studies are conducted. The first thing to consider is that nutrition researchers are studying an extremely difficult problem, since it is impossible to know what exactly and in what quantities people eat, unless they are locked in a room under supervision. Therefore, almost all nutrition studies are based on measurements of food intake according to the reports of the people themselves, who need to remember what they ate. The most common ways of such calculation are keeping a food diary, memory surveys and questionnaires on the frequency of food consumption [food frequency questionnaire, FFQ].

There are several options for ready-made FFQ, but the technique is the same for everyone: you need to ask people how often they eat a certain food and in what quantity. It's not always easy to remember everything you ate, even if it was yesterday. People usually underestimate the quantity and variety of what they ate, they may not admit that they ate something or incorrectly calculate the amount of food.

"As a result, it turns out that it is very difficult to conduct diet–related research," says Torin Block, CEO of NutritionQuest, a survey company. The company was founded by his mother, Gladys Block, a pioneer in this field, who began developing FFQ at the National Institute of Cancer Research. "There is no way to get rid of mistakes." And yet, according to him, there is a certain hierarchy of such questionnaires in terms of completeness. Food diaries are rated quite highly, and with them – daily surveys in which the administrator interviews the subject, and makes a catalog of everything he has eaten in the last 24 hours. But, as Blok says, "it is necessary to conduct several such sessions in order to get a complete picture of the average diet of the subject." Researchers are usually not very interested in what people ate yesterday or the day before yesterday – they need to find out what they eat regularly. Studies that have used daily surveys usually underestimate or overestimate the impact of food that people do not eat every day, because they record a small and unrepresentative period of time.

When I tried to keep a consumption diary, I discovered how right Blok was – it is strikingly difficult to make a picture of eating habits from the data collected in a few days. It so happened that I went to a conference that week, so I ate snacks and ate in restaurants, which is very different from home meals. My diary showed that one day before dinner I ate only a donut and two bags of chips. And what did I have for dinner? It was a delicious Indonesian seafood curry, but I couldn't give a list of all the ingredients here.

Another lesson I learned from short–term diary keeping is that the very process of tracking consumption can affect what you eat. When I knew that I would have to write it down, I was much more attentive to what I was eating, and sometimes as a result I didn't eat something because I was too lazy to write it down, or because I realized that I didn't need a second donut (or I didn't want to admit, that I ate it).

It's hard to deceive the instinct to lie about the food you eat, but FFQ strives to overcome the unrepresentativeness of brief nutrition records by evaluating what people consume over longer periods. When you read some article with a text like "blueberries prevent memory loss", the evidence was most likely taken from some FFQ. The questionnaire usually asks what the interviewee has eaten in the last three, six or 12 months.

To understand how these surveys work and how reliable they are, we hired Blok to try out a 6-month questionnaire from their company on me, my colleagues Anna Barry-Jester and Walt Hickey, as well as on a group of volunteers.

Some questions – how often do you drink coffee? – they were quite simple. Others have baffled us. Take tomatoes, for example. How often have I eaten them for six months? In September, when my garden was full of them, I ate cherry tomatoes like a candy baby. Maybe I ate 2-3 Cherokee tomatoes with vinegar and olive oil a day. But at the same time, from November to July, I could not eat tomatoes at all. And how do I answer such a question?

Questions about the volume of food puzzled everyone. In some cases, the survey gave us unusual, though useful, tips – for example, it depicted the approximate volume of half a cup, a whole cup and two cups of yogurt using photos of cups filled with sawdust. Other questions seemed absurd. "Who even knows what a cup of salmon meat or two cups of pork ribs looks like?" asked Walt.

Although the questionnaire was supposed to simply measure how much food we consume, sometimes it seemed that there was criticism in the questions – do we drink fatty milk, low-fat milk or skim milk? I noticed that when choosing from three options for the volume of dishes, I always tried to choose the average, regardless of the actual size of my portions.

Despite these difficulties, Anna, Walt, and I did our best to answer honestly and as fully as possible. After that, we compared the results. The questionnaire showed that fatty cheese and different types of alcohol were our main sources of calories.

Then our diets were divided. Walt lost 25 kg on a ketogenic diet, Anna eats quite a bit of protein, and I, according to FFQ, absorb almost twice as many calories as each of them.

Can these results be correct? Anna and I are practically the same height and weight; we could probably share clothes with each other. How can I eat twice as many calories as she does? Block acknowledged that accurate calorie counting is difficult to do, especially without food records for a long period of time, and if you start to understand individual nutrients, it gets even more confusing. He referred to a study from 1987, according to which, in order to fully assess the average calorie intake, it is necessary to collect nutrition data daily for an average of 27 days for men and 35 days for women. And some nutrients are even harder to track – for example, it takes 474 days for women to assess vitamin A intake. This suggests that our reports may be correct, or they may contain a bunch of errors.

Of course, memory–based measurements have their limitations," says Brenda Davey, professor of human nutrition at Virginia Tech. "But most of us nutrition experts think they have value." Calories are the hardest to measure, she says, noting evidence that people underestimate the amount of food they consume that they consider unhealthy, such as fatty foods or sugary foods. "But this does not mean that they underestimate all indicators. This does not mean that there are problems with measuring the fiber or calcium consumed."

The developers of the questionnaires understand that the answers are not perfect, and try to correct them with confirmatory studies that check the data from FFQ with data obtained by other methods – usually a survey about the food taken in the last day or a consumption diary for a longer period. The results of confirmatory studies, according to Block, allow researchers to take into account the variability of daily consumption.

Critics of FFQ, for example, Edward Archer, a computational psychology specialist at the Nutrition and Obesity Research Center at the University of Alabama at Birmingham, says that these confirmatory studies are just reasoning that does not go out of the logical circle. "You take one type of subjective report and validate it with another type of subjective report," he says.

Writing down everything you eat is harder than it might seem, says Tamara Melton, a nutritionist and an official representative of the Academy of Nutrition and Dietetics in Atlanta. Among other things, it's almost impossible to measure ingredients and portion sizes when you're not eating at home. "It's inconvenient. If you're at a business lunch, you won't take out your measuring cup."

When Anna, Walt, and I compared the calorie intake according to our FFQ with what we calculated thanks to our weekly consumption diaries, the results did not match. We had a hard time calculating our portion sizes for FFQ, and who knows which of the results was more accurate?

The difference in the estimate of calorie intake according to FFQ (green)
and consumption diaries (blue)

And while doubts about the accuracy of self-assessment of food intake have existed for decades, these debates have intensified recently, according to David Allison, director of the Nutrition and Obesity Research Center at the University of Alabama at Birmingham. Allison was the author of a 2014 expert report compiled by the Energy Balance Measurement working group, in which the use of "definitely inaccurate" measurement methods was called "unacceptable" for the development of health strategies, research and clinical practices. "In this case," the researchers wrote, "the saying 'at least something is better than nothing' needs to be changed to 'at least something is worse than nothing'."

The problems with the questionnaires are even deeper. They are not just unreliable, they produce huge amounts of data with many variables. The resulting cornucopia of possible combinations of variables makes it too easy to fit them to beautiful and incorrect results, as we learned by inviting readers to take FFQ and answer a few questions about themselves. We received 54 complete responses and looked for connections between the data in them – the way researchers look for connections between food and dangerous diseases. It was ridiculously easy to find such connections.

Our shocking new study found that:

Our FFQ gave out 1066 variables, and additional questions sorted the respondents by 26 possible characteristics (for example, right-handed or left-handed). This large amount of data allowed us to build 27716 regressions in just a few hours (full results can be viewed on GitHub). With such a set of possibilities, we were guaranteed to find "statistically significant" correlations that are unrelated to reality, says Veronica Wieland, a statistician who manages the Batell Center for Mathematical Medicine at the National Children's Hospital in Columbus, Ohio. Using a p-value of 0.05 or less to measure statistical significance is equivalent to a 5% error, says Wieland. And with 27716 regressions, 1386 false-positive results can be expected.

But false-positive results are not the only problem. According to Wieland, there was a high probability that we would find real correlations that were useless from a scientific point of view. For example, our experiment found that people who cut off fat from steak were more likely to be atheists than those who believed that fat was a gift from God. It is possible that this correlation is real, says Wieland, but this does not mean that it expresses the connection of cause and effect.

A preacher who advises parishioners not to cut off fat from meat so as not to lose faith may be ridiculed, but nutrition epidemiologists often make recommendations based on such unreliable evidence. A few years ago, Jorge Chavarro, a nutrition epidemiologist at the Harvard School of Public Health, recommended that women trying to get pregnant switch from low-fat foods to fatty foods, such as ice cream, based on data obtained from FFQ during a study of nurses. He and his colleague Walter Willett also wrote a book promoting a "fertility-boosting diet" based on these data. When I contacted Chavarro to ask how confident he was about the link between diet and fertility, he said that "of all the links we found, we were the least confident about this one." And of course, it was she who made the headlines.

Almost any product that you can imagine has been associated with some kind of health impact in peer-reviewed scientific papers using tools such as FFQ, says John Ioannidis, an expert on the reliability of research results from the Center for Innovation in Meta–Research at Stanford. In a 2013 analytical paper published in the American Journal of Clinical Nutrition, Ioannidis and colleagues randomly selected 50 common foods from a cookbook and found studies evaluating the association of each of them with cancer risk. It turned out that studies have found a link between 80% of foods – including salt, eggs, butter, lemons, bread and carrots – and cancer. Some of them indicated an increase in the risk of cancer, others – a decrease, but the impact of these products on health was "impossibly exaggerated," says Ioannidis, in the presence of a weak evidence base.

Products that increase or decrease the risk of cancer.
Estimates are given for 20 products, each of which was studied,
in at least 10 studies.

But the problems are not only statistical. According to Ioannidis, many of the research results were biologically unlikely. For example, a 2013 study found that people who consumed nuts three times a week had a 40% reduction in mortality. And if nuts really reduced the risk of death so much, it would be a revolutionary discovery, but this number is almost certainly greatly exaggerated, as Ioannidis told me. And without context, it doesn't make sense. Will a 90-year-old get the same benefits as a 60-year-old? How many days or years do you need to maintain a nut diet to activate its effect, and how long will it last? These are the questions people want answers to. But as our experiment has shown, it is very easy to use nutrition surveys to link products with some results, but it is very difficult to understand what these connections mean.

FFQ "are not perfect," Chavarro said, but there are almost no other options for today. "We may have reached the limitations of the current nutrition assessment methodology, and a major shift will be needed to improve the results."

Current research has another fundamental problem: we expect too much from them. We want to answer questions like, which is better for health – butter or margarine? Will eating blueberries keep my mind in good shape? Will I get bowel cancer because of bacon? But research uses memory-based measurements, and these tools are too crude to answer such specific questions.

Ioannidis said one reason may be that individual foods like saturated fats or antioxidants may have very little effect on the risk of disease. This conclusion follows from more thorough randomization checks. That is why the headlines often write about the relative risks – how many people got cancer in the group that consumed more bacon compared to those who did not eat it. The relative risks are almost always higher than the absolute ones, but in fact we are interested in the absolute risk (for example, your risk of getting cancer when consuming bacon). If, for example, one out of 10,000 people who have eaten the most bacon gets cancer, then compared to three people out of 10,000 people who have not eaten bacon, the difference will be threefold. But the difference in absolute risks – a chance of 0.01% against a chance of 0.03% – is tiny, and it will not be enough to change your eating habits.

The tendency to present research results as more accurate and important than they actually are also explains why we face such controversial headlines about products like coffee. "Big data reduces fake accurate results to noise," Ioannidis wrote in his 2013 analysis.

So we return to our original question: what is a healthy diet? We know the basics – enough calories and protein to keep the body from dying. We need micronutrients like vitamin C and iron. But on top of all this, we may attach too much importance to products, says Archer, a doctor from the Nutrition and Obesity Research Center. "There are cultures that abstained from fruits and vegetables, and existed perfectly for thousands of years," he says. Some populations live well today, eating very few vegetables, while others survive almost entirely on plant-based food. The bottom line of all this, according to Archer, is that our body adapts well and can perfectly tell us what it needs, if only we learn to listen to it.

And even if that's the case, I doubt we'll stop looking for secret health elixirs in our pantries and fridges. The reason for the huge demand for such research from the media and society is the same reason why scientists spend billions on conducting these studies. We live in a world in which terrible diseases constantly affect the people around us, and sometimes it happens suddenly. The natural reaction to a heart attack or cancer of a familiar person is to find ways to protect yourself from such a fate. We turn to food to gain a little bit of control. We can't influence what happens inside our cells, but we can influence what we put into our bodies. Science has yet to find the magic vitamin or nutrient that allows us to stay healthy, but we are clearly determined to continue our search.

Portal "Eternal youth" http://vechnayamolodost.ru  20.07.2017