Why are there errors in data from diet assessments?

Continuing in my trend of reviewing diet assessment methods, today I am reviewing an article written by Dr. Sharon Kirkpatrick.  Dr. Kirkpatrick is a World-Class scholar and a Canadian dietitian from Ontario. She has done some of the best work on diet assessment methodologies.

The article:

Kirkpatrick, Sharon I., et al. “Best practices for conducting and interpreting studies to validate self-report dietary assessment methods.” Journal of the Academy of Nutrition and Dietetics 119.11 (2019): 1801-1816.

Background

Author’s often state that the method they used to assess diet were validated.  For example, here is a quote from this articles methods section:

“Usual dietary intake was assessed at baseline with a validated, semiquantitative food-frequency questionnaire (27–29)…. The reproducibility and validity of this food-frequency questionnaire were described in detail previously (27–29); for example, the correlation coefficient for saturated fat intake assessed by the questionnaire versus 2 one-week diet records was 0.75.”

So what does validation mean?

According to Dr. Kirkpatrick, researchers need to choose the diet assessment tool based on the type of data they want to collect.  Specifically, they need ensure that their tool can accurately capture the component of the diet that the tool is measuring. 

Let’s move move back to our example article. In this study, the goal was to investigate how dietary fat and carbohydrates are associated with atherosclerosis in postmenopausal women.

So, according to Kirkpatrick, these researchers would need to choose a method that can measure dietary fat and carbohydrate intake in a way that is as close as possible to actual intake of these macronutrients.  And depending on the tool they use, tailor how they use the tool to minimize errors.

Tool validation is the process of making sure that the tool measures what you want it to. For our example, to measure dietary fat and carbohydrate intake, it would be imperative that the authors choose an assessment tool that accurately captured intake of all foods and beverages that contain fat and carbohydrate.

Types of Assessment Methods

This article grouped assessment tools into short-term and long term methods.

According to Kirkpatrick, short term tools are good at estimating the average usual intake within a group.  However, if the goal is to figure out what proportion of a group is consuming too much or too little of a nutrient, then it is better to collect multiple short-term measures and take an average.  When doing multiple short term measures, it is best to do them non-consecutively as this helps improve the quality of the results.

Long term assessments measure habitual intake, though how the foods are grouped together will change how granular the data produced will be. Long term measures tend to be population specific and it is not recommended to use these tools in different populations.  For example, in some parts of the world a significant source of fat may come from coconut milk and coconut products. In other parts of the world, significant sources of dietary fats may come from deep-fried foods.  If a food frequency questionnaire fails to ask about consumption of coconut milk or deep-fried foods, the results will not be accurate for the groups with high consumption of these foods.

Objective Measures of Dietary Intake

What people are actually eating can only truly be measured by direct observation or in a feeding study.  Though these methods aren’t considered “naturalistic” because people eat differently when they are being observed. Or, in the case of a feeding study, they are eating what the researchers provide them with.

Biomarkers are also considered objective measures of dietary intake because they do not rely on self-report.  Biomarkers can be used to test the accuracy of self-reported diet measures.  There are three types of biomarkers:

1. Recovery markers, such as doubly-labeled water, 24 hour urine nitrogen excretion
2. Predictive markers, such as 24 hour urine fructose
3. Concentration markers, such as serum carotenoids

Recovery markers are considered true markers of intake. Predictive markers are good at picking up dose response relationships (e.g. the more fructose consumed, the higher the urine fructose). Predictive markers do not have a 1:1 relationship with what they are predicting. Concentration markers demonstration correlations with intake. For example, those who consume more fruits and vegetables tend to have a higher serum carotenoid level.

Why are there errors in self-reported diet data?

Just as the other paper discussed, errors are grouped into two categories: Random and Systematic

There are things we can do to help reduce random errors. For example, conducting a 24 hour recall more than once on non-consecutive days can reduce random error.  But systematic errors are harder to fix. 24 hour recalls are more likely to suffer from random errors.  FFQs are more likely to suffer from systematic errors.  That being said all measures have some error.  This can making validation studies challenging because we can end up comparing one incorrect measure with another incorrect measure.

So back to our example

In the methods section of our example paper they state that they used an FFQ that had been validated against food records.  This means it was validated against a tool that isn’t perfect either. However, as the validation study used more than 1 food record helps improve our confidence in the results.

Take Aways

Observational studies will use different tools to estimate nutrient intake.  We expect there to be errors in the data generated from these tools, but we hope the error’s are too great.  This means we need to be cautious in our interpretation of observational studies.  And it also explains why we will sometimes see differences in outcomes in nutrition studies.