I am hoping this post can help slow the spread of misinformation, by bringing to light the nature of sensationalism in online news media, while touching on what to look for to draw your own conclusions from the many studies coming out in relation to COVID-19.
News headlines from most non-educational sources aim to be catchy, leaning sometimes heavily on shock value and current trends or concerns. The intent is not always to mislead, but to grab your attention so that you will go to the website with the article on it. The title is not always reflective of what is written in the article. Read the article and be mindful of the parameters of different studies.
Example- I clicked on an article from AOL about some scientists connecting lack of vitamin D in patients with severe coronavirus-related complications and mortality.
The title was , "New study suggests vitamin D is linked to COVID-19 mortality- how a common vitamin could become pivotal". Say you click on it, because aren't we all hoping and praying for breakthroughs in fighting or slowing down the virus?
The study is a retrospective study- as in, it is studying something that already happened, and we can only study if the thing correlated, not caused, another thing. Correlation is when two things happen at the same time. An example often used by college professors in introductory psychology courses- in childhood, as shoe size goes up, so does age. Are big feet the cause of aging?
Most people laugh at that. Of course the two things are just happening at the same time. If that were true, human beings would reach a point in which we would be immortal or close to it! But the same can be said of vitamin D pills and COVID-19 mortality rates. The article points out a trend, but not a cause. Two co-existing conditions- vitamin D levels and coronavirus severity.
Unless you are directly administering different test conditions to different groups of people in a controlled environment (in this case- taking at least two large groups of test subjects, purposefully administering vitamin D to one group and not to another, in a controlled environment), you cannot say one causes another, or even begin to draw anything but a spark of hope from the study.
Even when you do have a study when there is a strong link between two things, you must be careful to draw conclusions right away. Especially in the early stages of research. This is why we draw conclusions after months of repetitive study or years of studying something, not after one study. This is why I shouldn't walk away from reading the title of the article and thinking to myself, "I should stockpile on vitamin D, because it will lessen my symptoms if I catch the virus".
So... When is enough enough? When can I trust the results of a study? Right now, as a country, we are desperate for good news. Science knows that one study gives us an idea of what direction we can possibly go in for more research, while multiple peer-reviewed studies can show real promise. A common example- in some sleep studies, the test subjects sleep in a research setting, so that all of them are facing the same exact conditions in that place. Note- you must control for the possible side-effects of the new environment. Eventually, you replicate your findings in the real world, where there is less control. Example- lung cancer is highly correlated with smoking. It is one of the strongest correlations that exists in medical science, and has remained one of the strongest year after year. Therefore, we can draw the conclusion that smoking increases risk of lung cancer). Does all of this mean I'm supposed to be paranoid about drawing conclusions from any research study?
Just the ones that don't hit the right checkpoints. We can still draw conclusions from science, but science is not the result of a single research effort, but months to years of careful study and replication. Think of a research study like a doctor.
Do you trust your doctor before they've studied for their exams to test their knowledge? Before they've gone through medical school? When they've only practiced surgery and diagnosis on a mannequin? Before they've done their real-world training? We want well-seasoned professional workers. In the same way, we should always desire a well-seasoned research base.
This is barely scraping the surface of what to look out for when a "new study" suggests something. We like new studies and new conclusions because they can give us hope- that there is no such thing as "too much" caffeine, that some foods high in fat are better for us than other foods high in fat or, say, that there is a panacea for the poison that is this coronavirus. When you have a personal stake in the end results, be wary- wishful thinking has its place in mindless chit-chat, but not in trustworthy scientific study. Curiosity, creative problem solving and painstakingly replicating what little research we have on this virus will pay off, in time.
Be mindful of how you consume news media- especially exciting or newsbreaking studies. Hope for the best, but have a healthy sense of skepticism if a new study tells you what you want to hear.
 |
| Note that the image is from an advertisement- but it's the first thing you see when you click, so it looks like a guy who is pumped about taking some vitamin D! |
Unless you are directly administering different test conditions to different groups of people in a controlled environment (in this case- taking at least two large groups of test subjects, purposefully administering vitamin D to one group and not to another, in a controlled environment), you cannot say one causes another, or even begin to draw anything but a spark of hope from the study.
Even when you do have a study when there is a strong link between two things, you must be careful to draw conclusions right away. Especially in the early stages of research. This is why we draw conclusions after months of repetitive study or years of studying something, not after one study. This is why I shouldn't walk away from reading the title of the article and thinking to myself, "I should stockpile on vitamin D, because it will lessen my symptoms if I catch the virus".
So... When is enough enough? When can I trust the results of a study? Right now, as a country, we are desperate for good news. Science knows that one study gives us an idea of what direction we can possibly go in for more research, while multiple peer-reviewed studies can show real promise. A common example- in some sleep studies, the test subjects sleep in a research setting, so that all of them are facing the same exact conditions in that place. Note- you must control for the possible side-effects of the new environment. Eventually, you replicate your findings in the real world, where there is less control. Example- lung cancer is highly correlated with smoking. It is one of the strongest correlations that exists in medical science, and has remained one of the strongest year after year. Therefore, we can draw the conclusion that smoking increases risk of lung cancer). Does all of this mean I'm supposed to be paranoid about drawing conclusions from any research study?
Just the ones that don't hit the right checkpoints. We can still draw conclusions from science, but science is not the result of a single research effort, but months to years of careful study and replication. Think of a research study like a doctor.
Do you trust your doctor before they've studied for their exams to test their knowledge? Before they've gone through medical school? When they've only practiced surgery and diagnosis on a mannequin? Before they've done their real-world training? We want well-seasoned professional workers. In the same way, we should always desire a well-seasoned research base.
This is barely scraping the surface of what to look out for when a "new study" suggests something. We like new studies and new conclusions because they can give us hope- that there is no such thing as "too much" caffeine, that some foods high in fat are better for us than other foods high in fat or, say, that there is a panacea for the poison that is this coronavirus. When you have a personal stake in the end results, be wary- wishful thinking has its place in mindless chit-chat, but not in trustworthy scientific study. Curiosity, creative problem solving and painstakingly replicating what little research we have on this virus will pay off, in time.
A really "promising" study...
1.) Has results that are replicated by more than one group of scientists
Replication assures us that the results are not random and can be trusted. Plus, scientists learn from other studies and adjust their methodology as needed, to create more extensive and trustworthy studies.
2.) Who are of limited personal consequence and gain nothing from the test results. When someone has something to gain from the results of a study, we call it bias.
Example- be wary of a study finding promising results on a specific medication, when the researchers work for big pharma or the producer's company, and no other researchers are finding the same result as they are). They have something to gain from looking good. But if other studies by independent researchers replicate the same results, the results can be promising.
3.) Has control conditions (at least one group being subjected to something and another group to compare the first group to that receives none of that thing). All of this takes place in a controlled environment (neutral environment)- you want your test subjects to be subjected to as much of the same experience as humanly possible, so that no individual variance that you didn't catch can account for your conclusion.
4.) Finds the same promising result over a period of time
Months or years of studying the same subject only increases the trustworthiness of the results. Right now, we do not have the benefit of many months, or even one year, of data pertaining to the coronavirus.
The best example of how the passage of time helps solidify scientific findings as trustworthy- the correlation between smoking cigarettes and lung cancer. There is, perhaps, no stronger link in medical scienc than that of smoking and lung cancer rates. Because nothing about this correlation is changing over time, and study after controlled study shows the same results, we can confidently say that smoking is a cause of lung cancer.
No comments:
Post a Comment