One of my favourite sections of the newspaper is the Health section. I particularly enjoy reading articles about interesting scientific studies that reveal fruitful information about mental and physical well-being. Unfortunately, journalism and science are not always a match made in heaven, and it is not uncommon to read headlines and articles that are biased and poorly represent the practice of science. Hence, I was pretty disappointed by a recent article that made all the rounds in the media on Tuesday.
The story dealt with the impact that electronic devices have on people's sleep (see here). The title of the story I read was "Gadgets Ruining People's Sleep, Study Reveals," published in the Montreal Gazette. I was initially excited to read the story, but once I started reading I was quickly disappointed by the misleading title, and even more disappointed after learning more about the actual study.
The problem starts with the title, which makes a causal link between two variables - gadgets and sleep. Specifically, gadgets are causing problems with sleep. One of my pet peeves with science journalism is the use of causal statements in titles in relation to studies that are correlational in design. This is the case with the current story as the reader quickly learns that the study did not manipulate any variables via an experiment, so statements of causality cannot (should not) be made. It's misleading and incorrect.
Moving on from the title, the article is filled with statements and assertions that have little to insufficient evidence to support them. This partly stems from issues with the study itself and statements made by professionals associated with the study. For example, the study itself does not include much statistical analysis. Throughout the report (available online - not in a peer-reviewed journal), there are multiple comparisons made among groups of people and variables. Essentially all of the "statistical associations" found in the study are not based on statistics of probability.
For example, the authors note that people who frequently text in the hour before bedtime (21% of the sample) have more problems with getting a good night sleep. The authors then present two percentages (51% vs. 60%). The conclusion that texting is associated with less sleep is being made because one number (60%) is larger than another number (51%). The important question to be asked is this - how large does a difference in percentages have to be in order to conclude there is an association? Five percentage points (i.e., 55% vs. 50%)? Two percentage points?
The way that researchers often answer this question is by using inferential statistics. You use a statistic (like a correlation or t-test) to determine whether the differences are significant (i.e., whether they represent two different populations). No such statistics were performed in this study, so it is difficult to determine whether the results reflect actual differences or random findings. In the end, this study presents trends that are subjectively evaluated by the authors.
Even if the results were assessed with formal statistics, the next question would be "how large is the effect size?" Suppose that electronic devices were harming sleep - how big of an impact are they having (theoretically speaking)? It is important to know the size of the effect because it helps put the findings in perspective. For example, if cell phones are responsible for 50% of sleep problems, this would be a very serious issue. However, if cell phone use was correlated with problematic sleep, but only explained a very small portion of the association (ex; 1%), then it is not that important. As a researcher, you would then move on to examine other variables that play a greater role in problematic sleep. The effect size from the current study is unknown because there are no inferential statistics.
Another important issue to be considered is that of confounding variables. This was a survey study, not an experiment, so no statements of causality can be made. Whenever you find that two things are associated, as a researcher you must always bear in mind the possibility that the association is due to a third variable. In this case, perhaps people who are highly stressed are using electronic devices more frequently, and it is the stress that is causing the poor sleep, not the devices themselves. If this were true, you might find that people who have sleep problems are more likely to be working with a psychologist. Does seeing a psychologist cause sleep problems? I hope not. In any case, no definitive conclusions can be drawn from this study that gadgets play a direct role in sleep problems.
Other problems with this article include a quote from a Harvard professor that has no apparent scientific support. According to the article, Dr. Charles Czeisler asserts that the light from these instruments is suppressing melatonin in the brain, which causes problems with sleep. Of course, there is no research (or statement of research) offered to support this statement. In fact, research shows that someone would have to receive at least 366 lux(1) (a measure of light intensity) for the entire hour prior to sleeping in order for melatonin to be suppressed. This is impossible at least with cell phones, as they typically only emit between 43 and 207 lux.
The article then goes on to quote another person who warns that "the higher use of these potentially more sleep-disruptive technologies among younger generations may have serious consequences for physical health, cognitive development and other measures of well-being." Pretty serious stuff. But what exactly are these consequences? Surely, this statement is not only being made in response to the findings from this study. Other research? Who knows. 'Just be worried' seems to be the message.
Finally, the article notes that these "sleep-deprived gadget users were using caffeine and naps to cope with fatigue." The actual report only presents data on general caffeine use, not use among gadget users, making this sentence highly misleading.
Unfortunately, the picture painted by the article is that (a) Americans are using electronic devices, (b) these devices are ruining sleep, (c) the causal link is suppression of melatonin, (d) everyone is just trying to cope by drinking coffee, and (e) there could be all kinds of dire consequences from using these gadgets.
Having looked at the press release, and considering the quotes offered in the article, I must admit that blame should be shared for this poor piece of journalism. I found a number of other articles all over the internet on this story which might have all kinds of people (and perhaps health professionals) recommending that electronic gadgets be avoided in the evening to aid sleep.
Maybe I should just skip the Health section from now on.
The story dealt with the impact that electronic devices have on people's sleep (see here). The title of the story I read was "Gadgets Ruining People's Sleep, Study Reveals," published in the Montreal Gazette. I was initially excited to read the story, but once I started reading I was quickly disappointed by the misleading title, and even more disappointed after learning more about the actual study.
The problem starts with the title, which makes a causal link between two variables - gadgets and sleep. Specifically, gadgets are causing problems with sleep. One of my pet peeves with science journalism is the use of causal statements in titles in relation to studies that are correlational in design. This is the case with the current story as the reader quickly learns that the study did not manipulate any variables via an experiment, so statements of causality cannot (should not) be made. It's misleading and incorrect.
Moving on from the title, the article is filled with statements and assertions that have little to insufficient evidence to support them. This partly stems from issues with the study itself and statements made by professionals associated with the study. For example, the study itself does not include much statistical analysis. Throughout the report (available online - not in a peer-reviewed journal), there are multiple comparisons made among groups of people and variables. Essentially all of the "statistical associations" found in the study are not based on statistics of probability.
For example, the authors note that people who frequently text in the hour before bedtime (21% of the sample) have more problems with getting a good night sleep. The authors then present two percentages (51% vs. 60%). The conclusion that texting is associated with less sleep is being made because one number (60%) is larger than another number (51%). The important question to be asked is this - how large does a difference in percentages have to be in order to conclude there is an association? Five percentage points (i.e., 55% vs. 50%)? Two percentage points?
The way that researchers often answer this question is by using inferential statistics. You use a statistic (like a correlation or t-test) to determine whether the differences are significant (i.e., whether they represent two different populations). No such statistics were performed in this study, so it is difficult to determine whether the results reflect actual differences or random findings. In the end, this study presents trends that are subjectively evaluated by the authors.
Even if the results were assessed with formal statistics, the next question would be "how large is the effect size?" Suppose that electronic devices were harming sleep - how big of an impact are they having (theoretically speaking)? It is important to know the size of the effect because it helps put the findings in perspective. For example, if cell phones are responsible for 50% of sleep problems, this would be a very serious issue. However, if cell phone use was correlated with problematic sleep, but only explained a very small portion of the association (ex; 1%), then it is not that important. As a researcher, you would then move on to examine other variables that play a greater role in problematic sleep. The effect size from the current study is unknown because there are no inferential statistics.
Another important issue to be considered is that of confounding variables. This was a survey study, not an experiment, so no statements of causality can be made. Whenever you find that two things are associated, as a researcher you must always bear in mind the possibility that the association is due to a third variable. In this case, perhaps people who are highly stressed are using electronic devices more frequently, and it is the stress that is causing the poor sleep, not the devices themselves. If this were true, you might find that people who have sleep problems are more likely to be working with a psychologist. Does seeing a psychologist cause sleep problems? I hope not. In any case, no definitive conclusions can be drawn from this study that gadgets play a direct role in sleep problems.
Other problems with this article include a quote from a Harvard professor that has no apparent scientific support. According to the article, Dr. Charles Czeisler asserts that the light from these instruments is suppressing melatonin in the brain, which causes problems with sleep. Of course, there is no research (or statement of research) offered to support this statement. In fact, research shows that someone would have to receive at least 366 lux(1) (a measure of light intensity) for the entire hour prior to sleeping in order for melatonin to be suppressed. This is impossible at least with cell phones, as they typically only emit between 43 and 207 lux.
The article then goes on to quote another person who warns that "the higher use of these potentially more sleep-disruptive technologies among younger generations may have serious consequences for physical health, cognitive development and other measures of well-being." Pretty serious stuff. But what exactly are these consequences? Surely, this statement is not only being made in response to the findings from this study. Other research? Who knows. 'Just be worried' seems to be the message.
Finally, the article notes that these "sleep-deprived gadget users were using caffeine and naps to cope with fatigue." The actual report only presents data on general caffeine use, not use among gadget users, making this sentence highly misleading.
Unfortunately, the picture painted by the article is that (a) Americans are using electronic devices, (b) these devices are ruining sleep, (c) the causal link is suppression of melatonin, (d) everyone is just trying to cope by drinking coffee, and (e) there could be all kinds of dire consequences from using these gadgets.
Having looked at the press release, and considering the quotes offered in the article, I must admit that blame should be shared for this poor piece of journalism. I found a number of other articles all over the internet on this story which might have all kinds of people (and perhaps health professionals) recommending that electronic gadgets be avoided in the evening to aid sleep.
Maybe I should just skip the Health section from now on.
No comments:
Post a Comment