As researchers with a national lens, we hope this analysis might help policymakers and health officials to identify common problems with gathering the data and target improvements moving forward. https://t.co/wHdZBrf9ME
Finally, we look at data presentation. Design decisions affect accessibility, and we’ve seen states require users to navigate pesky buttons and toggles. We’ve also found that states struggle to communicate known backlogs and delays, which might lead to context getting lost.
Next, we look at how states make their data available. Some states report metrics in percentages, without raw numbers. Some provide granular data while others report only summaries. And some states don’t report some metrics at all.
First up: data definitions. More than one year in, many facets of COVID-19 data are still not standardized. States have defined metrics inconsistently, making summaries and comparisons difficult if not impossible.
With little guidance from the feds, states have had to make their own decisions about compiling and presenting COVID-19 data, leading to sweeping inconsistencies. Here are some of the key reporting problems we found. https://t.co/wHdZBrf9ME
Tip 6: Be cautious about what the data can say. If you’re trying to extract insights from the data itself, it can be very easy—especially within a headline—to make causal claims when only correlative evidence is available.
Tip 5: Get familiar with caveats. The most recent dates in epidemiological datasets are always incomplete—because, for example, the data points for people who died today won’t finish being reported for many days, weeks, or even months in the future.
Tip 3: Holiday and weather-related reporting issues happen when national or natural events occur across many states at once, and can mimic shifts in the pandemic. Look for holidays or major disruptions that might have artificially depressed—and then inflated—the data.
Tip 2: Data backlogs—and the “data dumps” that occur when those backlogs are resolved—can mimic major declines and then jumps, especially in cases, tests, and deaths. Look for explanations on state dashboards and call public health officials.
Tip 1: If you see dramatic movement in the data, look for contextual clues before interpreting it as a change in the pandemic. Day-of-week effects in data arranged by date of report produce predictable reporting swings over the course of each week.
Over the past few weeks, we’ve noticed that newsrooms of all sizes—and even some government agencies—have fallen into some of the data potholes that we’ve become familiar with in our year of wrangling public COVID-19 data.