A group of marketing researchers at University of Illinois at Urbana–Champaign decided to tackle the problem of gender bias by analyzing a peculiar statistical anomaly: hurricanes with female names tend to be more deadly than ones with male names. And that was only the first thing they did wrong.
Kiju Jung and colleagues published the results of their research this week in Proceedings of the National Academy of Sciences. Ed Yong sums up their findings, and lays out the problems with it:
Jung's team thinks that the effect he found is due to unfortunate stereotypes that link men with strength and aggression, and women with warmth and passivity. Thanks to these biases, people might take greater precautions to protect themselves from Hurricane Victor, while reacting more apathetically to Hurricane Victoria. "These kinds of implicit biases routinely affect the way actual men and women are judged in society," says Sharon Shavitt, who helped to design the study. "It appears that these gender biases can have deadly consequences."
But Jeff Lazo from the National Centre for Atmospheric Research disagrees. He's a social scientist and economist who has looked into the public communication of hurricane risk, and he thinks the pattern is most likely a statistical fluke, which arose because of the ways in which the team analysed their data.
Yong goes on to explain carefully what the researchers' findings were before exploring Lazo's objections. Here are a few of the things that Lazo pointed out to Yong:
They analysed hurricane data from 1950, but hurricanes all had female names at first. They only started getting male names on alternate years in 1979. This matters because hurricanes have also, on average, been getting less deadly over time. "It could be that more people die in female-named hurricanes, simply because more people died in hurricanes on average before they started getting male names," says Lazo ...
They included indirect deaths in their fatality counts, which includes people who, say, are killed by fallen electrical lines in the clean-up after a storm. "How would gender name influence that sort of fatality?" he asks. He also notes that the damage a hurricane inflicts depends on things like how buildings are constructed, and other actions that we take long before a hurricane is named, or even before it forms.
On top of all these basic failures to analyze statistics, this study could do lasting harm by undermining the legitimacy of studying gender bias.
Read more at Ed Yong's blog, Not Exactly Rocket Science