Over at Naked Facts we’ve been posting a series of seven questions to ask when you read the science news. #1 on the list: Where’s the proof?
“This seems almost too simple. But many people reading the news don’t pay attention to the one thing that really matters: Where’s the evidence that this ‘discovery’ is real? Never accept anything just because a reporter says so. Demand proof.”
Okay, I have to walk it like I talk it. And chance, as Pasteur said, favors the prepared mind. Two days after I posted the above, I read a review of a new book called Pandemic by Sonia Shaw, who has done a lot of good work writing about contagious diseases. In it, the New York Times reviewer, Jennifer Senior, did a good thing: She repeated a claim Shaw made in her introduction, then live-linked to the scientific study Shaw cited to support it. Here’s what Senior wrote:
“…she cites a study in which 90 percent of epidemiologists say they believe a global pandemic will sicken one billion and kill up to 165 million within the next two generations. Looking closely at the source of this statistic — a paper by the epidemiologist Larry Brilliant and three colleagues — I’m not convinced that things are quite so dire. (The number of epidemiologists he surveyed is actually quite small.) But…”
Then Senior goes on to the rest of the review. What hit me was Shaw’s Startling Claim: 1 billion sick! 165 million dead! Within two generations! So say 90% of epidemiologists! I track the medical news a little and had never before heard anything like these numbers. They were so specific, and they took such pride of place in the book, and they were from such a good writer, that they seemed instantly credible.
But they aren’t. At least not as far as I can tell.
The book reviewer’s caveat about the small sample size made me wonder about where those numbers came from, and the live link made it easy to check. Kudos to Jennifer Senior and the great grey Times.
So I read the paper. And guess what? No matter how many times I read it, I couldn’t find the numbers in Shaw’s Startling Claim. The study itself was sort of, well, minor. It was a bit long in the tooth for starters, based on data gathered in 2005 and published ten years ago (I think that’s like 60 in epidemiology years). As Senior noted, it was pretty small-scale — the entire study was based on an email survey of 19 “medical experts” and about the same number of non-experts. No telling how many of the “experts” were actually epidemiologists. Another red flag was the purpose of the study, which was designed to compare expert to non-expert opinions about bird flu, not to predict the chances of a pandemic. But the most important thing was this: The numbers Shaw used were not there. Not that I could see.
One more fishy thing: Shaw and Senior both referred to Larry Brilliant as the source of the Startling Statement. But he was only third of four authors on the paper. Usually the first author gets the credit. So why was Brilliant given top billing by Shaw? Turns out it’s not only because he has such a great name. I mean, really, Brilliant? No, turns out it’s because he’s a BIG DEAL in the world of science. Check him out: head of this, leader of that, TED speaker, Google ties, big government experience, big foundation backing, etc. etc. When Larry speaks, people listen.
So I did the next logical thing: I wrote Larry, Larry, what’s the source of those numbers? Haven’t heard back yet. But I also wrote the other three authors on the study, asking them if they could make the published data match Shaw’s Startling Statement.
And here’s where things get weird. . . .
(You can read Part II of the story here)