The End of Evidence-Based Medicine?

stains-cover-webA recent debate about statins in the British medical press — Lancet vs. BMJ! Cage fight! — has me wondering about a bigger question: the limits to what physicians and others in the health industry call “evidence-based medicine.” Let’s shorten that to EBM, and see what the problem is.

EBM is usually defined as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.” Hard to argue with that. Of course doctors should weigh what the current evidence says about a treatment before subjecting their patients to it. There may be questions about what is accepted as “evidence” and who defines what’s “best,” But that shouldn’t be too hard, either. Most physicians that I’ve talked to (and most people in general) would think that evidence means carefully conducted scientific studies, and best means most reproducible and consistent. As a science writer, I’m all in favor of basing things like medical treatment on scientific evidence. So yay EBM.

If there’s any place EBM should work, it’s in the arena of statins, the biggest-selling, most widely prescribed prescription medicines of our time. They are also the most studied, with hundreds of scientific papers assessing their risks and benefits.

So why are we still debating their proper use? It might come down to something that the admirable Ben Goldacre wrote a couple of years ago, “If there is any uncertainty at all about the risks and benefits of statins – and there is – then we have failed to competently implement the most basic principles of evidence based medicine.”  In other words, the cure for this problem is more and better EBM.

But I don’t think it’s quite that. At least not only that. I encourage interested readers to read both sides of the statin debate summarized in the first link, as well as the bits that it links to. You’ll soon find yourself immersed — as I have been for the past few months — in a morass of evidence, a range of opinions about what that evidence means, and what appears to me to be a wall circumscribing EBM, a hard limit on what we can expect from basing medical practice on science. More studies — more EBM — won’t solve the problem.

Because the problem is rooted in places that EBM can’t go. In human greed and fallability, in our proclivity to adopt hard positions and defend them, in our personal ideas of what’s good for our society, and the ways in which these ideas conflict with others. EBM is misused by those who might profit from a particular angle, and when the stakes are high enough, will rarely lead to a consensus accepted by all. The same factors are at play in even bigger questions, like climate change. No amount of “evidence” is going to convince “skeptics.”

I spent several months going over the evidence about statins, and published my own reading of the issue here. (Spoiler alert: They are good for some people, bad for others; ; cause a bunch of side effects that range from annoying to life-threatening; can save many lives in high-risk groups but are relatively useless in populations who are at low-to-moderate risk of heart disease. There are equally effective, less risky alternatives for many patients. Bottom line: Statins are way, way, way overprescribed.)

That’s my opinion. It conflicts with the opinion of others. And as much as I hate to say it, I have little faith that the answer to that conflict will come from EBM.

 

Well, This Is Nice

This is not me. It's Alan Alda, who, oddly enough, also won this award.

This is not me. It’s Alan Alda, who, oddly enough, also won this award for his work in science communication..

Just found out some very good news: the American Chemical Society has announced its 2017 national award winners, and I’m among them. They’re giving me the Grady-Stack Award for interpreting chemistry for the public, the organization’s highest communications prize.

I’m really happy for a couple of reasons. First, because some of the science writers I admire most in this world — like the talented and tireless Deborah Blum, nobelist/poet Roald Hoffman, the brilliant Isaac Asimov, the dean of all science journalists Victor Cohn, and the man who originally inspired me to enter this field, Boyce Rensberger — are among the past recipients. It’s an incredible honor to be ranked in their number.

And second, because this is a seal of approval from those I write about, the scientists whose labors and insights are explored in my books. It’s a mark that my writing is true to its subjects as well as its readers. And that gets to the heart of what we all try to do in this science writing game: To bring to life, as accurately and completely as possible, the excitement and drama of science, and its value to our society.

Like I said, I’m really happy about this.

The Naked Facts

stains-cover-webI’ve been spending a good part of the past eight months working on an experiment in publishing: The Naked Facts series. My son Jackson and I were sitting around one day, despairing of the wide range but generally poor quality of science information available on the internet to non-specialists. Say you’re looking for information on a drug you’ve been prescribed. Either you’re left with package-insert level info prepared by the drug company (and who reads those?), or some anti-drug screed written by alternative health practitioners, or chat rooms, or, if you know how to get them, research articles from journals. With rare exceptions either the info is slanted, or promotional, or too technical for non-specialists to read. So we decided to see if anyone would be interested in short, inexpensive, easy-to-read summaries of the best current knowledge about popular drugs and treatments, something that tried hard to be objective in approach, comprehensive in scope, but a quick, digestible summary of the essentials patients need to know to understand what they’re taking, and to have more productive conversations with their healthcare providers.

We’ve just published #6 in the series, on the world’s bestselling prescription drugs — statins. These are fascinating, sometimes miraculous, sometimes way overprescribed drugs, with a lot of pluses and minuses that most patients — and many physicians — don’t know about.

As an experiment in publishing, the results have been mixed so far. The books are just what I wanted: Simple, solid science, unlike anything else on the market. But sales have been slower than we want (perhaps because as a start-up company, we don’t have much money to put into marketing).

Breaking Water

IMG_1041My book The Alchemy of Air traces the history of one of humanity’s greatest discoveries: How to make bread out of air. The Haber-Bosch process does that by turning air into fertilizer, using nitrogen and hydrogen. But it also creates a lot of air pollution, almost all of it from the process used to make pure hydrogen.

When Bosch designed the first plant to turn air into fertilizer around 1912, he made his hydrogen by electrolytically splitting apart water molecules, releasing pure hydrogen and oxygen. It was a pretty clean process. But it was also costly, because it took a lot of electricity to split the water.

So the industry moved to another source of hydrogen, which it still uses today: natural gas. Especially given today’s low prices for natural gas, it’s a lot cheaper to use than water. No wonder the entire nitrogen fertilizer industry is built around it. Problem is, purifying the hydrogen from natural gas releases a lot of CO2 along the way, making the industry a big polluter.

There might be an answer on the horizon. The trick will be making it easier to get H from H20, making it possible to get back to Bosch’s original vision of water-produced hydrogen. And a novel new approach to doing that has just been published.  It is part of a wider effort to find catalysts that can make it easier for water to break into its constituent atoms. But this one is interesting because it’s temperature sensitive, allowing the catalyst to do part of its work effectively at one level, then do another part when the temperature is changed.

It’s no more than a theory at this time, but it’s an interesting theory. And it could open the door to a much, much cleaner fertilizer industry.

 

Always Think for Yourself

Pauling

Linus Pauling

I love it when a story has a happy ending. And this one does. Sort of.

Farrar, Straus & Giroux put out a new book a few weeks ago, Pandemic, by a good science/medical writer, Sonia Shah. To promote the book, they pulled some startling facts from Shah’s introduction and used them on the back cover and in other marketing material. Here are the “facts” as Shah presented them in her book: “90 percent of epidemiologists said that a pandemic that will sicken 1 billion, kill up to 165 million, and trigger a global recession that could cost up to $3 trillion would occur sometime in the next two generations.”

Startling numbers. Also very precise and factual sounding, and, well, science-y. Shah even footnoted her statement, citing a study done some years ago.

But turns out these are not “facts” at all. Ninety percent of epidemiologists have never said any such thing; there don’t appear to be any published studies at all to back up the statement. Turns out these “facts” are nothing more than guesses, extrapolations of  worst-case scenarios floated by a small group of attendees at a private meeting more than a decade ago. They were mentioned in a TED talk in 2006. And Shah picked them up from there.

I spent a little time trying to get to the source of the statement, only to find smoke. In other words, as Gertrude Stein said, “There is no there there.”

But now to the happy part(s):

  1. I got to indulge in detective work. I love that.
  2. I got to communicate with several bright, interesting people, from Larry Brilliant to Jennifer Senior. I like that.
  3. I got to explode some fake facts used to scare people into buying a book. Great.
  4. And Sonia Shah graciously acknowledged the error and pledged to correct it. This is the best of all.

There is also a not-so-happy part: I found out that fact-checking is not done, or is not done well, by most major book publishers.  This, to me, is even more startling than Shah’s statement. I have been writing books for three decades. I love the industry. I trust that books from major publishers will be accurate.  I assume that they have been checked for accuracy. But my assumptions, it turns out, are wrong.

Most book publishers don’t fact check books. They don’t. They really don’t.

Instead, most standard book contracts (mine included) shuffle the responsibility for accuracy onto authors. Editors focus on voice and flow and grammar. Publishers focus on sales. The only ones who have to take responsibility for factual accuracy, it seems, are writers.

Another illusion shattered. And a lesson learned.

What does this mean for readers? Well, caveat emptor, I suppose. As Linus Pauling said, “When an old and distinguished person speaks to you, listen to him carefully and with respect — but do not believe him. Never put your trust in anything but your own intellect. . . .  Always think for yourself.”

 

 

Chasing Smoke

imageMy simple question – where did the author of a recent science/medical book get her startling facts? – ended up with a deep dive into sources, a very enjoyable phone conversation, and a resolution of sorts.

Readers are directed to earlier posts here and here, which spell out how it all started.

Story begins with this startling line from Sonia Shah’s new book Pandemic: “90 percent of epidemiologists said that a pandemic that will sicken 1 billion, kill up to 165 million, and trigger a global recession that could cost up to $3 trillion would occur sometime in the next two generations.”

Pretty scary. So where did her numbers come from? All signs pointed to a guy named Larry Brilliant, coauthor of the paper Shah mistakenly cited, and a man with one of the best CVs ever, including stints as Ken Kesey’s traveling companion, Jerry Garcia’s personal doctor, world traveler, smallpox eradication pioneer, Google executive director, and head of various global health initiatives. A fascinating career.

I expect that Brilliant is very busy. So I was mildly surprised a few days ago when I got a phone message and an email from the man himself. He wanted to talk about the numbers that I had questioned.

We spoke yesterday. And I really enjoyed our talk, which ranged from the question at hand to digressions on everything from movies to think tanks to rock. Larry Brilliant seems like a great person.

But when it came to the source of Shah’s numbers, well . . .

Turns out that the figures she used were floated first at a private meeting ten years ago, during a time when bird flu (H5N1) was just starting to hit, jumping from poultry to humans and looking like something that might become the next Spanish Influenza, killer of millions. The meeting Brilliant convened in 2005, called Pandefense 1.0, was a private gathering of around 50 thinkers from various fields who wanted to get their hands around the potential scope of the threat. Among them was a investment expert from Canada who was among the first to project the economic impact of a pandemic. Her rough numbers for the U.S. and Canada were back-of-the-envelope projected to global levels to get to the $3 trillion figure. None of this was ever published in a professional journal.

As for the other numbers? They also came from Pandefense 1.0, via a survey of participants. The only published paper from the survey (the one cited by Shah as her source) does not include the numbers she used. Brilliant agrees her citation was wrong. Instead, he said, she should have cited a TED talk he gave in 2006. And — now we’re getting down to it — the numbers he used in that talk came from discussions at Pandefense. They were nothing more than educated guesses thrown around at a private meeting. They were never peer-reviewed. They never appeared in a published paper.

They might turn out to be right, but they are not science. (And they might turn out to be wrong: After years of frightening bird flu headlines and disaster scenarios, I was more than slightly surprised to find out that the actual global death toll from H5N1, to date, is 440.)

Brilliant plans to ask for a new survey that might really assess what epidemiologists think of the risk of a pandemic. I agree that one is needed. And until that’s done, the statement made in Shah’s book should be corrected.

(you can read the followup to this post here)

The Pandemic Problem, Part II

Yeah, this is not the book cover, but it's cool

Yeah, this is not the book cover, but it’s cool

Okay, continuing my little saga about looking for the facts behind the Startling Statement made by Sonia Shah in her new book, Pandemic. Please start by reading my first post about it, below.

And our story continues:

The problem starts when Shah makes this Startling Statement in the introduction to her new book: “90 percent of epidemiologists said that a pandemic that will sicken 1 billion, kill up to 165 million, and trigger a global recession that could cost up to $3 trillion would occur sometime in the next two generations.” Her cited source is a survey by epidemiologist Larry Brilliant. I read Brilliant’s paper and can’t find Shah’s numbers.

What does a curious science writer like me do? Write a bunch of emails, of course. I write all four authors of the original paper (Brilliant is third author among four) asking how that work can serve as the source of Shah’s numbers. I write Sonia Shah. I write the book reviewer at the New York Times who repeated the Startling Statement in her positive review of the book. When I don’t get a fast response from Shah (understandable, she’s on book tour)  I write her agent and  her editor at Farrar, Straus, and Giroux (tip for sleuths: you get these names from the Acknowledgments in Shah’s book).

Why do I spend all this time? Because I hate it when writers scare people with unsupported facts. There are probably a million facts in Shah’s book, and 99.999% of them are undoubtedly correct. But the Startling Statement is the big one, the nugget repeated in conversations, the elevator pitch that sells books. That one needs to be right.

Within a day or so I get a response from the original paper’s lead author, Wandi Bruine de Bruin, who confirms that the paper does not include the numbers in the Startling Statement, nor does she know of any other place where Larry Brilliant has published such numbers.

Which leads back to Brilliant himself. And here’s where we finally get to something like an answer. Brilliant gave a TED talk in 2009, halfway through which he mentioned the numbers in the Startling Statement. He said 1 billion sick, he said 165 million dead, he said it all. Shah appears to have picked it up from there, and the rest is history.

In other words, the Startling Statement is not from any scientific study (at least not one that anyone can point to yet; still haven’t heard from Brilliant), but sources to an unsupported couple of lines that a guy said in the middle of a TED talk. And that would be okay if Shah simply said where she got her numbers. But then the Startling Statement becomes something much less startling. More like, “Six years ago Larry Brilliant said in a speech that most epidemiologists probably believe that a pandemic will likely happen some time in the next five or six decades. . . ” Not exactly the stuff that book contracts are made of.

I hope this is not the end of the story. I hope Larry Brilliant can offer some real science to back up his statement. I hope Shah is a little more careful about her citations. I hope her publisher is a little more careful about fact-checking. But most of all, I hope that readers will demand real proof when they read stories about science.

(story continues here)

 

The Pandemic Problem

Naked Facts bannerOver at Naked Facts we’ve been posting a series of seven questions to ask when you read the science news. #1 on the list: Where’s the proof?

“This seems almost too simple. But many people reading the news don’t pay attention to the one thing that really matters: Where’s the evidence that this ‘discovery’ is real? Never accept anything just because a reporter says so. Demand proof.”

Okay, I have to walk it like I talk it. And chance, as Pasteur said, favors the prepared mind. Two days after I posted the above, I read a review of a new book called Pandemic by  Sonia Shaw, who has done a lot of good work writing about contagious diseases. In it, the New York Times reviewer, Jennifer Senior, did a good thing: She repeated a claim Shaw made in her introduction, then live-linked to the scientific study Shaw cited to support it. Here’s what Senior wrote:

“…she cites a study in which 90 percent of epidemiologists say they believe a global pandemic will sicken one billion and kill up to 165 million within the next two generations. Looking closely at the source of this statistic — a paper by the epidemiologist Larry Brilliant and three colleagues — I’m not convinced that things are quite so dire. (The number of epidemiologists he surveyed is actually quite small.) But…”

Then Senior goes on to the rest of the review. What hit me was Shaw’s Startling Claim: 1 billion sick! 165 million dead! Within two generations! So say 90% of epidemiologists! I track the medical news a little and had never before heard anything like these numbers. They were so specific, and they took such pride of place in the book, and they were from such a good writer, that they seemed instantly credible.

But they aren’t. At least not as far as I can tell.

The book reviewer’s caveat about the small sample size made me wonder about where those numbers came from, and the live link made it easy to check. Kudos to Jennifer Senior and the great grey Times.

So I read the paper. And guess what? No matter how many times I read it, I couldn’t find the numbers in Shaw’s Startling Claim. The study itself was sort of, well, minor. It was a bit long in the tooth for starters, based on data gathered in 2005 and published ten years ago (I think that’s like 60 in epidemiology years). As Senior noted, it was pretty small-scale — the entire study was based on an email survey of 19 “medical experts” and about the same number of non-experts. No telling how many of the “experts” were actually epidemiologists. Another red flag was the purpose of the study, which was designed to compare expert to non-expert opinions about bird flu, not to predict the chances of a pandemic. But the most important thing was this: The numbers Shaw used were not there. Not that I could see.

One more fishy thing: Shaw and Senior both referred to Larry Brilliant as the source of the Startling Statement. But he was only third of four authors on the paper. Usually the first author gets the credit. So why was Brilliant given top billing by Shaw? Turns out it’s not only because he has such a great name. I mean, really, Brilliant? No, turns out it’s because he’s a BIG DEAL in the world of science. Check him out: head of this, leader of that, TED speaker, Google ties, big government experience, big foundation backing, etc. etc. When Larry speaks, people listen.

So I did the next logical thing: I wrote Larry, Larry, what’s the source of those numbers? Haven’t heard back yet. But I also wrote the other three authors on the study, asking them if they could make the published data match Shaw’s Startling Statement.

And here’s where things get weird. . . .

(You can read Part II of the story here)

This One Graphic Says It All

One thing Twitter is good at is — yes, I’m tweeting more now that we’re trying to draw attention to the Naked Facts books — is the juxtaposition of otherwise random bits of information, allowing factoids to brush up against each other as if they were guests at a noisy, crowded cocktail party. Today’s example: Notice from the NYT and a link to the very first Science Times articles from 1978. That was the year I decided to become a science writer, enthused about raising the bar, bringing people better quality news. Yay! And right below it, from another source, was a tweet linking to a story about why science writers have so few friends (tongue in cheek, but still . . . ). Within that story was this great graphic:Medical_studies-05.0nicely illustrating the confusing overall message from today’s standard news coverage of health issues.  Nothing has changed, it seems. Final irony, there’s nothing better than Twitter itself at illustrating why people aren’t reading newspapers (thus killing many science writing positions) while adding to the noise, confusion, and attention-grabbing-at-the-expense-of-context attitude of most of today’s science coverage.

Naked Facts #2

abilify-cover-kdpThis is the second in a series of short books Monroe Press is doing on specific topics in science and medicine.  Our idea is to act as translators for the general public, deciphering and balancing the flood of science research and activist response so that non-scientist readers can quickly get the essentials.

The larger theme for the first titles is mind medicine, specifically antipsychotics — some of the most misunderstood, misprescribed, and profitable drugs ever made.