Ben Goldacre: Battling Bad Science

Summary

In this engaging talk, an epidemiologist demystifies epidemiology, explaining its role in discerning if something is beneficial or harmful in the real world. Using humorous examples from sensationalist headlines, the speaker illustrates how media often misrepresents scientific studies. He emphasizes the importance of proper evidence and trials, highlighting how even sophisticated industries manipulate data. The talk culminates in a critique of withheld medical data, stressing that transparency and full disclosure are crucial for informed decision-making in medicine.

Takeaways

• 🧬 Epidemiology is the science of understanding the impact of various factors on health in the real world, often through the lens of contradictory media headlines.
• 📰 Media often sensationalizes health-related findings, leading to confusion about the causes and preventions of diseases like cancer.
• 🤔 The importance of critically appraising evidence is emphasized, as real science involves evaluating the claims made by others, not just accepting authority.
• 🔍 The speaker highlights the flaws in relying on authority figures without proper accreditation and the potential for misinformation in health advice.
• 🍷 An example is given about the misleading claims regarding red wine's effect on health, showing the need for proper scientific studies involving human subjects.
• 📊 The limitations of observational studies are discussed, pointing out that correlation does not imply causation, as seen in the case of olive oil and wrinkle reduction.
• 🧪 The necessity of randomized controlled trials for accurate scientific conclusions is explained, with examples of flawed trial designs.
• 🤹‍♂️ The placebo effect is introduced as a powerful factor in medical treatments, influenced by beliefs and expectations, not just physical intervention.
• 💊 The pharmaceutical industry is critiqued for using deceptive practices in clinical trials to make new drugs appear more effective than they are.
• 📈 The concept of publication bias is introduced, showing how the selective reporting of positive trials can skew perceptions of a treatment's effectiveness.
• 🌐 The speaker concludes by emphasizing the importance of transparency and access to all trial data for making informed decisions in medicine.

Q & A

What is epidemiology and why is it important?

Epidemiology is the science of understanding whether something is good or bad for you in the real world. It's important because it helps us critically appraise evidence to make informed decisions about health and well-being.

Why do newspaper headlines often confuse the public about health advice?

Newspaper headlines often confuse the public because they tend to oversimplify or sensationalize scientific findings, leading to contradictory and misleading information about health issues like cancer prevention.

What is the problem with relying on authority in scientific research?

Relying on authority in scientific research is problematic because it disregards the need for evidence and reasoning. In science, it's the reasons behind a claim that matter, not the credentials of the person making the claim.

Why is the authority of Dr. Gillian McKeith questioned in the script?

Dr. Gillian McKeith's authority is questioned because her PhD is from a non-accredited correspondence course and her membership in a professional association is easily attainable without rigorous qualifications, casting doubt on her credibility.

What is the issue with using animal studies or laboratory experiments to draw conclusions about human health?

Animal studies or laboratory experiments, while useful for initial research, do not always translate to humans due to biological differences. They cannot definitively inform us about personal health risks or benefits.

What is the flaw in the design of the fish oil pills trial mentioned in the script?

The flaw in the fish oil pills trial design is the lack of a control group. Without a control group, it's impossible to isolate the effect of the fish oil pills on school performance and behavior.

Why are observational studies insufficient for determining the effectiveness of a treatment?

Observational studies are insufficient because they can show correlation but not causation. Factors other than the treatment may account for the observed effects, leading to misleading conclusions.

What is the significance of the placebo effect in medical trials?

The placebo effect is significant in medical trials because it demonstrates that patients' beliefs and expectations can influence the outcome of a treatment, which is why controlled trials with a placebo group are necessary.

How can the pharmaceutical industry manipulate trial results?

The pharmaceutical industry can manipulate trial results by conducting trials against placebos instead of the best available treatment, using suboptimal doses of competing drugs, or by selectively publishing positive results while withholding negative data.

What is publication bias and how does it affect the medical community's understanding of a treatment's effectiveness?

Publication bias occurs when only positive or significant results are published, while negative or insignificant results are withheld. This can skew the medical community's understanding of a treatment's effectiveness by creating an incomplete picture of the evidence.

Why is access to all trial data crucial for making informed medical decisions?

Access to all trial data is crucial because it provides a complete picture of a treatment's effectiveness and potential side effects. Without all the data, doctors and patients cannot make fully informed decisions about treatment options.

What is the ethical problem with the current state of medical research as described in the script?

The ethical problem is that important medical decisions are being made based on incomplete data. Withholding negative or inconclusive trial results prevents a full understanding of a treatment's risks and benefits, which is a significant breach of ethical responsibility in medical research.

Outlines

00:00

🔬 Introduction to Epidemiology and Misleading Headlines

The speaker, an epidemiologist, introduces epidemiology as the science of determining what is good or bad for health in the real world. He uses the example of misleading newspaper headlines, particularly those from the Daily Mail, which often categorize everyday items as causing or preventing cancer. He highlights the contradictions and absurd claims, such as coffee both causing and preventing cancer, and points out the potential political biases in such claims. The speaker emphasizes the importance of critically appraising evidence and explains that real science involves rigorous scrutiny of claims.

05:01

🧪 The Importance of Proper Trials and the Placebo Effect

The speaker discusses the necessity of proper scientific trials, highlighting the flaws in a high-profile trial of fish oil pills for improving school performance and behavior. He explains that a well-conducted trial should include a control group and account for the placebo effect. The placebo effect, influenced by cultural beliefs and expectations, can significantly impact trial outcomes. The speaker uses examples like the efficacy of sugar pills and saltwater injections to illustrate the placebo effect's power. He argues that many studies and headlines distort evidence for various purposes, including those by the pharmaceutical industry.

10:03

🔍 Publication Bias and Missing Data

The speaker addresses the critical issue of missing data in scientific research. He introduces the concept of a funnel plot to detect publication bias, where small negative trials are often withheld. The speaker shares personal experiences with the antidepressant reboxetine, revealing that 76% of its trials were withheld, skewing the perceived effectiveness of the drug. He emphasizes that missing data prevents a true understanding of a treatment's efficacy. The speaker also discusses the broader implications, such as the withholding of data on Tamiflu, a drug for flu complications, highlighting the ethical dilemma of making medical decisions without complete information. He concludes by advocating for transparency and the importance of scrutinizing the mechanics of scientific research to address these issues.

Keywords

💡Epidemiology

Epidemiology is defined as the branch of medical science that deals with the incidence, distribution, and control of diseases in a population. In the context of the video, it is portrayed as the science that helps determine if something is beneficial or harmful to health, often through the analysis of patterns, causes, and effects in populations. The speaker uses epidemiology to critique misleading health headlines and to emphasize the importance of evidence-based conclusions.

💡Evidence-based medicine

Evidence-based medicine refers to the practice of making medical decisions based on the best available scientific evidence from well-conducted research. The video discusses this concept to highlight the necessity of critically appraising evidence for claims made about health and disease, using examples of how people often get it wrong and the importance of proper scientific methodology.

💡Authority

In the video, 'authority' is discussed as the weakest form of evidence. It refers to the reliance on the status or credentials of an individual rather than on empirical data or logical reasoning. The speaker points out that in science, authority is not a substitute for sound reasoning or evidence, as illustrated by the example of Dr. Gillian McKeith, whose qualifications and health advice are questioned.

💡Placebo effect

The placebo effect is a phenomenon in which a patient experiences a perceived improvement in their condition due to their belief in the treatment's effectiveness, rather than the treatment itself. The video explains the placebo effect in the context of clinical trials, emphasizing its power and the need to control for it by using placebos in study design to isolate the actual effects of a treatment.

💡Publication bias

Publication bias is the tendency to publish studies with positive results while withholding or ignoring those with negative or null results. The video uses the concept of publication bias to criticize the selective reporting of trial outcomes, which can skew the perception of a treatment's effectiveness and mislead medical professionals and patients.

💡Randomized controlled trial (RCT)

A randomized controlled trial is a type of scientific experiment that aims to reduce bias when testing a new treatment by randomly assigning participants to either a treatment group or a control group. The video explains the importance of RCTs in providing reliable evidence and criticizes studies that lack proper controls or are designed with the expectation of a certain outcome.

💡Pharmaceutical industry

The pharmaceutical industry is the sector of the economy concerned with the research, development, marketing, and sale of drugs. In the video, the speaker discusses how the industry can sometimes manipulate evidence to favor their products, such as by conducting trials against placebos or by comparing new drugs to suboptimal doses of existing treatments.

💡Systematic review

A systematic review is a comprehensive analysis that uses explicit methods to identify, select, and critically appraise all research relevant to a specific topic, and to collect and analyze data from the studies that are included in the review. The video mentions the Cochrane Groups, which perform systematic reviews, to emphasize the importance of having access to all trial data for accurate health assessments.

💡Confidence in medical treatment

Confidence in medical treatment refers to the trust that patients and healthcare providers place in the effectiveness and safety of medical interventions. The video argues that confidence is undermined when data is withheld or manipulated, as it prevents informed decision-making based on complete information.

💡Transparency

Transparency in the context of the video refers to the openness and clarity with which medical research data is shared and made available. The speaker argues for greater transparency in the reporting of medical trials to ensure that all evidence, not just the favorable parts, is considered in medical practice.

💡Health claims

Health claims are statements made about the health benefits of a product or treatment. The video discusses the scrutiny required for such claims, using examples from media headlines that suggest contradictory effects of the same substances, like coffee causing and preventing cancer, to illustrate the need for evidence-based evaluation of these claims.

Highlights

Epidemiology is the science of how we know in the real world if something is good or bad for you.

Epidemiology is best understood through examples, often seen in newspaper headlines.

Newspapers often publish contradictory health claims, such as coffee both causing and preventing cancer.

Critically appraising evidence is a core part of real science and academic research.

The weakest form of evidence in science is authority; what matters are the reasons behind beliefs.

Some health experts have dubious credentials, such as non-accredited correspondence course PhDs.

Proper scientific evidence, such as studies on enzymes or observational studies, is crucial for valid conclusions.

Trials with no control groups, like the fish oil pill study, can produce misleading results.

The placebo effect demonstrates the power of beliefs and expectations in medical treatments.

Comparing new drugs to inadequate treatments, such as too high or too low doses of existing drugs, can skew results.

Industry-funded trials are more likely to give positive results than independently sponsored trials.

Missing data, such as negative trial results being withheld, can distort the true effectiveness of treatments.

Publication bias, where small negative trials disappear, is a significant issue in medical research.

Withholding trial data, as seen with antidepressants and the drug Tamiflu, prevents informed medical decisions.

Sunlight, or transparency, is the best disinfectant for addressing problems in scientific research.

Transcripts

So I'm a doctor, but I kind of slipped sideways into research,

and now I'm an epidemiologist.

And nobody really knows what epidemiology is.

Epidemiology is the science of how we know in the real world

if something is good for you or bad for you.

And it's best understood through example

as the science of those crazy, wacky newspaper headlines.

And these are just some of the examples.

These are from the Daily Mail.

Every country in the world has a newspaper like this.

It has this bizarre, ongoing philosophical project

of dividing all the inanimate objects in the world

into the ones that either cause or prevent cancer.

Here are some of the things they said cause cancer:

divorce, Wi-Fi, toiletries and coffee.

Some things they say prevent cancer:

crusts, red pepper, licorice and coffee.

So you can see there are contradictions.

Coffee both causes and prevents cancer.

As you start to read on, you can see

that maybe there's some political valence behind some of this.

For women, housework prevents breast cancer,

but for men, shopping could make you impotent.

(Laughter)

So we know that we need to start unpicking the science behind this.

And what I hope to show is that unpicking the evidence behind dodgy claims

isn't a kind of nasty, carping activity;

it's socially useful.

But it's also an extremely valuable explanatory tool,

because real science is about critically appraising the evidence

for somebody else's position.

That's what happens in academic journals,

it's what happens at academic conferences --

the Q&A session after a postdoc presents data is often a bloodbath.

And nobody minds that; we actively welcome it.

It's like a consenting intellectual S&M activity.

(Laughter)

So what I'm going to show you is all of the main things,

all of the main features of my discipline, evidence-based medicine.

And I will talk you through all of these and demonstrate how they work,

exclusively using examples of people getting stuff wrong.

We'll start with the absolute weakest form of evidence known to man,

and that is authority.

In science, we don't care how many letters you have after your name --

we want to know what your reasons are for believing something.

How do you know that something is good for us or bad for us?

But we're also unimpressed by authority because it's so easy to contrive.

This is somebody called Dr. Gillian McKeith, PhD,

or, to give her full medical title, Gillian McKeith.

(Laughter)

Again, every country has somebody like this.

She is our TV diet guru.

She has five series of prime-time television,

giving out very lavish and exotic health advice.

She, it turns out, has a non-accredited correspondence course PhD

from somewhere in America.

She also boasts that she's a certified professional member

of the American Association of Nutritional Consultants,

which sounds very glamorous; you get a certificate.

This one belongs to my dead cat, Hettie. She was a horrible cat.

You go to the website, fill out the form,

give them $60, it arrives in the post.

That's not the only reason we think this person is an idiot.

She also says things like eat lots of dark green leaves,

they contain chlorophyll and really oxygenate your blood.

And anybody who's done school biology remembers

that chlorophyll and chloroplasts only make oxygen in sunlight,

and it's quite dark in your bowels after you've eaten spinach.

Next, we need proper science, proper evidence.

So: "Red wine can help prevent breast cancer."

This is a headline from The Daily Telegraph in the UK.

"A glass of red wine a day could help prevent breast cancer."

So you find this paper, and find that it is a real piece of science.

It's a description of the changes in the behavior of one enzyme

when you drip a chemical extracted from some red grape skin

onto some cancer cells

in a dish on a bench in a laboratory somewhere.

And that's a really useful thing to describe in a scientific paper.

But on the question of your own personal risk of getting breast cancer

if you drink red wine,

it tells you absolutely bugger all.

Actually, it turns out that your risk of breast cancer

increases slightly with every amount of alcohol you drink.

So what we want are studies in real human people.

And here's another example.

This is from Britain's "leading" diet nutritionist in the Daily Mirror,

our second-biggest selling newspaper.

"An Australian study in 2001 found that olive oil,

in combination with fruits, vegetables and pulses,

offers measurable protection against skin wrinklings,"

and give the advice:

"If you eat olive oil and vegetables, you'll have fewer wrinkles."

They helpfully tell you how to find the paper,

and what you find is an observational study.

Obviously, nobody has been able to go back to 1930,

get all the people born in one maternity unit,

and half of them eat lots of fruit and veg and olive oil,

half of them eat McDonald's,

and then we see how many wrinkles you've got later.

You have to take a snapshot of how people are now.

And what you find is, of course:

people who eat veg and olive oil have fewer wrinkles.

But that's because people who eat fruit and veg and olive oil are freaks --

they're not normal, they're like you; they come to events like this.

(Laughter)

They're posh, they're wealthy, less likely to have outdoor jobs,

less likely to do manual labor,

they have better social support, are less likely to smoke;

for a host of fascinating, interlocking

social, political and cultural reasons,

they're less likely to have wrinkles.

That doesn't mean it's the vegetables or olive oil.

(Laughter)

So ideally, what you want to do is a trial.

People think they're familiar with the idea of a trial.

Trials are old; the first one was in the Bible, Daniel 1:12.

It's straightforward: take a bunch of people, split them in half,

treat one group one way, the other group, the other way.

A while later, you see what happened to each of them.

I'm going to tell you about one trial,

which is probably the most well-reported trial

in the UK news media over the past decade.

This is the trial of fish oil pills.

The claim: fish oil pills improve school performance and behavior

in mainstream children.

They said, "We did a trial.

All the previous ones were positive, this one will be too."

That should ring alarm bells:

if you know the answer to your trial, you shouldn't be doing one.

Either you've rigged it by design,

or you've got enough data so there's no need to randomize people anymore.

So this is what they were going to do in their trial:

They were taking 3,000 children,

they were going to give them these huge fish oil pills, six of them a day,

and then, a year later, measure their school exam performance

and compare their performance

against what they predicted their exam performance would have been

if they hadn't had the pills.

Now, can anybody spot a flaw in this design?

(Laughter)

And no professors of clinical trial methodology

are allowed to answer this question.

So there's no control group.

But that sounds really techie, right? That's a technical term.

The kids got the pills, and their performance improved.

What else could it possibly be if it wasn't the pills?

They got older; we all develop over time.

And of course, there's the placebo effect,

one of the most fascinating things in the whole of medicine.

It's not just taking a pill and performance or pain improving;

it's about our beliefs and expectations, the cultural meaning of a treatment.

And this has been demonstrated in a whole raft of fascinating studies

comparing one kind of placebo against another.

So we know, for example,

that two sugar pills a day are a more effective treatment

for gastric ulcers

than one sugar pill.

Two sugar pills a day beats one a day.

That's an outrageous and ridiculous finding, but it's true.

We know from three different studies on three different types of pain

that a saltwater injection is a more effective treatment

than a sugar pill, a dummy pill with no medicine in it,

not because the injection or pills do anything physically to the body,

but because an injection feels like a much more dramatic intervention.

So we know that our beliefs and expectations can be manipulated,

which is why we do trials where we control against a placebo,

where one half of the people get the real treatment,

and the other half get placebo.

But that's not enough.

What I've just shown you are examples

of the very simple and straightforward ways

that journalists and food supplement pill peddlers and naturopaths

can distort evidence for their own purposes.

What I find really fascinating

is that the pharmaceutical industry uses exactly the same kinds

of tricks and devices,

but slightly more sophisticated versions of them,

in order to distort the evidence they give to doctors and patients,

and which we use to make vitally important decisions.

So firstly, trials against placebo:

everybody thinks a trial should be a comparison

of your new drug against placebo.

But in a lot of situations that's wrong;

often, we already have a good treatment currently available.

So we don't want to know that your alternative new treatment

is better than nothing,

but that it's better than the best available treatment we have.

And yet, repeatedly, you consistently see people doing trials

still against placebo.

And you can get licensed to bring your drug to market

with only data showing that it's better than nothing,

which is useless for a doctor like me trying to make a decision.

But that's not the only way you can rig your data.

You can also rig your data

by making the thing you compare your new drug against

really rubbish.

You can give the competing drug in too low a dose,

so people aren't properly treated.

You can give the competing drug in too high a dose,

so people get side effects.

And this is exactly what happened

with antipsychotic medication for schizophrenia.

Twenty years ago, a new generation of antipsychotic drugs were brought in;

the promise was they would have fewer side effects.

So people set about doing trials of the new drugs against the old drugs.

But they gave the old drugs in ridiculously high doses:

20 milligrams a day of haloperidol.

And it's a foregone conclusion if you give a drug at that high a dose,

it will have more side effects, and your new drug will look better.

Ten years ago, history repeated itself,

when risperidone, the first of the new-generation antipsychotic drugs,

came off copyright, so anybody could make copies.

Everybody wanted to show their drug was better than risperidone,

so you see trials comparing new antipsychotic drugs

against risperidone at eight milligrams a day.

Again, not an insane dose, not an illegal dose,

but very much at the high end of normal.

So you're bound to make your new drug look better.

And so it's no surprise that overall,

industry-funded trials are four times more likely

to give a positive result

than independently sponsored trials.

But -- and it's a big but --

(Laughter)

it turns out,

when you look at the methods used by industry-funded trials,

that they're actually better than independently sponsored trials.

And yet, they always manage to get the result that they want.

So how does this work?

(Laughter)

How can we explain this strange phenomenon?

Well, it turns out that what happens

is the negative data goes missing in action;

it's withheld from doctors and patients.

And this is the most important aspect of the whole story.

It's at the top of the pyramid of evidence.

We need to have all of the data on a particular treatment

to know whether or not it really is effective.

There are two different ways you can spot whether some data has gone missing.

You can use statistics or you can use stories.

I prefer statistics, so that's what I'll do first.

This is a funnel plot.

A funnel plot is a very clever way of spotting

if small negative trials have disappeared, have gone missing in action.

This is a graph of all of the trials done on a particular treatment.

As you go up towards the top of the graph,

what you see is each dot is a trial.

As you go up, those are bigger trials, so they've got less error;

they're less likely to be randomly false positives or negatives.

So they all cluster together.

The big trials are closer to the true answer.

Then as you go further down at the bottom,

what you can see is, on this side, spurious false negatives,

and over on this side, spurious false positives.

If there is publication bias,

if small negative trials have gone missing in action,

you can see it on one of these graphs.

So you see here that the small negative trials

that should be on the bottom left have disappeared.

This is a graph demonstrating the presence of publication bias

in studies of publication bias.

And I think that's the funniest epidemiology joke you will ever hear.

(Laughter)

That's how you can prove it statistically.

But what about stories?

Well, they're heinous, they really are.

This is a drug called reboxetine.

This is a drug which I, myself, have prescribed to patients.

And I'm a very nerdy doctor.

I hope I go out of my way

to try and read and understand all the literature.

I read the trials on this.

They were all positive, all well-conducted.

I found no flaw.

Unfortunately, it turned out, that many of these trials were withheld.

In fact, 76 percent of all of the trials that were done on this drug

were withheld from doctors and patients.

Now if you think about it,

if I tossed a coin a hundred times,

and I'm allowed to withhold from you the answers half the times,

then I can convince you that I have a coin with two heads.

If we remove half of the data,

we can never know what the true effect size of these medicines is.

And this is not an isolated story.

Around half of all of the trial data on antidepressants has been withheld,

but it goes way beyond that.

The Nordic Cochrane Group were trying to get ahold of the data on that

to bring it all together.

The Cochrane Groups are an international nonprofit collaboration

that produce systematic reviews

of all of the data that has ever been shown.

And they need to have access to all of the trial data.

But the companies withheld that data from them.

So did the European Medicines Agency --

for three years.

This is a problem that is currently lacking a solution.

And to show how big it goes, this is a drug called Tamiflu,

which governments around the world

have spent billions and billions of dollars on.

And they spend that money on the promise that this is a drug

which will reduce the rate of complications with flu.

We already have the data

showing it reduces the duration of your flu by a few hours.

But I don't care about that, governments don't care.

I'm sorry if you have the flu, I know it's horrible,

but we're not going to spend billions of dollars

trying to reduce the duration of your flu symptoms by half a day.

We prescribe these drugs.

We stockpile them for emergencies

on the understanding they'll reduce the number of complications,

which means pneumonia and death.

The infectious diseases Cochrane Group, which are based in Italy,

has been trying to get the full data in a usable form

out of the drug companies,

so they can make a full decision

about whether this drug is effective or not,

and they've not been able to get that information.

This is undoubtedly the single biggest ethical problem

facing medicine today.

We cannot make decisions in the absence of all of the information.

So it's a little bit difficult from there

to spin in some kind of positive conclusion.

But I would say this:

I think that sunlight

is the best disinfectant.

All of these things are happening in plain sight,

and they're all protected by a force field of tediousness.

And I think, with all of the problems in science,

one of the best things that we can do

is to lift up the lid, finger around at the mechanics

and peer in.

Thank you very much.

(Applause)