Drug research and 'significance chasing,' in today's medical field

[Guest guest]

An interesting excerpt from " The Truth Wears Off " -- an examination of

research studies on today's drugs. Some of these drugs influence

appetite and/or weight change and may be of interest to RDs. - ne

" On September 18, 2007, a few dozen neuroscientists, psychiatrists, and

drug-company executives gathered in a hotel conference room in Brussels

to hear some startling news. It had to do with a class of drugs known as

atypical or second-generation antipsychotics, which came on the market

in the early nineties. The drugs, sold under brand names such as

Abilify, Seroquel, and Zyprexa, had been tested on schizophrenics in

several large clinical trials, all of which had demonstrated a dramatic

decrease in the subjects' psychiatric symptoms. As a result,

second-generation antipsychotics had become one of the fastest-growing

and most profitable pharmaceutical classes. By 2001, Eli Lilly's Zyprexa

was generating more revenue than Prozac. It remains the company's

top-selling drug.

" But the data presented at the Brussels meeting made it clear that

something strange was happening: the therapeutic power of the drugs

appeared to be steadily waning. A recent study showed an effect that was

less than half of that documented in the first trials, in the early

nineteen-nineties. Many researchers began to argue that the expensive

pharmaceuticals weren't any better than first-generation antipsychotics,

which have been in use since the fifties. 'In fact, sometimes they now

look even worse,' , a professor of psychiatry at the

University of Illinois at Chicago, told me.

" Before the effectiveness of a drug can be confirmed, it must be tested

and tested again. Different scientists in different labs need to repeat

the protocols and publish their results. The test of replicability, as

it's known, is the foundation of modern research. Replicability is how

the community enforces itself. It's a safeguard for the creep of

subjectivity. Most of the time, scientists know what results they want,

and that can influence the results they get. The premise of

replicability is that the scientific community can correct for these flaws.

" But now all sorts of well-established, multiply confirmed findings have

started to look increasingly uncertain. It's as if our facts were losing

their truth: claims that have been enshrined in textbooks are suddenly

unprovable. This phenomenon doesn't yet have an official name, but it's

occurring across a wide range of fields, from psychology to ecology. In

the field of medicine, the phenomenon seems extremely widespread,

affecting not only antipsychotics but also therapies ranging from

cardiac stents to Vitamin E and antidepressants: has a forthcoming

analysis demonstrating that the efficacy of antidepressants has gone

down as much as threefold in recent decades.

For many scientists, the effect is especially troubling because of what

it exposes about the scientific process. If replication is what

separates the rigor of science from the squishiness of pseudoscience,

where do we put all these rigorously validated findings that can no

longer be proved? Which results should we believe? Francis Bacon, the

early-modern philosopher and pioneer of the scientific method, once

declared that experiments were essential, because they allowed us to

'put nature to the question.' But it appears that nature often gives us

different answers. ...

" [ph Banks Rhine, a psychologist at Duke, came to call this trend

toward a reduction in the strength of proof for a theory he had

developed in the early nineteen-thirties] the 'decline effect.'

" According to Ioannidis, an epidemiologist at Stanford University,

the main problem is that too many researchers engage in what he calls

'significance chasing,' or finding ways to interpret the data so that it

passes the statistical test of significance - the ninety-five-per-cent

boundary invented by Fisher. 'The scientists are so eager to pass

this magical test that they start playing around with the numbers,

trying to find anything that seems worthy,' Ioannidis says. In recent

years, Ioannidis has become increasingly blunt about the pervasiveness

of the problem. One of his most cited papers has a deliberately

provocative title: 'Why Most Published Research Findings Are False.'

" The problem of selective reporting is rooted in a fundamental cognitive

flaw, which is that we like proving ourselves right and hate being

wrong. 'It feels good to validate a hypothesis,' Ioannidis said. 'It

feels even better when you've got a financial interest in the idea or

your career depends upon it. And that's why, even after a claim has been

systematically disproven' - he cites, for instance, the early work on

hormone replacement therapy, or claims involving various vitamins - 'you

still see some stubborn researchers citing the first few studies that

show a strong effect. They really want to believe that it's true.' ...

" The disturbing implication of a study [conducted in the late

nineteen-nineties by Crabbe, a neuroscientist at the Oregon Health

and Science University] is that a lot of extraordinary scientific data

are nothing but noise. The problem, of course, is that ... dramatic

findings are ... the most likely to get published in prestigious

journals, since the data are both statistically significant and entirely

unexpected. Grants get written, follow-up studies are conducted. The end

result is a scientific accident that can take years to unravel.

" This suggests that the decline effect is actually a decline of

illusion. While Karl Popper imagined falsification occurring with a

single, definitive experiment - Galileo refuted Aristotelian mechanics

in an afternoon - the process turns out to be much messier than that.

Many scientific theories continue to be considered true even after

failing numerous experimental tests. Verbal overshadowing might exhibit

the decline effect, but it remains extensively relied upon within the

field. The same holds for any number of phenomena, from the disappearing

benefits of second-generation antipsychotics to the weak coupling ratio

exhibited by decaying neutrons, which appears to have fallen by more

than ten standard deviations between 1969 and 2001....

" Such anomalies demonstrate the slipperiness of empiricism. Although

many scientific ideas generate conflicting results and suffer from

falling effect sizes, they continue to get cited in the textbooks and

drive standard medical practice. Why? Because these ideas seem true.

Because they make sense. Because we can't bear to let them go. And this

is why the decline effect is so troubling. Not because it reveals the

human fallibility of science, in which data are tweaked and beliefs

shape perceptions. (Such shortcomings aren't surprising, at least for

scientists.) And not because it reveals that many of our most exciting

theories are fleeting fads and will soon be rejected. (That idea has

been around since Kuhn.) The decline effect is troubling because

it reminds us how difficult it is to prove anything. We like to pretend

that our experiments define the truth for us. But that's often not the

case. Just because an idea is true doesn't mean it can be proved. And

just because an idea can be proved doesn't mean it's true. When the

experiments are done, we still have to choose what to believe. "

Author: Jonah Lehrer

Title: " The Truth Wears Off "

Publisher: The New Yorker

Date: December 13, 2010

Pages: 52-57