Re: What if...... Bonferoni

[Guest guest]

a, it's a very important, though, whether they considered the

Bonferroni problem. Those p values, as you probably know, quantify the

odds that a result could happen purely by chance if one's hypothesis

were false. Ie, say you have a coin, and you hypothesize that it's

biased towards tails; you toss it 5 times and get tails every time.

This tends to support your hypothesis, but there is also a 0.03125

chance of getting this result even if your hypothesis is false and the

coin is in fact perfectly fair. So, p = 0.03125 in this case. In

medicine we usually don't bother with a finding unless p < 0.05 -

that's just the traditional, somewhat arbitrary threshold for a decent

level of evidence, termed statistical significance.

The problem is that for every 20 comparisons you make, one (on

average) will come out statistically significant even if the

hypothesis is false. If I take one random group of 10 people and

compare the serum levels of 20 different cytokines to the levels found

in a second group of random people, one of the cytokines will look

like it is seriously different from one group to the other.

This is why you sometimes read a big trial with designated " primary

outcome measures " and some secondary outcome measures. There are

usually just a handful of primary measures - a lot less than 20. This

is to minimize the Bonferroni problem. It's understood that the

outcome of the trial is staked on those primary measures only - those

are the ones the researcher thinks are going to make the proof. The

secondary measures are just things that are interesting enough to be

examined. If they do come out to be statistically significant, they

can always be examined later, alone, under hypotheses of their own, in

a *separate* group of subjects (this makes them primary measures in

their own right in a new study). Of course, primary outcome measures

have to be designated before the study is executed. If you just pick

out something like HLA groups *after* the fact (I'm not saying that's

what your guys did, not having read it, but it happens), then you're

cherry picking. There are 400 other correlations they could have

looked for besides HLA, and if you examine for 400 correlations at p <

0.05, you will find 20 illusory correlations.

Reason I go into all this is that it's of more general significance.

Just because related results might take years to be completed, and be

published in separate papers, doesn't mean the Bonferroni problem

doesn't apply. Say a lab is interested in proving objectively that CFS

has a physical cause, so they look for physical differences from

controls. Labs tend not to publish negative results. If they have 20

ideas over the years and only one yields a statistically significant

result and gets published, then this doesn't do much to support the

hypothesis of a physical basis of CFS, because 20 comparisons applying

to that hypothesis were made at a p threshold (the traditional medical

research one) of 0.05. But, if after that research happens, the one

statistically significant finding is replicated in a separate group of

patients, THEN the physical basis hypothesis of CFS is strongly

supported.

That's one reason there's so much emphasis on separate replication in

science. And this whole thing is one big reason why there are soooo

many medical findings reported that end up not being replicated

successfully by anyone (much less by everyone).

In physics research, I'm told, the traditional threshold for

significance is not p = 0.05 ... instead it's much more stringent. If

I recall, it may be more like p = 0.001.

Here's a free paper from PLoS Medicine, provocatively titled " Why Most

Published Research Findings Are False " :

http://medicine.plosjournals.org/perlserv/?request=get-document & doi=10.1371%2Fjo\

urnal.pmed.0020124

I haven't read the entire paper (particularly not the math), but it

has even more ideas about why there is so much wrong information on

pubmed.

> > >Product Overview - MBP8298

> > >

> > >Product Information

> > >

> > >MBP8298 is a synthetic peptide that consists of 17 amino acids

> linked

> > >in a sequence identical to that of a portion of human myelin basic

> > >protein (MBP). MBP8298 has been developed for the treatment of

> > >multiple sclerosis (MS), and is based on over 26 years of

> research.

> > >

> > >MS is generally considered an autoimmune disease, caused by immune

> > >attack against normal components of the central nervous system.

> The

> > >specificity of the immune attack at the molecular level is

> determined

> > >in each case by the HLA type of the individual patient, and HLA

> type

> > >is known to be one factor that contributes to susceptibility to

> MS.

> > >The MBP8298 synthetic peptide is a molecular replicate of the site

> of

> > >attack that is dominant in MS patients with HLA haplotypes DR-2 or

> DR-

> > >4. These HLA types are found in 65-75% of all MS patients.

> > >

> > >The apparent mechanism of action of MBP8298 is the induction or

> > >restoration of immunological tolerance with respect to ongoing

> immune

> > >attack at this molecular site. High doses of antigen delivered

> > >periodically by the intravenous route are expected to suppress

> immune

> > >responses to the administered substance. The potential benefit of

> > >MBP8298 for any individual patient is therefore expected to be

> > >related to the extent to which his or her disease process is

> > >dominated by autoimmune attack at the site represented by this

> > >synthetic peptide .

> > >

> > >The results of phase II and long-term follow-up treatment of MS

> > >patients with MBP8298, recently published in the European Journal

> of

> > >Neurology (EJN), showed that MBP8298 safely delayed median time to

> > >disease progression for five years in progressive MS patients with

> > >HLA-DR2 or HLA-DR4 immune response genes.

> >

>

[Guest guest]

Thanks, , for giving the detailed explanation that I was too lazy to

provide. I'll add that although most concepts that are named after

someone are fairly complicated, in this case the math is simple: if

you're trying 20 different hypotheses, the Bonferroni correction is to

multiply your P value by 20. In this case, they got a P value of 0.01;

if they'd looked at seven different hypotheses, the correction would push

it to 0.07. (And if they had, they wouldn't necessarily tell you. The

hope of being able to sell a drug at a higher price per gram than gold

does tend to make people unscrupulous.)

The Bonferroni correction formula is derived, though, by assuming that

the hypotheses that are tested are independent of each other (in the

statistical sense of the word independent: the probability of any one of

them being correct does not depend on whether any of the others are

correct). For instance, in the humongous genes-for-diseases study that

described earlier, this isn't nearly true, since genes are

correlated with each other. So for that case, the Bonferroni correction

is too harsh. But it's a decent start.

By the way, for me it's more than just an abstract belief that MS is

caused by an infectious disease. There is a particular suspect:

Chlamydia pneumoniae (Cpn). It has been controversial, since many labs

have failed to duplicate the finding, by Stratton's lab at Vanderbilt,

that people with MS usually have Cpn in their cerebrospinal fluid,

detectable by PCR. But there was a competition held, where the

disagreeing labs were sent blinded samples of CSF from MS sufferers and

normal people; the Vanderbilt lab found Cpn in 75% of the MS sufferers

and 25% of the normals, while other labs mostly found none at all in

either group. There's quite a bit of other evidence, too -- enough so

that when I researched the subject a couple of years ago, after having an

MS-style attack of numbness in my feet, I was relieved to find that I

wouldn't have to try anything that was low-probability, but could go with

a technique (the anti-Cpn antibiotic protocols from Vanderbilt) that was

pretty sure to work. It did: I have had no further attacks, and have

regained a lot of mental acuity that I'd lost before the one attack.

(The loss had been so gradual that I was unaware of it until it was

reversed.)

On Sun, Jun 17, 2007 at 05:58:55PM -0000, wrote:

>

>a, it's a very important, though, whether they considered the

>Bonferroni problem. Those p values, as you probably know, quantify the

>odds that a result could happen purely by chance if one's hypothesis

>were false. Ie, say you have a coin, and you hypothesize that it's

>biased towards tails; you toss it 5 times and get tails every time.

>This tends to support your hypothesis, but there is also a 0.03125

>chance of getting this result even if your hypothesis is false and the

>coin is in fact perfectly fair. So, p = 0.03125 in this case. In

>medicine we usually don't bother with a finding unless p < 0.05 -

>that's just the traditional, somewhat arbitrary threshold for a decent

>level of evidence, termed statistical significance.

>

>The problem is that for every 20 comparisons you make, one (on

>average) will come out statistically significant even if the

>hypothesis is false. If I take one random group of 10 people and

>compare the serum levels of 20 different cytokines to the levels found

>in a second group of random people, one of the cytokines will look

>like it is seriously different from one group to the other.

>

>This is why you sometimes read a big trial with designated " primary

>outcome measures " and some secondary outcome measures. There are

>usually just a handful of primary measures - a lot less than 20. This

>is to minimize the Bonferroni problem. It's understood that the

>outcome of the trial is staked on those primary measures only - those

>are the ones the researcher thinks are going to make the proof. The

>secondary measures are just things that are interesting enough to be

>examined. If they do come out to be statistically significant, they

>can always be examined later, alone, under hypotheses of their own, in

>a *separate* group of subjects (this makes them primary measures in

>their own right in a new study). Of course, primary outcome measures

>have to be designated before the study is executed. If you just pick

>out something like HLA groups *after* the fact (I'm not saying that's

>what your guys did, not having read it, but it happens), then you're

>cherry picking. There are 400 other correlations they could have

>looked for besides HLA, and if you examine for 400 correlations at p <

>0.05, you will find 20 illusory correlations.

>

>Reason I go into all this is that it's of more general significance.

>Just because related results might take years to be completed, and be

>published in separate papers, doesn't mean the Bonferroni problem

>doesn't apply. Say a lab is interested in proving objectively that CFS

>has a physical cause, so they look for physical differences from

>controls. Labs tend not to publish negative results. If they have 20

>ideas over the years and only one yields a statistically significant

>result and gets published, then this doesn't do much to support the

>hypothesis of a physical basis of CFS, because 20 comparisons applying

>to that hypothesis were made at a p threshold (the traditional medical

>research one) of 0.05. But, if after that research happens, the one

>statistically significant finding is replicated in a separate group of

>patients, THEN the physical basis hypothesis of CFS is strongly

>supported.

>

>That's one reason there's so much emphasis on separate replication in

>science. And this whole thing is one big reason why there are soooo

>many medical findings reported that end up not being replicated

>successfully by anyone (much less by everyone).

>

>In physics research, I'm told, the traditional threshold for

>significance is not p = 0.05 ... instead it's much more stringent. If

>I recall, it may be more like p = 0.001.

>

>Here's a free paper from PLoS Medicine, provocatively titled " Why Most

>Published Research Findings Are False " :

>

>http://medicine.plosjournals.org/perlserv/?request=get-document & doi=10.1371%2Fj\

ournal.pmed.0020124

>

>I haven't read the entire paper (particularly not the math), but it

>has even more ideas about why there is so much wrong information on

>pubmed.

>

>

>

>

>

>

>

>

>> > >Product Overview - MBP8298

>> > >

>> > >Product Information

>> > >

>> > >MBP8298 is a synthetic peptide that consists of 17 amino acids

>> > >linked in a sequence identical to that of a portion of human myelin

>> > >basic protein (MBP). MBP8298 has been developed for the treatment of

>> > >multiple sclerosis (MS), and is based on over 26 years of research.

>> > >

>> > >MS is generally considered an autoimmune disease, caused by immune

>> > >attack against normal components of the central nervous system. The

>> > >specificity of the immune attack at the molecular level is

>> > >determined in each case by the HLA type of the individual patient,

>> > >and HLA type is known to be one factor that contributes to

>> > >susceptibility to MS. The MBP8298 synthetic peptide is a molecular

>> > >replicate of the site of attack that is dominant in MS patients with

>> > >HLA haplotypes DR-2 or DR- 4. These HLA types are found in 65-75% of

>> > >all MS patients.

>> > >

>> > >The apparent mechanism of action of MBP8298 is the induction or

>> > >restoration of immunological tolerance with respect to ongoing

>> > >immune attack at this molecular site. High doses of antigen

>> > >delivered periodically by the intravenous route are expected to

>> > >suppress immune responses to the administered substance. The

>> > >potential benefit of MBP8298 for any individual patient is therefore

>> > >expected to be related to the extent to which his or her disease

>> > >process is dominated by autoimmune attack at the site represented by

>> > >this synthetic peptide .

>> > >

>> > >The results of phase II and long-term follow-up treatment of MS

>> > >patients with MBP8298, recently published in the European Journal of

>> > >Neurology (EJN), showed that MBP8298 safely delayed median time to

>> > >disease progression for five years in progressive MS patients with

>> > >HLA-DR2 or HLA-DR4 immune response genes.

>> >

>>

>

>

[Guest guest]

> The Bonferroni correction formula is derived, though, by assuming that

> the hypotheses that are tested are independent of each other (in the

> statistical sense of the word independent: the probability of any one

of

> them being correct does not depend on whether any of the others are

> correct).

Thanks, I did not know that.