Techniques to read studies (was: Re: Can fat be converted to glucose?)

[Guest guest]

-

>In science there is a 5% chance of a result being 'statistically

>significant' just through random chance. This means that if you have

>a list of 20 things to try, odds are that at least one of them will

>be significant even though it doesn't mean anything. It would be like

>flipping a coin 10 times and getting 7 heads and 3 tails. That

>doesn't mean you've got a biased coin!

Actually, there are statistical formulas for determining the margin for

error (which is essentially what gives you the chance that a seemingly

significant result won't actually be significant). It's not a flat 5%

regardless of sample size.

-

[Guest guest]

> -

>

> >In science there is a 5% chance of a result being 'statistically

> >significant' just through random chance. This means that if you

have

> >a list of 20 things to try, odds are that at least one of them will

> >be significant even though it doesn't mean anything. It would be

like

> >flipping a coin 10 times and getting 7 heads and 3 tails. That

> >doesn't mean you've got a biased coin!

>

> Actually, there are statistical formulas for determining the margin

for

> error (which is essentially what gives you the chance that a

seemingly

> significant result won't actually be significant). It's not a flat

5%

> regardless of sample size.

You might be thinking of power analysis. It tells you what the

probability is that the results of an expirement are due to chance or

not, based on the relative risk and sample size.

For example, supporters of the cholesterol theory sometimes explain

away the failure of all non-statin cholesterol lowering drugs by

there not lowering cholesterol as much as the statins. After 30

clinical trials of non-statins, with 80,000 patient-years (IIRC), and

an average of 10% reduction in cholesterol, the results are a

relative risk of 1.03 - a slight increase. Since we expect each

percentage of cholesterol reduction to translate to a 2% reduction in

mortality, we would expect a relative risk 0.8 for the non-statin

takers. A power analysis would tell us what the chances are of having

a relative risk of 1.03 with that many patients when we expect a

relative risk of 0.8

I've been meaning to run the numbers on it, but haven't gotten around

to it.

Statistical significance is completely differant. It has nothing to

do with sample size, its just a number chosen by the scientists to

help decide if the results are likely to be due to chance or not. It

is independant of sample size, so as sample size goes up, smaller and

smaller effects will become significant - even if they have no

clinical or scientific importance.

IMO, clinical trials should stop using statistical significance

levels and instead establish significant relative risk thresholds.

Epidemiology should move from 5% and 1% thresholds to 0.05% and 0.01%

thresholds. That way large clinical trials would not be in danger of

declaring scientifically unimportant phenomina as statistically

significant, and epidemiology would not suffer so badly from the

multiple comparison problem. But unless those levels are set

prospectively, it does no good and I doubt it will change.