Techniques to read studies (was: Re: Can fat be converted to glucose?)

[Guest guest]

> he

> cites Okinawa Japanese as an example -- they are the

> longest living people and are healthy, cognitive, etc.

> They have about 65% or so carbs in their diet.

>

ok, I've redone this post a couple times now. Its one of my hot

topics. Comparing nations is what is called an 'ecological study',

and they aren't used very much in science anymore, for reasons that

will become clear. When you do these kinds of studies there are two

problems:

Problem 1: confounds

There are hundreds of differances between the US and okinawa. Which

one(s) of those hundreds of differances is the one that makes the

differance?

When you have hundreds of differant data points (in this case we have

one data point: okinawa), you can use a statistical technique called

multiple regression to try to compare differant factors. You

could 'adjust' for fish, infant mortality, refined carbs and a couple

other factors and then see how predictive a high carb diet is.

So when you read that stuff, ask if they did a multiple regression,

and what other factors they regressed for. Then think of confounds,

like perhaps they didn't regress for sugar or trans-fats. You can

only regress for so many factors.

Ecological studies have largely unused in science because there are

way more differances between nations (or subsets of nations), then

you could ever possibly regress for. Its literally impossible to find

the real cause from the hundreds of differances.

Instead 'epidemiology' is favored, in which people from a similar

background (say, one town) are compared. You get thousands of people

instead of dozens of countries, and fewer differances to have to

adjust for. The result is better data.

Problem 2: Multiple Comparisons

Ok, Okinawans live longer. Why? It could be the: high carb diet, high

fiber diet, low unrefined carb diet, fish, soy, rice, vegetable,

dietary antioxidants, pork, lack of trans etc... If you want to make

a full list you could come up with hundreds of differant factors

which might " prove " your pet theory.

In science there is a 5% chance of a result being 'statistically

significant' just through random chance. This means that if you have

a list of 20 things to try, odds are that at least one of them will

be significant even though it doesn't mean anything. It would be like

flipping a coin 10 times and getting 7 heads and 3 tails. That

doesn't mean you've got a biased coin!

Scientists cope with this by doing 'prospective studies'. This means

that the whatever you measure (perhaps whether carb consumption is

correlated to longevity) is stated before the study, and you will

publish a negative result if there is no relationship. No more trying

out hundreds of differant things and only reporting what you find to

be statistically signifacant. That's called retrospective study.

So ask if its a prospective study. Be very, very skeptical of

retrospective studies. All ecological studies are retrospective.

Of course in this day there are dozens or even hundreds of such

studies. And with the 5% chance, you'd expect that a few of them will

show a correlation just through random chance. So you have to know

the full body of the literature. In epidemiology that's really tough,

because differant studies collect differant sets of data. This means

you should be skeptical of even prospective epidemiology unless you

really know the overall body of research.

Finally here are some general questions to ask:

1. How strong is the correlation? Cigarrette smokers have 20 times

the rate of lung cancer. That's a correlation that suggests causality!

2. How consistant is the correlation? For example, high cholesterol

isn't a risk factor for the elderly - the ones must likely to have a

heart attack. An inconsistant correlation suggests that its *not*

causal.

3. How about overall mortality? Sometime studies just report coronary

mortality. That usually means that the factor isn't a risk factor for

overall mortality, which means that its probably not causal.

4. Is the relationship dose-responsive? Do people with a little bit

have a little bit of the risk? In this example, no. Swizterland,

austria and greece are in the top 5 longest lived nations with high

fat diets. Meanwhile many nations have very low fat diets and very

low life expectancies.

And to repeat the above:

5. was it prospective?

6. What do the other similar studies find?

7. What factors did it multiply regress for? Did they miss anything

that you would like to add?

This post is a mess, but I'd have to spend several weeks revising to

make it good. Hopefully someone will find some small value in it.