Probabilities, genetic testing, and doctors

Skeptic magazine features a regular medical column by a doctor, Harriet Hall.  This month (subscription required), she talks about how patients demand too much certainty from doctors, when the science is often unsettled and doctors are often imperfect.  Mostly good stuff, except one of her points bothered me:

"Direct-to-consumer genetic testing can be misleading.  ... Testers only look for specific SNPs (single nucleotide polymorphisms) and report probabilities based on imperfect information.  They may report that people with your SNP are 30% more likely to develop Parkinson's disease than people with other SNPs.  But disease is not destiny*.  Even if you have the gene for that disease, that gene may or may not be expressed.  Gene expression depends on environmental and epigenetic factors and on interactions with other genes.  Our access to genetic information currently exceeds our understanding of what that information actually means."

[* I think she means "But genetics is not destiny."]

Maybe I'm misunderstanding her point, but ... her argument does not debunk genetic testing probabilities.  It *supports* them.

If what Dr. Hall means is that having a certain SNP doesn't necessarily mean you'll get Parkinson's ... well, of course not.  It only means you have a 30% higher chance than you would otherwise, as stated.  If that's her argument, she's obviously just attacking a straw man.  I'm going to assume that's not really her argument.

In which case, what I think she's saying is something like this (my paraphrase):

"People with gene X get Parkinson's 30% more often than average.  But, if you have gene X, it doesn't mean that you're *certain* to have a 30% higher probability, in the sense that a weighted coin has a 30% higher probability of landing heads.  Gene X might interact with gene Y.  If you have both X and Y, you might have a 90% greater chance.  But if have X and not Y, you may be completely average.  Or, in connection with other genes, you might even have a *lower* than average chance of getting Parkinson's!"

But even if that's true -- and it's almost certain that it is, that gene X acts in combination with other genetic traits -- it doesn't change the fact that you DO have a 30% increased probability of winding up with Parkinson's.  Because you still don't know if you're in the 90% group, or the 0% group.  

----

Here's an analogy.  God has a collection of red urns and blue urns.  Each urn has 20 coins in it.  The red urns have 20 fair coins.  The blue urns have 17 14 fair coins, and 3 6 two-headed coins.

Your DNA determines which urn you draw a coin from.  You draw a coin without looking.  At some later date, you'll flip the coin.  If the coin lands heads, you eventually get Parkinson's.  

Your chance of getting Parkinson's from the red urn is 50%.  Your chance of getting Parkinson's from the blue urn is 65%.

You take a direct-to-consumer DNA test, and it says you drew from the blue urn.  You say, "Oh, no, I have a 30% higher Parkinson's probability than someone who's DNA tested red!"  You are correct.  But, as I read it, Dr. Hall is saying, "No, that's misleading.  You might wind up having drawn a fair coin, and still be at 50%.  You might be normal!"

Well, yes, but ... that doesn't change the fact that, without knowing which coin you actually wind up with,  your probability is still 65%, because 65% of blue urn drawers will get Parkinson's, *including* the ones who are normal.  So, you are correct in being worried!

----

What I think might be going on, with this argument, is that Dr. Hall isn't actually thinking of the probability of Parkinson's.  She's thinking of an intermediate result, which kind of coin was drawn.  

The 65% chance of Parkinson's is the combination of 

(a) having drawn an unfair coin and getting Parkinson's for sure; or
(b) having drawn a fair coin and getting Parkinson's at a normal rate.

If you look at it that way, you might think: "how can you state flat out that you're at higher risk for Parkinson's, when there's an 85% chance you drew a fair coin and are completely normal?"

That argument implies that you can't say anything about the probability of getting Parkinson's unless you know what coin you drew.  That's not correct.  It's not how probabilities work.   

What's important is your overall risk of heads, not your overall risk of getting a high-probability coin.  I think what's going on is that we're not so much worried about getting Parkinson's as we are about *being at high risk* for Parkinson's.  A "normal" risk bothers us as ... well, almost zero, because we're just used to it, we tolerate it.  But a "bigger than normal" risk sets off alarm bells.  

What the Hall argument is for is making someone feel better by addressing that cognitive fallacy.  It says, "yes, if you look at it that way, you're a higher risk, but ... that's really just a big pool of normal people with a small minority with MUCH higher risk.  You're probably just one of the normals, so don't worry about it."

-----

To see the fallacy another way ... 

You're playing russian roulette.  There are two guns, with six chambers each.  One of the chambers in one of the guns has a bullet.  You've picked a gun at random, and spun the cylinder.  You're due to pull the trigger when you reach middle age.

You think, "I have a 1 in 12 chance of dying".

Now, experts have done some analysis, and they've noticed that chamber 3 winds up containing the bullet twice as often as any other chamber.  They're not sure why, but they know it's a real effect, and not random.

You now sign up to get your random selection "tested," and it comes back that you wound up with chamber 3.  You are distressed.  You think, "Instead of a 1 in 12 chance of dying, I'm down to 1 in 7.  I'm 71 percent more likely to die than I thought!!"

But, the doctor says, "No, that's misleading!  You may have chosen the empty gun, in which case your chance is zero!"

That, obviously, is BS, just by common sense.  But the doctor states it in a form where the missing common sense is harder to notice.  Something like:

"You don't know your chance is down to 1 in 7.  Whether you die depends not just on the chamber, but on the interaction between the chamber and other factors, like the gun.  If you don't have the "Gun A" gene, the "chamber 3" gene will not be "expressed," and you won't have any chance of dying at all.  Our access to the information of what chamber you have currently exceeds our understanding of what that information will really mean when you pull the trigger."

Sorry, it's still 1 in 7.   

If half the people are zero, and half the people are 1 in 3.5, and nobody knows which group you're in ... you're 1 in 7.  The fact that you *might* be in the zero group doesn't change that fact.

------

It's interesting that the author of this article specifically mentions "direct-to-consumer genetic testing."  Recently, in the US, the FDA banned the company "23andMe" from supplying genetic information to customers, on the grounds that test results are a "medical device for the diagnosis and prevention of disease."

That actually seems flimsy to me, First, that information is a "device"; you'd have to ban all medical books from the library, on that grounds.  Second, the information doesn't diagnose a disease, it just gives you information about your probability of contracting that disease.  Third, if you really wanted to crack down on diagnosis, you'd ban Consumer Reports magazine, which recently ran an article on how to tell a cold from the flu.  Fourth, the information is really no different, in kind, than the information that Parkinson's is hereditary.  The sentence, "Your parents died of it, so you might be higher risk" is certainly not a "medical device."  Fifth ... well, I'll stop here, but, obviously, I could go on for a while.

And, the "information" is not that complicated.  I purchased the service last year, before the ban, and I still have access to my results.  Among other things, my genes suggest I have six times the normal chance of contracting Type 1 Diabetes.  What they told me was something like this (my paraphrase):

"On average, 1.0 in 100 males will develop Type I Diabetes in their lifetimes.  We estimate your risk at 6.0 percent, which is six times as high.

"Why do we think that?  Academic study X found that one of your particular gene combinations was related to an 18% increase in risk.  Study Y found another one of your combinations was related to a 4% decrease.  A third study  was related to a 400% increase.  And so on.  Overall, it works out to 6 times the chance.

"Here's a few sentences on the biological details in the studies, the presumed mechanism by which the genes translate to diabetes, if you care, and to make sure you get the idea that we understand the science.

"Also, we survey our members in hopes of mining the data to find empirical connections.  In this case, we haven't made any of our own discoveries yet.

"That's all we know.  Remember, and there are other factors that contribute to whether you get diabetes -- like environment and lifestyle -- so don't go assuming that you're going to get the disease just because you have this genetic makeup."

Not that complicated, and pretty well explained.  What should I do with the information?  Well, for my part, knowing that my risk is six times as high (6 percent probability from birth, but only 2 percent from my current age), I might, you know, keep an eye on it, especially because they told me my type 2 diabetes risk is also a little high.

But, they didn't try to sell me anything, or tell me what to do, or suggest treatment, or anything.  They do, at some point, suggest genetic counselling, or talking to my doctor, if the results bother me.

And, they're not a fraud or anything.  I think the information and probabilities, are, for the most part, correct.  I trust that 23andMe got it right.  They identified things that I've heard run in my family.  And, they even found me an actual relative I had never heard of, based on DNA profile alone.

-----

Going off-topic here, but what's the FDA's problem?  

It might be a turf war.  According to this article, the FDA is p*ssed off that 23andMe didn't respond deferentially enough to their investigation.  And, doctors tend to think that anything to do with disease needs to go through them, as gatekeepers.  But, never mind that, and let's just look at the rationales they actually give.

Mostly, they think that patients are too uneducated to handle the information:

Robert Field, a Wharton health care management lecturer, believes the 23andMe technology would not have generated so much regulatory concern if it had been marketed to doctors instead of consumers. "Any kind of genetic testing has to be combined with professional counseling to do the patient any real good," he notes. "The concern is that when you do a home test, you’re not going to get that counseling, and you’re not going to know how to act appropriately on the results." If the test had been marketed to doctors instead, Field adds, "you would have built into the process the professional advice needed."

Well, that's kind of arrogant, isn't it, that we need a doctor to tell us what "6 times as high a risk" means?  I mean, doctors may be expert in diagnosing diabetes, and treating it, but do they also somehow have some god-given expertise in explaining probabilities?  In fact, from the same article:

"Most of the physicians said they didn't know what they were going to do with that kind of information, [medical-genetics professor Reed] Pyeritz says."

I mean, seriously, if you think that counselling is needed ... I have a degree in statistics.  I think, you know, *I* should be the counsellor.  In fact, I think the FDA should ban doctors from advising patients on risk without a trained statistician in the room. 

Now, I don't mean to trivialize the customers' confusion about what the results actually mean.  In the forums on the 23andMe site, I've seen a lot of posts like, "They were wrong.  They told me I had only a 1% chance, but I was diagnosed last week."  Or, "Oh my God, I'm ten times the [1 in a million] risk for disease Y, I'm going to die!"

But, aside from those obvious cases, I suspect doctors might be *worse* at evaluating the information, because they understand the medical side too much.  If I tell you that you have a 1 in 7 chance of dying, you get it.  But if I tell you that you have a 1 in 7 chance of dying and then tell you the rules of the Russian Roulette game ... now, you have knowledge with which to rationalize your disbelief.  "How can they misleadingly say I'm at a higher risk?  I may have the empty gun!"  Even though that extra knowledge should make you MORE certain that the 1 in 7 is correct.

In articles on the web, I read different doctors making that same argument, that, after all, there are many genes that cause Parkinson's (or whatever), and those haven't been discovered yet, so how can the results be accurate?

But they can.  And they can, for the same reason that they tell overweight people that they're a higher risk for a heart attack, even though there are multiple causes of that, too.  

Anyway ... I've gone on too long about this, which was meant to be just a statistical post.  Still, I think the faulty statistical argument presents an excellent example of how doctors overreach -- in this case, to push to make it illegal for anybody but them to obtain and interpret my own genetic information, even though they clearly don't know how to interpret it themselves.

Knowing how to diagnose and treat Parkinson's Disease is something in which medical professionals have expertise.  Understanding and interpreting probabilities about Parkinson's -- even those based on genetic testing -- is not.



-----

UPDATE: Part II is here.