Probabilities, genetic testing, and doctors, part II

(Part I is here)

------

Kira Peikoff ordered "direct-to-consumer" genetic tests from three competing companies.  In some cases, they gave her results that were very different from each other.  This led Peikoff to think that maybe she got ripped off, or that the firms aren't able to deliver what they promise.  In a New York Times article, she writes,

"At a time when the future of such companies hangs in the balance, their ability to deliver standardized results remains dubious, with far-reaching implications for consumers."

But: I think her concern stems from a misunderstanding of how the probabilities work.

The provider "23andMe" -- the one recently shut down by the FDA -- reported to Peikoff that she had a higher-than-normal risk of contracting psoriasis, twice the normal chance.  But a rival company, Genetic Testing Laboratories (GTL), told her she had a much *lower* risk -- 80% less than average.  

The two companies differed by a factor of ten, a proverbial "order of magnitude".  Clearly, those results can't both be right, can they?

Well, actually, they can, because GTL tested more genes than 23andMe.

In the illustration that accompanies the article, we can see that GTL tested eight sets of genes: HLA, IL12B, IL23R, Intergenic_1q21, SPATA2, STAT2, TNFAIP3, and TNIP1.

The article doesn't say what genes 23andMe tested, but, in my own report, my result is based on only 3 tests: HLA-C, IL12B, and IL23R.

So, it's quite reasonable that the two analyses would give different results, since they're based on different information. And, they're both correct, as far as they go.  If all you have is the three genes that 23andMe looked at, it's reasonable to say that your risk is twice normal.  The extra genes that GTL tested provided more information, and more information always changes an estimate.  

This is the essence of Bayesian reasoning: start with your prior, and update your beliefs based on new information.

------

You flip two coins, and leave them covered.  You ask a statistician what the chance is that you got to heads.  He says, "one in four."  That is the correct answer.

Then, you call in a second statistician, and you uncover the first coin, which turns out to be a head.  You ask the same question.  The second statistician says, "one in two".  That is again the correct answer.

But the first statistician was not wrong.  He was absolutely correct.  It's perhaps counterintuitive.  I mean, he said "one in four," and now we know the answer is "one in two".  How could he have been right?  You can argue that he did the best he could with the information available, and it's not his fault that he was wrong, but ... his answer wasn't right.

But his answer WAS right.  That's because the two statisticians were asked two different questions.  The first one was asked, "what's the chance that both coins landed heads?"  The second one was asked, "what's the chance that both coins landed heads given that we know the first one did?"

-----

Doesn't this at least demonstrate that 23andMe is being lax in its testing, not using enough information?  No, it doesn't. Any information is better than none.  23andMe costs $99 and uses saliva.  The GTL test costs $259 and uses blood.  I'm sure if you wanted to spend $1000, you could find even more genes to test.

Say you're buying car insurance.  Company A asks if you use a seat belt.  You say no, and they quote you a high rate. You go to company B.  They secretly shadow you around for a week, and discover that you're actually such a safe and cautious driver that it completely cancels out your non-seatbelt-risk, and they quote you a lower rate.  

Was Company A wrong in quoting you a high rate?  No, they weren't.  It was the right answer for the information they had.  Unless you fault them for not following you around to get the information a better estimate.  If you do choose to fault them for that, then you have to fault every real-life risk estimate ever made, because there's always more information you can get if you take the time to uncover it.  Risk estimates *always* change with additional relevant information, which is what Bayes' Theorem is all about.


-----

This is a variation of the thinking I argued against last post: the idea that since there's always more information -- including  information that hasn't even yet been discovered -- the information we do have is incomplete, and therefore not relevant.  From Peikoff's article:

"Imagine if you took a book and you only looked at the first letter of every other page,” said Dr. Robert Klitzman, a bioethicist and professor of clinical psychiatry at Columbia. (I [reporter Peikoff] am a graduate student there in his Master of Bioethics program.) "You’re missing 99.9 percent of the letters that make the genome. The information is going to be limited.""

Again: the information is limited, but still useful -- like the fact that you don't wear a seatbelt.  If gene X is linked to double the risk, it's not reasonable to say, "well, we might later find that gene Y turns off gene X, so don't worry about it."

Interestingly, in the same article, another bioethicist implicitly contradicts Klitzman!  Arthur L. Caplan, director of medical ethics at New York University, writes,

"If you want to spend money wisely to protect your health and you have a few hundred dollars, buy a scale, stand on it, and act accordingly."

That completely contradicts Dr. Klitzman, doesn't it?  Klitzman is saying, "if you don't have all the information on risk factors, the genetic information you do have isn't useful."  Caplan is saying, "if you don't have all the information on risk factors, the obesity information you do have is still very important."

What's going on, where the same story can quote two opposite arguments without noticing the contradiction?  I think, maybe, it's the fallacy of mental accounting.  There's the obesity mental account, and the DNA mental account. We have full knowledge how fat you are, so we should consider what we know.  But we have only partial knowledge of your DNA, so we have to ignore what we know.

Except, probabilities don't work that way.  They don't keep separate mental buckets.  If there are 100 independent DNA datapoints, and 1 obesity datapoint, the laws of probability treat them the same, as 101 datapoints.  

It's like, if you roll 101 dice, but 100 are blue and only one is red ... the first blue die is just as useful in predicting the overall total as the first (and only) red one.

Sure, my obesity might give me twice the risk of disease X. But if a gene you looked at gives you three times the risk ... you should be more worried than me, even if you only looked at one gene, and even if your other 999 genes might cancel it out.  

That's just how probabilities work.  

Probabilities, genetic testing, and doctors

Skeptic magazine features a regular medical column by a doctor, Harriet Hall.  This month (subscription required), she talks about how patients demand too much certainty from doctors, when the science is often unsettled and doctors are often imperfect.  Mostly good stuff, except one of her points bothered me:

"Direct-to-consumer genetic testing can be misleading.  ... Testers only look for specific SNPs (single nucleotide polymorphisms) and report probabilities based on imperfect information.  They may report that people with your SNP are 30% more likely to develop Parkinson's disease than people with other SNPs.  But disease is not destiny*.  Even if you have the gene for that disease, that gene may or may not be expressed.  Gene expression depends on environmental and epigenetic factors and on interactions with other genes.  Our access to genetic information currently exceeds our understanding of what that information actually means."

[* I think she means "But genetics is not destiny."]

Maybe I'm misunderstanding her point, but ... her argument does not debunk genetic testing probabilities.  It *supports* them.

If what Dr. Hall means is that having a certain SNP doesn't necessarily mean you'll get Parkinson's ... well, of course not.  It only means you have a 30% higher chance than you would otherwise, as stated.  If that's her argument, she's obviously just attacking a straw man.  I'm going to assume that's not really her argument.

In which case, what I think she's saying is something like this (my paraphrase):

"People with gene X get Parkinson's 30% more often than average.  But, if you have gene X, it doesn't mean that you're *certain* to have a 30% higher probability, in the sense that a weighted coin has a 30% higher probability of landing heads.  Gene X might interact with gene Y.  If you have both X and Y, you might have a 90% greater chance.  But if have X and not Y, you may be completely average.  Or, in connection with other genes, you might even have a *lower* than average chance of getting Parkinson's!"

But even if that's true -- and it's almost certain that it is, that gene X acts in combination with other genetic traits -- it doesn't change the fact that you DO have a 30% increased probability of winding up with Parkinson's.  Because you still don't know if you're in the 90% group, or the 0% group.  

----

Here's an analogy.  God has a collection of red urns and blue urns.  Each urn has 20 coins in it.  The red urns have 20 fair coins.  The blue urns have 17 14 fair coins, and 3 6 two-headed coins.

Your DNA determines which urn you draw a coin from.  You draw a coin without looking.  At some later date, you'll flip the coin.  If the coin lands heads, you eventually get Parkinson's.  

Your chance of getting Parkinson's from the red urn is 50%.  Your chance of getting Parkinson's from the blue urn is 65%.

You take a direct-to-consumer DNA test, and it says you drew from the blue urn.  You say, "Oh, no, I have a 30% higher Parkinson's probability than someone who's DNA tested red!"  You are correct.  But, as I read it, Dr. Hall is saying, "No, that's misleading.  You might wind up having drawn a fair coin, and still be at 50%.  You might be normal!"

Well, yes, but ... that doesn't change the fact that, without knowing which coin you actually wind up with,  your probability is still 65%, because 65% of blue urn drawers will get Parkinson's, *including* the ones who are normal.  So, you are correct in being worried!

----

What I think might be going on, with this argument, is that Dr. Hall isn't actually thinking of the probability of Parkinson's.  She's thinking of an intermediate result, which kind of coin was drawn.  

The 65% chance of Parkinson's is the combination of 

(a) having drawn an unfair coin and getting Parkinson's for sure; or
(b) having drawn a fair coin and getting Parkinson's at a normal rate.

If you look at it that way, you might think: "how can you state flat out that you're at higher risk for Parkinson's, when there's an 85% chance you drew a fair coin and are completely normal?"

That argument implies that you can't say anything about the probability of getting Parkinson's unless you know what coin you drew.  That's not correct.  It's not how probabilities work.   

What's important is your overall risk of heads, not your overall risk of getting a high-probability coin.  I think what's going on is that we're not so much worried about getting Parkinson's as we are about *being at high risk* for Parkinson's.  A "normal" risk bothers us as ... well, almost zero, because we're just used to it, we tolerate it.  But a "bigger than normal" risk sets off alarm bells.  

What the Hall argument is for is making someone feel better by addressing that cognitive fallacy.  It says, "yes, if you look at it that way, you're a higher risk, but ... that's really just a big pool of normal people with a small minority with MUCH higher risk.  You're probably just one of the normals, so don't worry about it."

-----

To see the fallacy another way ... 

You're playing russian roulette.  There are two guns, with six chambers each.  One of the chambers in one of the guns has a bullet.  You've picked a gun at random, and spun the cylinder.  You're due to pull the trigger when you reach middle age.

You think, "I have a 1 in 12 chance of dying".

Now, experts have done some analysis, and they've noticed that chamber 3 winds up containing the bullet twice as often as any other chamber.  They're not sure why, but they know it's a real effect, and not random.

You now sign up to get your random selection "tested," and it comes back that you wound up with chamber 3.  You are distressed.  You think, "Instead of a 1 in 12 chance of dying, I'm down to 1 in 7.  I'm 71 percent more likely to die than I thought!!"

But, the doctor says, "No, that's misleading!  You may have chosen the empty gun, in which case your chance is zero!"

That, obviously, is BS, just by common sense.  But the doctor states it in a form where the missing common sense is harder to notice.  Something like:

"You don't know your chance is down to 1 in 7.  Whether you die depends not just on the chamber, but on the interaction between the chamber and other factors, like the gun.  If you don't have the "Gun A" gene, the "chamber 3" gene will not be "expressed," and you won't have any chance of dying at all.  Our access to the information of what chamber you have currently exceeds our understanding of what that information will really mean when you pull the trigger."

Sorry, it's still 1 in 7.   

If half the people are zero, and half the people are 1 in 3.5, and nobody knows which group you're in ... you're 1 in 7.  The fact that you *might* be in the zero group doesn't change that fact.

------

It's interesting that the author of this article specifically mentions "direct-to-consumer genetic testing."  Recently, in the US, the FDA banned the company "23andMe" from supplying genetic information to customers, on the grounds that test results are a "medical device for the diagnosis and prevention of disease."

That actually seems flimsy to me, First, that information is a "device"; you'd have to ban all medical books from the library, on that grounds.  Second, the information doesn't diagnose a disease, it just gives you information about your probability of contracting that disease.  Third, if you really wanted to crack down on diagnosis, you'd ban Consumer Reports magazine, which recently ran an article on how to tell a cold from the flu.  Fourth, the information is really no different, in kind, than the information that Parkinson's is hereditary.  The sentence, "Your parents died of it, so you might be higher risk" is certainly not a "medical device."  Fifth ... well, I'll stop here, but, obviously, I could go on for a while.

And, the "information" is not that complicated.  I purchased the service last year, before the ban, and I still have access to my results.  Among other things, my genes suggest I have six times the normal chance of contracting Type 1 Diabetes.  What they told me was something like this (my paraphrase):

"On average, 1.0 in 100 males will develop Type I Diabetes in their lifetimes.  We estimate your risk at 6.0 percent, which is six times as high.

"Why do we think that?  Academic study X found that one of your particular gene combinations was related to an 18% increase in risk.  Study Y found another one of your combinations was related to a 4% decrease.  A third study  was related to a 400% increase.  And so on.  Overall, it works out to 6 times the chance.

"Here's a few sentences on the biological details in the studies, the presumed mechanism by which the genes translate to diabetes, if you care, and to make sure you get the idea that we understand the science.

"Also, we survey our members in hopes of mining the data to find empirical connections.  In this case, we haven't made any of our own discoveries yet.

"That's all we know.  Remember, and there are other factors that contribute to whether you get diabetes -- like environment and lifestyle -- so don't go assuming that you're going to get the disease just because you have this genetic makeup."

Not that complicated, and pretty well explained.  What should I do with the information?  Well, for my part, knowing that my risk is six times as high (6 percent probability from birth, but only 2 percent from my current age), I might, you know, keep an eye on it, especially because they told me my type 2 diabetes risk is also a little high.

But, they didn't try to sell me anything, or tell me what to do, or suggest treatment, or anything.  They do, at some point, suggest genetic counselling, or talking to my doctor, if the results bother me.

And, they're not a fraud or anything.  I think the information and probabilities, are, for the most part, correct.  I trust that 23andMe got it right.  They identified things that I've heard run in my family.  And, they even found me an actual relative I had never heard of, based on DNA profile alone.

-----

Going off-topic here, but what's the FDA's problem?  

It might be a turf war.  According to this article, the FDA is p*ssed off that 23andMe didn't respond deferentially enough to their investigation.  And, doctors tend to think that anything to do with disease needs to go through them, as gatekeepers.  But, never mind that, and let's just look at the rationales they actually give.

Mostly, they think that patients are too uneducated to handle the information:

Robert Field, a Wharton health care management lecturer, believes the 23andMe technology would not have generated so much regulatory concern if it had been marketed to doctors instead of consumers. "Any kind of genetic testing has to be combined with professional counseling to do the patient any real good," he notes. "The concern is that when you do a home test, you’re not going to get that counseling, and you’re not going to know how to act appropriately on the results." If the test had been marketed to doctors instead, Field adds, "you would have built into the process the professional advice needed."

Well, that's kind of arrogant, isn't it, that we need a doctor to tell us what "6 times as high a risk" means?  I mean, doctors may be expert in diagnosing diabetes, and treating it, but do they also somehow have some god-given expertise in explaining probabilities?  In fact, from the same article:

"Most of the physicians said they didn't know what they were going to do with that kind of information, [medical-genetics professor Reed] Pyeritz says."

I mean, seriously, if you think that counselling is needed ... I have a degree in statistics.  I think, you know, *I* should be the counsellor.  In fact, I think the FDA should ban doctors from advising patients on risk without a trained statistician in the room. 

Now, I don't mean to trivialize the customers' confusion about what the results actually mean.  In the forums on the 23andMe site, I've seen a lot of posts like, "They were wrong.  They told me I had only a 1% chance, but I was diagnosed last week."  Or, "Oh my God, I'm ten times the [1 in a million] risk for disease Y, I'm going to die!"

But, aside from those obvious cases, I suspect doctors might be *worse* at evaluating the information, because they understand the medical side too much.  If I tell you that you have a 1 in 7 chance of dying, you get it.  But if I tell you that you have a 1 in 7 chance of dying and then tell you the rules of the Russian Roulette game ... now, you have knowledge with which to rationalize your disbelief.  "How can they misleadingly say I'm at a higher risk?  I may have the empty gun!"  Even though that extra knowledge should make you MORE certain that the 1 in 7 is correct.

In articles on the web, I read different doctors making that same argument, that, after all, there are many genes that cause Parkinson's (or whatever), and those haven't been discovered yet, so how can the results be accurate?

But they can.  And they can, for the same reason that they tell overweight people that they're a higher risk for a heart attack, even though there are multiple causes of that, too.  

Anyway ... I've gone on too long about this, which was meant to be just a statistical post.  Still, I think the faulty statistical argument presents an excellent example of how doctors overreach -- in this case, to push to make it illegal for anybody but them to obtain and interpret my own genetic information, even though they clearly don't know how to interpret it themselves.

Knowing how to diagnose and treat Parkinson's Disease is something in which medical professionals have expertise.  Understanding and interpreting probabilities about Parkinson's -- even those based on genetic testing -- is not.



-----

UPDATE: Part II is here.

Stop revering doctors

Warning: Non-sports post. And, this is only tangentially related to the recent "should employers' insurance companies have to pay for employees' birth control" controversy, or Rush Limbaugh's response. My arguments do not depend on which side of that debate is correct. They apply to only one small aspect. (If you're not familiar with the debate, Google "Sandra Fluke".)

----

A little while ago, Steven E. Landsburg decided to post about some of economic issues surrounding the question of employer coverage of contraceptives. He was critical of Fluke for demanding free birth control without providing a good economic reason why. He then got in trouble from the president of the University of Rochester, where he teaches.

Afterward, reporters started calling him for comment. One reporter asked him,

"Do you think that “reasons” accepted by an economist deserve more weight and respect than “reasons” a medical doctor might have for recommending that birth control be universally covered by medical insurance?"

Landsburg replied,

Yes, absolutely. Here’s why: Economists are trained to look at all the consequences of a decision before passing judgment; doctors tend to focus only on some kinds of consequences (those directly related to health) while ignoring others (for example, the many other effects that flow from raising people’s taxes or insurance premiums). ...

Economists have thought long and hard about how to make sure we do that. We don’t always get it right, but at least we’ve got a framework for it. Doctors don’t.

Landsburg is, of course, absolutely correct.

Why would this reporter, or, anyone, think that a doctor was as qualified to talk about an economic question as an economist is?

Because we respect doctors too much. We respect them well beyond the scope of their expertise. We somehow think that they're better and smarter than the rest of us, and we have this unspoken feeling that their opinions have extra weight, because of their higher class and status.

It's certainly not that we need their subject matter expertise to argue the question. Because, suppose Sandra Fluke had stepped up before Congress to demand that the car companies include oil changes in their warranty coverage. And economists disagreed that that was a good idea. And suppose the reporter had asked,

"Do you think that “reasons” accepted by an economist deserve more weight and respect than “reasons” an auto mechanic might have for recommending that oil changes be universally covered by warranty?"

That would be laughable. It's equally laughable when it's a doctor.

I think it's all a matter of pecking order. Doctors have high status, and auto mechanics have low status.

Economists have high status too, but significantly lower status than doctors. To see that, imagine the situation reversed, where Landsburg said something about medicine -- say, about why birth control works, biochemically -- and a doctor corrected him. And the reporter asked the doctor,

"Do you think that “reasons” accepted by a doctor deserve more weight and respect than “reasons” an economist might have for understanding how birth control works, biochemically?"

That question answers itself. The other one should have, too.

-----

P.S. A previous post about doctors overstepping their expertise is here.