Monday, November 28, 2011

Why p-value isn't enough, reiterated

Question 1:

People are routinely tested for disease X, which 1 in 1000 people have overall. It is known that if the person has the disease, the test is correct 99% of the time. If the person does not have the disease, the test is also correct 99% of the time.

A patient goes to his doctor for the test. It comes out positive.

What is the probability that the patient has the disease?



Question 2:

Researchers routinely run studies to test unexpected hypotheses (such as: can outside prayer help cure disease?), of which 1 in 1000 tend to be true overall. It is known that if a hypothesis is true, a study correctly finds statistical significance 99% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 99% of the time.

A researcher tests one such unexpected hypothesis. He finds statistical significance.

What is the probability that the hypothesis is true?



--------

Hat Tip: Inspired by Jeremy's last paragraph of comment #25, here.

--------

P.S. Answer to question 1 (very slightly modified question, but the same answer) at my previous post, here.


Labels: , ,

2 Comments:

At Tuesday, November 29, 2011 11:59:00 AM, Anonymous mettle said...

If I may add a third:

Question 3:

Researchers more often run studies to test hypotheses that are possibly true (such as: do hearing aids help speech perception? did humans arrive on the American continent more than 14000 years ago? does dopamine improve parkinsons symptoms?), of which somewhere between 2 in 10 and 8 in 10 tend to be true overall. Indeed, this is what most of science does, since if something is 99.9% true or 99.9% untrue, most people won't bother testing it. It is known that if a hypothesis is true, a study correctly finds statistical significance 95% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 95% of the time.

A researcher tests one such unexpected hypothesis. He finds statistical significance.

What is the probability that the hypothesis is true?

 
At Tuesday, November 29, 2011 12:06:00 PM, Blogger Phil Birnbaum said...

Mettle: I like your question!

I wish press reports (and the papers themselves) would discuss the answer to question 2 when the hypothesis is unexpected, and the answer to your question 3 when the hypothesis seems reasonable.

Hell, even if they ignored question 2 entirely, and discussed question 3 even when the hypothesis was obviously off the wall, it'd still be an improvement.

 

Post a Comment

<< Home