Why p-value isn't enough, reiterated
Question 1:
People are routinely tested for disease X, which 1 in 1000 people have overall. It is known that if the person has the disease, the test is correct 99% of the time. If the person does not have the disease, the test is also correct 99% of the time.
A patient goes to his doctor for the test. It comes out positive.
What is the probability that the patient has the disease?
Question 2:
Researchers routinely run studies to test unexpected hypotheses (such as: can outside prayer help cure disease?), of which 1 in 1000 tend to be true overall. It is known that if a hypothesis is true, a study correctly finds statistical significance 99% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 99% of the time.
A researcher tests one such unexpected hypothesis. He finds statistical significance.
What is the probability that the hypothesis is true?
--------
Hat Tip: Inspired by Jeremy's last paragraph of comment #25, here.
--------
P.S. Answer to question 1 (very slightly modified question, but the same answer) at my previous post, here.
Labels: academics, bayes, statistics
posted by Phil Birnbaum @ 11/28/2011 06:48:00 PM
2 comments
2 Comments:
If I may add a third:
Question 3:
Researchers more often run studies to test hypotheses that are possibly true (such as: do hearing aids help speech perception? did humans arrive on the American continent more than 14000 years ago? does dopamine improve parkinsons symptoms?), of which somewhere between 2 in 10 and 8 in 10 tend to be true overall. Indeed, this is what most of science does, since if something is 99.9% true or 99.9% untrue, most people won't bother testing it. It is known that if a hypothesis is true, a study correctly finds statistical significance 95% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 95% of the time.
A researcher tests one such unexpected hypothesis. He finds statistical significance.
What is the probability that the hypothesis is true?
Mettle: I like your question!
I wish press reports (and the papers themselves) would discuss the answer to question 2 when the hypothesis is unexpected, and the answer to your question 3 when the hypothesis seems reasonable.
Hell, even if they ignored question 2 entirely, and discussed question 3 even when the hypothesis was obviously off the wall, it'd still be an improvement.
Post a Comment
<< Home