### Why p-value isn't enough, reiterated

Question 1:

People are routinely tested for disease X, which 1 in 1000 people have overall. It is known that if the person has the disease, the test is correct 99% of the time. If the person does not have the disease, the test is also correct 99% of the time.

A patient goes to his doctor for the test. It comes out positive.

What is the probability that the patient has the disease?

Question 2:

Researchers routinely run studies to test unexpected hypotheses (such as: can outside prayer help cure disease?), of which 1 in 1000 tend to be true overall. It is known that if a hypothesis is true, a study correctly finds statistical significance 99% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 99% of the time.

A researcher tests one such unexpected hypothesis. He finds statistical significance.

What is the probability that the hypothesis is true?

--------

Hat Tip: Inspired by Jeremy's last paragraph of comment #25, here.

--------

P.S. Answer to question 1 (very slightly modified question, but the same answer) at my previous post, here.

Labels: academics, bayes, statistics

## 2 Comments:

If I may add a third:

Question 3:

Researchers more often run studies to test hypotheses that are possibly true (such as: do hearing aids help speech perception? did humans arrive on the American continent more than 14000 years ago? does dopamine improve parkinsons symptoms?), of which somewhere between 2 in 10 and 8 in 10 tend to be true overall. Indeed, this is what most of science does, since if something is 99.9% true or 99.9% untrue, most people won't bother testing it. It is known that if a hypothesis is true, a study correctly finds statistical significance 95% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 95% of the time.

A researcher tests one such unexpected hypothesis. He finds statistical significance.

What is the probability that the hypothesis is true?

Mettle: I like your question!

I wish press reports (and the papers themselves) would discuss the answer to question 2 when the hypothesis is unexpected, and the answer to your question 3 when the hypothesis seems reasonable.

Hell, even if they ignored question 2 entirely, and discussed question 3 even when the hypothesis was obviously off the wall, it'd still be an improvement.

Post a Comment

<< Home