## Monday, November 28, 2011

### Why p-value isn't enough, reiterated

Question 1:

People are routinely tested for disease X, which 1 in 1000 people have overall. It is known that if the person has the disease, the test is correct 99% of the time. If the person does not have the disease, the test is also correct 99% of the time.

A patient goes to his doctor for the test. It comes out positive.

What is the probability that the patient has the disease?

Question 2:

Researchers routinely run studies to test unexpected hypotheses (such as: can outside prayer help cure disease?), of which 1 in 1000 tend to be true overall. It is known that if a hypothesis is true, a study correctly finds statistical significance 99% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 99% of the time.

A researcher tests one such unexpected hypothesis. He finds statistical significance.

What is the probability that the hypothesis is true?

--------

Hat Tip: Inspired by Jeremy's last paragraph of comment #25, here.

--------

P.S. Answer to question 1 (very slightly modified question, but the same answer) at my previous post, here.

Labels: , ,

At Tuesday, November 29, 2011 11:59:00 AM,  mettle said...

If I may add a third:

Question 3:

Researchers more often run studies to test hypotheses that are possibly true (such as: do hearing aids help speech perception? did humans arrive on the American continent more than 14000 years ago? does dopamine improve parkinsons symptoms?), of which somewhere between 2 in 10 and 8 in 10 tend to be true overall. Indeed, this is what most of science does, since if something is 99.9% true or 99.9% untrue, most people won't bother testing it. It is known that if a hypothesis is true, a study correctly finds statistical significance 95% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 95% of the time.

A researcher tests one such unexpected hypothesis. He finds statistical significance.

What is the probability that the hypothesis is true?

At Tuesday, November 29, 2011 12:06:00 PM,  Phil Birnbaum said...