Why p-value isn't enough, reiterated
People are routinely tested for disease X, which 1 in 1000 people have overall. It is known that if the person has the disease, the test is correct 99% of the time. If the person does not have the disease, the test is also correct 99% of the time.
A patient goes to his doctor for the test. It comes out positive.
What is the probability that the patient has the disease?
Researchers routinely run studies to test unexpected hypotheses (such as: can outside prayer help cure disease?), of which 1 in 1000 tend to be true overall. It is known that if a hypothesis is true, a study correctly finds statistical significance 99% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 99% of the time.
A researcher tests one such unexpected hypothesis. He finds statistical significance.
What is the probability that the hypothesis is true?
Hat Tip: Inspired by Jeremy's last paragraph of comment #25, here.
P.S. Answer to question 1 (very slightly modified question, but the same answer) at my previous post, here.