Privileging the null hypothesis

The fallacy of "privileging the hypothesis" occurs when you concentrate on one particular possibility, without having any logically correct reason for preferring it to all the others.

The general idea is: if you want to put forth a hypothesis for consideration, it's incumbent on you to explain why that particular hypothesis is worthy of attention, and not any other ones.  The "Less Wrong" website gives an example:

"Privileging the hypothesis" is the fallacy committed by a creationist who points out a purported flaw in standard evolutionary theory, and brings forward the Trinity to fill the gap - rather than a billion other deities or a trillion other naturalistic hypotheses. Actually, without evidence already in hand that points to the Trinity specifically, one cannot justify raising that particular hypothesis to explicit attention rather than a trillion others.

I think this is right.  And, therefore, I think that many statistical studies -- the traditional, frequentist, academic kind -- are committing this fallacy on a regular basis.

Specifically: most of these studies test whether a certain variable is significantly different from zero.  If it isn't, they assume that zero is the correct value.

That's privileging the "zero" hypothesis the same way the example privileges the "Trinity" hypothesis, isn't it?  The study comes up with a confidence interval, which includes zero, but, by definition, includes an uncountable infinity of other values.  And then it says, "since zero is possible, that's what we're going to assume." 

That's not the right thing to do -- unless you can explain why the "zero" hypothesis is particularly worthy.

Often times, it obviously IS worthy.  Carl Sagan wrote that "extraordinary claims require extraordinary evidence."  If a non-zero coefficient is an extraordinary claim, then, of course, there's no problem.

For instance, suppose a subject in an ESP study gets 3 percent more correct guesses than you'd expect by chance.  That's not significantly different than zero percent.  In that case, you're absolutely justified in assuming the real effect is zero. 

"ESP exists" is an extraordinary claim.  A p-value of, say, .25 is not extraordinary evidence.  So, you're not "privileging the hypothesis," in the sense of giving it undeserved consideration.  You do have a logically correct reason for preferring it.

------

But ... that's not always the case.  Suppose you want to test whether scouts can effectively evaluate NHL draft prospects.  So you find 50 scouts, and you randomly choose two prospects, and ask them which one is more likely to succeed in the NHL.  If scouting were random, you'd expect 25 out of the 50 scouts to be correct.  Suppose it turns out that 27 are correct, which, again, isn't statistically significant. 

Should you now conclude that scouts' picks are no better than random chance -- that scouts are worthless?

I don't think you should. 

Because, why not start with a different null hypothesis, one that says that scouts are always 54.3 percent right?  If you do that, you'll again fail to find statistical significance.  Then, just like in the "zero" case, you say, "there's no evidence that 54.3 percent is wrong, so we will assume it's right."

That second one sounds silly, doesn't it?  It's obvious that a null hypothesis "scouts are good for exactly 4.3 percent" is arbitrary.  But, "Scouts are no good at all" seems ... better, somehow.

Why should we favor one over the other?  Specifically: why do we judge that this null hypothesis is good, but that other null hypothesis is bad?

It's not just the number zero.  Because, obviously, we can easily set up this study so that the null hypothesis is 50 (percent), or 25 (out of fifty), and we'll still think that's a better hypothesis than 54.3 percent.

Also, you can set up any hypothesis you want, to make the null zero.  Suppose I want to "prove" that third basemen live exactly 6.3 percent longer than second basemen.  I say, "John Smith believes third basemen live 6.3 percent longer.  So I built that into the model, and added another parameter for how much John is off by.  And, look, the other parameter isn't significantly different from zero.  So, while others might suggest that the other parameter should be negative 6.3 percent, there's no proof of that.  So we should assume that it's zero, and therefore that third basemen live 6.3 percent longer than second basemen."

That should make us very uncomfortable.

So if it's not the *number* zero, is it, perhaps, but the hypothesis of zero *effect*?  That is, the hypothesis that a certain variable doesn't matter, regardless of whether we represent that with a zero or not. 

I don't think that's it either.  Suppose I find a bunch of random people, and use regression to predict the amount of money in their pocket based on the number of nickels, dimes, and quarters they're carrying.  And the estimate for nickels works out to 4 cents, but with an SD of 5 cents, so it's not statistically significantly different from zero.

In this case, nobody would assume the zero is true.  Nobody would say, "nickels do not appear to influence the amount of money someone is carrying."  

It would be obvious that, in this case, the null hypothesis of "zero effect" isn't appropriate.

-----

So what is it?  Well, as I've argued before, it's common sense.  The null hypothesis has to be the one that human experience thinks is very much more likely to be true.  And that's often zero. 

If you're testing a possible cancer drug, chances are it's not going to work; even after research, there are hundreds of useless compounds for every useful one.  So, the chance that this one will work is small, and zero is reasonable. 

If people had ESP, we'd see a lot of quadrillionaires in the world, so common sense suggests a high probability that ESP is zero. 

But what about scouting?  Why does it seem OK to use the null hypothesis of zero?

Perhaps it's because zero just seems like it should be more likely than any other single value.  It might still be a longshot, that scouts don't know what they're doing -- maybe you consider it a 5 percent chance.  That's still higher than the chance that scouts are worth exactly 2.156833 percentage points.  Zero is the value that's both more likely, and less arbitrary.

But ... still, it depends on what you think is common sense, which is based on your experience.  If you're an economist who's just finished reading reams of studies that show nobody can beat the stock market, you might think it reasonable that maybe scouts can't evaluate players very well either. 

On the other hand, if you're an NHL executive, you feel, from experience, that you absolutely *know* that scouts can often see through the statistics and tell who's better.  To you, the null hypothesis that scouts are worth zero will seem as absurd as the null hypothesis that nickels are worth zero.

What happens, then, when a study and comes up with a confidence interval of, say, (-5 percent, 30 percent)?  Well, if the null hypothesis were zero, the researcher might say, "scouts do not appear to have any effect."  And the GM will say, "That's silly.  You should have used the null hypothesis of around 10 percent, which all us GMs believe from experience and common sense.  Your confidence interval actually fails to reject our null hypothesis too."

Which is another reason I say: you have to make an argument.  You can't just formulaically decide to use zero, and ignore common sense.  Rather, you have to argue that zero is an appropriate default hypothesis -- that, in your study, zero has *earned* its status.

But ... for scouting, I don't think you can do that.  There are strong, plausible reasons to assume that scouts have value.  If you ignore that, and insist on a null hypothesis of zero, you're begging the question.  You're committing the fallacy of privileging your null hypothesis.