### Two nice statistical analogies

I'm always trying to find good analogies to help explain statistical topics. Here's a couple of good ones I've come across lately, that I'll add to the working list I keep in my brain.

**1.**

Here's Paul Bruno, explaining why r-squared is not necessarily a good indicator of whether or not something is actually important in real life:

"Consider 'access to breathable oxygen'. If you crunched the numbers, you would likely find that access to breathable oxygen accounts for very little – if any – of the variation in students' tests scores. This is because all students have roughly similar access to breathable oxygen. If all students have the same access to breathable oxygen, then access to breathable oxygen cannot 'explain' or 'account for' the differences in their test scores.

"Does this mean that access to breathable oxygen is unimportant for test scores? Obviously not. On the contrary: access to breathable oxygen is very important for kids’ test scores, and this is true even though access to breathable oxygen explains ≈ 0% of their variation."

Great way to explain it, and an easy way to understand why, if you want to see if a factor is important in a "breathable oxygen" kind of way, you need to look at the regression coefficient, not the r-squared.

**2.**

This sentence comes from Jordan Ellenberg's mathematics book, "How Not To Be Wrong," which I talked a bit about last post:

"The significance test is the detective, not the judge."

I like that analogy so much I wanted to start by putting it by itself ... but I should add the previous sentence for context:

"A statistically significant finding [only] gives you a clue, suggesting a promising place to focus your research energy.The significance test is the detective, not the judge." [page 161, emphasis in original]

(By the way, Ellenberg doesn't put a hyphen in the phrase "statistically-significant finding," but I normally do. Is the non-hyphen a standard one-off convention, like "Major League Baseball?")

(UPDATE: this question now answered in the comments.)

The point is: one in twenty experiments would produce a statistically-significant result just by random chance. So, statistical significance doesn't mean you can just leap to the conclusion that your hypothesis is true. You might be one of that "lucky" five percent. To be really confident, you need to wait for replications, or find other ways to explore further.

I had previously written an analogy that's kind of similar:

"Peer review is like the police deciding there's enough evidence to lay charges. Post-publication debate is like two lawyers arguing the case before a jury."

Well, it's really the district attorney who has the final say on whether to lay charges, right? In that light, I like Ellenberg's description of the police better than mine. Adding that in, here's the new version:

"Statistical significance is the detective confirming a connection between the suspect and the crime. Peer review is the district attorney deciding there's enough evidence to lay charges. Post-publication debate is the two lawyers arguing the case before a jury."

Much better, I think, with Ellenberg's formulation in there too.

Labels: Jordan Ellenberg, peer review, r-squared, significance, statistical significance, statistics

## 7 Comments:

Regarding your question about whether to hyphenate "statistically significant" when used as a compound modifier -- the general convention is that a hyphen is unnecessary when the modifier is an adverb ending in "-ly" because the suffix indicates to the reader that it is a modifier. See e.g., https://en.wikipedia.org/wiki/Compound_modifier#Exceptions_2

Thanks, Bart! That appears to be a standard English grammar rule of which I was unaware.

Still looks strange to me without the hyphen, but I guess I'll have to get used to it.

Is the hyphen allowed? You say "unnecessary," and Wikipedia says "not requir[ed]", which suggests I can still get away with using it.

It seems to me the oxygen situation is just a result of selection effects.

If we consider the universe of possible students, those without access to breathable oxygen have performed so poorly that they've fallen out of our sample.

I'm not sure if this is just the same point as arguing in favor of the regression coefficient (maybe it is?).

Well, you can't even get a regression coefficient for oxygen if it's exactly the same for every student. If not -- if some classrooms are 20.95% and some are 20.96% and some are 20.94% -- you'll get a near-zero coefficient that's not statistically significant.

In this case, the real problem is fitting a linear model to a non-linear effect. Oxygen doesn't matter much above 20 percent, but then it matters a lot once you start dropping below 20.

So, maybe, oxygen is a good example for something being vitally important but with a very low r-squared, but NOT a good example for using the coefficient.

For that, I'll stick with an example I used in an old post, that "pianos dropping on heads doesn't explain much of the difference in death rates." The low r-squared doesn't imply that pianos falling on heads aren't deadly -- but the regression coefficient shows they are.

The oxygen example isn't good, because a regression analysis is only valid within the range it is calculated in. So if the sample includes only rooms with oxygen between 20.94% and 20.96%, the regression doesn't tell you anything about what happens if the oxygen drops to 19%.

This really helped a lot and I got a great mark in my test thanks to this website.

It a good chance for students to acquire new knowledge!

Post a Comment

<< Home