R-squared and "twenty questions"

(This is a follow-up to two previous posts on r-squared.)

------
 
You've probably played the game "Twenty Questions." Here's how it works.  I choose a subject, which can be anything I want -- "baseball glove," or "Hillary Clinton".  Then, you have twenty "yes/no" questions to try to figure out what it is.

To win the game, you try to narrow it down as fast as you can, and as much as possible.  To start, you might ask the traditional question: "is it bigger than a bread box?"  That one question won't tell you what it is immediately, but it starts narrowing it down.  According to Wikipedia, other good questions are, "can I put it in my mouth?" and "does it involve technology for communications, entertainment, or work?"

Some questions are obviously bad.  Starting off by asking, "is it a DVD of a Sylvester Stallone movie?" is a waste of a question.  The answer is probably "no," in which case you're left pretty much where you started.  Of course, if it's a "yes," you're almost certain to get it, but the chances of that "yes" are pretty slim.

-----

So, now, here's a variation of the game.  This time, I'm going to pick a random American person.  Your job is to guess his or her 2011 income, and come as close as you can. 

Instead of twenty questions, I'm only going to give you one, for now.  However, it doesn't have to be a "yes/no" question -- if you want, it can be any question that can be answered by a number.  (You can't ask specifically about the salary, though.)  Once you get the answer, you take your guess at the income. 

What kind of question do you ask? 

Well, one good question might be, "how many years of education does the person have?"  You can then go by the general rule that the more education, the more likely they were to have a higher salary.

Or, you can ask, "what's the person's IQ?"  Again, you can assume that the higher a person's people's intelligence, the more likely they are to have a higher income.

But if you ask, "did they win a lottery jackpot last year?", that's a waste.  It's the like the Stallone DVD question.  Most of the time, the answer is no, and you get barely any useful information.  It's just not worth asking, just on the off-chance that you get a yes.

-----

That all makes sense, right?  Well, if you understand the strategy of the game of "Twenty Questions," you understand r-squared.  Because, r-squared is really just a measure of how good your question is.  Seriously -- the correspondence between the two is almost perfect.  The better the question, the higher the r-squared; and, the higher the r-squared, the better the question.

If you were to run a regression of IQ against income, you'd probably wind up with a decent r-squared -- maybe, I don't know, .15 or something.  That means that if you know a random person's IQ, you can knock 15 percent off your average error squared.  Maybe if you were completely ignorant, you would guess $30,000 for everyone.  But if you know IQ, you can guess $20,000 for low, $30,000 for average, and $40,000 for high, and your guesses would be closer. 

But, the bad question, the lottery question: the r-squared of that might only be .001.  Originally, you guessed $30,000.  Now, if you find out they didn't win the lottery, you guess $29,999, and are a tiny bit closer, on average.  If you find out they *did* win the lottery, then you guess, say, $5 million, and you come a lot closer that you would have before.  But that happens very infrequently -- so infrequently that you're square error is still going to be well over 99.99 percent of what it was without the question.

-----

I said the analogy between the game and r-squared was *almost* perfect.  If you care, here's how to make it exact:

After you ask the IQ question, you're given a table of all 300,000,000 people in the US, with their income and answer to your question (IQ).  Then, before I tell you the random person's IQ, you have to decide in advance what you're going to answer for each possible IQ, and your decision has to have each point of IQ worth the same amount of income (that is, it has to be linear, since it's a linear regression). 

Once you've decided, I give you the IQ, and we figure your answer, and your negative score is the square of how much you missed it by.

Under those rules, the analogy is exact: the r-squared exactly corresponds to how good a question you asked.

(Oh, and if you want to actually ask twenty questions instead of one ... that's just a multiple regression with 20 variables.)

-----

In the past, I've been critical of analyses that find a low r-squared, and assume that, therefore, there's only a weak relationship.  For instance, I've written about the study that found, in MLB, an r-squared of .18 for team payroll vs. team wins.  The authors of the study then said something like, "The r-squared is low.  Therefore, there's not much of a relationship.  Therefore, salary doesn't lead to wins."

Well, that's not right.  It's like saying, for the lottery example, "The r-squared is low.  Therefore, winning the lottery doesn't lead to more money."

That's obviously incorrect. 

The r-squared does NOT measure the direct relationship between the variables.  It just measures how good a question it is to ask about the one variable.

But, the thing is, if what you really want is the relationship between winning the lottery and getting rich ... well, that's easy.  Just look at the regression equation!

If you do that regression, the one on lottery winnings that gives you an r-squared of .001, you'll wind up with an equation like

Expected salary = $30,000 + $5,000,000 if they won the lottery.

It gives you exactly what you want -- winning the lottery is worth $5 million.  Why would you focus on the r-squared, when the exact answer is right there?  In fact, the r-squared is completely irrelevant!

I think the reason we sometimes focus on the r-squared, though, is that we make a false assumption.  It is true that (a) if you have a high r-squared, you have a strong relationship.  But it is NOT necessarily true that (b) if you *don't* have high r-squared, you *don't* have a strong relationship.  I think that maybe we just assume because (a) is true, (b) is also true.  But it's not.

------

So, in summary, three different ways to think about it:

--- One

The regression equation answers, "how much does winning the lottery affect income?"  [lots.]

The r-squared answers, "is asking about the lottery a good "twenty questions" way to help estimate income?"  [not very.]

--- Two

The regression equation answers, "how much does winning the lottery affect income?"  [lots.]

The r-squared answers, "when people differ in income, how much of that is because some of them won the lottery?" [not much.]

--- Three

The regression equation answers, "if you change the value of the lottery variable from "no" to "yes," how much does income change?  [lots.]

The r-squared answers, "if you change the value of the lottery value from one random person's to another random person's, how much does income change?"  [not much -- two random people are probably both "no", so the change is usually zero.]

------

If you've got any more good ones, let me know, and I'll add them in.