Can money buy meat?

If you want to have meat in your diet, you have to spend money in the grocery store.  At least, that's the conventional wisdom.

But is that really true, or is it just a myth?  Let's look at the evidence.

I took a (made up) random sample of 30 shoppers in my local supermarket earlier this year.  I ran a regression to predict the total amount of meat they had, from the total amount of money they spent.  It did turn out that the York family, the one who spent the most money by far, did get the most meat.  And, that there was a positive slope, meaning that spending more money leads to more meat.

However, there was one very important issue: the link between meat and money was not statistically significant.  In other words, we can't argue that money spent and meat obtained are actually related to each other in 2012.

It's easy to understand why we got this result.  Some of the lowest-spending families wound up with a lot of meat -- one was stocking up for a BBQ, and one owned a cattle ranch.  And a few rich-spending families barely had any meat in their houses at all -- they paid a lot for only a few ounces of filet mignon.

But 2012 isn't typical.  When I (pretended that I) did the same experiment for other years, I got statistically significant results.  But even for those years, explanatory power is quite low.  Only 17 percent of the variation in meat over the last 25 years is explained by variation in spending.  So much of the variation in meat obtained is not explained by how much money was spent.

And if you look at each year individually -- as the following table (with made-up numbers) illustrates -- the power of money buying meat seems to vary quite a bit:

2012: not significant
2011: r-squared = .17, p = .01
2010: r-squared = .13, p = .04
2009: r-squared = .21, p = .02
2008: r-squared = .10, p = .06
2007: r-squared = .25, p = .00
2006: r-squared = .29, p = .00
2005: r-squared = .24, p = .00
2004: r-squared = .29, p = .00
2003: r-squared = .18, p = .02
2002: r-squared = .20, p = .01
2001: r-squared = .10, p = .04
2000: r-squared = .10, p = .04
1999: r-squared = .50, p = .00
1998: r-squared = .47, p = .00
1997: r-squared = .22, p = .01
1996: r-squared = .34, p = .00
1995: not significant
1994: r-squared = .16, p = .07
1993: r-squared = .09, p = .09
1992: not significant
1991: not significant
1990: not significant
1989: not significant
1988: r-squared = .18, p = .00

From 1996 to 2001, supermarket spending and meat were statistically linked each and every year.  However, explanatory power varied.  If we look at shoppers before 1993, we see four years where where the relationship was again not significant.

So here is the big question: Why is the relationship not stronger?  One would think that as shoppers spend more, they would wind up with more meat.  But, often, that's not what we see in the data.

One issue is that you can get meat at other places than the supermarket -- butchers, say, or gifts, or the slaughter of animals you own yourself.  Another issue is that it's hard to predict what shoppers will buy any given week. 

But, does the result from 2012 show that spending and meat will not be statistically related in future?  We don't know.  But what we *do* know is that spending does not guarantee a shopper more meat.

That's the nature of shopping.  Sometimes you don't get enough meat, and, it seems, no amount of spending can change that reality.

--------

So: do you believe me?  Do you believe that how much meat you have in 2012 doesn't depend on how much money you spend?  I hope not.

What, specifically, is wrong with the logic?  Lots of things, many of which I've written about before.

------

1.  Even if you don't get a statistically-significant relationship between spending and meat, that does NOT mean that you "can't argue that money spent and meat obtained are actually related to each other".  Of course you can!  Lack of significance just means that, in one specific, narrow, sense, you don't have enough grounds to assert a relationship *on this evidence alone*. 

But, of course, there's LOTS of other evidence that meat and spending are related.  For one thing, there's a big sign in front of the steaks, that says, "$7.99 per pound."  For another thing, millions of people will tell you that they have successfully exchanged money for meat.

You can only argue that there's no relationship if you choose to ignore all those things.  Which, I hope, you wouldn't.

2.  The implicit assumption in the argument is that every year is different.  That is: money bought meat in 1993 and 1994, but not in 1992 or 1995.  Why would you assume that, that the nature of shopping changes so often and so much that you can buy meat in 1994, but not 1995?  If we're going to assert that, we need some kind of explanation of how that could be plausible.

3.  Also, if you're interested in statistical significance, shouldn't you care about checking if 1994 and 1995 are actually significantly different?  What do you do if there's not statistically significant difference between them, as there probably isn't?  How can you say money bought meat one year, but not the other, when the p value of the difference is very high? 

You have a contradiction:

1994 is significantly different from zero
1995 is NOT significantly different from zero
1994 is NOT significantly different from 1995.

Isn't it just as reasonable to say there's no difference, than to say that money bought meat in 1994 but not 1995?  Even if you're depending on statistical significance, you still have to make an argument.

4.  Why use "different from zero" as your significance criterion anyway?  In this particular situation, there is no real reason to think that zero is more likely than any other value -- and, in fact, there's very, very good reason to believe it's different, unless you have good reason to believe that big spenders don't buy more meat than the guy in the express lane with one item. 

In some cases, like whether prayer cures cancer, a default of zero makes sense.  But not here.  Saying, "we'll assume money can't buy meat until we see strong evidence otherwise" ... well, that's just privileging your hypothesis.

5.  If you get a value that's significant in the real world sense, but isn't statistically significant, you need more data.  You can say, "I don't have enough evidence."  You can say, "there isn't enough evidence HERE."  But you can't just assume that there's no relationship.  Otherwise, it would be easy to argue that smoking is harmless.  You just do a double-blind study that's really small.  And then you say, "even though 40 percent of the five smokers got lung cancer, and only 20 percent of the five non-smokers got cancer, we got an r-squared of only .1, and that's not statistically significant.  So, there's no evidence that smoking causes cancer."

Yes, the evidence of THAT study is weak.  But that's because the study is too small.  Twice the risk of cancer is plausible, and important, and you can't just dismiss it because you deliberately designed your study the way you did.  And there are lots and lots of other studies showing a link, and a biological mechanism by which it happens.

If you did that study, and you deliberately ignore all the other evidence, than it's fair to say that YOU can't conclude that smoking causes cancer.  But WE can certainly conclude it. 

Similarly, if all you know is that within the dataset of your 30 individuals, the correlation between meat and spending is low ... YOU can conclude you don't have evidence that meat can be bought.  But WE cannot, because WE have other evidence: we've been to a supermarket.  We know something about how the market for meat works.

6.  Even noting that the r-squareds jump around a bit -- and that the jumping around is statistically significant -- that doesn't necessarily mean that the relationship between money and meat has changed.  The r-squared depends not just on the relationship, but on the scattering of the values in the actual dataset. 

So an increasing r-squared could simply indicate a larger variation in overall spending.  Think about it ... if some families spend $1, and some spend $1000, it should be easier to notice the relationship between spending and meat, which means a large r-squared.  But if everyone spends exactly $100, it's going to be harder -- a lower r-squared -- even if money buys the same amount of meat as always.

So when you see a changing r-squared, you can't really be sure what's going on.  It would be better to look at the coefficient estimate of the regression equation, rather than the r-squared.

In fact, for any arbitrarily low r-squared, I can construct a dataset where the coefficient is as statistically significant as you like, and meat costs any amount per pound you like.  (I thought I wrote about this fact before, but I can't find it.)

7.  Even though an r-squared less than .10 may look small intuitively, it probably isn't.  A low-looking r-squared can be very important in real life.  You can't just say ".10 is small".  You have to *argue* that, in context, it's small.

If you did a regression of suicide vs. life expectancy, the r-squared would be at least as small as the ones here.  But suicide and life expectancy are most definitely linked. 

You have to interpret the r-squared for what it is.  It's not really an indicator of how easily money buys meat.  It's a measure of how well you can predict meat from money, *relative to all the other things* that help you predict meat*. 

If cancer kills a million people, and suicide kills 10, the r-squared between suicide and life expectancy will be low, because suicide is being compared to cancer.  That's true even though a single suicide has a bigger effect on life expectancy than a single case of cancer.

8.  You'll notice how large the relationship really is if you look at the r, instead of the r-squared.  The square root of .17 is .41.  That means that for every standard deviation difference in money spent, you get 41% of a standard deviation in meat obtained.  That's a pretty strong association: if you move two inches to the right on the supermarket-spending bell curve, you move 2/5 of an inch to the right on the meat curve.

9.  The r-squared doesn't really tell you whether meat CAN be increased by increasing supermarket spending.  It tells you how much meat WAS increased with supermarket spending.  Obviously, you'd expect an imperfect correlation.  People use money on all kinds of things -- TVs, cars, tofu, vegetables.  They get meat from sources -- their own animals, gifts, butchers -- other than supermarkets.  And, they buy different kinds and forms of meat, at various prices: steaks, hamburger, spam, TV dinners, dog food, and so on.

Given all that variation, *of course* you're going to find a less-than-100% correlation between supermarket spending and meat purchased.  That doesn't mean that there's no cause-and-effect relationship of deliberately spending more money and getting more meat, at the margin.

This is easier to understand if we look at something other than meat -- say, hair. 

Hair CAN be bought for money.  If you're bald, and you want to have hair, you can write a check to Hair Club For Men, and they'll add hair to your head.  But if you look at whether hair HAS BEEN bought for money, very little of it has -- most of it we got free, from God.  The r-squared between "hairs on head" and "money spent" is low, because most hair is not bought for money, and most money is not spent on hair. 

But if you have money, and you choose to buy hair, you'll get it.  

Same for meat.

------

And so, the botttom line is: even if we get a legitimately small correlation, you CANNOT say that "no amount of spending can buy meat."  That's exactly like noting that the correlation between shooting yourself in the head and lifespan is small, and saying, "no amount of shooting yourself in the head can change your life expectancy." 

That's just not true, because it's just not what r-squared means.

-------






(Inspiration: this Freakonomics post.)