Stumbling on Wins: Are NBA rebounds consistent because of talent or opportunities?
In David Berri and Martin Schmidt's "Stumbling on Wins," the authors paraphrase JC Bradbury on what makes a useful player-evaluation statistic. They write,
"First, one must look at how the measure connects to current outcomes. Then, one must look at the consistency of the measure over time."
Fair enough. But there's a third criterion that the authors need to add.
To see why, take, for instance, saves in baseball. By the first criterion, saves are obviously important -- that's why teams put their best reliever in the stopper role. By the second criterion, saves are very consistent -- for Yankee pitchers over the last 15 years, there's a very high correlation between saves last year and saves this year. There's a much higher year-to-year correlation for saves than any other measure -- ERA, WHIP, DIPS, even strikeouts.
Does that mean that saves are the most useful way to assign value to a reliever? Does it really mean that Mariano Rivera, with 30 saves, is fifteen times as talented at saving than some other guy in the bullpen with two saves? Of course not. The number of saves depends mostly on opportunities. And opportunities are not a characteristic of the player -- they're a characteristic of the manager, who decides how to assign the workload. Yankee pitchers are not consistent because Mariano Rivera has ten or more times as much "save talent" than any other Yankee. Rather, they're consistent because Yankee managers are consistent in giving Mariano almost all the save opportunities.
So, I propose:
Third, one should look at how much the measure is a true reflection of the player's talent, and how much is a measure of
factors outside the player's controlother factors unrelated to talent, such as opportunities.
(Note: above update 3/10/10 after suggestion from Guy in the comments.)
The reason I bring this up is that Berri and Schmidt use the first two criteria to defend why they assign the value of rebounds to the player who grabbed the ball:
"When we look at consistency, ... we see that 90% of the variation in a player's per-minute rebounds is explained by a player's per-minute rebounds the previous season. There appear to be no statistics in baseball or football that are as consistent as rebounds in basketball."
But that doesn't mean that rebounds are a useful statistic. They could be like saves -- it could be that the consistency is due to consistency of *opportunities*, not talent. And many people, myself included, have argued that, that certain players position themselves to compete for rebounds, and others do not. If player X is the designated "rebound guy" on the team, year after year, that would explain the consistency without providing evidence of talent.
If Berri and Schmidt are using the high r-squared to defend their hypothesis that rebounds are talent, then they don't succeed. Indeed, I think the high r-squared shows the opposite. Given that there's a certain amount of binomial randomness in who gets any particular rebound, there's a limit to how much consistency you'd be able to see if everyone had the same number of opportunities. The exceedingly high r-squared is an indication that the cause is probably more than just talent.
I should explain that better. Here's a baseball example. Suppose you computed the year-to-year correlation in hits among players who had at least 400 AB. The r-squared wouldn't be 1, because players don't hit the same every year. Someone who got 150 hits last year might get 160 next year, and vice-versa. Almost everyone would be in the 100 to 200 range, clustering maybe around 150. And you'd get an r-squared of maybe 0.2 (I'm guessing).
Now, suppose you include *every* player, not just those with 400AB. Now, players are much more likely to have similar results than last year. You get your typical regular who has 150 this year and 160 next year. Then you have your utility player who has 40 one year and 27 the next year. And you have your pitchers, who have 8 hits last year and 11 this year.
And so you have an r-squared that's much higher, maybe .7 or more. But the jump in r-squared is measuring consistency of *opportunity*, not talent.
So when you have one argument that rebounds are almost all talent, and another argument that rebounds have a huge component in there that reflects opportunity -- and then you get a high r-squared -- that result better supports the second argument, not the first.
Anyway, that's my main point. While I'm here, a couple of other smaller things I disagree with in that section of the book (pages 33 to 39):
1. The authors list the r-squareds for different measures in various sports; they find that their correlations for basketball are higher than other sports, and therefore argue that NBA statistics are more useful than others. But as I have pointed out before, you can't just use the raw r-squared or correlation coefficient as a measure of persistence of talent. The r-squared is dependent on many other factors -- most notably (as Tango has also pointed out many times), the length of a season. The authors found an r-squared of QB completion percentage of 24%, but a 90% r-squared for rebounding. That doesn't necessarily mean anything on its own. That's because the QB numbers are over 16 games and maybe a few hundred attempts, whereas the rebounding numbers are over 81 games and several thousand attempts. You just can't compare raw r-squared values that way, without first interpreting them.
2. When the authors say "there are no statistics in baseball as consistent as rebounds" ... well, they didn't include saves. I don't know for sure if saves have a higher r-squared or not, but I'd certainly be willing to bet they do.
3. The authors do indeed mention that the football season is shorter than the basketball season, but they don't seem to realize that that fact, in and of itself, affects the r-squareds. Instead, they have two alternative explanations. The first is that football statistics depend more on teammates than basketball statistics do -- which doesn't seem unreasonable, even without evidence backing it up.
But their second argument I'm not sure about. Berri and Schmidt argue that another reason professional football players are inconsistent is because of lack of experience. Why lack of experience? Because football players play only 16 games a season, so they're less experienced than basketball players, who play 81. Moreover, basketball players probably played pickup basketball every day as teenagers, while football players had to wait for organized leagues, because they couldn't just get a few friends together and play a real football game. So NBA players are more experienced because they've played a lot more basketball in their lives than NFL players have played football.
Well, it's probably true that NBA players have spent more time in games than NFL players, but I'm not sure why that's important. Why does playing fewer games (but still a lot of games -- a regular lot, rather than a huge lot) make you less consistent?
If I shoot foul shots for 15 minutes every day for a decade, and you shoot foul shots for 30 minutes every day for a decade, it would be expected that you'd be better than me. So maybe suppose I have more talent, so that even with less practice, I'm as good as you. Now: why would you really be more consistent than me? We're both 70% shooters, say. For me to be less consistent, I'd have to have more 60% years and more 80% years, while you'd hover closer to 70% every year. Why would that be the case? I suppose it's possible, but it doesn't seem plausible to me. Where's the evidence? Why would it matter that we got to the same point with different amounts of practice time?
Would I be more variable day to day, too, so I'd wind up having more 60% games and more 80% games? If that were true, if I'm sometimes 80% and sometimes 60%, my shots will be clustered together more than average. That means I'm more likely to make a shot after I've made my previous shot, and I'm more likely to miss a shot after I've missed the previous shot. That's the equivalent of saying that inexperienced players have a "hot hand" effect. But given that numerous "hot hand" studies have failed to find any effect, doesn't that suggest that all players are equally (binomially) consistent within their level of talent?
Now, I suppose you can make the argument that because of inexperience, football players are more likely to still be learning their technique, so they might be continuously improving. In that case, you might see a QB go from 20% to 25% in some measure more often than a basketball player goes from 20% to 25% in a similar measure. But if that were true, wouldn't the QB be improving throughout his entire career, given that he plays only 16 games a season? In that case, he'd still be improving into his 30s, so his age-related dropoff would be mitigated, and he would look *more* consistent later in his career. So there would be a balance: young players appearing less consistent between seasons, and old players appearing more consistent. The result should be a wash.
So I just don't understand how any inconsistency caused by "inexperience" would happen.
It looks to me like the authors are looking at the raw r-squareds, and then coming up with possible explanations for why they differ. But, as I said, they miss what is by far the biggest explanation, which is simply sample size. It's just the nature of how correlations work that the smaller the sample, the more luck dominates the results, and the lower the season-to-season r-squareds. I bet if you looked more closely than just listing correlation coefficients, you'd discover the difference in opportunities accounts for almost all the difference right there.
We can do a quick calculation.
The authors found that NFL QB completion percentage had a year-to-year r-squared of .24. Suppose that's because you have 24 points of variance caused by talent, and 76 points of variance caused by luck.
Now, suppose you played 80 games in an NFL season instead of 16 -- five times as many games, and close to the 82 games that the NBA plays. Now you'd still have 24 points of variance caused by talent, but only one-fifth the original variance caused by luck, which works out to 15.2 points. That would give you an r-squared of (24/39.2), or .61. That fits right in to what you get for similar NBA year-to-year r-squareds:
.47 NBA field goal percentage
.59 NBA free throw percentage
.61 NBA turnovers per minute
.61 QB completion percentage (projected)
.68 NBA steals per minute
.75 NBA points per minute
See? It's just opportunities. Those other explanations, about teammates an inexperience, might be factors too. But they're minor factors at best, and, without evidence, they're just speculation.
In fairness, the authors may have evidence for them that they're not telling us about. They don't say that the apparent inconsistency "may" be caused by inexperience, or that they "suspect" or "wonder" if that's the cause. Rather, they say:
"The inconsistency with respect to football statistics can be traced to two issues: inexperience and teammate interactions." [emphasis mine.]
So they imply they traced the effect, but they don't say *how* they did the tracing. So while I'm currently very skeptical that the apparent "inconsistency" is anything more than just straight sample size, I'm still willing to look at the authors' evidence, when they choose to show it.