Thursday, April 24, 2008

Racial bias and NBA referees -- a follow-up study

In comments to one of my posts on the Hamermesh study, commenter Guy mentioned a study on a similar subject, NBA referees and race. I hadn't seen that paper before.

The article is called "
Racial Bias in the NBA: Implications in Betting Markets." It's a follow-up to last year's famous Joe Price/Justin Wolfers study (which I reviewed back then in three parts). This one, now also co-authored by Tim Larsen (so I'll refer to the authors as "LPW"), examines the impact of race on NBA betting markets -- that is, the Vegas line.

The paper comes to similar conclusions to last year's study – that referees appear to be either biased in favor of players of their own race, or biased against players of a different race (or both – it's impossible to tell which). Also, it suggests a profitable betting strategy to take advantage of this bias.

However, I disagree that the study has found referee bias. The way I see it, there is only one finding in this paper that suggests such a bias, and it's significant only at the 10% level. I'll get to that finding in a bit, but I'm going to start by listing some of the paper's findings that do NOT have to do with racial bias. Some of these, actually, are very interesting findings, and I don't think I've seen them before.


1. Black referees favor the home team more than white referees.

In Panel C of Table 2 of the paper, the authors do a regression on the home team's winning margin, against the relative racial composition of the two teams. On average, the home team's winning margin is:

3.167 when there are three white referees
3.541 when there are two white referees [and one black]
3.631 when there is one white referee [and two black]
4.591 when there are no white referees [so three black].

The differences are not statistically significant, but it is interesting that each one is higher than the last, and that the extremes are significant in a basketball sense (1.4 points difference is a lot).

2. White players, on average, are better than black players.

Again, in the same table, the authors show the point differential between when one team has 100% black players, and the other team has 100% white players. Actually, this is extrapolated from real life – a more realistic explanation is that the study found what happened when one team has (say) one "extra" white player, and multiplied it by 5. In any case, here are the results, expressed in more points for the all-white team (relative to the all-black team):

6.128 more points when all three referees are white
5.140 more points when two referees are white
3.673 more points when one referee is white
0.903 more points when no referees are white.

Obviously, "whiter" teams, overall, are better than "blacker" teams. A 100% white team seems to be about 5 points better than a 100% black team, so the difference is about 1 point per "extra" white player.

Also, there's some evidence here of the racial bias the authors are searching for. It does look like white refs are easier on white teams (and vice-versa) This is indeed suggestive. However, none of the differences are statistically significant. The standard errors of the four estimates above are 1.1, 0.9, 1.3, and 3.2 points, respectively.

3. The Vegas line underestimates white teams' chances of winning.

We know that from Panel D of Table 2, where LPW show how the point spread varies by the race of referees. When one team has 100% white players, and the other has 100% black players, the spread in favor of the white team is:

2.976 points when all three referees are white
3.593 points when two referees are white
2.413 points when one referee is white
2.144 points when no referees are white.

Compare these spread numbers to the actual numbers above: except for the no-white-refs case (which is only 3% of games), the spread underestimates how much the white teams will win by. In real life, the difference was about 5 points. In the spread, it's about 3 points.

This appears to contradict the hypothesis that betting markets are efficient – they appear to be a couple of points off in these cases.

Since the Vegas line doesn't try to pick the correct score, but, rather, tries to pick the spread at which the betting on both sides will be even, the most likely explanation here is that bettors, as a group, are slightly biased in favor of teams with more black players. That could be because black players are more likely to be superstars (and have more fans who bet on their teams). It could be that cities with more basketball bettors happen to have more black players. It is also possible that basketball fans are biased against white players (although I prefer not to accuse anyone of racial bias, even bias on who to be a fan of, except when all other explanations have been considered and found wanting).


Okay, those are three statistical effects the study found that have nothing to do with racial bias among refs. I'll repeat them:

1. Black refs tend to slightly favor the home team.
2. White players tend to be slightly better than black players.
3. The betting line underrates whiter teams.


Now, let's go to the bias tests.

In Table 3 of the paper, LPW run a regression to predict various aspects of their sample of games. They correct for the race of the teams and the races of the referees. Then, after making those corrections, they look at what's left – specifically, what happens when the race of the refs matches the race of the players, and when it doesn't.

(Technical note: they define the "same-race" parameter as

% white refs * (% home black players - % visitor black players)

This reaches its minimum of -1.00 when (a) all the refs are white, (b) all the home players are white, and (c) all the road players are black. It's +1.00 when (b) and (c) are reversed.)

Here are the differences in this most extreme case (all white team, all black team, all white refs):

A 16% additional chance of beating the spread
An extra 3.3 points relative to the spread
An extra 4.1 points relative to the other team

(The difference between the 3.3 and 4.1 comes from the spread itself being 0.8 points lower.)

These seem like big differences. However, none of these numbers are highly significant. All three are a little less than 2 standard deviations away from zero, so they're only significant at the 10% level.

In terms of basketball, are they significant? Well, in the extreme case, yes. But in real life, you're not going to have these extreme conditions very often, if at all. No team is 100% white, and no team is 100% black. More typically, the home team might have one extra white player (out of 5, adjusted for expected playing time), and there might be two white refs on average. That would give a parameter of (66% * 20%), which is about .13. So the effects would be 13% of the ones above That means the extra chance of beating the spread would be closer to one percentage point, not 16.

Still, that's something, even though (as I said) statistically significant only at the 10% level.


Now, for Table 4, the authors suggest a betting strategy: bet on the team with the greatest racial similarity to the refereeing crew. If there are more white refs than blacks, bet on the whiter team. Otherwise, bet on the blacker team.

Here's the percentage of bets you'll win:

3 white refs: 51.37%
2 white refs: 50.99%
1 white refs: 50.39%
0 white refs: 52.53%

In all cases, you win more than half your bets!

However: remember finding number 3: the betting line underrates whiter teams. The "3 white refs" and "2 white refs" cases involve betting on the whiter team. So it's likely that what's happening here is not referee bias, but Vegas line inefficiency! Betting on the whiter team is generally a better than 50-50 shot.

The other two cases are not as easily explained – now, you're betting on the blacker team, and those should be slightly below-average bets. But they're not – they're better than even wagers. What's going on?

Probably statisical insignificance. The "0 white refs" sample is very small, and the 52.53% figure is only one standard error from 50%. As for the "1 white ref" sample, there's an interesting result: the more black players you're betting on, the lower your chance of winning the bet. If the blacker team is blacker by more than half a player (out of 5), your winning percentage is less than 50%. It's only when one team is *slightly* blacker – less than half a player – that the bet is a winning one. This is certainly not consistent with the idea that it's racial bias causing the effect.

In the 3-white-refs and 2-white-refs cases, the odds of winning go up the more white players there are on the team you're betting. Since this is consistent with white players being better, you can't really tell if there's referee bias happening there too.


Finally, we come to Table 5, where the authors consider various betting strategies.

They propose a simple rule: wait for an all-black or all-white refereeing crew. Then, bet on the team whose racial composition better matches the referees. Doing this, they find you win 51.48% of bets. This is significant at the 5% level (a tiny bit over 2 SD).

But, by this rule, one team might be only very, very slightly blacker (or whiter) than the other. Shouldn't you improve your odds if you also wait for a large discrepancy in team race? It turns out that you do. If you don't bet until you find a race difference is over 10% (one half a player), you win 51.82% of your bets.

If you wait for a 20% advantage (one player), you move up to 54.34%. If you wait for a 30% advantage, it's 56.30%. And if you wait for a 40% advantage, you'll only bet 160 times in 14 years. But you'll win 61.88% of those bets!

Here's all this in a table. Remember, these are games with three same-race refs only:

51.48% -- all games
51.82% -- 10% player race advantage
54.34% -- 20% player race advantage
56.30% -- 30% player race advantage
61.88% -- 40% player race advantage

The first three of these are statistically significant at 5%; the fourth one is significant at 1%. The last one is also significant at 1% (although the authors didn't mark it as such).

So does all this significance indicate racial bias? Nope.

Remember that there are many more white referees than black. Three white refs happens 28.1% of the time, but, because there are so few black referees, three black refs in one game happens only 3.0% of the time. So if you wait for three same-race refs, over 90% of the time, those three refs will be white.

And since you bet on teams that match the referees, that means that 90% of the time, you'll be betting on the whiter teams. And we saw, from number 3, that the Vegas line underrates white teams. So you're probably winning because your teams are whiter, not because of the referees.

To check for sure, you need to (for instance) find all the games, not just the ones with same-race refs, where the race difference is 30% or more. Then see what happens when you bet on the whiter team. I bet it would be not too far off from the 56.30% observed with only white refs. And I'd bet the difference between the two would not be statistically significant.


Bottom line: in this study, the only evidence of same-race bias came in Table 3. The difference one extra white player makes (assuming two white refs) is:

-- an extra .022 chance of beating the spread
-- an extra .54 points relative to the opponent

That has reasonable basketball significance – half a point. But even so, I hesitate to accuse anyone of racial bias on the basis of a significance level of only 10%.

Labels: , ,


At Thursday, April 24, 2008 10:55:00 AM, Anonymous Anonymous said...

Phil: More spot-on analysis. I hope you will send Justin W. a note with this analysis, and invite him to post a reply here. Would be an interesting exchange.

Your point about white teams beating the spread is probably the most important, and it’s quite surprising to me that the authors failed to note this. I’d say it’s the more important and interesting “market failure” identified in their data. If we look at games in which there is a non-trivial difference in racial makeup of the teams (>10%), you would win 50.87% of the time by betting on the team racially similar to the refs, but you would win 51.19% if you just bet on the whiter team in those same games! It’s true that betting on the ref/team similarity is slightly better than betting on white teams over all games (51.01 vs 50.69), but that’s mainly a function of teams that are just slightly more black (<.1 spread) winning 52.5% in front of majority black ref crews (probably a fluke result, as you say).

It would also be interesting to know if white teams’ overperforming vs. the spread is the result of racial bias by bettors, as you speculate, or some other factor. For example, it also appears possible that stronger teams may beat the spread more than 50% of the time, and on average whiter teams will be the stronger team. The authors don’t report wins-vs-spread based on teams’ win%, but Table 1 shows that away-favorites (who will always be the stronger team, given impact of home court advantage) beat the spread 51.01% of the time. So that could be a factor.

I understand why the authors are interesting in beating the spread, as an exploration of efficiency of betting markets. But to test the underlying claim of ref racial bias, it’s also important to know if “whiter” teams win more than they “should” as the # of white refs increases. Here is the winning % of the “whiter” team, based on racial difference of teams and ref crew (based on authors’ table 4, collapsing categories to avoid cells with very small n): [Phil: help me with formatting if you can!]

TmDif 3WR 2WR 0-1WR All
0-.1 50.89 50.75 47.53 50.02
.1-.2 49.29 51.45 51.67 50.91
>.2 54.36 50.83 49.54 51.51

All>.1 51.73 51.17 50.65 51.19
All 51.37 50.99 49.35 50.69

There’s a hint of racial impact here – mostly in the 3WR/>.2 cell -- but it’s very hard to say it demonstrates a clear pattern of bias. For example, under majority black crews, a clear racial disparity actually results in more wins for the whiter team. The authors’ conclusion that swapping out 1 white ref for 1 black ref in a typical game would change the win% for each team by 1% seems overstated. If we look at games where the teams really differ racially (spread >.1), it’s more like .5%.

To really interpret the table, we’d want to know the expected win probability of the whiter teams in each cell, based on the two teams’ seasonal win% at home/away. I hope Justin will provide that at some point. However, if the whiter team’s true win expectancy is close to 50.7 in all cells, it may be that at least in the case of all-white crews (and maybe all-black crews) we’re seeing some bias.

Finally, I think the question of single-race crews is an interesting one for the authors to look at. It appears that much – perhaps all – of any bias occurs there, and one can imagine why that might be true. I believe that Wolfers and Price have the NBA’s own data, with data on individual refs. If they can, it would be useful to look to see if refs behave differently when matched with 2 other refs of their own race, vs. mixed-race crews.

At Thursday, April 24, 2008 11:21:00 AM, Anonymous Anonymous said...

Oops, a correction. We shouldn't expect to see the same win expectancy of 50.7 for the whiter team in all cells, even if distribution of matchups is perfectly random, because it should vary based on racial gap of teams (for example, 51.5% if racial gap >.2).

* *

One other observation on the table: in games with a typical racial spread (.1 to .2, n=4390), the win% for the whiter team actually falls as number of white refs increases.

At Monday, April 28, 2008 11:13:00 AM, Blogger Phil Birnbaum said...

Hi, Guy,

sorry for not replying until now ... I never got Blogger's e-mail announcing your comments. Give me an hour or two to digest.

At Monday, April 28, 2008 1:24:00 PM, Blogger Phil Birnbaum said...

Okay, I'm back.

>If we look at games in which there is a non-trivial difference in racial makeup of the teams (>10%), you would win 50.87% of the time by betting on the team racially similar to the refs, but you would win 51.19% if you just bet on the whiter team in those same games!

Good way of putting it ... I didn't catch that!

I also like the theory that whiter teams beat the spread because *stronger* teams beat the spread. I always prefer non-racial explanations to racial ones, unless the KKK or explicit affirmative action are involved.

Thanks for the table (which I don't know how to format, sorry). It's good to split the rows into independent categories instead of overlapping ones. Non-independent rows (like in Table 4) can fool people (read: me) into seeing more of a trend than actually exists.

At Monday, April 28, 2008 2:06:00 PM, Anonymous Anonymous said...

Phil: I think Table 4 gives you the answer to this question you posed: "To check for sure, you need to (for instance) find all the games, not just the ones with same-race refs, where the race difference is 30% or more. Then see what happens when you bet on the whiter team." If I've done the math right, the white team wins 52.93% of these games overall. That compares to 56.76% in all-white-ref games. I don't know if that's a significant difference or not. But I don't think you can simply use the authors' standard errors to draw a conclusion. For example, they report a standard error of 2.34% on 451 all-white-ref games with player racial disparity >30%. But I beleive that's only correct if the 451 observations are independent. That's almost certainly not true. In reality, there have probably been a handful of very "white" NBA teams since 1992 that account for a large proportion of the games in this cell (and to a lesser degree, some 95% black teams may be overrepresented as well). It's not like players are randomly assigned to a new team each day, as the refs are.

Now, there's no reason to assume the 30%+ more white teams playing in front of 3 white refs were any better than the white teams playing in front of majority-black ref crews. But since we're probably only talking about a few teams with such a high white proportion, there could easily be a disparity. That's why I hope Justin will provide the expected white win% for each cell (or the actual win% for the teams).

* *

I also wanted to add a comment on the magnitude of the apparent bias. The impact of one extra white ref in games with a meaningful disparity (>.1) appears to be about +.005 for the whiter team. Let's assume the average racial gap in these games is about .2, or about one player. So a 1-ref change means a white player gets an extra .005 win per game for his team. Presumably, the most fair racial split for ref crews would be 50%, so all players get same-race preference exactly half the time (leaving aside players who are neither white nor black). Since ref crews are now 67% white, that means a white player today gets a "bias bonus" of .0025. That means about .15 wins per season for a full-time player, or one win every 7 years.

At Monday, April 28, 2008 2:18:00 PM, Blogger Phil Birnbaum said...

Excellent, teams 30% whiter (they must have used Tide!) win 52.93% overall, vs. 56.76% with three white refs.

With a standard error of 2.34%, that's less than significant.

I agree with you in theory that the 2.34% figure should be higher because the results may not be completely indpendent, but I'd bet that the sample size is high enough that any difference is small. I need to think a bit more about that, though.

At Monday, April 28, 2008 2:34:00 PM, Anonymous Anonymous said...

I confess I'm not sure about how to deal with the independence issue. But it's not far-fetched to think that 3 or 4 squads, each playing together for several years, played in almost all of the 30%+ disparity games (which of course requires a minimum 30% white composition). Let's say that "white" teams A and B are .650teams, which C and D are .400 teams. It's not hard to imagine that, just by chance, the all-white crew games had more A and B games (and perhaps weaker "black" opponents).

I'm not saying that's the case. I don't know. If I had to guess, I'd say there's probably a little bias when you assemble single-race crews (but little or none in mixed-race crews). But I don't think we know that based on the data we've seen so far.

* *

Man, I must really be getting old. I can barely read the word verification characters on this site.

At Monday, April 28, 2008 2:51:00 PM, Blogger Phil Birnbaum said...

Yes, it's possible that the all-white crews had better white teams, but there were 451 of those games. Even if (say) 251 were against the 2 good teams, and 200 against the 2 bad, that's only 25/451 extra games of a .200 difference. That's 1/9 of .200, which is .022.

And that's a pretty extreme case. Plus, you're not comparing it to 50/50: you're comparing it to what might have happened if there were more than 4 teams of mostly white guys.

So I'm not too worried about that, but, sure, it's something you could check out.


I think they made the word verifications harder. Someone told me he thought he saw news that someone had figured out software to crack some of the easier ones -- maybe that has something to do with it.


Post a Comment

<< Home