Monday, May 04, 2009

NBA's debunking of referee bias flawed, says researcher

A couple of years ago, Joseph Price and Justin Wolfers came out with a study (.pdf) that found a bit of racial bias among NBA referees. The more white referees on the court, the more fouls were called against black players (relative to white players). And vice-versa: more black referees meant relatively more fouls against whites. (The vice-versa has to be true by definition, since the white referees can only be judged relative to their black peers.)

At the time, I summarized the Price/Wolfers study here, here, and here.

When that study came out, the NBA wasn't pleased, and David Stern commissioned a counter-study to refute it. I haven't seen that NBA study, but David Berri has, and he recently wrote about it on his blog. Apparently, it's very amateurish: according to Berri, the researchers estimated the results twice, once with a dummy variable for black refs, and again with a dummy variable for white refs. But, as Berri points out, that will just give the same results -- it doesn't matter which way you define the dummy variable, and "that suggests that the person doing the work for the NBA didn't understand dummy variables."

I wish I had a copy of the NBA study ... I haven't been able to find it online, and I think I read somewhere that it was only distributed to a select group of readers. One of them was Joseph Price, the author of the original study. Berri's post was based on a visit Price made to Berri's class. In addition to the dummy variable issue, Berri reports that Price said "that was just the beginning" of the problems with the NBA study, but gives no further details.

Anyone know where the NBA response can be found? I'd love to take a look at it firsthand.

P.S. Berri gets in a shot at non-academic researchers:

"Unfortunately, the quality of work offered by the consulting firm [the NBA hired] was consistent with what you sometimes see in on-line studies. In other words, it wasn’t very good. In fact, much of it consisted of mistakes you would not expect an undergraduate in econometrics to make."

Labels: , , ,


At Tuesday, May 05, 2009 8:53:00 AM, Anonymous Jake Russ said...


I followed this story while it was going on but assumed the NBA's study would never see the light of day.

If anyone out there knows where it is, make that two of us (and I'm sure many more) who would like to see it.

I didn't read much into the Price/Wolfers results. It is a solid piece of academic work, but I'm sure there is a lot more going on as Phil covered. The NBA however looks awful for blowing this out of proportion and then trying to bury the research with some hidden "study." The NBA had a chance to endear itself to the growing community of stat-minded-people and win some new fans. Now I'm sure they've got people out there pouring over public data looking for something else to throw in the NBA's face.

At Tuesday, May 05, 2009 12:37:00 PM, Anonymous Guy said...

This article mentions that ESPN had a review conducted of the NBA's study: You might try contacting the article's author, and/or the Univ. of Chicago researcher they employed, to see if either would provide the study. It also sounds like Wolfers wrote an evaluation of the NBA study -- it would be interesting to see that as well.

I would think the NBA microdata could shed light on some interesting questions -- such as whether same-race bias is exhibited by white refs, black refs, or both -- that the Wolfers/Price study couldn't answer (although oddly enough, this didn't stop Wolfers and Price from concluding that same-race bias advantages white players.)


I would still like to see the data from Wolfers and Price on team win%, for racial composition of teams crossed with racial composition of ref crews. That is, how often does the "whiter" team win in front of 0, 1, 2 and 3 white refs? And similarly, show us performance stats for both white and black players, broken down by type of ref crew. The fact that they have never revealed this data leaves me skeptical of their regression results purporting to show a relatively large impact of referee race on wins and losses (beyond what foul rates can explain).

At Tuesday, May 05, 2009 12:46:00 PM, Blogger Phil Birnbaum said...

Guy, excellent suggestion. I'll try getting the paper that way. I also have plans to reread the original, and my original comments ... gotta get around to that too.

At Tuesday, May 05, 2009 1:23:00 PM, Anonymous Guy said...

Also look at Price-Wolfers' follow up study focusing on the betting implications of their first study. You had a post or two on that as well.


Post a Comment

<< Home