Tuesday, August 14, 2007

The Hamermesh study: what percentage of pitches are biased?

The Hammermesh study on racial bias among umpires (which I posted about here) concluded that “a given called pitch is approximately 0.34 percentage points more likely to be called a strike if the pitcher and umpire match race/ethnicity.” But I don’t think they actually say what percentage of calls are biased. I’m tried to figure this out for myself, without using regression. It depends on the assumptions you make; under the assumptions I’ll show you in a bit, I get 0.13%.


I’m going to start with a simple example. Suppose you have only white umpires and black umpires, and only white pitchers. The white umpires call 31% strikes, but the black umpires only call only 30% strikes. We don’t know yet whether this is bias or just random fluctuation. But we can say that for whatever reason, the pitchers are advantaged by 1 percentage point with a white umpire, relative to a black umpire.

But suppose it were bias. What percentage of (called) pitches would be affected?

Your first inclination might be to say that 1% of pitches are affected when there’s a white umpire, but none are affected when there’s a black umpire.

But, wait. We don’t know what direction the bias goes. It’s possible that all the bias is from the black umpire. He should be calling 31% strikes, but he’s only calling 30%. In that case, 1% of pitches are affected when there’s a black umpire, but 0% when there’s a white umpire.

There’s still another case. Perhaps the pitcher should actually 30.5% strikes, and both umpires are biased by 0.5%. In that case, 0.5% of pitches are affected when there’s a black umpire, and 0.5% are affected when there’s a white umpire.

Indeed, there is an infinity of possibilities. Maybe the black umpire is biased 0.3% and the white 0.7%. Maybe the black umpire is biased 2%, and the white umpire is biased negative 1%. And so on.

So we can’t say what percentage of pitches are biased for which umpire. We can’t even say what percentage of total pitches are biased. In the last case I suggested, where the black umpire was biased 2% and the white umpire was biased 1% in the other direction, there would be 1.5% of total pitches biased (assuming the white and black umpires called equal numbers of pitches). But in all the other cases I used as examples, it would be only 0.5%.

So it all depends on your assumptions of where the bias is.


OK, now let’s move to a real-life example, one where the umpires don’t call equal numbers of pitches. In fact, I’ll take the “white pitcher” column of the paper’s Table 3. Here it is:

White umpires: 32.06% strikes.
Hspnc umpires: 31.91% strikes.
Black umpires: 31.93% strikes.

Now, again, let’s suppose all the differences between umpires are bias. That means we want equal numbers for all umpires. Since the overall strike rate for white pitchers was 32.05% (from Table 2), we might choose that. So we *really* want the table to look like:

White umpires: 32.05% strikes.
Hspnc umpires: 32.05% strikes.
Black umpires: 32.05% strikes.

How do we make that happen? Well, we can subtract called strikes from the first case, and add them to the second and third cases:

White umpires: subtract 0.01 from the original 32.06
Hspnc umpires: add 0.14 to the original 31.91
Black umpires: add 0.12 to the original 31.93.

This will make all the percentages equal to 32.05%.

Since we assumed the differences represented bias, that means that

0.01% of white umpires’ calls were biased;
0.14% of hispanic umpires’ calls were biased;
0.12% of black umpires’ calls were biased.

Those numbers depended on our choice of 32.05% as the baseline. Another choice we can make is to use 32.06% as the baseline instead, which assumes that both hispanic and black umpires should call the same 32.06% strikes as the white umpires. In that case, you need to add 37 strikes to the hispanic umpires, and 61 strikes to the black umpires. That means

0.00% of white umpire calls are biased
0.15% of hspnc umpire calls are biased
0.13% of black umpire calls are biased

Which do you like? Either: there’s no real reason to choose one over the other, and no way to prove which is more accurate.

We could go extreme the other way, by assuming that the hispanic umpires are correct, and the “real” percentage is 31.91%. In that case,

0.15% of white umpire calls are biased
0.00% of hspnc umpire calls are biased
0.02% of black umpire calls are biased

Do you like that better? It’s arbitrary still.

Arbitrary or not, my feeling is that we should find a pattern where all three groups of umpires seem about equally biased. That way, we don’t have to single out one group. Maybe we can choose 31.99 as our estimate of the “true” value. That would mean:

0.07 of white umpire calls are biased
0.08 of hspnc umpire calls are biased
0.06 of black umpire calls are biased.

This is my favorite, because it spreads the blame around; in the absence of evidence that one group is “guiltier” than another, this seems like the ethical default assumption.

Now, back to the original question: what percentage of *all pitches* are biased? From this last distribution of bias, it looks like about 0.07% of pitches are biased (regardless of who the umpire is). That’s one out of every 1,400 pitches.

I’ll quickly do the Hispanic and Black umpires too. For Hispanic, it looks like 31.15% might be a pretty good stab at the “real” strike percentage, the one that makes all the umpires look equally biased. That means that for Hispanic pitchers,

0.32% of white umpire calls are biased
0.35% of hspnc umpire calls are biased
0.28% of black umpire calls are biased.

Since white umpires are an overwhelming majority, the overall average of these numbers is probably 0.32%.

For black pitchers, let’s use 30.69% as the base:

0.08% of white umpire calls are biased
0.08% of hspnc umpire calls are biased
0.09% of black umpire calls are biased

That’s about 8% overall for the black pitchers.

Summarizing the three groups of pitchers:

White pitchers: 0.07% of calls are biased
Hspnc pitchers: 0.32% of calls are biased
Black pitchers: 0.08% of calls are biased

Hispanic pitchers are 25% of the total, and black pitchers about 5%. If we weight the three groups accordingly, we get:

Overall: 0.13% of all umpire calls are biased. That’s about 1 pitch in 750.

Again, this is subject to assumptions, not all of which might be true:

-- we assume that all umpires have an equal propensity to be biased for a particular race of pitche

-- we assume that 100% of the discrepancies actually seen represent bias. This is almost certainly not true, because there is inherent random variation in what kinds of pitches umpires will see.

-- more importantly, we assume that bias exists. I’m still not convinced that the amounts of variation seen in the study are statistically significant.

But it appears that if you do accept the above assumptions, it follows that at most 1 pitch in 700 will be biased. Unless I’ve screwed up the logic somewhere.

Labels: ,


At Wednesday, August 15, 2007 2:31:00 PM, Blogger Tangotiger said...

I'm with you.

It should also be pointed out that this is called pitches, which represents, according to the paper, 53% of all pitches.

A top-end pitcher will be in the 3500 pitch range, meaning about 3 pitches will be biased for the entire season, according to Phil's contention.

Also, there are THREE Hispanic umpires and 5 Blacks. I don't see how you can have an "average Black" umpire based on 5 of them. Did the authors introduce a parameter to see if there were any outliers among these groups of umpires? I'd hate to think Eric Gregg would make up 20% of a group average, and then paint the entire group as being Black.

At Wednesday, August 15, 2007 3:44:00 PM, Blogger Brian Burke said...

Has anyone considered the possibility that neither black nor white umpires are biased and there is just one or a handful of culprits. How bad could a single umpire's bias be before it was noticable?

Perhaps that level of bias is within the variance of all umpires.

Perhaps there is a black umpire with a bias against a particular pitcher , and that pitcher happens to be white. In other words, it's not a racial bias, but the individual bias crosses race. Could that account for the observed differences?

At Wednesday, August 15, 2007 5:31:00 PM, Blogger Phil Birnbaum said...

Yeah, I think you're right ... it seems more likely that there are *some* culprits than that all umpires of race X are similar.

In fairness, I don't think the study precludes this ... it just concludes there's bias, but I think it leaves open the question of how the bias is distributed among the umpires.

That's a good point by Tango, wondering if the authors looked at the individual umpires. That definitely seems like something you'd want to look at.

Of course, if you found that an outlier was causing the effect, it would no longer be an economics paper.


Post a Comment

<< Home