The Hamermesh study: what percentage of pitches are biased?

The Hammermesh study on racial bias among umpires (which I posted about here) concluded that “a given called pitch is approximately 0.34 percentage points more likely to be called a strike if the pitcher and umpire match race/ethnicity.” But I don’t think they actually say what percentage of calls are biased. I’m tried to figure this out for myself, without using regression. It depends on the assumptions you make; under the assumptions I’ll show you in a bit, I get 0.13%.

---

I’m going to start with a simple example. Suppose you have only white umpires and black umpires, and only white pitchers. The white umpires call 31% strikes, but the black umpires only call only 30% strikes. We don’t know yet whether this is bias or just random fluctuation. But we can say that for whatever reason, the pitchers are advantaged by 1 percentage point with a white umpire, relative to a black umpire.

But suppose it were bias. What percentage of (called) pitches would be affected?

Your first inclination might be to say that 1% of pitches are affected when there’s a white umpire, but none are affected when there’s a black umpire.

But, wait. We don’t know what direction the bias goes. It’s possible that all the bias is from the black umpire. He should be calling 31% strikes, but he’s only calling 30%. In that case, 1% of pitches are affected when there’s a black umpire, but 0% when there’s a white umpire.

There’s still another case. Perhaps the pitcher should actually 30.5% strikes, and both umpires are biased by 0.5%. In that case, 0.5% of pitches are affected when there’s a black umpire, and 0.5% are affected when there’s a white umpire.

Indeed, there is an infinity of possibilities. Maybe the black umpire is biased 0.3% and the white 0.7%. Maybe the black umpire is biased 2%, and the white umpire is biased negative 1%. And so on.

So we can’t say what percentage of pitches are biased for which umpire. We can’t even say what percentage of total pitches are biased. In the last case I suggested, where the black umpire was biased 2% and the white umpire was biased 1% in the other direction, there would be 1.5% of total pitches biased (assuming the white and black umpires called equal numbers of pitches). But in all the other cases I used as examples, it would be only 0.5%.

So it all depends on your assumptions of where the bias is.

---

OK, now let’s move to a real-life example, one where the umpires don’t call equal numbers of pitches. In fact, I’ll take the “white pitcher” column of the paper’s Table 3. Here it is:

White umpires: 32.06% strikes.
Hspnc umpires: 31.91% strikes.
Black umpires: 31.93% strikes.

Now, again, let’s suppose all the differences between umpires are bias. That means we want equal numbers for all umpires. Since the overall strike rate for white pitchers was 32.05% (from Table 2), we might choose that. So we *really* want the table to look like:

White umpires: 32.05% strikes.
Hspnc umpires: 32.05% strikes.
Black umpires: 32.05% strikes.

How do we make that happen? Well, we can subtract called strikes from the first case, and add them to the second and third cases:

White umpires: subtract 0.01 from the original 32.06
Hspnc umpires: add 0.14 to the original 31.91
Black umpires: add 0.12 to the original 31.93.

This will make all the percentages equal to 32.05%.

Since we assumed the differences represented bias, that means that

0.01% of white umpires’ calls were biased;
0.14% of hispanic umpires’ calls were biased;
0.12% of black umpires’ calls were biased.

Those numbers depended on our choice of 32.05% as the baseline. Another choice we can make is to use 32.06% as the baseline instead, which assumes that both hispanic and black umpires should call the same 32.06% strikes as the white umpires. In that case, you need to add 37 strikes to the hispanic umpires, and 61 strikes to the black umpires. That means

0.00% of white umpire calls are biased
0.15% of hspnc umpire calls are biased
0.13% of black umpire calls are biased

Which do you like? Either: there’s no real reason to choose one over the other, and no way to prove which is more accurate.

We could go extreme the other way, by assuming that the hispanic umpires are correct, and the “real” percentage is 31.91%. In that case,

0.15% of white umpire calls are biased
0.00% of hspnc umpire calls are biased
0.02% of black umpire calls are biased

Do you like that better? It’s arbitrary still.

Arbitrary or not, my feeling is that we should find a pattern where all three groups of umpires seem about equally biased. That way, we don’t have to single out one group. Maybe we can choose 31.99 as our estimate of the “true” value. That would mean:

0.07 of white umpire calls are biased
0.08 of hspnc umpire calls are biased
0.06 of black umpire calls are biased.

This is my favorite, because it spreads the blame around; in the absence of evidence that one group is “guiltier” than another, this seems like the ethical default assumption.

Now, back to the original question: what percentage of *all pitches* are biased? From this last distribution of bias, it looks like about 0.07% of pitches are biased (regardless of who the umpire is). That’s one out of every 1,400 pitches.

I’ll quickly do the Hispanic and Black umpires too. For Hispanic, it looks like 31.15% might be a pretty good stab at the “real” strike percentage, the one that makes all the umpires look equally biased. That means that for Hispanic pitchers,

0.32% of white umpire calls are biased
0.35% of hspnc umpire calls are biased
0.28% of black umpire calls are biased.

Since white umpires are an overwhelming majority, the overall average of these numbers is probably 0.32%.

For black pitchers, let’s use 30.69% as the base:

0.08% of white umpire calls are biased
0.08% of hspnc umpire calls are biased
0.09% of black umpire calls are biased

That’s about 8% overall for the black pitchers.

Summarizing the three groups of pitchers:

White pitchers: 0.07% of calls are biased
Hspnc pitchers: 0.32% of calls are biased
Black pitchers: 0.08% of calls are biased

Hispanic pitchers are 25% of the total, and black pitchers about 5%. If we weight the three groups accordingly, we get:

Overall: 0.13% of all umpire calls are biased. That’s about 1 pitch in 750.

Again, this is subject to assumptions, not all of which might be true:

-- we assume that all umpires have an equal propensity to be biased for a particular race of pitcher.

-- we assume that 100% of the discrepancies actually seen represent bias. This is almost certainly not true, because there is inherent random variation in what kinds of pitches umpires will see.

-- more importantly, we assume that bias exists. I’m still not convinced that the amounts of variation seen in the study are statistically significant.

But it appears that if you do accept the above assumptions, it follows that at most 1 pitch in 700 will be biased. Unless I’ve screwed up the logic somewhere.