Chopped liver II

David Berri and J.C. Bradbury have a new paper out. Called "Working in the Land of the Metricians," it purports to be a guide for Ph.D. sports economists in how to interact with sabermetricians.

The subject is an important one, but, sadly, the paper isn't very constructive. There's a good quote-by-quote critique of the paper at Tango's blog, which I agree with substantially enough that I can just refer you there. (Also, since the paper is gated, you can get a good feel for what's happening in it by reading Tango's take.)  

UPDATE: the paper is now available in full (.pdf) at Dave Berri's site.

For my part, I'll just concentrate on one major point -- that Berri and Bradbury still treat the non-academic community's work as if it barely has any value at all. And they're pretty forthright about it. Despite paying lip service to the idea that maybe non-academic researchers have *something* to contribute, the authors remain wilfully blind to 98 percent of the sabermetric community's contribution to the field.

Indeed, Berri and Bradbury explicitly refuse to recognize that the active research community has any expertise at all:

"Birnbaum considers sabermetricians to be "no less intelligent than academic economists" and superior to economists in their understanding of baseball. This statement reveals a curious worldview. On one hand, the aspect that is universal across both groups -- members of both communities have been devoted sports fans since an early age -- is considered unique to the nonacademic sports anaysts. On the other hand, when it comes to the aspect that is unique to academics -- academia normally involves many years of advanced training and requires its participants to be judged competent by their peers in a "publish or perish" environment -- metricians demand equal recognition. In our view, this mentality begets misplaced confidence."

Get what they're saying here? Bill James, Tom Tango, Mitchel Lichtman, Andy Dolphin, Pete Palmer, those guys hired by baseball teams, all those other sabermetric researchers -- they're not baseball experts at all. They're *just sports fans*. How could they possibly know more about analyzing baseball than any other fan, unless they've formally studied econometrics?

It's astonishing that Berri and Bradbury could possibly believe that economists, even sports economists, know more about baseball than these guys, who have built their careers around analyzing the game. Equally astonishing is their implication that we *are* less intelligent than they are. At first, I thought that they didn't realize what they were saying. But, no, it seems pretty clear that they *do* believe it.

What has become apparent in sabermetricians' debates with Bradbury (and, to a lesser extent, Berri, who doesn't engage us as much) is that they disagree with us on almost every point we make, and that they seem to be uncomfortable arguing informally. I can't recall a single time that either of them has conceded that they're wrong, even on a small point. Economists are supposed to be fond of hypotheticals, simple models, and playful arguments to illustrate a point (see this Paul Krugman column), but Bradbury and Berri, not so much. Attempts to try to describe their models with simplified analogies are usually met with detailed rebukes about how we don't understand their econometric methods.

In that light, it's easier to see where they're coming from. They believe that (a) only formal, peer-reviewed research counts as knowledge, and (b) all us non-peer-reviewed people have been wrong every time we've disagreed with their logic. If both of those were actually true, they'd be right -- we'd just be ignorant sports fans who don't have any expertise and need to be educated.

As for the sabermetric findings that they actually use, like DIPS and OPS ... Berri and Bradbury seem to consider them a form of folk wisdom that the unwashed baseball fans managed to stumble upon, and argue that economists should not accept them until they've been verified by regression methods that would pass peer review and be publishable in academic journals.

Back in 2006, in the post that Bradbury and Berri quoted above, I accused Bradbury of ignoring the findings of sabermetricians in one of his papers. At the time, I thought perhaps I was too harsh. I was wrong. This paper shows that he truly believes that, as non-peer-reviewed sports fans with no special expertise, our research findings are unworthy of being cited.

Indeed, in Bradbury's exposition (I am assuming all the arguments in the baseball portion of the current paper are Bradbury's, although Berri is still listed as co-author), he treats the history of sabermetric knowledge as if it were mostly a series of academic papers. That's absurd; it's indisputable that, conservatively, at least 90 percent of our sabermetric knowledge came from outside academia. If you were writing the history of research about Linear Weights, what would you include? Think about it for a minute.

Ready? Here's how Bradbury sees it:

--In 1992, academic A.A. Blass published a study where he estimated linear weights by regression.

--In 2005, academic Ted Turocy published a paper which highlighted the "omitted variable" bias making the results not as accurate as they could be.

--In 1963, academic George Lindsey had published a paper with a rudimentary form of the same equation, but not using regression.

--In 1984, sabermetricians John Thorn and Pete Palmer "popularized" and "updated" Lindsey's work.
--In 2003, academics Jim Albert and Jay Bennett compared the two approaches.

Got it? Four academic papers (Albert/Bennett is actually their book, Curve Ball, but no matter) and Thorn/Palmer. On top of that, the only mentions of Thorn and Palmer are their "updating" and "popularizing". Each of the academics' work, on the other hand, is described in some detail.

Is that how you would characterize the state of knowledge, that the history of accumulated knowledge about Linear Weights comes from these five academics and a cursory contribution from Pete Palmer? I mean, come on. There's a huge literature out there if you look outside academia, including studies on various improvements to the original.

It gets worse, in the DIPS discussion.

Bradbury starts off by reviewing the seminal Gerald Scully paper. In 1974, Scully (a famed sports economist who passed away in 2009) published a paper that tried to value players' monetary contributions to their teams. Regrettably, he used strikeout-to-walk ratio as a proxy for the pitcher's value to his team's success on the field. That's not a particularly good measure of the value of a pitcher; ERA is much better. Even before sabermetrics, everyone knew that, including casual baseball fans and Joe Morgan. And if they didn't, Bill James would have made it clear to them in his Abstracts, so there was no excuse for a baseball researcher to not be aware of that after, say, 1983.

And Bradbury acknowledges that, that ERA was perceived to be better than K/BB ratio. Does he cite common sense, conventional wisdom, or Bill James? Nope. He cites two academic papers from the 1990s. No, really:

"... [Andrew] Zimbalist (1992b) and [Anthony] Krautmann (1999) argue that ERA is a better measure of pitcher quality ..."

So, before 1992, nobody else argued it? Well, OK, if you say so.

Anyway, with that established, Bradbury continues. In 2001, Voros McCracken came along, and, in an "essay" on the "popular sabermetric Web site Baseball Prospectus," he "suggested" that pitchers have little control over what happens to balls in play. At this point, Bradbury checks the correlation of a pitchers' BABIP in consecutive seasons, and finds it's fairly low (.24).

"This supports McCracken's assertion," he writes.

Okay, thanks for the verification! But, er, actually, it's not like the sabermetric community was just sitting on its hands the past eight or nine years, staring at McCracken's hypothesis and wondering if someone would ever come along and tell them if it was true. Sabermetricians of all sorts have done huge volumes of work to refine and prove versions of the DIPS hypothesis. For a long time, you couldn't hit any sabermetric website without DIPS this and DIPS that hitting you from every angle, with all kinds of theories about it tested and studied.

It's perfectly fine that Bradbury verifies McCracken with his quick little regression here, but why wouldn't he acknowledge everyone else, by, for instance, saying that his study is an example of the kinds of confirmation studies that the sabermetric community has been doing since 2001? In light of the rest of the article, which implies that sabermetricians are just unsophisticated baseball fans, that omission would reasonably lead readers to incorrectly assume that Bradbury's little study is the first of its kind.

But, instead, Bradbury ignores all those years of sabermetric study of the DIPS issue, just because it wasn't academically published or peer-reviewed. That's as wrong now as it was when I wrote about it in 2006 -- especially in an essay that's ostensibly suggesting that academics can benefit from outside research.

The DIPS approach is as well accepted in sabermetrics as the Coase Theorem is accepted in economics. The difference is, if I published something mentioning Coase's hypothesis, and then published a study "supporting" it without citing any other study or mentioning that it's a canon of the economics literature, Bradbury would go ballistic, laying into my ignorance like ... like Tiger Woods on a supermodel. (Sorry.) The other way around, though, and it's all OK.

Anyway, that's just the prelude -- this is the point where Bradbury's argument gets really bizarre.

Why did Bradbury bring up DIPS? Because it shows that, instead of using ERA to evaluate the skill of a pitcher, it's better to just use walks, strikeouts, and home runs allowed, thus eliminating a lot of random chance from the pitcher's record. That part is absolutely fine. But then he comes back to Scully.

Remember when Scully did his 1974 study that used strikeout-to-walk ratio as a measure of a pitcher's value? And everyone agreed that he should have used ERA instead? Well, now, hang on! McCracken has shown us that if we ignore a pitcher's balls in play, we can get a better measure of his talent than if we use ERA. Eliminating balls in play leaves only BB, K, and HR. That means strikeouts and walks are really important. And, in turn, that means that Scully was actually correct back in 1974 when he emphasized strikeouts and walks by using K/BB ratio as his statistic of choice!

To make this bizarre argument Bradbury ignores the fact you have to combine K and BB in a very specific way, or it doesn't work well at all, and that K/BB ratio is still worse than ERA. And HR are important too. But, argues Bradbury, at least Scully was right in that it had to be K and BB. Maybe he wasn't completely correct, but he had the right idea.

Why did Gerald Scully choose to value pitchers by K/BB? According to Bradbury, it wasn't because he just gave it a guess. He did it because he intuitively anticipated that DIPS was true. Voros McCracken, the non-academic sabermetrician, just served later to confirm Scully's original insight.

See? The academics were right all along!

If that sounds ludicrous, it is. I really hope you don't believe what I'm saying here ... I hope you're thinking that nobody who could actually make that argument with a straight face, that Gerald Scully's choice of K/BB ratio as his metric is an anticipation of a completely different, complex, 12-part formula for DIPS ERA that happens to partly depend on K and BB. I hope you believe that I'm making it up.

But I'm not. That's actually what Bradbury argues! Here are the quotes:

"If a measure varies considerably for an individual player over time, it is likely that the measure is heavily polluted by luck ... Scully appeared to understand this point with his choice of the strikeout-to-walk ratio to measure pitcher quality.

"Later research from outside academia [McCracken] confirmed Scully's original approach and suggested [also] including home-run prevention when evaluating pitchers. ...

"Consequently, we see Scully's general approach is confirmed by a metrician, demonstrating what the nonacademic sports research community can contribute to sports economics research."

Could there be a sillier, more self-serving rationalization?

Oh, and by the way ... it seems that Scully wasn't even a baseball fan at the time he started writing his study. How does Bradbury think Scully was able to anticipate a result that wouldn't become apparent until 27 years later? A result, moreover, that pertained to a sport Scully probably didn't know that much about, in a field of science that didn't even exist at the time, that wouldn't emerge until years of study by non-academic researchers, and that was so surprising it shocked even the most veteran researchers in the field?

Maybe it was that "years of advanced training" in economics -- obviously so much more valuable than the non-expertise in sabermetrics that Voros McCracken, a mere baseball fan, couldn't possibly have had.