Tuesday, August 19, 2008

Analysis in new JQAS paper makes no sense

The new issue of JQAS came out a couple of weeks ago, and I'm starting to go through some of the articles. The first one kind of floored me. It's the longest paper I've seen in JQAS so far, at 64 pages. It's ostensibly about measuring the effects of NFL coaching. But I've started reading it, and it makes no sense at all.

Here's the link. It's "Quantifying NFL Coaching: a Proof of New Growth Theory," by Kevin P. Braig.

The paper starts out with some baseball arguments, but they're more numerology than sabermetrics. It starts by implying that, because the batter is one person, and the pitcher and catcher are two people, that the battery enjoys a 2-1 edge over the hitter. Also, you need four bases to score a run, and have only two outs to do so – which happens to be another 2-1 edge. (Braig knows you get three outs, but figures that since you're retired after the third out, you really only have two outs to expend.)

So baseball has an intrinsic 2-1 structure. And that's why, as Braig triumphantly notes, the historic Major League on-base percentage happens to be .333. See, it's two outs for the defense for every hit by the offense!

"On-base percentage confirms the battery's 2-1 design edge … in other words, hitters have succeeded at the *exact rate* that one would expect by taking outs from the hitters at a 2-1 rate."

That, of course, is ridiculous – the three numbers have nothing do with each other, and that each works out to a 2:1 ratio is nothing but coincidence.

Moving on, Braig figures out what would happen if a team gets on base at .333 – a regular .333, repeatedly alternating an "on base" with two outs. It turns out that if you assume that (a) the "on base" is a single, and (b) all runners advance one base on an out, you wind up alternating innings where one run scores (with a .400 OBP) with innings where no run scores (with a .250 OBP). From this, he concludes,

"These models show that baseball success emerges from the hitters' ability to make a base 40% of the time in an inning … as the hitters' OBP in an inning approaches .400, the hitters approach scoring 1 run."

Um, why is that? Why should we expect that the contrived example, that has so very little resemblance to real baseball, should give a correct result?

So far every step in this process is completely wrong, including this next one: Braig concludes that you can measure a team's "offensive efficiency" by dividing its OBP by .400. So the 1927 Yankees were 95.25% efficient (.381/.400).

And finally, one last leap of logic: because .333 is the "standard," its 2:1 ratio built into the structure of baseball, it must be that the contribution of the hitters ("human capital") to the results is only the difference between his OBP and .333. So the 1927 Yankee hitters, who were .048 above .333, were responsible for only 15% of the total offensive efficiency of the team.

At this point I pretty much stopped reading – all this took only the first seven pages out of 65. And there are additional glaring absurdities, even in those first few pages, that I haven't touched on here. This has got to among the worst psuedo-analyses that I've seen anywhere, never mind in a peer-reviewed publication.

If any intrepid readers want to check out the whole paper, to see if the football discussion has anything of value in it, please report back.

Labels: ,


At Tuesday, August 19, 2008 11:28:00 PM, Anonymous Anonymous said...

I'm laughing so hard! I had the same reaction and was going to do my own post but, like they say, never get into an argument with a crazy person.

Did Louis Farrakhan write that paper?

I think this is evidence of the 'complexification' of stupid ideas, usually perpetrated by academics. If you make simple (and often mistaken) ideas complicated and technical enough, people might be intimidated into just taking your word.

The funniest part is in the title--"Proof."

At Wednesday, August 20, 2008 8:15:00 PM, Anonymous Anonymous said...

Before we start tarring academics here, the author isn't an academic. He's a lawyer, whose primary qualification seems to be he was the defense attorney for the arraignment of Julio Castillo, the Peoria pitcher who threw the ball last month that struck a fan during a Peoria-Dayton brawl.

JQAS also isn't truly a fully academic journal, either. It's probably somewhat more scholarly than, say, the SABR annual publications, but much less scholarly than most academic journals.

At Wednesday, August 20, 2008 8:28:00 PM, Blogger Phil Birnbaum said...

Right, the author didn't present himself as an academic, and no tarring of academics is stated or implied. My criticisms apply to the article only, not to any group the author may belong to, whether that be academics, non-academics, lawyers, or people named Kevin.

However, I think it's fair to wonder why a quality, peer-reviewed journal like JQAS (BTW, I'd call it an academic journal, and I bet the JQAS editors would too) chose to publish the paper.

At Wednesday, August 20, 2008 8:49:00 PM, Blogger Ted said...

Phil, I was addressing the first commenter, not your original post.

I agree with the last point in your comment completely.

At Wednesday, August 20, 2008 11:21:00 PM, Blogger Phil Birnbaum said...

Oh, okay.

At Friday, August 22, 2008 12:30:00 PM, Blogger Brian Burke said...

I meant that the type of unnecessary jargon and hyper-complexity in the paper is usually found in academic sources. (Though not often to the degree in this paper.) I did not mean that most academics engage in that sort of thing. I apologize if I offended any professors out there.

At Friday, August 22, 2008 1:13:00 PM, Blogger The Football Know-it-all said...

Don't worry. I've had to listen to endless academics manufacturing terms and "discovering" phenomena by creating a new word, just to get their papers published. Offend away.

At Friday, August 22, 2008 1:51:00 PM, Blogger Keith Law said...

Hoax? First thing that came to mind was Sokal's hoax.

At Saturday, August 23, 2008 7:10:00 PM, Anonymous Anonymous said...

Hoax or not (and likely not), it sadly makes the same statement about the editors of JQAS.

At Monday, August 25, 2008 8:27:00 AM, Anonymous Anonymous said...

I too thought it was a hoax and immediately also thought of sokal's hoax. It makes me worry that JQAS doesn't properly peer review papers. I think someone of standing in the sabremetric community shoudl write to the journal and ask if this is a hoax.

At Monday, August 25, 2008 1:29:00 PM, Blogger Hawerchuk said...

Ouch. I'm a hockey reviewer for JQAS and I'm surprised to see something like this in the journal. The journal is typically rigorous and we make sure that articles show an understanding of the sport and aren't just statistics applied to a problem that the authors didn't understand.

At Monday, September 08, 2008 1:34:00 AM, Anonymous Anonymous said...

"Rigorous" indeed. You people should relieve yourselves with your right hands more frequently ... and then plot volume vs. frequency to, you know, establish a correlation.

I'm laughing so hard, too.


Post a Comment

<< Home