Thursday, July 16, 2009

Why don't reporters consult sabermetricians?

Here's a completely misguided article about baseball production and salary. It's from Forbes magazine, which should know better.

The piece tries to figure out which players are the most overpaid in baseball. To get a measure of production, they compare a player's raw stats to the average at his position. The article doesn't give an actual formula, but it says that

"we compared the major offensive stats--batting average, home runs, runs batted in and OPS ... of each player to the league average for starters at his position. Then we did the same with salaries."

The idea being, if player X hits .300, and the league average is .270, player X's salary should be 11% higher than average.

That's just plain wrong -- a .300 hitter is a lot more than 11% more valuable than a .270 hitter. You have to measure from replacement value. If the best available minor-leaguer would hit .240, then you can see that the .300 hitter is actually twice as valuable as the .270 hitter -- he gains 60 points over replacement, as compared to 30 points. (This doesn't mean his salary should be double; only the part of the salary above replacement level should be double.)

Of course, batting average isn't a very good proxy for performance; OPS is much better (although there are other measures evan even more accurate). And if you're going to include OPS, like the Forbes article does, why bother including the other statistics anyway?

And, since we're already shooting fish in a barrel ... why would you evaluate a hitter by his RBI? We've known better than to do that for, what, around 30 years now.


This brings up a question, which Tom Tango asks on "The Book" blog: How did this article make it into print? Shouldn't Forbes, which is a pretty sophisticated business magazine, know better than to run a financial analysis that's wrong on so many levels?

The answer, I think, is that writers and editors' don't care as much about accuracy when it comes to baseball. I can think of several reasons, off the top of my head. Feel free to suggest more in the comments.

1. People don't think of sabermetrics as a formal field of knowledge, one with subject-matter experts who should be consulted. Forbes would never have dabbled in this kind of layman analysis in the field of, say, engineering. Can you imagine an article on which bridges are most overpriced, based on dividing the price by the total length of the bridge compared to the average length of bridges? That would never happen. Forbes would consult an engineer, who would tell them that there are many, many factors affecting how much it costs to build a bridge, and that their analysis was simplistic, silly, and very wrong.

2. My perception is that among most of the media (the Wall Street Journal among the notable exceptions), sabermetricians are seen as fringe researchers, nerds with the "blogging out of their parents' basement" stereotype. Certainly a veteran financial reporter should be more expert at player valuation than a unpaid 24-year-old blogger? Well, no. In this field, it turns out that the experts are among those with the fewest formal credentials. And that might be difficult for the mainstream media to accept.

3. It's only baseball. Remember when Bill James, in the 1991 Baseball Book, reamed out David Halberstam for some awful inaccuracies in "Summer of '49". James wrote,

"It is frightening to think that Halberstam, one of the nation's most respected journalists, is this sloppy in writing about war and politics, yet has still been able to build a reputation simply because nobody has noticed.

"What seems more likely is that Halberstam, writing about baseball, just didn't take the subject seriously. He just didn't figure that it *mattered* whether he got the facts right or not, as long as we was just writing about baseball."

I think that's what's happening here. Baseball is frivolous, Forbes probably thinks, and so it's not worth caring about getting it right. It's easy to forget, when you're dealing with a subject you don't take seriously, that there are other people who do. Freakonomics thought it was the first to talk about baby-naming trends, and was criticized by serious experts whom it would have been good to consult. I've done things like that myself, assuming that the subject is so trivial that I must be just as much an expert as anyone else. It's usually not the case.

4. Sabermetrics is a field composed more of logic than fact; sabermetric analysis sounds like opinion, not science. What treatments work for Hepatitis? I don't know, and Forbes reporters don't know; it's a question of medicine, a question of fact. We can't even guess. You have to ask a doctor; they know.

But when it comes to evaluating a player's performance ... well, if you don't think about it too deeply, it seems like you don't need to know anything. Sure, just divide the salary by the numbers ... it seems reasonable and logical, doesn't it, that if you drive in 20 percent more runs you should make 20 percent more money? It doesn't seem like there's an underlying fact there, something that an expert has to tell you. You can come up with something plausible in your head.

And if someone tells you you're wrong, so what? It doesn't look like it's a question that has a right answer. If you can find two financial analysts, one who says a stock is worth $30, the other who says the same stock is worth $10, and it's OK to stick them on consecutive pages of Forbes ... then who's to say that figuring out the value of a player is any different? It's just one opinion instead of another.

This isn't a problem just in sabermetrics ... a few years ago, I read an economist (I think it was Steven Landsburg) with the same complaint. He said (and I'm paraphrasing) that economics is such that everyone thinks they understand it as well as people who have studied it for years -- people who wouldn't dream of having an opinion on a medical issue will confidently come forth with solutions -- wrong solutions -- to whatever economic problem is on today's agenda.

And, again, I think that's because a lot of it is logic. And logic is hard -- it's hard to reason through an argument than to memorize a fact. There are lots of arguments that sound plausible, but are, in fact, flawed. It takes work to figure out which is which, especially when you don't have enough background to be able to avoid the common fallacies, when you might be too proud of your own flawed logic to accept that you might have left something out, and when you might, ignorantly, be rejecting certain principles that sound silly to you but that the field has empirically found to be correct.

It's funny, too, because reporters will be willing to publish their half-baked theories without a second thought -- but when it comes to a specific mathematical calculation, they'll often cite a source, even when the calculation is simple enough to state as fact. It's as if they know their math *facts* might be shaky, but believe their mathematical *arguments* are beyond error.

Labels: , ,


At Thursday, July 16, 2009 8:04:00 AM, Blogger Jeff J said...

Great post.

By suggesting theat Forbes should know better, is it possible you're giving them too much credit? The author's twitter from two days ago:

"Nice op-ed by Gov. Palin on the absurdities of cap & trade energy plan."

He's clearly an imbecile.

And Landsburg's point would carry a lot more weight if there weren't such diverse opinions within the field of economics itself. Many aspects are poorly understood by both the laymen and the experts.

At Thursday, July 16, 2009 9:42:00 AM, Anonymous Nobrainer said...

"I think that's what's happening here. Baseball is frivolous, Forbes probably thinks, and so it's not worth caring about getting it right."

That's probably true, but I wouldn't limit this type of behavior on their part to just baseball. They put together all kinds of various rankings (seems readers love articles in list form) where they just put together some criteria that sound reasonable and then just go from there.

At Friday, July 17, 2009 10:00:00 PM, Blogger Chone Smith said...

Economics is a tough one. Among experts you still have widely different opinions of how the economy works. It really should not be called a science, certainly not compared to something like physics, where experts can agree on things like the weight of a distant planet to the 10th decimal place.

Right now I have mostly contempt for the opinions of most of the "leading economists" of the day, the ones heading the federal reserve, widely reported in media, or in academics.

I don't know if any policies I would recommend would work any better, but I do know that they have been reliably wrong on just about everything so far, first denying a problem existed, then underestimating the impact of it, and next being clueless about the effects of their solutions.

At Saturday, July 18, 2009 7:15:00 AM, Blogger Ted said...

I was drenched yesterday morning by a huge thunderstorm that wasn't supposed to happen, so it would seem physics isn't a science yet, by that definition.

The problem here is that the media (again, our favorite whipping boys here) uses "economics" to mean two different things. Economics is a science with well-understood principles. Full stop. But, when the general public reads about "economists," you're really reading about "policy analysts," who indeed have economic training, but who are applying economic principles to complex problems -- just like weather forecasters are applying physics principles to complex problems. Yet, when the weather forecasters whiff on today's forecast, noone concludes that physics is bogus -- in part because of the language choices, where we use a completely different term for atmospheric studies and don't just call them all "physicists."

Sabermetrics is economics. I don't mean "is like" economics; it is literally part of economics. Phil's OP is right on target in taking Forbes to task, because it posts an argument that violates basic principles of economics, so the analogy is perfect.

At Wednesday, July 22, 2009 1:48:00 PM, Blogger Jeff J said...

Economics is at best a social science. Full stop.

Rather than get into the details (which, I suspect, wouldn't convince Ted anyway), I'll just point to Google.

At Monday, August 17, 2009 1:09:00 PM, Anonymous pawnking said...

Speaking of poor logic, your straw man arguement of "people who wouldn't dream of having an opinion on a medical issue will confidently come forth with solutions" is a good example of poor logic. People have an opinion of medical issues all the time, even without any medical training whatsoeever. I put it that this statement is so self-obvious as to need no example, but if you disagree please tell me so and I will substantiate.

If your main arguement is the media suffers from poor logic, please try to use good logic. I enjoyed your article, but I cannot take it seriously due to this.

At Monday, August 17, 2009 1:12:00 PM, Blogger Phil Birnbaum said...

I meant a technical medical issue, like the best kind of surgery to to remove a tumor. That sort of thing, the kind of thing that you really can't have an opinion unless you study it first.

At Sunday, September 27, 2009 8:10:00 PM, Anonymous Anonymous said...

You're dead on with #3. It's just baseball. Unless you're on the payroll making recommendations to Billy Beane or Theo Epstein a deep analysis of baseball statistics is a thorough waste of time. Especially if you're a journalist at Forbes.

Don't get me wrong: If you're a fan and a data junkie it's good fun and quite fulfilling but, ultimately it's a trivial subject.

However, I agree with your premise, the responsible thing to do would be to consult Tom Tango...but it's just baseball and 98% of Forbes readers don't give a rats a$$ about Tom Tango. (No offense to the man, he does great work.)


Post a Comment

<< Home