Who is a sabermetrician?
J.C. Bradbury says he's not a sabermetrician:
"... I shouldn’t be considered a sabermetrician. I have never claimed to be a member of this community. What I do is apply my knowledge from my economics training and experience with analyzing data to issues in baseball."
I find this kind of confusing – If JC analyzes issues in baseball, statistically, then *of course* he's a sabermetrician. Perhaps there's a confusion in regards to the definition – does sabermetrician mean "one who does sabermetrics," or "one who is expert in the field of sabermetrics"? I suppose JC could be the one but not the other – if I do some analysis using economic reasoning, I'm doing economics, but that doesn't mean I'm an economist.
In any case, where all this came from is a post by Rob Neyer, pointing out that JC and Tom Tango are in strong disagreement on the Raul Ibanez signing. JC has Ibanez being worth $46 million over three years, while Tango has him worth only about $10 million. That is, to say the least, a substantial difference.
Rob uses this example to show how there are, and always will be, major sources of disagreement in the sabermetric community. I'm not completely with Rob on this one – I think conflicts of this magnitude are fairly rare. My take is that JC has a methodology for valuing players that most other sabermetricians think is incorrect (I, myself, disagreed with him here). As Mitchel Lichtman says in one of the comments,
"There are plenty of experts with respect to projections, and I don't think - in fact I know - that any of them will project his value at anything close to 14mm per year."
If that's the case, that JC's opinion is an outlier, then it might not be that there's long-term disagreement, but simply that somebody – perhaps JC, perhaps everyone else -- is just plain wrong. And, in legitimate science, when someone is wrong, it usually gets corrected pretty quick.
-----
JC's post also goes on to argue that academic research in sabermetrics is more reliable than informal online discussions:
"... much what passes for research within [the sabermetric] community is not sufficiently rigorous to reach the conclusions often claimed. There are many academic researchers from a variety of fields who have significantly advanced the understanding of baseball that receive scant mention in the sabermetric community. For example, Michael Schell’s Baseball’s All-Time Best Sluggers is the most thorough treatise on hitting ever written; yet, few individuals mention his work or attempt to replicate his methods. You rarely see economists Gerald Scully or Tony Krautmann mentioned when attempting to value players, despite the fact that their methods were published in reputable peer-reviewed economics journals, where established experts vetted their work. Academics are not always right, but I believe the checks ensure they are more likely to reach correct conclusions than informal online discussions."
Not to beat a dead horse, but, as I have written before, I'm not very impressed with the output of the academic community, when it comes to sabermetrics. I think the peer review to which JC refers is often not capable of spotting flawed analysis. For my own part, I would certainly prefer to have a paper of my own vetted by Tango or MGL than by an anonymous academic economist who is unlikely to have a strong grounding in sabermetrics. I think experienced amateur sabermetricians are much more likely to find legitimate flaws in my work than professional economists.
However, I haven't seen the academic examples JC mentions. It's certainly possible that Schell, Scully and Krautmann have done work that we non-academics are ignoring. Are any readers familiar with Schell's book, or the Scully and Krautmann valuation methods? If someone can point me to the relevant journal articles, I'll post a review.
Labels: academics, economics, peer review
posted by Phil Birnbaum @ 12/18/2008 04:59:00 PM
16 comments
16 Comments:
The citations you requested sir:
Krautmann, Anthony C. "What's Wrong with Scully-Estimates of a Player's Marginal Revenue Product." Economic Inquiry 37 (1999): 396-381.
MacDonald, Don N., and Morgan O. Reynolds. "Are Baseball Players Paid Their Marginal Products?" Managerial and Decision Economics 15 (1994): 443-457.
"Pay and Performance in Major League Baseball"
Gerald W Scully
American Economic Review, 1974, vol. 64, issue 6, pages 915-30
Enjoy.
As someone nearing the completion of his Ph.D. in economics who has also done a fair bit of sabermetric research on the side, I tend to think that both academic economists and trained sabermetricians need each other. Too much of sabermetrics ignores decision theory and basic econometric techniques to account for unobservables, and too much of academic economists' work on sabermetrics ignores the discoveries within the data that sabermetricians have found. Tom Tango has a very economic methodology, I should say, whether he explicitly states that is what he is doing or not.
Jake: Thanks very much, I'll check those out!
Phil, Schell's book is a favorite of mine (in the analytical field I think it's similar to Win Shares because it has a lot of methodology and some good discoveries along the way.). I wrote a review of it a few years ago and THT also has an online database of his ratings.
Thanks, Studes! I just checked, and it turns out that the public library 300 yards from my house has a copy checked in. I'll pick it up tomorrow.
MattS,
Too much of sabermetrics ignores decision theory and basic econometric techniques to account for unobservables
Can you go into more detail on this? Certainly in terms of free agent signings, the type of analysis that Tango does is just the starting point. We have to consider the player's range of possible performance levels, the team's plan B if they don't sign the player, etc. I can guess from the name that decision theory would have a lot to say on this subject.
Re: techniques to account for unobservables, I'm ignorant but interested. Can you give an example?
No doubt (in my mind) that the field of sabermetrics and the breadth of work in that field can greatly benefit from several other disciplines, including economics.
In fact, if I were versed in economics or econometrics, or even if I were more highly trained in statistics, I would be able to do better work, or at least work with more breadth and depth.
The "problem" is that there are not a whole lot of people who are well-versed in some or all of these disciplines AND are experienced and "expert" sabermetricians (having lots of experience and being good at something are not necessarily the same thing of course).
I believe that most of the academicians who delve into sabermetrics only dabble in it. People like Tango, Phil, myself, and even some of the much younger crowd, like David Gassko, have infinitely more experience and expertise with sabermetrics, than most, if not all, of these academicians who occasionally write about baseball analysis. Even someone like Andy Dolphin, who is a brilliant statistician and mathematician, does not spend much time on sabermetrics anymore, I don't think. And he certainly has another full-time job which I assume takes up much of his time.
That is the "problem."
Someday, perhaps, there will be "professional sabermetricians," who are trained in statistics and/or econometrics and specializing in baseball research, in which case, we will have the best of both worlds, or at least an optimal combination in which to do great baseball research.
In order for that to occur, however, that will likely require a lucrative and somewhat expansive job market for sabermetricians. When there is no market (in $) for something, you are only going to have it exist as a "cottage industry", by and large.
So the bottom line now is that while there is some good work being done by academicians, MOST of the good work is done by the "amateur" sabermetricians and much or most of it is being published not in academic journals, but in informal and semi-formal venues, like the interweb and books.
Whose work is "better" pound for pound (the academician or amateur sabermetrician)? I don't know.
MGL
I agree that much could be gained by online sabremetricians gaining better training in economics and econometrics and that much could be gained by greater interactions between the online community and academics. Many of the problems I see in academic work is that the author fails to have a complete understanding of baseball, which would better inform the analysis.
One key issue in econometrics (and sabremetrics) is often identifying (or attempting to identify) causal effects in masses of nonexperimental data. As a result, econometricians have developed a wide array of techniques for trying to isolate these effects. Many of these depend on trying to estimate and/or remove the impact of unobservables, mentioned in a post above.
A classic example from labor economics--we see the correlation between income/wealth and education. Does education cause income increases? Or is there an latent or unobserved variable (determination, IQ, time preference) that results in people both gaining greater education and greater income? Getting better data is often a solution, but in some cases statistical techniques can get us closer to the truth, too.
So, while academics would benefit from a better understanding of the subtleties of baseball, the online sabremetricians could benefit from the decision modeling and sophisticated methods used in econometrics.
I think both communities have value. I am always amazed at the insights and inventiveness of the sabr crowd. Often, however, their analysis is closer to exploratory in nature, rather than of the detailed kind you would find in academics. I would not underestimate the amount of work done in academics. Google scholar shows close to 20,000 articles since 2003 on baseball and economics. While often using better statistical approaches, the answers may not be "better" because of the weakness I've mentioned.
Both groups ae doing a ton of good solid work.
Good comments, allround.
My one issue with JC's comment is that it reads almost as if that the academic process will yield better results. Clearly, there is fantastic work being done without care of an academic process. And alot of this work is done in a collaborative manner, as each person is inspired by others.
The only precondition I would say that an academic and a sabermetrician needs is a deep understanding of baseball. Too often, I see an academic (and sometimes sabermetricians) be a slave to the regression, even if it runs counter to something intuitive about baseball.
Another bias I see is that while a sabermetrician will be all too eager to read the work of anyone out there, including those from academia, I don't see the same eagerness the other way. Academia will rarely reference work done outside the academic world, essentially marginalizing 90% of the advancements in sabermetrics.
I also see a similar level of ignorance (disproportionately) from the older generation, whose knowledge of sabermetrics stopped when Palmer and James stopped publishing regularly.
The "underground" (i.e., internet, which is really the mainstream at this point) is where most of the great work is happening.
Yes, we'd love to have a stronger knowledge of economic and statistical disciplines. But, as MGL said, we have day jobs too. But, that slows us down just a tiny bit. It's like having -4 glasses, when you really need -4.5.
Lack of knowledge of baseball itself though is like wearing sunglasses instead of prescription glasses.
Speaking of being a slave to regression, I have my name on this paper:
B. Alamar, G. Desjardins, J. Ma and L. Ruprecht “Who Controls the Plate:
Isolating the Pitcher/Batter SubGame” Journal of Quantitative Analysis in Sports (July 2006)
It comes up with the deeply-unintuitive conclusion that the batter-pitcher matchup is 62%-controlled by the batter.
My name's on there because I modeled fielding, but I disagree with the conclusion. In fact, my own (much simpler) analysis found essentially a 50/50 split between pitcher and batter.
I don't think any peer reviewers objected to the conclusion. But it's an example of regression leading you astray from intuitive results.
I don't know if that conclusion is unintuitive, Hawker. We know that a pitcher has much less control than the batter over the BABIP and HR/FB rates. Unless there's a coresponding area of the batter-pitcher matchup where the pitcher has more control than the batter, it makes sense that the batter is a better determinate of the outcome of the batter-pitcher matchup.
Yes, that paper that Hawerchuk linked to was discussed on my site, and one of the authors (or two of them actually) stopped by for a little chat. I told them they were wrong, and demonstrated why they were wrong.
"Peer" does not have to mean "academic". "Peer" would also include the subject matter experts. Just because I can't follow all the stats they were doing doesn't mean that I can't see what they were doing was wrong, calling them out on it, and having a discussion with them to tell them so.
BTW, can anyone leave an example of a specific academic paper that isn't getting the recognition it deserves? Not just a citation, but explain what the paper tells us, what the valuable new knowledge is that the sabermetric community is ignoring or reinventing.
I think the point J.C. was trying to make is being missed. What he's saying -- and what I think I've said in at least one conversation with Phil and probably others reading this blog -- is that motivation and audience differs.
Where a non-academic sabermetrician sees a lack of grounding in baseball, what may be (often?) the case is that the work is targeted to a different audience. A professional economist or game theorist (or whatever field) who studies baseball is primarily motivated by investigating how principles from their field of endeavor operate (or don't) within baseball, and when he writes about it, he is going to write primarily to the audience of others in his field. In doing so, the challenge is to explain the study by giving enough information about the particular application (baseball) so readers in the field can understand it -- but just enough. That means that (1) the authors are necessarily going to leave out things about baseball in the writing, but it doesn't follow that the writer's knowledge of baseball is lacking, and (2) the models and paradigms of analysis are going to come primarily from the audience's field of endeavor, and not from baseball.
(As an aside, baseball's lack of universality makes it much harder to publish about baseball in economics. Tennis and soccer papers of roughly equivalent quality get published in better journals, simply because they're more familiar to a broad audience.)
Like J.C., I would say that I'm not a sabermetrician, either. I'm a game theorist who studies strategic interactions in sports, particularly baseball. If we go by Bill James' original definition of "sabermetrics" as "the search for objective facts about baseball," then I'm not a sabermetrician, since I wasn't motivated by learning objective facts about baseball, but objective facts about the way people make strategic decisions.
It'd be a question of semantics except for the audience issue. I'd write very differently to a group of professional sabermetricians, should such a field ever arise in significant numbers, than I would to a group of professional economists. The identity question does have practical implications.
Off the top of my head no specific papers, Phil, but I rarely see online researchers cite any of the work of David Berri (other than wages of wins) or Anthony Krautman or Martin Schmidt or John Burger or Steve Walters or Rodney Fort or many of the other top sports economists. While their work is often on the MR part of the MRP equation, while online researchers tend to focus on the MP part of that, the two are needed for things like evaluating contracts, which we see a lot of around this time of year. Yet little to none of the economic research on that is ever mentioned.
In response to the question asked of me earlier, I would say that there are a few times that I think decision theory could help guide sabermetrics. Most of these are just examples of when decision theory can help explain certain statistical phenomena.
The best example that I can think of is Tom Tango's critcism of Major League Equivalencies to determine how well a player would have played in the major leagues based on his minor league statistics. The problem is that those players who split their time within a season between the major and minor leagues were brought up to the major leagues explicitly because someone in their organization thought that they were ready. There are two unobservable biases here: (1) The player was likely a bit lucky to be in this position-- someone who lined out 12 times in a row this week is less likely to be called up than someone who hit 12 linedrives and went 9 for 12 in the process, and probably more importantly, (2) Something less numeric made the scouts think that player was ready. Take the example of R.J. Swindle, a lefty with a 51 MPH curveball called up to the major leagues by the Phillies this year. Upon reaching the majors, he looked pathetic. What did work on minor league hitters did not work on major league hitters. It's likely that a 51 MPH curveball will get a lot of bad untrained hitters out. David Wright was not hearing any of that. There are certain things (that scouts know) that give players comparative advantages against major league hitters. Certain weaknesses are more likely to be exploited by major league hitters and other weaknesses are likely to be exploited more equally by major or minor league hitters. There are econometric techniques that try to correct for unobservables, and while I don't really know the detailed process by which MLEs were created, I doubt they come up with this stuff.
Will Carroll of Baseball Prospectus has done studies to determine the effect of throwing a lot of innings on the probability of being injured. One conclusion he came to was that minor league innings have no noticeable statistical effect on being injured. This is counterintuitive and wrong. The most likely explanation is that organizations knew those pitchers who were promoted to he major leagues after throwing minor league innings and knew that they were less likely to be injured. In fact, those pitchers who only pitched a lot of innings in the minor leagues were ones that were perceived to be less risky by their organizations. Simply saying that throwing a lot of innings has no effect is missing the point-- that pitchers who throw a lot of innings were allowed to do so because they were less likely to be injured (according to their organizations). If the data doesn't seem to show a negative effect on the probability of being injured, then that is evidence of cause.
A key aspect of who gets playing time is another important thing that decision theory could help understand. Frequently, people notice changes in a players' lefty/righty splits and conclude something is different about their performance. However, certain hitters who struggle against lefties might only bat against the weakest lefties-- using their statistics without considering this decision masks this effect. For instance, a statistic that was frequently cited around the time of the World Series was that the Phillies actually hit LHP better than RHP. This effect was grossly overexaggerated by the fact that so many of the Phillies plate appearances against lefties were by Chase Utley and Ryah Howard (over 28% of the team's plate appearances, compared to their combined 20% of the team's plate appearances against righties). Managers make decisions in a nonrandom fashion and too frequently these statistics are treated as the results of natural experiments.
I read somewhat recently that groundball pitchers have poorer K/BB ratios than flyball pitchers. This was stated somewhat baldly, but it also requires decision theoretic analysis-- there are pitchers with abilities to induce groundballs and pitchers with abilities to command the strike zone. Pitchers who are weaker in one area will not get to pitch unless they are strong in the other. This negative correlation is merely a result of that fact.
Academics clearly lack knowledge of the intricacies of baseball that sabermetricians have as well as facts in the data-- how many economists doing baseball research know about the DH penalty or PH penalty? All in all, the benefit to combining forces is huge.
Post a Comment
<< Home