Charlie Pavitt: what's the false positive rate for MLB steroids tests?

This guest post is by Charlie Pavitt. Take it away, Charlie ...

-----

I have a question about drug testing, which for the sake of this blog I will restrict to that for steroids by professional baseball leagues although it is just as relevant for all performance enhancing drugs (PEDs) by all sports organizations in general. I’ve had this question for awhile, but this is a good time to ask it, given that the Tour de France was recently completed (with the usual disqualifications for PEDs along the way) and the Olympics is about to begin (with, leading up to the Games, the same).

No drug test is perfect. All lead to some percentage of false negatives (users whom the test misses) and false positives (innocents whom the test implies are guilty). The latter is what specifically concerns me. In our court of law, one is supposedly innocent until proven guilty. But every few weeks I read of another minor leaguer suspended for a couple months for a positive test. Given that every player knows about the tests, I wonder how many are really still using. Yeah, I can see the possibility that some of the more pampered of them, who have never been punished for anything they’ve done in the past decade because their athletic skill has made them sacrosanct in their communities, probably imagine they can use and get away with it. But I’m not at all convinced that they are all guilty. And, given the publicity it generates, a reported false positive would result in a drug-free player pretty much permanently tarred-and-feathered (Rafael Palmeiro comes to mind; I’m not saying that he wasn’t a user, I have no idea either way, but talk about a positive image permanently crashing down to earth in an eyeblink…).

I just did a bit of web searching (googled “drug testing baseball false positive”) and didn’t find a ton of helpful information on it. One website mentions a study with a huge 14% false positive rate. I’m not the only one concerned (see here, among others).

So here’s my specific, two-part question: What is the false positive rate for the test (assuming there is only one) used in professional baseball? And what precautions are there against false positives (at the very least, there should be a blood sample divided in two, with the second tested in case the first comes up positive)? If anyone has these answers, it is not I’d like to know. More importantly, I’d like the general baseball-fan public to know also.

--Charlie Pavitt

Charlie Pavitt on "Redskins"

Here's occasional guest blogger Charlie Pavitt commenting on the use of the nickname "Redskins."  He invites intelligent responses, as always.

------

Along with others who are troubled by the commercial use of the term "Redskins" in reference to the Washington area football team, I was surprised by the results of the survey indicating that about 75% of Native American respondents did not find the term insulting, and only about 10% did (I don’t remember the exact numbers). Now one thing I have learned over my years as a social scientist is to be suspicious of surprising survey results. I automatically think in terms of biased question writing, but I did have the opportunity to read the question, which was worded something like (again memory fails) the following:

Do you or do you not find the term “Redskin” insulting?

I don’t find that biased, so I can’t quibble with the reported results on that issue.

But I wish we could find out how that sample of Native Americans would have responded to two other questions. One was suggested by a statement in response made by an activist on this issue (third memory flaw – forgotten name):

Are you or are you not comfortable with the commercial use of the term “Redskin” by the Washington area football team?

Is that not the issue that is really at stake here?  The second comes from a remark by a sportswriter made on ESPN (fourth memory failure):

Would you or would you not be insulted if a Caucasian referred to you personally as a “redskin”?

My guess here is that a large proportion would find that insulting.  And if I am correct about that guess, then the term is insulting no matter the results of the question that was actually asked.

One more issue – I am perturbed by those who defend the use of the term because of its “tradition.” In a roughly parallel circumstance, more and more people have realized that the “tradition” of flying the Confederate States flag on state government grounds is insulting, because the reason why there was a Confederate States independent of the United States was to protect the institution of slavery. (And I don’t want to hear about “states’ rights”; the reason why the Confederate states wanted their “rights” was to protect the institution of slavery.) And there is a tradition in part of Africa to mutilate female genitalia.  I could go on and on with examples, but I’ll stop here; “tradition” is no reason for continuing a bad practice.

Anyway, I do think that some of the discussion of this issue is overblown; two hundred years of ill-treatment has left the Native American nations with far worse problems than a football team’s commercial trademark. And maybe I am wrong about the answers one would get from the two survey questions suggested above. But it doesn’t matter to me. I shall continue my practice of referring to the “Washington area football team” by that title rather than their commercial trademark.  And I invite all like-minded people to do the same.

-- Charlie Pavitt

"Who controls the plate?" A review by Charlie Pavitt

Benjamin Alamar, Jeff Ma, Gabriel M. Desjardins, and Lucas Ruprecht, Who Controls the Plate? Isolating the Pitcher/Batter Subgame, Journal of Quantitative Analysis in Sports, Volume 2 Issue 3, 2006, Article 4.

Benjamin Alamar is the editor of the on-line JQAS, and, based on the evidence provided in this article, undoubtedly a very competent researcher. Having said this, this effort is a textbook case of what happens when very competent researchers with little understanding of advances on statistical baseball research attempt to contribute to our knowledge. The authors make two fundamental errors: first, they ignore past research on the topic, and second, they use ill-chosen performance measures in their estimations.

The issue at stake here is, quoting the abstract, “to determine the percentage of the outcome of an at bat that is controlled by a pitcher and the percentage that is controlled by the batter.” They seem to start off well enough. First, using play-by-play data for 2001 to 2003 obtained from STATS, they estimated expected run value for all base-out situations and then determined through multiple regression which game situation variables had an impact on scoring; the significant variables were out, base, league, ninth inning, extra inning, batter lineup position, and park effect. The use of the regression coefficient for each of these variables allowed for a more precise final determination of expected run values. Second, to increase accuracy for expected run values on plays ending with batted balls other than home runs, they combined this information with hit location data, specifically the odds of getting a hit based on where the ball was batted. Third, the results of all this were regressed on pitcher and batter indices to see which accounted for more variance. It is in their choice of these indices where the study goes to pieces. For batters, they choose strikeouts per plate appearance and home runs per plate appearance. These are useful measures, but seem incomplete without some further ways of distinguishing batters (both Babe Ruth and Dave Kingman struck out a lot and hit a lot of homers, but to say the least they had a few differences). For pitchers, they chose strikeouts per home run allowed and outs per bases allowed. The latter is relevant but biased by team defense; the former makes absolutely no sense to me at all. At no point do the authors provide rationale for these choices. Further, the whole exercise ignores bases on balls as a predictor. Anyway, the authors conclude that batters are responsible for 62% of expected run value and pitchers are responsible for the remaining 38%.

Time to editorialize. First, statistical baseball research as a discipline has a lot to learn from other sciences about the cumulative nature of knowledge. Far too many studies are performed in a historical vacuum, and as a consequence even well done efforts are too often wasted in constantly reinventing the wheel. Researchers are either ignorant of or unwilling to cite previous relevant work (case in point; as Phil Birnbaum noted in his review in a recent "By the Numbers", most of the research in the Baseball Prospectus folks’ Baseball Between the Numbers is well-done wheel reinvention). Alamar et al. are academics who should no better, but they have cited no past efforts to compare the impact of pitching and hitting on team performance. I am aware of the two studies attempting what Alamar et al. tried found in the academic literature.

In An Empirical Estimation of a Production Function: The Case of Major league Baseball, American Economist, Volume 25 Number 2, Fall 1981, pages 19-23, Charles E. Zech used the following indices to measure player performance: batting average and home runs to represent batting, stolen bases to stand in for speed, total fielding chances as an indicator of fielding, strikeout to walk ratio for pitching, and career manager won-loss record and years of experience to measure managing. Zech determined that batting accounted for 6 times more variance in team won-loss than pitching; fielding and managing had no significant impact.

Also, there's An Acturial Analysis of the Production Function of Major League Baseball, Journal of Sport Behavior, Volume 11 Number 2, 1988, pages 99-112. Michael D. Akers and Thomas E. Buttross purposely patterned their study after Zech’s, although they replaced total fielding chances with fielding average. They discovered that both hitting, as measured by batting average alone, and managing, as measured by manager’s career won-loss record only, were better predictors than pitching, as measured by strikeout to walk ratio. Again, managing and fielding impacts were small.

Given the similar findings of hitting being more important than pitching, Alamar et al.’s research might have some value as a replication of these past efforts with measurement indices and data sets, increasing our confidence in the validity of the conclusion. But this brings us to the second problem; choice of indices to represent batting and pitching. At the time of Zech’s work, all that researchers had to work with were the standard measures of pitching, batting, and fielding that predated sabermetric work. Even by the time of Akers/Buttross, all that was available other than the standard measures was a couple of years of raw Project Scoresheet data. But even then, we knew that on-base average was a better measure than batting average for getting on base and slugging average better than home runs alone for power, and that we knew that we could do a better job measuring fielding with some measure of plays made than with fielding average. Strikeout to walk ratio is actually a pretty good index for representing pitching, but now we know we can do better by adding home runs allowed to the mix. The point is, Alamar et al. are responsible for knowing this. Their choice of measurement indices is indefensibly ignorant.

In conclusion, Alamar et al. is a lot of sound and fury signifying nothing.

-- Charlie Pavitt

Charlie Pavitt writes reviews of sabermetric studies for "By the Numbers." (Click here, scroll down for current and back issues.) He also maintains a sabermetric bibliography.

Charlie Pavitt: On some implications of the Adam LaRoche situation

Charlie Pavitt occasionally guest posts here ... been a while, but he's back! Here's Charlie's thoughts on Adam Laroche and education.  

-----

Let me begin this essay by stating that if it is true that the White Sox promised Adam LaRoche full clubhouse access for his son Drake, then they are wrong to reverse themselves on that promise now; and if LaRoche only agreed to sign the contract because of the promise, he has good reason to feel betrayed.

But this essay is not about this specific issue. It is about something more general.  My understanding is that LaRoche made a public statement that included something like the following: School is not so important, you can learn more about life in the baseball clubhouse than in school. Even if I am wrong about what LaRoche did or did not say, the question deserves consideration, so I want to discuss my thinking about this general issue and not about this specific case.

You most certainly can learn something about life in the baseball clubhouse. Here are three important things that you can learn: First, that a group of men from drastically different backgrounds (different races/ethnicities/social classes/religions/etc.) can work together in harmony in pursuit of a shared goal. Second, that a group of men from drastically different backgrounds can forge close friendships. Third, that success in one’s pursuits requires what I call the three D’s: desire, dedication, discipline. These are all extremely valuable lessons, and I suppose there are some other things that you can learn in the baseball clubhouse that I cannot think of right now.

But there are many things that are also important in life that you cannot learn in the baseball clubhouse.

First, you cannot learn how to interact in a mature enough manner with women to work in harmony with them in pursuit of a shared goal and to forge close friendships. I have no idea about now, but certainly in the “old days” a baseball clubhouse was anything but a good training ground for learning how to treat women as potential co-workers or non-sexual friends.

Second, you cannot learn how to interact in a mature enough manner with people with a different sexual orientation than you to work in harmony with them in pursuit of a shared goal and to forge close friendships. In this case, it is pretty clear that instances of disparaging treatment are still occurring, although happily the response to these instances has been to demand that the perpetrators grow up and act like adults. I would like to add that it is likely that there are quite a few gay major league baseball players right now who do not feel comfortable enough with how their teammates would react to come out of the closet, given how few past players have felt comfortable enough to do so even after retirement.

Third, you cannot learn how to interact in a mature enough manner with people with disabilities, be they of sight or hearing or physical limitations or psychiatric problems or low “intelligence,” whatever that is, to work in harmony with them in pursuit of a shared goal and to forge close friendships. I will say that baseball players have as a group been sensitive to and supportive of people such as these and also to other players with analogous issues (e.g., alcoholism), which is laudable.  But the issue here is whether one can learn this sensitivity in a baseball clubhouse as well as you can in school.

Fourth, you cannot learn how to interact in a mature enough manner with straight men with no disabilities, but a different temperament than one usually finds among baseball players, to work in harmony with them in pursuit of a shared goal and to forge close friendships.  I am thinking of men who think like artists or musicians or writers (or academics like me), who are basically non-competitive and wish to collaborate with everyone and not only one’s teammates. I am also thinking of men who have made financial sacrifices to dedicate their lives to the betterment of others; those who work in non-profits for pitiful wages, school teachers spending part of their much-lower-than-deserved income on their classroom and students because their school is so badly underfunded. I would not expect active disparagement of such people in the clubhouse, in fact if anything baseball players probably respect the hard work and achievements of such people. But I doubt they learned that attitude in the baseball clubhouse, and in any case the issue is once again whether the ability to understand those mindsets well enough to work alongside and becoming truly friendly with such people is better learned in a baseball clubhouse than in a school.

Fifth, you can learn the sorts of things that can make you a responsible public citizen and contribute to your community and to your nation; for example, someone who votes based on well-thought-out values and relevant knowledge rather than emotion and hearsay.  I am not saying that baseball players as a group do not have adequate public citizen skills, just like the general citizenry my guess is that some do and some don’t, but that you can learn those skills in school far better than in a baseball clubhouse. 

These are the sorts of things that you can learn in a school that you cannot learn in a baseball clubhouse. And, I might add, you can also learn in school about the three important things I listed above that you can learn in a clubhouse; in fact, you can learn them better. You can learn to pursue a goal with or become friends with people from drastically different backgrounds, and you can learn about the importance of the three D’s, and you can learn them better in a school because you are actually participating and not just observing others, as would be generally the case in the clubhouse. Note that unlike the clubhouse, you are learning these skills while interacting with people your own age rather than 10/20 years older. Research has conclusively shown that, while basic language and communication skills are originally learned from intense interaction with one’s immediate family, they are practiced and perfected through intense interaction with one’s age peers.

Now let me add one other thing that you can learn in a school that you cannot learn in a baseball clubhouse. You can learn a marketable skill other than playing baseball.  Again, I want to consider the general issue and stay away from the specific LaRoche situation; my understanding is that Drake is being home-schooled and that Adam LaRoche does many things other than play baseball and I trust that Drake is participating and learning from them. If a player thinks that the baseball clubhouse is a more educational environment for a son than a school, the player is thinking as if there was no question that the son was also going to spend their lives in the baseball clubhouse.  ut what if the son’s interests and temperament are more in line with the examples I mentioned above: the musician/artist/writer/academic, the person who values other’s betterment over personal wealth?  Or even if the son has the desire and temperament to play baseball, but not the skill?  In particular, the latter type of son is woefully unprepared for work, not just for the nuts and bolts of the job but also for working in tandem with women, gay men, and those of different temperament. Or does the player think that, in that instance, it is fine for the son to live his life off the fortune the player is making?

Now, I understand that Drake LaRoche is being home-schooled, and for all I know he is learning about the things I am concerned about, and whether or not this is true in this instance is beside the general point. And I am not saying that boys should never spend any time in a baseball clubhouse, even if it means missing a little school. In conclusion, I am saying the following:

A baseball team that promises a player that the player’s son can totally share the baseball player life at the detriment of schooling is performing a great disservice to the son.

A baseball player who wants his son to totally share the baseball player life at the detriment of schooling is performing a great disservice to his son.

A baseball team that realizes the detriment to the son that the promise has caused and reverses itself on the promise certainly deserves censure for breaking a promise, but is performing a great service to the son.

And a baseball player in the latter situation who retires because of it and, in so doing, insures educational experiences for the son beyond the clubhouse is performing a great service to the son. 

-- Charlie Pavitt

Charlie Pavitt: Steroids and the Hall of Fame

This guest post is from occasional contributor Charlie Pavitt. Here's a link to some of Charlie's previous posts.

------

I am writing today about a much-discussed topic, performance enhancing drugs and Baseball Hall of Fame enshrinement. My goal is not to defend a particular opinion about it, but rather to attempt to lay out five possible positions and some strengths and weaknesses each has. In fact, one reason why I will not defend a particular opinion is that, given these strengths and weaknesses, I am torn among several of the options.

But before I start, a few preliminaries. First, research of which I am aware provides strong evidence that steroid use significantly increases offensive performance, whereas there is little if any evidence that human growth hormone has any impact.

Second, none of this is new. Ancient Greek athletes took then-known stimulants before competitions, and nobody back then batted an eye.

                

Third, one must be careful throwing rocks when one’s own house could potentially, in a different context, be made of glass. When I was in graduate school, if someone had come to me and whispered, "Hey man, I have this pill you can take every day that will make you read, write, and think more quickly and efficiently," I would have been sorely tempted to partake.  In fact, one of my grad school cohort-mates imagined a situation in which you took a pill that provided you with the information you are supposed to learn from assigned reading, with lighter doses for undergraduate students and heavier doses for us grad students. Mighty tempting fantasy.

                

Fourth, and this is critical: Before throwing rocks, one needs to defend the claim that there is something wrong with taking performance enhancing drugs.  The fact that it may be illegal is, in my view, irrelevant, as many illegal items are not only harmless but helpful. For example, without getting into the marijuana debate, it is the case that any use of hemp has been illegal in some places, despite its many many positive applications. And taking something into the body to improve athletic performance is often a good thing. After all, a person can improve athletic performance by eating better, and perhaps taking supplements of necessary vitamins and minerals in moderation. So what’s the difference?  Here’s an argument for that difference; eating better and taking supplements in moderation promotes overall health whereas taking steroids (and overdosing on vitamins/minerals) does quite the opposite. The early deaths of many professional rasslers (I reserve the word "wrestlers" for the real sport), perhaps some football players (Lyle Alzado?), and two well-known baseball players (more on this later) has been linked with steroid use. 

One could then make the claim that it is the use of a substance that causes bodily harm that warrants rejection from the HOF. After all, the criteria for entry include "Integrity, sportsmanship, and character" along with "record, playing ability," and "contributions to the team(s) on which the player played."  

So, the argument continues, PED use is contrary to the former three criteria.  I think the best angle for this argument is that it sets the wrong example for others, particularly young people, whereas eating well and getting one’s vitamins/minerals sets the right example. Fair enough. But: Lots of HOF players were smokers or used chewing tobacco, and Babe Ruth certainly did not set a good dietary example by reportedly eating multiple hot dogs just before games.  And speaking of setting bad examples, if there is anybody enshrined who does not deserve it for absence of integrity etc., it is Adrian "Cap" Anson, who was proactive in the successful attempt to get Moses Fleetwood Walker, the first African-American major league baseball player, banned for the color of his skin.

So this argument leads to a slippery slope. But let us assume that we accept it.  Here are five possible responses, ordered from most lenient to most strict.

Position Number One: Let everyone in. The argument here is that great performers deserve entry no matter why they performed greatly. Buttressing this position is the seeming fact that until the public response to Jose Canseco’s confession among other events forced action, the powers-that-be in MLB’s establishment knew what was going on and intentionally turned a blind eye to it. After all, fans like offense, particularly home runs, and attendance was swelling, so all seemed right with the world. So if that was what baseball was in those days, goes the argument for this position, one must accept it and its great performers no matter what.  ne strength of this argument is that one must not make always-problematic non-performance-related judgments about players, as we will must for the other positions on this issue to be discussed in turn below.  The problem with this argument is that it is contrary to the "integrity, sportsmanship, and character" criteria, condones unhealthy behavior, and as such if anything encourages youthful copy cats.

                

Position Number Two: Let everyone in who deserves enshrinement independently of PED use. This implies that one likely accepts Alex Rodriguez, Roger Clemens, and Barry Bonds, because if one mentally subtracts the PED-fed "value-added" part of their performance, they are still HOF material. But one rejects those who would not have reached performance criteria otherwise: Mark McGwire and Rafael Palmeiro among others come to mind.  Perhaps this makes some sense, but one is still condoning bad behavior by allowing in known users while making questionable judgments about whose performance would have been "good enough" without PEDs.

                

Position Number Three: Ban known users. So Bonds, Clemens, McGwire, Palmeiro, Sammy Sosa, Manny Ramirez, and some others who reached supposed HOF performance levels are out. Also some who approached HOF levels and might otherwise deserve consideration (Miguel Tejada, Jason Giambi) get none. In so doing, we clear the deck of those guilty of poor integrity etc.  Also, it might allow us to consider those whose performance would have reached criteria in another era; think Fred McGriff, who hit as many homers as Lou Gehrig.  But what about those suspected of use? Take Jeff Bagwell for an example. Although there is no clear evidence of his use and he has steadfastly denied it, he did get a lot bigger fairly quickly, hit way more homers than anyone originally expected, and associated closely with the first known user alluded to earlier, Ken Caminiti, whose early death has been partly linked with use. If we lower our performance criteria to allow for McGriff, it also allows for Bagwell.  So now we’ve admitted someone who may have been as guilty as Bonds et al. but whose usage (if any) has not been publicly verified. Setting the in versus out boundary is a pretty questionable judgment call.

                

Position Number Four: Ban everyone either known or rumored to be users.  So Bagwell and Gary Sheffield and perhaps Juan Gonzalez if you think he reached HOF performance levels and maybe even David Ortiz are out, and Mike Piazza should not have been recently admitted. Now we know we’ve kept the HOF free of those with poor integrity etc. But at least in the U.S. court of law one is considered innocent until proven guilty. Take Jeff Bagwell.  Although he got a lot bigger fairly quickly, hit way more homers than anyone originally expected, and associated with a known user, there is no clear evidence of his use and he has steadfastly denied it. Again, setting the in versus out boundary is a pretty questionable judgment call.

                

Position Number Five: Not only ban everyone suspected, but kick out anyone currently in who is suspected. Now we are sure everyone in baseball had the proper integrity etc.  Out goes Mike Piazza. Further, and this is the second player I alluded to at the beginning of this essay, out goes Kirby Puckett.  Jose Canseco fingered him, plus the physical problems that ended his career along with those that ended his life along with his violent post-baseball behavior sure seem to be signals of steroid use. In addition, in a 2002 article, statistician Scott Berry calculated that Puckett’s jump from no home runs his rookie year (1985) to four his sophomore year to 31 his junior year was the most unlikely performance increase in the history of MLB, with an odds of one in 100 million, much greater than similar jumps made by other known or suspected users. But this is all indirect evidence, there are other explanations for all of it (maybe Kirby started taking his vitamins or radically changed his swing between the 1986 and 1987 seasons). And if we kick out Kirby, should we kick out Adrian Anson? (Actually, I think we should, but that’s a side issue here.)  How about Ty Cobb for his racism (to be expected, natural attitudes for a Georgian in his time)?  Babe Ruth for eating all those hot dogs? I do not believe I have heard anyone support this position, but I suppose someone could.

                

So – as I noted at the top, I am frankly torn among several of these options. If I had a vote, my heart would point me toward Position Three, but my head would tell me that it would be hard to rationally defend relative to some of the others (particularly Four). Anyway, I hope that I’ve laid out at least some of the arguments on either side well enough that readers can have an intelligent discussion about it and maybe even add some arguments to my list, and that those who are SURE that their position, whichever it is, is obviously correct think twice about its weaknesses along with its strengths.

-- Charlie Pavitt

New issues of "By the Numbers" now available

Two new issues of “By the Numbers,” the SABR Statistical Analysis Newsletter that I edit, are now available at my website, www.philbirnbaum.com .

The August issue contains:

-- a review of two recent “Chance” articles by Charlie Pavitt;
-- an analysis by Victor Wang of the historical value breakdown between OBP and SLG;
-- a steroid-related study by Yoshio Muki and John Hanks, showing that sluggers today see their power diminish much less slowly with age than they used to.

The November issue contains:

-- a Charlie Pavitt review of the JQAS recent paper on error rates;
-- an article by Gary Gillette and Pete Palmer on measuring middle relief performance, complete with updated data for 2006;
-- a study by Abbott Katz on the consistent historical relationship between at-bats and batting average.

<bleg>

If you’re wondering why the August and November issues are coming out in January, it’s because we were short of material until recently. If you’re interested in contributing an article, we’d be pretty darn grateful. Even if you’ve already had a blog post on your work, or published it online, we’d be interested in having it. Our only rule is that we won’t publish an exact duplicate of something that’s appeared elsewhere – if you want to do some revisions, or update your data for a more recent year, or anything else that constitutes value added, we’re happy.

To see the kinds of articles we’ve used, check out the many back issues at the above link. (People have said that the February, 1999 issue was one of the best.)

“By the Numbers” goes out to about 1,000 SABR members – 900 by download, 100 by postal mail – of which some are “famous” established sabermetricians. I suspect that most BTN readers don’t frequent online sabermetrics sites, so you might consider a contribution as a way to reach a new audience.

</bleg>

Thanks, and hope you enjoy the issues.

Charlie Pavitt reviews a hitting streak study

I [Charlie Pavitt] am writing this in response to Trent McCotter’s piece on hitting streaks from the 2008 Baseball Research Journal. I want to begin by commending Trent on this fine piece of work. In short, a series of Monte Carlo tests revealed that the number of actual hitting streaks of lengths beginning with 5 games and ending with 35 games or more between 1957 and 2006 was, in each case, noticeably greater than what would have been expected by chance. It is always good to see evidence inconsistent with our “received wisdom.” What I have to say here in no way attempts to contradict his research findings. My problem is with his attempt to explain them.

Trent first proposed three “common-sense” explanations for what he found. The first was that a batter might face relatively poor pitching for a significant stretch of time, increasing the odds of a long streak. But, in his words (page 64), “the problem with this explanation is that it’s too short-sided; you can’t face bad pitching for too long without it noticeably increasing your numbers, plus you can’t play twenty games in a row against bad pitching staffs, which is what would be required to put together a long streak.” He then goes on (page 65) “The same reasoning is why playing at a hitter-friendly stadium doesn’t seem to work either, since these effects don’t continue for the necessary several weeks in a row.” His third “common-sense explanation” is that, as hitting overall is thought to be better during the warm months, hitting streaks may be more common than expected during June through August. This is because, and this is critical (page 65), “hitting streaks are exponential…a player who hits .300 for two months will be less likely to have a hitting streak than a player who hits .200 one month and .400 the next...[because]…hitting streaks tend to highly favor batters who are hitting very well, even if it’s just for a short period.” This is absolutely correct. Unlike the first two proposed explanations, in this case Trent looked for relevant evidence, claiming that he looked for more streaks in June, July or August and found no more than in May. Trent, how about April and September?

Anyway, rejecting all three of these, Trent then proposed two possible psychological explanations. The first is that hitters aware of a streak intentionally change their approach to go for more singles, particularly when the streak gets long; and he has evidence that longer streaks occur less randomly than shorter ones, which would occur under this assumption (players would more likely think about keeping their streak going when it was long ongoing). The second is that hot hands really exist, and his claimed evidence is that taking games out of his random sample in which the player does not start increases the number of predicted hitting streaks, bringing it more in line with the number that actually occurred. Makes sense; a hitting streak is easier to maintain the more at bats one has in a game. He proposes that this could reflect real life because managers would start a player proportionally more often when he was hitting well. True, but we should keep in mind that the same statistical effect for starting games would occur whether there is a hot hand or not. In other words, I don’t think his evidence is very telling.

I want to be very clear here about my position on this issue. I have absolutely no problem with the suggestion that players’ performance is impacted by psychological factors; I don’t see how they aren’t. My problem is with the way in which those suggestions are treated. If we are serious about sabermetrics as a science, then we have to meet the standards of scientific explanation. As esteemed philosopher Karl Popper pointed out in his now-classic 1934 book "The Logic of Scientific Discovery," if a proposed explanation for observations is impossible to disconfirm, then we can’t take it seriously as scientific explanation. This is my problem with Trent’s treatment. Let us suppose that rather than finding more hitting streaks than chance would allow, Trent had found fewer. He could then say that the reason for this is that batters crumble under the stress of thinking about the streak and perform worse than they would normally. If Trent found no difference, he could then say that batters are psychologically unaffected by their circumstance. The point is that this sort of attempted explanation can be used to explain anything, and given our present store of knowledge about player psychology they are impossible to evaluate. Again, Trent’s proposals may be correct, but we can’t judge them, so we can’t take them as seriously as Trent appears to.

In contrast, the first three proposed explanations can be disconfirmed, so we can take them more seriously. Trent claims to have disconfirmed the third, but we need to know about April and September. But the real issue I have is with his dismissal of the first two, because he did not apply the logic in their case that he correctly applied for his “hot weather” proposal. Let me begin with the first. A batter does not have to face a bad pitching staff in consecutive games for his odds of a hitting streak to increase. Let us suppose that a batter faces worse pitching than average during only 10 of 30 games in May and makes up for it by facing worse pitching than average during 20 of 30 games in June. We use the same exact logic that Trent used correctly for the “hot weather” proposal; his odds of having a batting streak, which would occur during June, would be greater than another batter that faced worse pitching than average during 15 games in May and 15 games in June. The same explanation goes for hitter-friendly and hitter-unfriendly ballparks, and is strengthened in this case because of well-supported known differences in ballpark effects. If a player’s home field was hitter-friendly and, during a stretch of time, many of his road games were in hitters’ parks, he could easily have 20 or more games in this context in a given month.

I have no idea whether either of these two explanations for Trent’s findings is correct. But the difference between these and his psychological proposals is that we could test these two and not those he favors. Given the importance of Trent’s original findings, I would obviously like to see that happen. And I would very much like it if we remain very careful about not taking our psychological speculations too seriously.

Charlie Pavitt: Open the Hall of Fame to sabermetric pioneers

This guest post is from occasional contributor Charlie Pavitt. Here's a link to some of Charlie's previous posts.

-----

Induction into the National Baseball Hall of Fame (HOF) is of course the highest honor available to those associated with the game.  When one thinks of the HOF, one first thinks of the greatest players, such as the first five inductees in 1936 (Cobb, Johnson, Matthewson, Ruth, and Wagner). But other categories of contributors were added almost immediately; league presidents (Morgan Bulkeley, Ban Johnson) and managers (Mack, McGraw) plus George Wright in 1937, pioneers (Alexander Cartwright and Henry Chadwick) in 1938, owners (Charles Comiskey) in 1939, umpires (Bill Klem) and what would now be considered general managers (Ed Barrow) in 1953, and even union leaders (Marvin Miller, this year for induction next year). There is an additional type of honor associated with the HOF for contributions to the game; the J. G. Taylor Spink Award (given by the Baseball Writers Association of America) annually since 1962, the Ford C. Frick Award for broadcasters annually since 1978, and thus far five Buck O’Neill Lifetime Achievement Awards given every three years since 2008.  Even songs get honored ("Centerfield", 2010; "Talkin' Baseball", 2011).

But what about sabermetricians? Are they not having a major influence on the game?  Are there not some who are deserving of an honor of this magnitude?

I am proposing that an honor analogous to the Spink, Frick, and O’Neill awards be given to sabermetricians who have made significant and influential contributions to the analytic study of baseball. I would have called it the Henry Chadwick Award to pay tribute to the inventor of the box score, batting average, and earned run average, but SABR has already reserved that title for its award for research contributions, a few of which have gone to sabermetricians but most to other contributors. So instead I will call it the F. C. Lane award, not in reference to Frank C. Lane (general manager of several teams in the 1950s and 1960s) but rather Ferdinand C. Lane, editor of the Baseball Magazine between 1911 and 1937. Lane wrote two articles for the publication ("Why the System of Batting Should Be Reformed," January 1917, pages 52-60; "The Base on Balls," March 1917, pages 93-95) in which he proposed linear weight formulas for evaluating batting performance, the second of which is remarkably accurate.

I shall now list those whom I think have made "significant and influential contributions to the analytic study of baseball" (that phrase was purposely worded in order to delineate the intent of the award). The HOF began inductions with five players, so I will propose who I think should be the first five recipients:

George Lindsay

Between 1959 and 1963, based on data from a few hundred games either he or his father had scored, George Lindsay published three academic articles in which he examined issues such as the stability of the batting average, average run expectancies for each number of outs during an inning and for different innings, the length of extra-inning games, the distribution of total runs for each team in a game, the odds of winning games with various leads in each inning, and the value of intentional walks and base stealing. It was revolutionary work, and opened up areas of study that have been built upon by generations of sabermetricians since.

 

Bill James

Starting with his first-self-published Baseball Abstract back in 1977, James built up an audience that resulted in the Abstract becoming a conventionally-published best seller between 1982 and 1988.  During those years, he proposed numerous concepts – to name just three, Runs Created, the Pythagorean Equation, and the Defensive Spectrum – that have influenced sabermetric work ever since.  But at least if not more important were his other contributions.  He proposed and got off the ground Project Scoresheet, the first volunteer effort to compile pitch-by-pitch data for games to be made freely available to researchers; this was the forerunner and inspiration for Retrosheet. During the same years as the Abstract was conventionally published, he oversaw a sabermetric newsletter/journal, the Baseball Analyst, which provided a pre-Internet outlet for amateur sabermetricians (including myself) who had few if any other opportunities to get their work out to the public.  Perhaps most importantly, his work was the first serious sabermetric (a term he coined) analysis many of us saw, and served as an inspiration for us to try our hand at it too. I might add that calls for James to be inducted into the Hall itself can be found on a New York Times article from January 20, 2019 by Jamie Malinowski and the Last Word on Baseball website by its editor Evan Thompson.

Pete Palmer

George Lindsay’s work was not readily available. The Hidden Game of Baseball, written by Palmer and John Thorn, was, and included both a history of previous quantitative work and advancement on that work in the spirit of Lindsay’s. Palmer’s use of linear-weight equations to measure offensive performance and of run expectancies to evaluate strategy options were not entirely new, as Lane and Lindsay had respectively been first, but it was Palmer’s presentation that served to familiarize those that followed with these possibilities, and as with James these were inspirations to many of us to try our hands at baseball analytics ourselves.  Probably the most important of Palmer’s contributions has been On-base Plus Slugging (OPS), one of the few sabermetric concepts to have become commonplace on baseball broadcasts.

 

David Smith

I’ve already mentioned Project Scoresheet, which lasted as a volunteer organization from 1984 through 1989. I do not wish to go into its fiery ending, a product of a fight about conflict of interest and, in the end, money.  Out of its ashes like the proverbial phoenix rose Retrosheet, the go-to online source for data describing what occurred during all games dating back to 1973, most games back to 1920, and some from before then. Since its beginning, those involved with Retrosheet have known not to repeat the Project’s errors and have made data freely available to everyone even if the intended use for that data is personal financial profit. Dave Smith was the last director of Project Scoresheet, the motivator behind the beginning of Retrosheet, and the latter’s president ever since. Although it is primed to continue when Dave is gone, Retrosheet’s existence would be inconceivable without him.  Baseball Prospectus’s analyst Russell Carleton, whose work relies on Retrosheet, has made it clear in print that he thinks that Dave should be inducted into the Hall itself.

 

Sean Forman

It is true that Forman copied from other sources, but no matter; it took a lot of work to begin what is now the go-to online source for data on seasonal performance. Baseball Reference began as a one-man sideline for an academic, and has become home to information about all American major team sports plus world-wide info on “real” football. 

 

-----

Here are two others that I believe should eventually be recipients.

Sherri Nichols

Only two women have been bestowed with HOF-related awards; Claire Smith is a past winner of the Spink Award and Rachel Robinson is a recipient of the O’Neill Award.  Sherri Nichols would become the third. I became convinced that she deserved it after reading Ben Lindbergh’s tribute, and recommend it for all interested in learning about the "founding mother" of sabermetrics. I remember when the late Pete DeCoursey (I was scoring Project Scoresheet Phillies games and he was our team captain) proposed the concept of Defensive Average, for which (as Lindbergh’s article noted) Nichols did the computations. This was revolutionary work at that time, and laid the groundwork for all of the advanced fielding information we now have at our disposal.

 

Tom Tango

Tango has had significant influence on many areas of sabermetric work, two of which have joined Palmer’s OPS as commonplaces on baseball-related broadcasts. Wins Above Replacement (WAR) was actually Bill James’s idea, but James never tried to implement it. Tango has helped define it, and his offensive index wOBA is at the basis of the two most prominent instantiations, those from Baseball Reference (alternatively referred to as bWAR and rWAR) and FanGraphs (fWAR).  Leverage was an idea whose time had come, as our blogmaster Phil Birnbaum came up with the same concept at about the same time, but it was Tango’s usage that became definitive. His Fielding Independent Pitching (FIP) corrective to weaknesses in ERA is also well-known and often used. Tango currently oversees data collection for MLB Advanced Media, and has done definitive work on MLBAM’s measurement of fielding (click here for a magisterial discussion of that topic).

There are some historical figures that might be deserving; Craig Wright, Dick Cramer, and Allan Roth come to mind as possibilities. Maybe even Earnshaw Cook, as wrong as he was about just about everything, because of what he was attempting to do without the data he needed to do it right (see his Percentage Baseball book for a historically significant document). Perhaps the Award could also go to organizations as a whole, such as Baseball Prospectus and FanGraphs; if so, SABR should get it first.

Charlie Pavitt on recent clutch hitting studies

I am writing this entry in order to put my two cents in on an essay written by Bill James and published in Baseball Research Journal 33 (2005) a couple years back called “Underestimating the Fog.” It got a lot of response at the time, most notably from Jim Albert and Phil Birnbaum in By the Numbers, Volume 15 Number 1 (see page 3 for Jim, page 7 for Phil). Phil’s response was specifically relevant to one of the claims Bill made in that essay: that the conclusions made by previous analysts that clutch hitters as such don’t exist were premature. Bill believed the existence of clutch hitters to be an open question; Phil attempted to provide evidence that we have good reason to believe that there is no such thing. I wish to make some comments on the validity of Bill’s claim. Although I will use the clutch hitting case as an example, my comments are also relevant to analogous issues such as “hot and cold streaks aren’t real” (which Bill also claimed to be an open question in his essay).

I write this from the standpoint of the traditional logic of hypothesis testing (apologies to those readers already familiar with this way of thinking). In this tradition, one proposes a “research hypothesis” (e.g., clutch hitting exists) and a corresponding “null hypothesis” voicing its opposite (e.g., clutch hitting does not exist). One then compares one’s data to these hypotheses, and finds support for whichever hypothesis the data more closely reflects. The problem is that no matter what the data look like, one can never be sure that one’s finding is accurate, because, due to the law of chance, fluky data is always a possibility. It could be that in the real world there is no such thing as clutch hitting but that it appears to exist in one’s data, leading to an incorrect conclusion in favor of the research hypothesis. This is like flipping a fair coin 20 times and getting 17 heads (which I have seen occur twice in a course exercise I have used) and concluding that one’s coin is biased; statisticians call this “Type I error,” although it would have better been called Type I bad luck. It could also be that in the real world there is such a thing as clutch hitting but it does not show up in one’s data, leading to an incorrect conclusion in favor of the null hypothesis. This is like flipping a biased coin 20 times and, despite the bias, getting 10 heads; this is known as Type II error, although again it is more bad luck than mistake.

On any single occasion, one can never be sure whether one’s findings are accurate or not. However, in a given research situation, we can use laws of probability to estimate the odds that Type I or Type II error would occur, and use these estimates as the basis for our decision concerning which hypothesis should gain support. If our data appears to support the research hypothesis and the calculated probability of Time I error is very small (one’s “significance level,” traditionally in the social and behavioral sciences, less than five percent), then we claim support for that research hypothesis, although we know there is some chance that claim is wrong. If our data appears to support the null hypothesis and the calculated probability of Type II error is very small, then we can analogously claim support for the null hypothesis with the corresponding proviso.

The point I wish to make concerns the second of these possibilities. Although there have been two recent cases in which researchers claim to have found an impact, both of which I will comment on later, research as a general rule has supported the null hypothesis that clutch hitting as a skill does not exist. However, our trust in this conclusion has to be tempered by the chance for Type II error. Now, rather than Type II error, the odds of supporting the null hypothesis in error, we tend to think about the issue in terms of its complement; statistical power, the odds of correctly supporting the null hypothesis. In this case, that would be the odds of finding evidence for clutch hitting in a given data set assuming it really exists. Now, statistical power (and Type II error) is determined by three different factors. First, significance level, which is as noted earlier traditionally .05 in the social and behavioral sciences. If one become more lenient and makes it say .10, then in so doing one increases one’s statistical power. The problem with doing so is that one increases the odds of Type I error; as a consequence, we don’t normally muck around with significance level. Second, one’s sample size; the more data we can work with, the more random fluctuations in data are likely to cancel one another out, making it more likely to find something that is there. Third, the strength of the effect itself. This is called “effect size.”

Let us turn now to clutch hitting. To be clear, the issue is not whether there is such a thing as a clutch hit; of course there is. The issue is whether there are certain players that have some special talent such that they are consistently better in the clutch than in “normal” game situations, whereas other players do not have this talent. If clutch hitting as an inherent ability has a significant factor in performance, it would be easier to find than if it its impact is weak. Given the difficulty people have had in finding a clutch hitting effect, if it actually exists, it must have a very small effect size. Given that we don’t mess with significance level, we would need to increase our sample size as much as feasible if we want to increase our power and the resulting odds of finding an effect if it exists. Having said this: If we run a study and find no evidence for clutch hitting, we will be confident in that conclusion to the extent that statistical power is high and, saying the same thing a different way, Type II error is low. We can calculate that power, based on our sample size, our significance level, and the effect size of clutch hitting. We know our significance level, we know our sample size, but we do not know our effect size. We can only guestimate it. We can never prove that clutch hitting does not exist as an inherent talent, no matter what our data look like, because our guestimate of its effect may be too large. Herein lies the fog that Bill alluded to.

Now, let’s get to what Phil Birnbaum did. Bill James had claimed that the Dick Cramer method for studying the clutch hitting issue, looking at year-to-year correlations in clutch performance (see the 1977 Baseball Research Journal), is invalid, because one cannot assume anything based on random data. In his article, using the Elias definition of a “late inning pressure situation” to distinguish clutch from non-clutch plate appearances, Phil computed the statistical power of this type of test assuming various effect sizes. This is a good way to deal with the fog issue; show the possibilities and then let the reader decide. For example, based on his data, if the correlation in clutch performance from season to season was a trifling .08, he would have found it 97.7% of the time; as he did not find it, if such an effect exists, it must be even weaker than that. Assuming the validity of his data set and analysis, what cannot be denied from his work is that if clutch hitting does exist as a talent, then its effect size must be extremely small. The moral of our story: Bill James was right about the following: we will never be in the position to definitively say that an effect does not exist. This is because, no matter how large our sample, our analysis will assume some effect size, and the effect could always be smaller than our assumption. But Bill did not understand that we can estimate the odds that there is no effect given different conceivable effect sizes. And at some point, we can also place ourselves in the position of concluding that if such-and-such an effect does exist, it is too small for have any appreciable impact on the game.

Let me now turn to the two recent claimed demonstrations of a clutch hitting effect. One is by Nate Silver in the Baseball Prospectus folks’ Baseball Between the Numbers. I like Nate’s general method; rather than making a clear distinction between clutch and no-clutch situations, Nate used the concept of situational leverage (the likelihood that the outcome of a given game situation will determine which team wins the game) to estimate the degree of clutch-ness in batter’s plate appearances, which could then be compared to the player’s overall performance to see if he tended to do better or worse as situational leverage increased. Turning a dichotomous clutch/non-clutch distinction into a gradated degree of clutch-ness scale (in technical terms, from ordinal measurement into interval measurement) improves the subtlety of one’s measure, which in effect is another way to improve one’s statistical power. Now, apologies for getting a bit technical; using a sample of 292 players, Nate found a correlation of .33 in their “situational hitting” (his term, which is probably better then “clutch” given his method) between the first and last half of their careers, which is significant at better than .001; in other words, Type I error rate is less than one in a thousand. Finally, some evidence that clutch hitting might indeed exist as a skill. Nonetheless, Nate is very careful to downplay the effect, guestimating that it may account for 2 percent of the impact of offense on wins.

Tom Tango, Mitchel Lichtman, and Andrew Dolphin also devote a chapter to the issue in The Book. They unfortunately use the dichotomous clutch versus non-clutch distinction despite the fact that most of the analyses in their book rely on situational leverage, and they claim to have found an effect but do not provide any relevant data for the rest of us to examine (other than lists of players with good and bad clutch hitting indices, which does not rate as evidence that the effect is non-random). So I cannot judge the validity of their conclusion one way or another. This chapter is one of the few weak points of an otherwise impressive body of work.

-- Charlie Pavitt

A Response to Win Shares: A Partial Defense of Linear Weights

In a new article written for this blog, Charlie Pavitt defends the Linear Weights system against some Bill James criticisms:

"One of the impressions that reading [Win Shares] left me with is the seemingly constant attacks Bill makes in this book against Pete Palmer’s Linear Weights player evaluation system ... I believe these attacks to be at least partly misguided, in that, at least implicitly, Pete’s system is attempting to do something quite a bit different than Bill’s system is."

Click for the full article, "A Response to Win Shares: A Partial Defense of Linear Weights."

For those of you not familiar with Charlie's work, he regularly writes reviews of sabermetric studies for "By the Numbers." (Click here, scroll down for current and back issues.) He also maintains an indispensable sabermetric bibliography.

Charlie Pavitt: A long-winded rant concerning the evaluation of fielding

By Charlie Pavitt

I start with Branch Rickey’s famous quote from his (and Allan Roth’s) ground-breaking Life Magazine statistical-analytic article “Goodby to Some Old Baseball Ideas” (August 2, 1954); “there’s nothing anybody can do with fielding” (this and subsequent quotes all from page 83). Rickey/Roth realized that fielding averages were “utterly worthless as a yardstick” because they say nothing about fielding range. But their conclusion that “fielding could not be measured” is surprising given their insights into evaluating batting and pitching. There’s quite a bit we can do with fielding. But we’ve not gotten as far in this regard as we have with batting and pitching; and when I see many of the evaluation methods used by some of our otherwise-best analysts, I am disappointed. My goal in this blog is to rant long-windedly about how I think we ought to be thinking about fielding evaluation in the first place.

Let’s start with the basics. We make progress in evaluating an aspect of baseball when we can successfully break the aspect into its component skills, measure each skill, and then combine these measurements in a meaningful common metric. Take batting prowess. There are four major component skills; the ability to get base-hits, the ability to hit for power, the ability to coax base on balls, and the ability to steal bases without being caught. One factor common among the four that suggests a metric is that successful performance occurs when bases are gained without the loss of outs. Each can be measured fairly easily in that way, and the measurements combined, resulting in tools such as OPS (sans the steals) and total average, both of which work fairly well despite their simplicity. Another common metric is number of runs gained, leading to runs created and a large set of regression-based methods.

Defense as a whole works analogously; it is relatively simple to concoct measures of bases or runs given up relative to either outs gained or opportunity (e.g., innings played). The trick is to distinguish the pitching and fielding parts of the equation. Recent work starting with Voros McCracken’s insight implies that pitchers are best evaluated through examination of events that fielders cannot influence. This means that pitching prowess has the following three components; the ability to hit the strike zone (as measured by walks and hit-by-pitches), the ability to miss bats (as measured by strikeouts), and the ability to keep batted balls in play (as measured by home runs allowed). Measures based only on these skills would follow; the Baseball Prospectus people have proposed just that (DIPS ERA) in their recent Baseball Between the Numbers book. (I should add that pitchers do have some influence on batted balls in play, which implies a fourth skill; but this influence is far less than we thought pre-McCracken, and at this time I would agree with those recommending that we ignore this influence for the time being.)

This leaves batted balls in play as the responsibility of the fielder, so that the evaluation of what fielders do with these is the issue at hand. Can we distinguish the component skills? (We need to exclude catcher defense here, which is a whole ‘nother matter.) Surely they include range and sure-handedness on batted balls and throwing ability. For middle infielders, one must consider adeptness as the middle man in a double play; for first basemen, the knack of picking up throws in the dirt. The trick lies in measuring each of them and then combining them into a common metric. This is a challenging project. Baseball differs from football, basketball, soccer, etc. in that it is an individual sport in a team context; i.e. its outcomes are primarily due to the pitcher versus hitter matchup. However, with fielding, particularly infielding, team coordination matters. As a whole it makes sense to credit fielders with the out when they successfully field a ball in their territory. But what do we do with the second out in a double play? Should the fielder get full credit, or the middleman, or should they split it? What do we do with assists from the outfield, especially when cut-off men are involved?

In order to make this rant particularly long-winded, I shall continue with a bit of history. Back in the March 1976 issue of "Baseball Digest", Bill James proposed the seemingly-novel idea that we measure infielding by the number of putouts and assists the infielder makes per game (in so doing reinventing, a hundred years later, an identical but soon-forgotten measure credited to Al Wright). Range factor was clearly an advance over fielding percentage, but it was laced with problems. First, it intermixes different skills without previous reflection concerning each. Infielders amass putouts and assists both through fielding batted balls and through participation in double plays and force outs, but these are reflective of different skills, only the first of which is directly relevant to range. I did a couple of studies in which I attempted to solve this problem by measuring infielding purely by assists, under the assumption that they were a purer measure of range than putouts. Bill published both, respectively in issues 24 (June 1986) and 31 (August 1987) of the "Baseball Analyst", although I believe that he disagreed with my method given the display of range that can be shown when infielders catch pop-ups far from their position. And I knew full well that many assists are racked up as a double-play middleman. Second, and these were the issues that my two studies were really about, range factor ignored the fact that pitching staffs with high strikeout totals limit infielder opportunities to field balls; pitching staffs with a high proportion of innings taken by lefthanded pitchers will face a preponderance of righthanded batters, leading to proportionally more grounders to the third baseman and shortstop and fewer to the second and first baseman, when compared to pitching staffs with few lefty innings. I presented this material at a SABR convention near Washington D.C., if I remember correctly in 1986; during my presentation, an audience member noted that pitching staff groundball vs. fly ball tendencies have analogous implications. Interestingly enough, John Dewan assumed the pitching-handedness bias and presented fielding measures adjusted for this problem at the same SABR convention, for Dewan beginning a concern with this issue that has continued to this time.

It was obvious that if we wanted to measure fielding plays made on batted balls independently of participation in double plays and free of biases due to pitching staff tendencies, we would have to go beyond the standard statistical measures of fielding and use play-by-play data to measure the proportion of balls hit into the portion of the ballpark for which each position is responsible that are successfully fielded. Fortunately, at about this time Project Scoresheet was beginning to supply the needed data, and analysts started using it for this purpose. The earliest effort of which I am aware was Pete DeCoursey’s work on what he called defensive average, first published in the March 1989 issue of a (sadly) short-lived publication called the "Philadelphia Baseball File." I believe others among “amateur” statistical analysts continued in this vein, and would be happy to hear from readers who have information on anybody doing this work during the 1990s. As for the “professionals,” and probably thanks to Dewan, the STATS annual Player Profiles books during the 1990s included a measure called zone rating, which unfortunately gave credit for two plays for fielded balls turned into double plays, in so doing conflating two different skills.

What are the lessons I think we should take home from all this? Let’s start with two do nots. First, do not use the standard indices, because no matter how well they are massaged they do not provide valid information. An example of this is the Defense ratings appearing in the Baseball Prospectus group’s annual. They are not always clear about their methods; from a description in "Baseball Between The Numbers" (page 97), it seems that Clay Davenport’s version of fielding runs begins with the standard measures and then adjusts them for park factor and the pitching staff tendencies mentioned above. As far as I can tell, they do not take the double play problem into account, but otherwise these adjustments are right-headed. But the method doesn’t seem to work. If you glance through their books, their Defense ratings for players differ wildly from year to year, at least by eye-ball analysis far more than random factors would allow. And they don’t trust their own numbers, regularly making verbal comments clearly inconsistent with their own calculations. For two examples from the 2007 book: on page 418 they ask whether Chris Duncan is “the single worst defensive outfielder in modern memory,” but his 2006 ratings are slightly above average (+1) in both left and right field; on page 381, they wittily call Pat Burrell “the Zeno’s Paradox Outfielder, in that no matter how close he seems to be to catching the ball he’s only halfway there,” but his 2006 rating (-2) isn’t all that bad. An interesting case, of course, is, the normally-maligned Derek Jeter. According to Davenport’s numbers, after years of futility (-12 in 1999, -22 in 2000, -17 in 2001, -19 in 2002, -15 in 2003) Jeter improved to -4 in 2004 and became a good shortstop the past two seasons (+12 in 2005, +7 in 2006), and this change is the main theme in Chapter 3-1 of "Baseball Between The Numbers" as a result of having Alex Rodriguez next to him. As I will describe below, we have good evidence that Jeter’s defense has not improved, and, while I like most of what BP does, I don’t trust their fielding numbers for a second.

Second, if you are going to combine indices for the different skills involved in fielding, do not do so arbitrarily. The example here is Bill James. I admire what he attempted to do in his Win Shares book, but much of it is based on what seem to be arbitrary decisions that make no sense to me. To begin, pitching is given 67.5 percent of the credit for defense and fielding the remaining 32.5 percent; the reader is never told where these numbers come from. The division of this 32.5 across positions is performed according to criteria that the author himself admits to be arbitrary. Ratings for each position are made in the context of their different skills; here is the method for infielders:

Second base - forty points double plays, thirty points assists, twenty points error percentage, and ten points putouts.
Third base - fifty points assists, thirty points errors, ten points sacrifice hits allows, and ten points double plays.
Shortstops - forty points assists, thirty points double plays, twenty points error percentage, and ten percent putouts.

There is no indication of where these numbers come from: why double plays are the most important part of second base play, why putouts are irrelevant to third base and so low for the other positions; could this be a late recognition that I was correct more than fifteen years before the book was written about removing putouts from range factor? Unless and until we get a convincing rationale for these proportions, as with the BP work I don’t trust any of Bill’s ratings for a second.

What would I like to see? First and foremost, I would like to see all measurements of range and sure-handedness based on play-by-play data. Dewan has continued work in this regard with his Baseball Info Solutions; his book The Fielding Bible is a gold mine of valuable data on defense. I might add that Dewan’s work makes plain Jeter’s continued defensive shortcomings; in 2005, he ranked 31st in Dewan’s metric among 32 rated shortstops. David Pinto’s Probabilistic Model of Range and Mitchel Lichtman’s Ultimate Zone Rating are basically identical with Dewan’s work in this regard.

But more generally, I think it is possible to come up with a fielding metric that does a fairly good job of evaluating most aspects of fielding (catchers excluded) in the context of either bases or runs. Beginning with range and sure-handedness, turning a measure such as Dewan’s or Pinto’s into either a base or run measure should be easy; Lichtman already calculates Ultimate Zone Rating in run metrics. As for the other aspects of fielding: in Volume 10 Number 3 of SABR’s Baseball by the Numbers, Clem Comly proposed a nice method (Average Arm Equivalent Method, or ARM) for evaluating the number of runs outfielders either save or cost their team based on their number of assists relative to their number of opportunities to throw out baserunners. As ARM is already in a run metric, it would merely have to be summed with Ultimate Zone Rating for outfielders. We do need to come up with a good method for dividing up responsibility for the second out on double plays; are there any out there of which I am unaware? I know that Pinto has recently put some attention to the double-play problem. I admit that the first baseman’s ability to turn errant throws into outs gets shortchanged here; I’m not sure whether any of the currently available play-by-play data provides enough detail for us to enter that into the equation. It may not be perfect, but contra Rickey/Roth there’s quite a bit we can do with fielding.

May, 2012, "By the Numbers" now available

The May, 2012 issue of "By the Numbers" is now available.  

The issue (.PDF) is here.  Back issues are on the SABR website, and also on my website here.

It features four articles:

-- Charlie Pavitt: "Academic Research: Correcting the Aging Bias"

-- Phil Birnbaum: "Review: "Basic Ball""

-- Donald A. Coffin: "Trends in "Strategy" in Major League Baseball"

-- Kevin D. Dayaratna and Steven J. Miller: "First Order Approximations of the Pythagorean Formula"

Hope you enjoy the issue! 

May, 2014 issue of "By the Numbers"

"By the Numbers" is the statistical analysis publication of SABR (The Society for American Baseball Research).  

A new issue is now available. Here's the SABR link (.pdf). If that doesn't work, you can always find it at my own website.  If you like it and want back issues, the SABR link is here.

There are three articles in the issue.

-- First, Charlie Pavitt reviews "The Sabermetric Revolution," the recent book by Ben Baumer and Andrew Zimbalist.

-- Next, Don Coffin argues that the biggest statistical change in baseball, over the decades, is something other than home runs.

-- Finally, John F. McDonald tries some variations on the Pythagorean and "10 runs equals one win" estimators, to see if accuracy can be improved.

May issue of "By the Numbers" released

The new issue of "By the Numbers," the SABR statistical analysis newsletter, is now available at my website here. It's dated May, 2007, but it's still new.

It contains articles by Charlie Pavitt, J.P. Caillault, David Roher, Russell Carleton, Fred Worth, and Andrew Boslett/Matt Hoover/Thomas Pfaff.

As always, all back issues are available at my website, philbirnbaum.com .

New Issue: By the Numbers

The just-released May issue of “By the Numbers,” the sabermetrics newsletter of the Society for American Baseball Research (SABR), is now available at my website. I am the editor, so I won’t be writing reviews of these articles, just these summaries:

“Academic Research: Pitcher Luck vs. Skill” by Charlie Pavitt: a review of two academic studies by Jim Albert, which talk about how to break down a pitcher’s record between luck and skill components.

“The Wages of Wins – Right Questions, Wrong Answers” by Phil Birnbaum (me): a review of the recent book on sports sabermetrics/econometrics.

“The Interleague Home Field Advantage” by Eric Callahan, Thomas J. Pfaff, and Brian Reynolds: a study that shows that the home field advantage in interleague games is significantly (in the baseball sense, not the statistical sense) higher than in other games.

“Best of the Ball Hawks” by Tom Hanrahan: a study that evaluates the best centerfielders of all time using Win Shares and the Baseball Prospectus ratings.

We are always looking for material for future issues – please e-mail me if interested in contributing.

Are umpires biased in favor of star pitchers?

Are MLB umpires are biased in favor of All-Star pitchers? An academic study, released last spring, says they are. Authored by business professors Braden King and Jerry Kim, it's called "Seeing Stars: Matthew Effects and Status Bias in Major League Baseball Umpiring."

"What Umpires Get Wrong" is the title of an Op-Ed piece in the New York Times where the authors summarize their study. Umps, they write, favor "higher status" pitchers when making ball/strike calls:

"Umpires tend to make errors in ways that favor players who have established themselves at the top of the game's status hierarchy."

But there's nothing special about umpires, the authors say. In deferring to pitchers with high status, umps are just exhibiting an inherent unconscious bias that affects everyone: 

" ... our findings are also suggestive of the way that people in any sort of evaluative role — not just umpires — are unconsciously biased by simple 'status characteristics.' Even constant monitoring and incentives can fail to train such biases out of us."

Well ... as sympathetic as I am to the authors' argument about status bias in regular life, I have to disagree that the study supports their conclusion in any meaningful way.

-----

The authors looked at PITCHf/x data for the 2008 and 2009 seasons, and found all instances where the umpire miscalled a ball or strike, based on the true, measured x/y coordinates of the pitch. After a large multiple regression, they found that umpire errors tend to be more favorable for "high status" pitchers -- defined as those with more All-Star appearances, and those who give up fewer walks per game. 

For instance, in one of their regressions, the odds of a favorable miscall -- the umpire calling a strike on a pitch that was actually out of the strike zone -- increased by 0.047 for every previous All-Star appearance by the pitcher. (It was a logit regression, but for low-probability events like these, the number itself is a close approximation of the geometric difference. So you can think of 0.047 as a 5 percent increase.)

The pitcher's odds also increased 1.4 percent for each year of service, and another 2.5 percent for each percentage point improvement in career BB/PA.

For unfavorable miscalls -- balls called on pitches that should have been strikes -- the effects were smaller, but still in favor of the better pitchers.

I have some issues with the regression, but will get to those in a future post. For now ... well, it seems to me that even if you accept that these results are correct, couldn't there be other, much more plausible explanations than status bias?

1. Maybe umpires significantly base their decisions on how well the pitcher hits the target the catcher sets up. Good pitchers come close to the target, and the umpire thinks, "good control" and calls it a strike. Bad pitchers vary, and the catcher moves the glove, and the umpire thinks, "not what was intended," and calls it a ball.

The authors talk about this, but they consider it an attribute of catcher skill, or "pitch framing," which they adjust for in their regression. I always thought of pitch framing as the catcher's ability to make it appear that he's not moving the glove as much as he actually is. That's separate from the pitcher's ability to hit the target.

2. Every umpire has a different strike zone. If a particular ump is calling a strike on a low pitch that day, a control pitcher is more able to exploit that opportunity by hitting the spot. That shows up as an umpire error in the control pitcher's favor, but it's actually just a change in the definition of the strike zone, applied equally to both pitchers.

3. The study controlled for the pitch's distance from the strike zone, but there's more to pitching than location. Better pitchers probably have better movement on their pitches, making them more deceptive. Those might deceive the umpire as well as the batter. 

Perhaps umpires give deceptive pitches the benefit of the doubt -- when the pitch has unusual movement, and it's close, they tend to call it a strike, either way. That would explain why the good pitchers get favorable miscalls. It's not their status, or anything about their identity -- just the trajectory of the balls they throw. 

4. And what I think is the most important possibility: the umpires are Bayesian, trying to maximize their accuracy. 

Start with this. Suppose that umpires are completely unbiased based on status -- in fact, they don't even know who the pitcher is. In that case, would an All-Star have the same chance of a favorable or unfavorable call as a bad pitcher? Would the data show them as equal?

I don't think so. 

There are times when an umpire isn't really sure about whether a pitch is a ball or a strike, but has to make a quick judgment anyway. It's a given that "high-status" control pitchers throw more strikes overall; that's probably also true in those "umpire not sure" situations. 

Let's suppose a borderline pitch is a strike 60% of the time when it's from an All-Star, but only 40% of the time when it's from a mediocre pitcher.

If the umpire is completely unbiased, what should he do? Maybe call it a strike 50% of the time, since that's the overall rate. 

But then: the good pitcher will get only five strike calls when he deserves six, and the bad pitcher will get five strike calls when he only deserves four. The good pitcher suffers, and the bad pitcher benefits.

So, unbiased umpires benefit mediocre pitchers. Even if umpires were completely free of bias, the authors' methodology would nonetheless conclude that umpires are unfairly favoring low-status pitchers!

----

Of course, that's not what's happening, since in real life, it's the better pitchers who seem to be benefiting. (But, actually, that does lead to a fifth (perhaps implausible) possibility for what the authors observed: umpires are unbiased, but the *worse* pitchers throw more deceptive pitches for strikes.)

So, there's something else happening. And, it might just be the umpires trying to improve their accuracy.

Our hypothetical unbiased umpire will have miscalled 5 out of 10 pitches for each player. To reduce his miscall rate, he might change his strategy to a Bayesian one. 

Since he understands that the star pitcher has a 60% true strike rate in these difficult cases, he might call *all* strikes in those situations. And, since he knows the bad pitcher's strike rate is only 40%, he might call *all balls* on those pitches. 

That is: the umpire chooses the call most likely to be correct. 60% beats 40%.

With that strategy, the umpire's overall accuracy rate improves to 60%. Even if he has no desire, conscious or unconscious, to favor the ace for the specific reason of "high status", it looks like he does -- but that's just a side-effect of a deliberate attempt to increase overall accuracy.

In other words: it could be that umpires *consciously* take the history of the pitcher into account, because they believe it's more important to minimize the number of wrong calls than to spread them evenly among different skills of pitcher. 

That could just as plausibly be what the authors are observing.

How can the ump improve his accuracy without winding up advantaging or disadvantaging any particular "status" of pitcher? By calling strikes in exactly the proportion he expects from each. For the good pitcher, he calls strikes 60% of the time when he's in doubt. For the bad pitcher, he calls 40% strikes. 

That strategy increases his accuracy rate only marginally -- from 50 percent to 52 percent (60% squared plus 40% squared). But, now, at least, neither pitcher can claim he's being hurt by umpire bias. 

But, even though the result is equitable, it's only because the umpire DOES have a "status bias." He's treating the two pitchers differently, on the basis of their historical performance. But King and Kim's study won't be able to tell there's a bias, because neither pitcher is hurt. The bias is at exactly the right level.

Is that what we should want umpires to do, bias just enough to balance the advantage with the disadvantage? That's a moral question, rather than an empirical one. 

Which are the most ethical instructions to give to the umpires? 

1. 

Make what you think is the correct call, on a "more likely than not" basis, *without* taking the pitcher's identity into account.

Advantages: No "status bias."  Every pitcher is treated the same.

Disadvantages: The good pitchers wind up being disadvantaged, and the bad pitchers advantaged. Also, overall accuracy suffers.

2. 

Make what you think is the correct call, on a "more likely than not" basis, but *do* take the pitcher's identity into account.

Advantages: Maximizes overall accuracy.

Disadvantages: The bad pitchers wind up being disadvantaged, and the good pitchers advantaged.

3. 

Make what you think is the most likely correct call, but adjust only slightly for the pitcher's identity, just enough that, overall, no type of pitcher is either advantaged or disadvantaged.

Advantages: No pitcher has an inherent advantage just because he's better or worse.

Disadvantages: Hard for an umpire to calibrate his brain to get it just right. Also, overall accuracy not as good as it could be. And, how do you explain this strategy to umpires and players and fans?


Which of the three is the right answer, morally? I don't know. Actually, I don't think there necessarily is one -- I think any of the three is fair, if understood by all parties, and applied consistently. Your opinion may vary, and I may be wrong. But, that's a side issue.

------

Getting back to the study: the fact that umpires make more favorable mistakes for good pitchers than bad pitchers is not, by any means, evidence that they are unconsciously biased against pitchers based on "status." It could just as easily be one of several other, more plausible reasons. 

So that's why I don't accept the study's conclusions. 

There's also another reason -- the regression itself. I'll talk about that next post.




(Hat tip: Charlie Pavitt)