### Did "The Book" really find evidence for clutch hitting?

For a long time, the most thorough sabermetric studies showed no evidence for the idea that "clutch hitting" exists -- that some players can "turn it on" more than others when the situation is particularly important. Dick Cramer's 1977 study, which compared batters' 1969 clutch performances to those in 1970, found only a very slight tendency for clutch hitters to repeat. That conclusion was criticized by Bill James in his recent "Underestimating the Fog," but better analyses have existed for many years. Pete Palmer's study in 1990 (.pdf, page 6) compared the actual distribution of players' clutch stats to what would be observed if clutchiness were completely random; it found almost an exact match. Then, in 2005, Tom Ruane did the same thing, but for a much larger population of batters, and came up with a similar result.

But three years ago, in "The Book," authors Tom Tango, Mitchel Lichtman, and Andy Dolphin used a different technique (and, I think, even more data), and came up with a different answer. They found that a tendency to clutch hitting does exist, and has a standard deviation of .008 points of OBA. That is, one out of every six batters will hit more than .008 (8 points) better in the clutch than overall; and, by symmetry, one in six players will hit 8 points *worse* in the clutch.

As far as I know, the authors never published their study in full, and their book gives only an outline of how they did it. But, still, I think I was able to figure out their method -- or at least a method that's probably close to what they did -- and I don't have the same confidence in their conclusions that their book does.

I have two disagreements with their study. First, that they used OBA instead of batting average; second, and more seriously, their result of .008 ~~is not statistically significant~~ is significant only at the 14% level, which is only moderate evidence against the competing view that clutch talent does not exist.

First, OBA. The difference between OBA and BA is mostly a matter of including walks. Walks are certainly important, and if you're trying to measure a player's ability or performance, on-base percentage is a much better measure than batting average. But when it comes to clutch, the traditional question is about *hitting* in the clutch, not *walking* in the clutch.

To my knowledge, ability to draw a base on balls in clutch situations has not been studied. But, unlike hitting, it wouldn't be surprising to find that some players are "better" at it than others. Take Barry Bonds, for example. In clutch situations, Bonds was more likely to be walked. (Here are his career splits.)

Of course, Bonds' walks were mostly intentional, and "The Book" omitted the IBB from its totals. But, still, if Bonds was much more likely to be walked, you'd think he'd also have been more likely to be pitched around; and so he'd draw more unintentional walks in clutch situations as well. Maybe there weren't as many "semi-intentional" bases on balls as intentional ones, but, still, a small number would be enough to account for a chunk of a standard deviation of .008.

For instance: suppose on every team the best hitter increases his OBA by about 17 points (.017) in the clutch, because of the semi-intentional walk, and the worst hitter decreases his OBA by the same 17 points. If the other 7 batters are exactly the same in clutch situations, and only these two are different, that's enough to give you an SD of almost exactly .008.

What's 17 points in practice? It's an increase of about 17 walks per 600 PA. And if a typical hitter gets 60 clutch PA a season, you're talking about 1.7 extra walks for one player on the team, and 1.7 fewer walks for a second player. That difference of 3.4 walks total is enough to give you the SD of .008 that the authors found.

That seems pretty realistic, and reasonable, doesn't it? Well, maybe not; I've artificially decided that only two players on the team are affected, which makes the variance move a lot more for 3.4 walks than it would if every player had some tendency. But, still, intuitively, it does seem like a small effect for walks could explain the whole thing.

And that means:

-- several studies have found no clutch ability in batting average;

-- "The Book" found clutch ability in on-base percentage;

-- intuitively, "clutch walking" would seem to be able to account for everything "The Book" found.

So, with that being the state of the evidence, I am inclined to believe that the evidence still suggests that clutch hitting skill doesn't exist, but "clutch walking" skill does.

----------

But even if the authors had used batting average instead of OBP, and got the same result, the result isn't statistically significant. That's not just my conclusion, but also theirs; they say, on page 102,

"... we can merely state that there is a 68% probability that [the clutch talent SD] is between 3 and 12 points."

Since a 68% probability is 1 SD each way, the authors seem to be implying a standard error of about 4.5 points. That means a 95% confidence interval is about 9 points either way -- which includes zero.

Actually, I get an even wider confidence interval using my method (which might actually be the same as theirs). Let me go through it. For those of you who don't care about the math, you can skip this smaller print.

-- Math/details start here --

The study said that it included 848 players, with an average 2450 PA in non-clutch situations, and 200 in clutch situations. So I created 848 identical players with those numbers, and gave each player exactly zero clutch ability. Every player had an OBA of .340.

From the binomial distribution, the SD of each player's OBA over the non-clutch 2450 PA is .00957. The SD of each player's OBA over the clutch 200 PA is .0335. The SD of the difference between the two is the square root of the sums of the squares, which is .03484. That's 34.84 points of OBA.

That's the variance only due to randomness, or luck. If there truly is variance in players' *talent* for clutch hitting, the observed variance would be higher. How much higher? Well, if you assume that talent and luck are independent, then, as the authors often point out on their blog,

Variance (observed) = variance (talent) + variance (luck)

Since the authors concluded a talent variance of 8 points squared, we can assume that

Variance (observed) = 8 points squared + 34.84 points squared

Which means that

Variance (observed) = 35.75 points squared

Since the SD is the square root of the variance, we get

SD(observed) = 35.75 points

So, presumably, in their population of 848 players, the authors observed the SD of the clutch difference was 35.75 points.

Now, if there really was no such thing as clutch ability, how often would we observe an SD of more than 35.75 points due to luck alone, when the expected number is only 34.84? To check, I ran a simulation, and the answer was: about 14% of the time.

That's obviously not significant, 14%.

Another way to check: the SD of the simulated variance was about .88 of a point. The difference between 35.75 and 34.84 is about .91 of a point. So the observed difference was almost exactly 1 SD from zero. Again, that's not significant.

If we look for a 68% confidence interval like the authors had, 1 SD on each side, we get (34.87, 36.63). That means a 68% confidence interval for clutch talent is 0.1 to 11.3 points. That's different than what the authors gave -- 3 to 12 points -- but I'm not sure why.

Either way, the observed effect is certainly not statistically significant.

-- math/details end here --

To restate my conclusions for those who skipped the math:

The effect "The Book" found is about 1 SD from zero, which is certainly not statistically significant. It's at the 14% level, not the

-------

So, to sum up:

-- two previous studies found no evidence of clutch talent in batting average;

-- Tango/mgl/Dolphin found a small measure of clutch talent, but it wasn't statistically significant.

From that alone, I'd say our conclusion still has to be: not evidence to assume clutch talent. But if you add:

-- Tango/mgl/Dolphin's non-significant result included clutch walks, which common sense strongly suggests *do* vary by player,

Then, to me, that removes most of the last bit of doubt. I think that even if the effect they found is real, there's a really good chance it's caused by walks.

Hey, guys, how about running the study again using batting average?

(UPDATE: some statements on statistical significance replaced by something more accurate.)

## 13 Comments:

Just for interest, here's how often the simulation came up with various estimates for clutch SD:

12 points or more: 2% of the time

8-12 points: 12% of the time

3-8 points: 29% of the time

0-3 points: 6% of the time

No estimate, due to the observed variance being less than the theoretical: 51% of the time

Actually, my summary may be a bit overstated. The Palmer and Ruane studies did not quantify their effects in terms of an SD of talent; they just observed that the distribution intuitively looked like what you'd get if there were no such thing as clutch talent. You might argue that those studies *could* be consistent with a clutch talent distribution of something other than zero, and I suppose you could find a way to check.

I'm pretty sure that the Ruane data is not consistent with a talent SD of .008 (in batting average), but I haven't done any work to check that.

Also, since those two previous studies used a subset of the data that this one used, it's not really true that this study contradicts those studies. This study doesn't just add new data that contradicts the old data -- rather, it adds new data and implicitly re-evaluates the old data at the same time.

So my implied argument, "the old data still contradicts your new data" does have a plausible answer.

But the other arguments -- that (a) the confidence interval is wide and includes zero, and (b) walks are likely to be causing a big chunk of the effect -- are still strong enough that I don't think there's enough evidence to say that "clutch hitting" (by its usual definition) exists.

Didn't they use wOBA in The Book, not OBA? How does that change the results?

They found an SD of 8 points for OBA, and 6 points for wOBA.

To me, that suggests that walks are a large part of the difference, because when you add other stuff and rescale to the same scale as OBA, you get a smaller effect.

Phil

Part of my SABR con clutch presentation in Bostong looked at how OBP changes in various situations. I calculated z-scores the same way as Palmer. So it takes into account how much different a guy did in the clutch and also takes into account if their is a league wide differential between the clutch situation and the non-clutch situation. I looked at all the guys who had 6000+ PAs from 1987-2001. I had 70 guys and 5% of that is about 4. So you normally expect 4 guys to have a z-score of 1.96 or more (or less for -1.96)

Here are the z-scores for OBP with RISP (IBBs included)

Barry Bonds 6.347

Tim Raines 4.502

Paul Molitor 4.352

Chili Davis 4.188

Joe Carter 3.638

Mark McGwire 3.581

Wally Joyner 3.374

Tony Gwynn 3.356

Tony Fernandez 3.337

Eddie Murray 3.321

Ken Caminiti 3.161

David Justice 2.893

Robin Ventura 2.827

Terry Pendleton 2.765

Cal Ripken 2.588

John Olerud 2.383

Jay Buhner 2.378

Albert Belle 2.185

Jeff Bagwell 2.107

Tino Martinez 2.069

Ken Griffey Jr. 2.001

Steve Finley -2.000

Omar Vizquel -2.549

Now RISP with IBBs taken out

Tony Fernandez 3.914

Tim Raines 3.822

Jay Buhner 3.073

Jay Bell 2.999

Wally Joyner 2.937

David Justice 2.626

Cal Ripken 2.511

Ken Caminiti 2.448

Mark McGwire 2.419

Greg Vaughn 2.345

Todd Zeile 2.320

Now close and late with IBBs included

Mark Grace 2.667

Edgar Martinez 2.451

Tony Gwynn 2.044

Tino Martinez 1.986

Travis Fryman -2.557

Now close and late with IBBs taken out

Edgar Martinez 2.216

Mark Grace 2.144

Tino Martinez 2.029

B.J. Surhoff -2.063

Travis Fryman -2.158

Ken Caminiti -2.390

So there are more than 4 guys in each case, even in the cases with IBBs taken out. But we never know who got there by luck and who really was clutch at getting on base. Also, we don't know if there were semi-intentional walks.

Then look at RISP with IBBs taken out. Allo the z-scores are positive and there's a bunch

Cy, thanks. Your comment got cut off, there might be a size limit.

So what we have is: with walks taken out, two studies show almost no clutch effect. With (non-intentional) walks in, two studies (yours, and The Book's) show a very slight clutch effect.

So it seems to me like there is a clutch effect for walks, and that's what's being found.

No, it did not get cut off. Bunch was my last word. I really don't know for sure what the z-scores mean. Notice with RISP that taking out IBBs drastically lowers the significant cases and with IBBs taken out for close and late situations there are at most two clutch hitters since we would expect 4 and there are 6 significant z-scores.

But can't you remember the announcers always commenting on how good Tony Fernandez was at taking a walk with RISP?

Well, we know that some players are "clutch" IBBers, since they get walked a lot in critical situations. That doesn't really tell us about clutch hitting in general.

I also believe that there are guys who are clutch walkers ... maybe Tony Fernandez is one of them. That would explain why you got 6 significant results instead of 4 ... that's the kind of effect you'd expect if there were a slight clutch walk talent.

Phil/Cyril: Couldn't the "clutch" BBs be a function of the pitcher's decision of how best to pitch to these hitters in clutch situations, rather than a skill of the hitters'? It seems to me too much of a coincidence that Cyril's hitters are almost all good-to-great hitters -- exactly the type of hitters who pitchers should walk more than average in clutch PAs.

Back to the Book study: I would add one more potential objection to Phil's: the observed=real+random method assumes that all variance beyond binomial variation equals an underlying skill. But there are many other sources of variance: some hitters may have had many/few clutch PAs at home; some may have had many/few with the platoon advantage; some may have had more than average clutch PAs when facing a pitcher for the 3rd time in the game.

I think to use this method, researchers first need to determine the variance in players' performance under two set of conditions we know to be random, like day of the month. If the clutch/non-clutch variance is greater than the odd/even date variance, for example (and using equal sample sizes), then that's evidence of clutch talent. But I suspect it may not be any larger.

I find the 5% cutoff for statistical significance to be completely arbitrary, so I think it's fair to say that the study in The Book found "some" evidence for clutch hitting.

Guy,

I agree with your points. I definitely was not saying that those z-scores proved anything. I think that Tom Ruane many years ago on the SABR list did the kind of thing you suggest (ood/even). Maybe it is online somewhere. Here is a link to one of his studies. Not sure if has what I was thinking of

http://retrosheet.org/Research/RuaneT/clutch_art.htm

Cy

Guy,

Yes, I agree that "clutch walkers" are probably that way because of the behavior of the pitchers.

For "The Book" study, they did normalize for quality of pitcher and lefty/righty. But I agree with you that there might be other factors.

FWIW, I added a bit of randomness to every PA in my simulation (each PA's randomness was independent of all the others). It didn't change the results hardly at all.

James: sure, what they found is certainly "some evidence." My argument is that, especially in the light of other studies that found nothing, a 15% significance level isn't enough to declare that clutch hitting has been found.

Post a Comment

<< Home