The hot hand debate vs. the clutch hitting debate

In the "hot hand" debate between Guy Molyneux and Joshua Miller I posted about last time, I continue to accept Guy's position, that "the hot hand has a negligible impact on competitive sports outcomes."

Josh's counterargument is that some evidence for a hot hand has emerged, and it's big. That's true: after correcting for the error in the Gilovich paper, Miller and co-author Adam Sanjurjo did find evidence for a hot hand in the shooting data of Gilovich's experiment. They also found a significant hot hand in the NBA's three-point shooting contest. 

I still don't believe that those necessarily suggest a similar hot hand "in the wild" (as Guy puts it), especially considering that to my knowledge, none has been found in actual games. 

As Guy says,

"Personally, I find it easy to believe that humans may get into (and out of) a rhythm for some extremely repetitive tasks – like shooting a large number of 3-point baskets. Perhaps this kind of “muscle memory” momentum exists, and is revealed in controlled experiments."

-------

Of course, I keep an open mind: maybe players *do* get "hot" in real game situations, and maybe we'll eventually see evidence for it. 

But ... that evidence will be hard to find. As I have written before, and as Josh acknowledges himself, it's hard to pinpoint when a "hot hand" actually occurs, because streaks happen randomly without the player actually being "hot."

I think I've used this example in the past: suppose you have a 50 percent shooter when he's normal, but he turns in to a 60 percent shooter when he's "hot," which is one-tenth of the time. His overall rate is 51 percent.

Suppose that player makes three consecutive shots. Does that mean he's in his "hot" state? Not necessarily. Even when he's "normal," he's going to have times where he makes three consecutive shots just by random luck. And since he's "normal" nine times as often as he's "hot," the normal streaks will outweigh the hot streaks.

Specifically, only 19 percent of three-hit streaks will come when the player is hot. In other words, four out of five streaks are false positives.

(Normally, he makes three consecutive shots one time in 8. Hot, he makes three consecutive shots one time in 4.63. In 100 sequences, he'll be "normal" 90 times, for an average 11.25 streaks. In his 10 "hot" times, he'll make 2.16 streaks. That's about a 4:1 ratio.)

Averaging the real hotness with the fake hotness, the player will shoot 51.9 percent after a streak. But his overall rate is 51.0 percent. It takes a huge sample size to notice the difference between 51 percent and 51.9 percent.

Even if you do notice a difference, does it really make an impact on game decisions? Are you really going to give the player the ball more because his expectation is 0.9 percent higher, for an indeterminate amout of time?

-------

And that's my main disagreement with Josh's argument. I do acknowledge his finding that there's evidence of a "muscle memory" hot hand, and it does seem reasonable to think that if there's a hot hand in one circumstance, there's probably one in real games. After all, *part* of basketball is muscle memory ... maybe it fades when you don't take shots in quick succession, but it still seems plausible that maybe, some days you're more calibrated than others. If your muscles and brain are slightly different each day, or even each quarter, it's easy to imagine that some days, the mean of your instinctive shooting motion is right on the money, but, other days, it's a bit short.

But the argument isn't really about the *existence* of a hot hand -- it's about the *size* of the hot hand, whether it makes a real difference in games. And I think Guy is right that the effect has to be negligible. Because, even if you have a very large change in talent,  from 50 percent to 60 percent -- and a significant frequency of "hotness", 10 percent of the time -- you still only wind up with a 0.9 percent increased expectation after a streak of three hits. 

You could argue that, well, maybe 50 to 60 percent understates the true effect ... and you could get a stronger signal by looking at longer streaks.

That's true. But, to me, that argument actually *hurts* the case for the hot hand. Because, with so much data available, and so many examples of long streaks, a signal of high-enough strength should have been found by now, no? 


-------

This debate, it seems to me, echoes the clutch hitting debate almost perfectly.

For years, we framed the state of the evidence as "clutch hitting doesn't exist," because we couldn't find any evidence of signal in the noise. Then, a decade ago, Bill James published his famous "Underestimating the Fog" essay, in which he argued (and I agree) that you can't prove a negative, and the "fog" is so thick that there could, in fact, be a true clutch hitting talent, that we have been unable to notice.

That's true -- clutch hitting talent may, in fact, exist. But ... while we can't prove it doesn't exist, we CAN prove that if it does exist, it's very small. My study (.pdf) showed the spread (SD) among hitters would have to be less than 10 points of batting average (.010). "The Book" found it to be even smaller, .008 of wOBA (a metric that includes all offensive components, but is scaled to look like on-base percentage). 

To my experience, a sizable part of the fan community seizes on the "clutch hitting could be real" finding, but ignores the "clutch hitting can't be any more than tiny" finding. 

The implicit logic goes something like, 

1. Bill James thinks clutch hitting exists!
2. My favorite player came through in the clutch a lot more than normal!
3. Therefore, my favorite player is a clutch hitter who's much better than normal when it counts!

But that doesn't follow. Most strong clutch hitting performances will happen because of luck. Your great clutch hitting performance is probably a false positive. Sure, a strong clutch performance is more likely to happen given that a player is truly clutch, but, even then, with an SD of 10 points, there's no way your .250 hitter who hit .320 in the clutch is anything near a .320 clutch hitter. If you did the math, maybe you'd find that you should expect him to be .253, or something. 

Well, it's the same here, with the hot hand:

1. Miller and Sanjurjo found a real hot hand!
2. Therefore, hot hand is not a myth!
3. My favorite player just hit his last five three-point attempts!
4. Therefore, my player is hot and they should give him the ball more!

Same bad logic. Most streaks happen because of luck. The streak you just saw is probably a false positive. Sure, streaks will happen given that a player truly has a hot hand, but, even then, given how small the effect must be, there's no way your usual 50-percent-guy is anything near superstar level when hot. If you had the evidence and did the math, maybe you'd find that you should expect him to be 52 percent, or something.

-------

For some reason, fans do care about whether clutch hitting and the hot hand actually happen, but *don't* care how big the effect is. I bet psychologists have a cognitive fallacy for this, the "Zero Shades of Grey" fallacy or the "Give Them an Inch" fallacy or the "God Exists Therefore My Religion is Correct" fallacy or something, where people are unwilling to believe something into existence -- but, once given license to believe, are willing to assign it whatever properties their intuition comes up with.

So until someone shows us evidence of an observable, strong hot hand in real games, I would have to agree with Guy:

"... fans’ belief in the hot hand (in real games) is a cognitive error."  

The error is not in believing the hot hand exists, but in believing the hot hand is big enough to matter. 

Science may say there's a strong likelihood that intelligent life exists on other planets -- but it's still a cognitive error to believe every unexplained light in the sky is an alien flying saucer.

Guy Molyneux and Joshua Miller debate the hot hand

Here's a good "hot hand" debate between Guy Molyneux and Joshua Miller, over at Andrew Gelman's blog.

A bit of background, if you like, before you go there.

-----

In 1985, Thomas Gilovich, Robert Vallone, and Amos Tversky published a study refuting the "hot hand" hypothesis, which is the assumption that after a player has recently performed exceptionally well, he is likely to be "hot" and continue to perform exceptionally well.

The Gilovich [et al] study showed three results:

1. NBA players were actually *worse* after recent field goal successes than after recent failures;

2. NBA players showed no significant correlation between their first free throw and second free throw; and

3. In an experiment set up by Gilovich, which involved long series of repeated shots by college basketball players, there was no significant improvement after a series of hits.

-----

Then, in 2015-2016, Joshua Miller and Adam Sanjurjo found a flaw in Gilovich's reasoning. 

The most intuitive way to describe the flaw is this:

Gilovich assumed that if a player shot (say) 50 percent over the full sequence of 100 shots, you'd expect him to shoot 50 percent after a hit, and 50 percent after a miss.

But this is clearly incorrect. If a player hit 50 out of 100, then, if he made his (or her) first shot, what's left is 49 out of 99. You wouldn't expect 50%, then, but only about 49.5%. And, similarly, you'd expect 50.5% after a miss.

By assuming 50%, the Gilovich study set the benchmark too high, and would call a player cold or neutral when he was actually neutral or hot.

(That's a special case of the flaw Miller and Sanjurjo found, which applies only to the "after one hit" case. For what happens after a streak of two or more consecutive hits, it's more complicated. Coincidentally, the flaw is actually identical to one that Steven Landsburg posted for a similar problem, which I wrote about back in 2010. See my post here, or check out the Miller paper linked to above.)

------

The Miller [and Sanjurjo] paper corrected the flaw, and found that in Gilovich's experiment, there was indeed a hot hand, and a large one. In the Gilovich paper, shooters and observers were allowed to bet on whether the next shot would be made. The hit rate was actually seven percentage points higher when they decided to bet high, compared to when they decided to bet low (for example, 60 percent compared to 53 percent).

That suggests that the true hot hand effect must be higher than that -- because, if seven percentage points was what the participants observed in advance, who knows what they didn't observe? Maybe they only started betting when a streak got long, so they missed out on the part of the "hot hand" effect at the beginning of the streak.

However, there was no evidence of a hot hand in the other two parts of the Gilovich paper. In one part, players seem to hit field goals *worse* after a hit than after a miss -- but, corrected for the flaw, it seems (to my eye) that the effect is around zero. And, the "second free throw after the first" doesn't feature the flaw, so the results stand.

------

In addition, in a separate paper, Miller and Sanjurjo analyzed the results of the NBA's three-point contest, and found a hot hand there, too. I wrote about that in two posts in 2015. 

-------

From that, Miller argues that the hot hand *does* exist, and we now have evidence for it, and we need to take it seriously, and it's not a cognitive error to believe the hot hand represents something real, rather than just random occurrences in random sequences. 

Moreover, he argues that teams and players might actually benefit from taking a "hot hand" into account when formulating strategy -- not in any specific way, but, rather, that, in theory, there could be a benefit to be found somewhere.

He also uses an "absence of evidence is not evidence of absence"-type argument, pointing out that if all you have is binary data, of hits and misses, there could be a substantial hot hand effect in real life, but one that you'd be unable to find in the data unless you had a very large sample. I consider that argument a parallel to Bill James' "Underestimating the Fog" argument for clutch hitting -- that the methods we're using are too weak to find it even if it were there.

------

And that's where Guy comes in. 

Here's that link again. Be sure to check the comments ... most of the real debate resides there, where Miller and Guy engage each other's arguments directly.

A new "hot hand" study finds a plausible effect

There's a recent baseball study (main page, .pdf) that claims to find a significant "hot hand" effect. Not just statistically significant, but fairly large:

"Strikingly, we find recent performance is highly significant in predicting performance ... Furthermore these effects are of a significant magnitude: for instance, ... a batter who is “hot” in home runs is 15-25% more likely (0.5-0.75 percentage points or about one half a standard deviation) more likely to hit a home run in his next at bat."

Translating that more concretely into baseball terms: imagine a batter who normally hits 20 HR in a season. The authors are saying that when he's on a hot streak of home runs, he actually hits like a 23 or 25 home run talent. 

That's a strong effect. I don't think even home field advantage is that big, is it?

In any case, after reading the paper ... well, I think the study's conclusions are seriously exaggerated. Because, part of what the authors consider a "hot hand" effect doesn't have anything to do with streakiness at all.

------

The study took all player seasons from 2000 to 2011, subject to an AB minimum. Then, the authors tried to predict every single AB for every player in that span. 

To get an estimate for that AB, the authors considered:

(a) the player's performance in the preceding 25 AB; and
(b) the player's performance in the rest of the season, except that they excluded a "window" of 50 AB before the 25, and 50 AB after the AB being predicted.

To make this go easier, I'm going to call the 25 AB sample the "streak AB" (since it measures how streaky the player was). I'm going to call the two 50 AB exclusions the "window AB". And, I'm going to call the rest of the season the "talent AB," since that's what's being used as a proxy for the player's ability.

Just to do an example: Suppose a player had 501 AB one year, and the authors are trying to predict AB number 201. They'd divide up the season like this:

1. the first 125 AB (part of the "talent AB")
2. the next 50 AB, (part of the "window AB)
3. the next 25 AB (the "streak AB")
4. the single "current AB" being predicted
5. the next 50 AB, (part of the "window AB")
6. the next 250 AB (part of the "talent AB").

They run a regression to predict (4), based on two variables:

B1 -- the player's ability, with is how he did in (1) and (6) combined
B2 -- the player's performance in (3), the 25 "streak AB" that show how "hot" or "cold" he was, going into the current AB.

Well, not just those two -- they also include the obvious control variables, like season, park, opposing pitcher's talent, platoon, and home field advantage. 

(Why did they choose to exclude the "windows" (2) and (5)? They say that because the windows occur so close to the actual streak, they might themselves be subject to streakiness, and that would bias the results.)

What did the study find? That the estimate of "B2" was large and significant. Holding the performance in the 375 AB "ability" sample constant, the better the player did in the immediately preceding 25 "streak" AB, the better he did in the current AB.

In other words, a hot player continues to be hot!

------

But there's a problem with that conclusion, which you might have figured out already. The methodology isn't actually controlling for talent properly.

Suppose you have two players, identical in the "talent" estimate -- in 375 AB each, they both hit exactly 21 home runs.

And suppose that in the streak AB, they were different. In the 25 "streak" AB, player A didn't hit any home runs. But player B hit five additional homers.

In that case, do you really expect them to hit identically in the 26th AB? No, you don't. And not because of streakiness -- but, rather, because player B has demonstrated himself to be a better home run hitter than player A, by a margin of 26 to 21. 

In other words, the regression coefficient confounds two factors -- streakiness, and additional evidence of the players' relative talent.

-------

Here's an example that might make the point a bit clearer.

(a)  in their first 10 AB -- the "talent" AB -- Mario Mendoza and Babe Ruth both fail to hit a HR.
(b)  in their second 100 AB -- the "streak" AB -- Mendoza hits no HR, but the Babe hits 11.
(c)  in their third 100 AB -- the "current" AB -- Mendoza again hits no HR, but the Babe hits 10.

Is that evidence of a hot hand? By the authors' logic, yes, it is. They would say:

1. The two players were identical in talent, from the control sample of (a). 
2. In (b), Ruth was hot, while Mendoza was cold.
3. In (c), Ruth outhit Mendoza. Therefore, it must have been the hot hand in (b) that caused the difference in (c)!

But, of course, the point is ... (b) is not just evidence of which player was hot. It's also evidence of which player was *better*. 

-------

Now, the authors did actually understand this was an issue. 

In a previous version of their paper, they hadn't. In 2014, when Tango posted a link on his site, it took only two-and-a-half hours for commenter Kincaid to point out the problem (comment 6).  (There was a follow-up discussion too.)

The authors took note, and now realize that their estimates of streakiness are confounded by the fact that they're not truly controlling for established performance. 

The easiest way for them to correct the problem would have been just to include the 25 AB in the talent variable. In the "Player A vs. Player B" example, instead of populating the regression with "21/0" and "21/4", they could easily have populated it with "21/0 and "25/4". 

Which they did, except -- only in one regression, and only in an appendix that's for the web version only.

For the published article, they decided to leave the regression the way it was, but, afterwards, try to break down the coefficient to figure out how much of the effect was streakiness, and how much was talent. Actually, the portion I'm calling "talent" they decided to call "learning," on the grounds that it's caused by performance in the "streak AB" allowing us to "learn" more about the player's long-term ability. 

Fine, except: they still chose to define "hot hand" as the SUM of "streakiness" and "learning," on the grounds that ... well, here's how they explain it:

"The association of a hot hand with predictability introduces an issue in interpretation, that is also present but generally unacknowledged in other papers in the area. In particular, predictability may derive from short-term changes in ability, or from learning about longer-term ability. ... We use the term “hot hand” synonymously with short-term predictability, which encompasses both streakiness and learning."

To paraphrase, what they're saying is something like:

"The whole point of "hot hand" studies is to see how well we can predict future performance. So the "hot hand" effect SHOULD include "learning," because the important thing is that the performance after the "hot hand" is higher, and, for predictive purposes, we shouldn't care what caused it to be higher."

I think that's nuts. 

Because, the "learning" only exists in this study because the authors deliberately chose to leave some of the known data out of the talent estimate.

They looked at a 25/4 player (25 home runs of which 4 were during the "streak"), and a 21/0 player (21 HR, 0 during the streak), and said, "hey, let's deliberately UNLEARN about the performance during the streak time, and treat them as identical 21-HR players. Then, we'll RE-LEARN that the 25/4 guy was actually better, and treat that as part of the hot hand effect."

-------

So, that's why the authors' estimates of the actual "hot hand" effect (as normally understood outside of this paper) are way too high. They answered the wrong question. They answered,

"If a guy hitting .250 has a hot streak and raises his average to .260, how much better will he be than a guy hitting .250 who has a cold streak and lowers his average to .240?"

They really should have answered,

"If a guy hitting .240 has a hot streak and raises his average to .250, how much better will he be than a guy hitting .260 who has a cold streak and lowers his average to .250?"

--------

But, as I mentioned, the authors DID try to decompose their estimates into "streakiness" and "learning," so they actually did provide good evidence to help answer the real question.

How did they decompose it? They realized that if streakiness didn't exist at all, each individual "streak AB" should have the same weight as each individual "talent AB". It turned out that the individual "streak AB" were actually more predictive, so the difference must be due to streakiness.

For HR, they found the coefficient for the "streak AB" batting average was .0749. If a "streak AB" were exactly as important as important as a "talent AB", the coefficient would have been .0437. The difference, .0312, can maybe be attributed to streakiness.

In that case, the "hot hand" effect -- as the authors define it, as the sum of the two parts -- is 58% learning, and 42% streakiness.

-------

They didn't have to do all that, actually, since they DID run a regression where the Streak AB were included in the Talent AB. That's Table A20 in the paper (page 59 of the .pdf), and we can read off the streakiness coefficient directly. It's .0271, which is still statistically significant.

What does that mean for prediction?

It means that to predict future performance, based on the HR rate during the streak, only 2.71 percent of the "hotness" is real. You have to regress 97.29 percent to the mean. 

Suppose a player hit home runs at a rate of 20 HR per 500 AB, including the streak. During the streak, he hit 4 HR in 25 AB, which is a rate of 80 HR per 500 AB. What should we expect in the AB that immediately follows the streak?

Well, during the streak, the player hit at a rate 60 HR / 500 AB higher than normal. 60 times 2.71 percent equals 1.6. So, in the AB following the streak, we'd expect him to hit at a rate of 21.6 HR, instead of just 20.

-------

In addition to HR, the authors looked at streaks for hits, strikeouts, and walks. I'll do a similar calculation for those, again from Table A20.

Batting Average

Suppose a player hits .270 overall (except for the one AB we're predicting), but has a hot streak where he hits .420. What should we expect immediately after the streak?

The coefficient is .0053. 150 points above average, times .0053, is ... less than one point. The .270 hitter becomes maybe a .271 hitter.

Strikeouts

Suppose a player normally strikes out 100 times per 500 AB, but struck out at double that rate during the streak (which is 10 K in those 25 AB). What should we expect?

The coefficient is .0279. 100 rate points above average, times .0279, is 2.79. So, we should expect the batter's K rate to be 102.79 per 500 AB, instead of just 100. 

Walks

Suppose a player normally walks 80 times per 500 PA, but had a streak where he walked twice as often. What's the expectation after the streak?

The coefficient here is larger, .0674. So, instead of walking at a rate of 80 per 500 PA, we should expect a walk rate of 85.4. Well, that's a decent effect. Not huge, but something.

(The authors used PA instead of AB as the basis for the walk regression, for obvious reasons.)

---------

It's particularly frustrating that the paper is so misleading, because, there actually IS an indication of some sort of streakiness. 

Of course, for practical purposes, the size of the effect means it's not that important in baseball terms. You have to quadruple your HR rate over a 25 AB streak to get even a 10 percent increase in HR expectation in your next single AB. At best, if you double your walk rate over a hot streak, you walk expectation goes up about 7 percent.

But it's still a significant finding in terms of theory, perhaps the best evidence I've ever seen that there's at least *some* effect. It's unfortunate that the paper chooses to inflate the conclusions by redefining "hot hand" to mean something it's not.




(P.S.  MGL has an essay on this study in the 2016 Hardball Times. My book arrived last week, but I haven't read it yet. Discussion here.)

Can a "hot hand" turn average players into superstars?

Last post, I reviewed a study by Joshua Miller and Adam Sanjurjo that found a "hot hand" effect in the NBA's Three-Point Contest. In addition to finding a hot hand, the authors also showed how some influential previous studies improperly underestimated observed streakiness because of the incorrect way they calculated expectations. 

I agreed that the previous studies were biased, and accepted that the authors found evidence of a hot hand in the three-point contest. But I was dubious that you can use that evidence to assume a hot hand in anything other than a "muscle memory" situation.

Dr. Miller, in comments on my post and follow-up e-mails, disagreed. In the comments, he wrote,

"The available evidence shows big effect sizes. Should we infer the same effect in games, given we have no known way to measure them? It is certainly a justifiable inference."

Paraphrasing Dr. Miller's argument: Since (a) the original "no hot hand" studies were based on incorrect calculations, and (b) we now have evidence of an actual hot hand in real life ... then, (c) we should shift our prior for real NBA games from "probably no hot hand" to "probably a significant hot hand."

That's a reasonable argument, but I still disagree.

------

There are two ways you can define a "hot hand":

1. Sometimes, players have higher talent ("talent" means expected future performance) than other times. In other words, some days they're destined to be "hot," better than their normal selves.

2. When players have just completed a streak of good performance, they are more likely to follow it with continued good performance than you'd otherwise expect.

Call (1) the "hot hand talent" hypothesis, and (2) the "streakiness" hypothesis. Each implies the other -- if you have "good days," your successes will be concentrated among those good days, so you'll look streaky. Conversely, if your expectation is to exhibit streakiness, you must be "better in talent" after a streak than after a non-streak.

I think the two definitions are the same thing, under certain other reasonable assumptions. At worst, they're *almost* the same thing.

However, we can observe (2), but not (1). That's why "hot hand" studies, like Miller/Sanjurjo, have to concentrate on streaks.

----

The problem is: it takes a *lot* of variation in talent (1) to produce just a *tiny bit* of observed streakiness (2). 

Observed streakiness is a very, very weak indicator of a varation in talent. That's because players also go on streaks for a lot of other reasons than that they're actually "hot" -- most importantly, luck.

In the three-point contest study, the authors found an average six percentage point increase in hit rate after a sequence of three consecutive hits, from about 53 percent to 59 percent. As Dr. Miller points out, the actual increase in talent when "hot" must be significantly higher -- because not all players who go HHH are necessarily having a hot hand. Some are average, or even "cold," and wind up on a streak out of random luck. 

If only half of "HHH" streaks are from players truly hot at the time, the true "hot hand" effect would have to be double what's observed, or 12 percentage points.

Well, 12 points is huge, by normal NBA standards. I can see it, maybe, in the context of muscle memory, like the uncontested, repeated shots in the Miller/Sanjurjo study -- but not in real life NBA action.

What if there were a 12-point "hot hand" effect in, say, field goal percentage in regular NBA games? Well, for all NBA positions, as far as I can tell, the difference between average and best is much less than 12 points. That would mean that when an average player is +12 points "hot," he'd be better than the best player in the league. 

Hence my skepticism. I'm willing to believe that a hot hand exists, but NOT that it's big enough to turn an average player into a superstar. That's just not plausible.

------

Suppose you discover that a certain player shoots 60% when he's on a three-hit streak, and 50% other times. How good is he when he's actually hot? Again, he's not "hot" every time he's on a streak, because streaks happen often just by random chance. So, the answer depends on *how often* he's hot. You need to estimate that before you can answer the question. 

Let's suppose we think he's hot, say, 10 percent of the time.

So, to restate the question as a math problem:

"Joe Average is normally a 50 percent shooter, but, one time in ten, he is "hot", with a talent of P percent. You observe that he hits 60% after three consecutive successes. What's your best estimate of P?"

The answer: about 81 percent.

An 81 percent shooter will make HHH about 4.25 times as often as a 50 percent shooter (that's 81/50 cubed). That means that Joe will hit 4.25 streaks per "hot" game for every one streak per "normal" game.

However: Joe is hot only 1/9 as often as he is normal (10% vs. 90%). Therefore, instead of 425 "hot" HHH for every 100 "regular" HHH, he'll have 425 "hot" HHH for every *900* "regular" HHH.

Over 1325 shots, he'll be taking 425 shots with an expectation of 81 percent, and 900 shots with an expectation of 50 percent. 

Combined, that works out to 794-for-1325, which is the observed 60%.

Do you really want to accept that the "hot hand" effect turns an ordinary player into an 81-percent shooter? EIGHTY-ONE PERCENT? 

But that's what the assumptions imply. If you argue that:

-- player X is 50% normally;
-- player X is "hot" 10 percent of the time;
-- player X is expected to hit 60% after HHH

Then, it MUST FOLLOW that

-- player X is 81% when "hot".

To which I say: no way. I say, nobody is an 81% shooter, ever -- not Michael Jordan, not LeBron James, nobody. 

To posit that the increase from 50% to 60% is reasonable, you have to assume that an average player turns into an otherworldly Superman one day in ten, due to some ineffable psychological state called "hotness."  

-----

You can try tweaking the numbers a bit, if you like. What if a player is "hot" 25 percent of the time, instead of 10 percent? In that case,

-- player X is 71% when "hot".

That's not as absurd as 80%, but still not very plausible. What if a player is "hot" fully half the time? Now,

-- player X is 64.6% when "hot". 

That's *still* not plausible. Fifteen points is still superstar territory. Do you really want to argue that half the time a player is ordinary, but the other half he's Michael Jordan? And that nobody would notice without analyzing streaks?

Do you really want to assume that the variation in talent within a single player is wider than the variation of talent among all players?.

-----

Let's go the other way, and start with an intuitive prior for what it might mean to be "hot." My gut says, at most, maybe half an SD of league talent. You can go from 50th to 70th percentile when everything is lined up for you -- say, from the 15th best power forward in the league,  to the 9th best. Does that sound reasonable?

In the NBA context, let's call that ... I have no idea, but let's guess five percentage points.* And let's say a player is "hot" one time in five. 

(* A reader wrote me that five percentage points is a LOT more than half an SD of talent. He's right; my bad. Still, that just makes this part of the argument even stronger.)

So: if one game in five, you were a 55% shooter instead of 50%, what would you hit after streaks?

-- For 1000 "hot" shots, you'd achieve HHH 166 times, and hit 91.3 of the subsequent shots.

-- For 4000 "regular" shots, you'd achieve HHH 500 times, and hit 250 of the subsequent shots.

Overall, you'd be 341.3 out of 666, or 51.25%.

In other words: a hot hand hypothesis that posits a reasonable (but still significant) five-point talent differential expects you're only 1.25 percentage points better after a streak. 

Well, you need a pretty big dataset to make 1.25 points statistically significant. 30,000 attempts would do it: 6000 when "hot" and 24,000 when not hot.*

(* That's using binomial approximation, which underestimates the randomness, because the number of attempts isn't fixed or independent of success rate. But never mind for now.)

And even if you had a sample size that big, and you found significance ... well, how can you prove it's a "hot hand"? It's only 1.25 points, which could be an artifact of ... well, a lot of things other than streakiness.

Maybe you didn't properly control for home/road, or you used a linear adjustment for opponent quality instead of quadratic. Maybe the 1.25 doesn't come from a player being hot one game in five, but, rather, the coach using him in different situations one game in five. Or, maybe, those 20 percent of games, the opposition chose to defend him in a way that gave him better shooting opportunities. 

So, it's going to be really, really hard to prove a "hot hand" effect by studying performance after streaks.

------

But maybe there are other ways to analyze the data.

1. Perhaps you could look at player streaks in general, instead of just what happens in the one particular shot after a streak. That would measure roughly the same thing, but might provide more statistical power, since you'd be looking at what happens during a streak instead of just the events at the end. 

Would that work? I think it would at least give you a little more power. Dr. Miller actually does something similar in his three-point paper, with a "composite statistic" that measures other apsects of a player's sequences.

2. Instead of just a "yes/no" for whether to count a certain shot, you could weight it by the recent success rate, or the length of the streak, or something. Because, intuitively, wouldn't you expect a player to be "hotter" after HHHHHH than HHH? Or, even, wouldn't you expect him to be hotter after HHMHMHHMHHHMMHH than HHH? 

I'm pretty sure that kind of thing has been done before, that there are studies that try to estimate the next shot from the success rate in the X previous shots, or some such.

------

But, you can't fight the math: no matter what, it still takes ten haystacks of "hot hand talent" variation to produce a single needle of "streakiness." There just isn't enough data available to make the approach work. 

Having said that ... there's a non-statistical approach that theoretically could work to prove the existence of a real-life hot hand. 

In his e-mails to me, Dr. Miller said that basketball players believe that some of them are intrinsically streakier than others -- and that they even "know" which players those were. In an experiment in one of his papers, he found that the players named as "streaky" did indeed wind up showing a larger "hot hand" effect in a subsequent controlled shooting test.

If that's the case (I haven't read that paper yet), that would certainly be evidence that something real, and observable, is happening.

And, actually, you don't need an a laboratory experiment for this. Dr. Miller believes that coaches and teammates can sense variations in talent from body language and experience. If that's the case, there must be sportswriters, analysts, and fans who can do this too.

So, here's what you do: get some funding, set up a website, and let people log on while watching live games to predict, in real time, which players are currently exhibiting a hot hand. If even one single forecaster proves to be able to consistently choose players who outperform their averages, you have your evidence.  

-----

I'd be surprised, frankly, if anyone was able to predict significant overachievement in the long run. And, I'd be shocked -- like, heart attack shocked -- if the identified "hot" players actually did perform with "Superman" increases in accuracy. 

As always, I could be wrong. If you think I *am* wrong, that the "hot hand" is even half as significant a factor in real life as it is in the three-point contest, I think this would easily be your best route to proving it.

A "hot hand" is found in the NBA three-point contest

A recent paper provides what I think is rare, persuasive evidence of a "hot hand" in a sporting event.

The NBA Three-Point Contest has been held annually since 1986 (with the exception of 1999), as part of the NBA All-Star Game event. A pair of academic economists, Joshua Miller and Adam Sanjurjo, found video recordings of those contests, and analyzed the results. (.pdf)

They found that players were significantly more likely to make a shot after a series of three hits than otherwise. Among the 33 shooters who had at least 100 shots in their careers, the average player hit 54 percent overall, but 58 percent after three consecutive hits ("HHH").  

(UPDATE: the 58 percent figure is approximate: the study reports an increase of four percentage points after HHH than after other sequences. Because the authors left out some of the shots in some of their calculations (as discussed later in this post), it might be more like 59% vs. 55%, or some such. None of the discussion to follow depends on the exact number.)

The authors corrected for two biases. I'll get to those in detail in a future post, but I'll quickly describe the most obvious one. And that is: after HHH, you'd expect a *lower than normal* hit rate -- that is, an apparent "mean-reverting hand" -- even if results were completely random. 

Why? Because, if a player hit exactly 54 of 100 shots, then, after HHH, the next shot must come out of what remains -- which is 51 remaining hits out of 97 remaining shots. That's only 52.6 percent. In other words, the hit rate not including the "HHH" must obviously be lower than the hit rate including "HHH". 

That might be easier to see if you imagine that the player hit only 3 out of 100 shots overall. In that case, the expectation following HHH must be 0 percent, not 3 percent, since there aren't enough hits to form HHHH!

After the authors corrected for this, and for the other bias they noted, the "hot hand" effect jumped from 4 percentage points to 6. 

------

UPDATE: Joshua Miller has replied to some of what follows, in the comments.  I have updated the post in a couple of places to reflect some of his responses.

------

That's a big effect, a difference of 6 percentage points. Maybe it's easier to picture this way:

Of the 33 players, 25 of them shot better after HHH than their overall rate. 

In other words, the "hot hand" beat the "mean-reverting hand" with a W-L record of 25-8. With the adjustments included, the hot hand jumps to 28-5.

------

Could the result be due to something other than a hot hand? Well, to some extent, it could be selective sampling of players.

In the contest, players shoot 25 attempts per round. To get to 100 attempts, and be included in the study, a shooter has to play at least four rounds in his career.  (By the way, here's a YouTube video of the 2013 competition.)

In any given contest, to survive to the next round, a player needs to do well in the current round. That means that players who got enough attempts were probably lucky early. That might select players who concentrated their hits in early rounds, compared to the late rounds, and create a bit of a "hot hand" effect just from that.

And I bet that's part of it ... but a very small part. Even if a player shot 60/60/50 in successive rounds, just by luck, that alone wouldn't be nearly enough to show an overall effect of 6 percentage points, or even 4, or (I think) even 1.

UPDATE: The authors control for this by stratifying by rounds, Dr. Miller replies.

------

One reason I believe the effect is real is that it makes much more intuitive sense to expect a hot hand in this kind of competition than in normal NBA play.

In each round of the contest, players shoot 5 consecutive balls from the same spot on the court, in immediate succession. That seems like the kind of task that would easily show an effect. It seems to me that a large part of this would be muscle memory -- once you figure out the shot, you just want to do exactly the same thing four more times (or however many balls you have left once you figure it out). 

After those five balls, you move to another spot on the arc for another five balls, and so on, and the round ends after you've thrown five balls from each of five locations. However, even though the locations move, the distances are not that much different, so some of the experience gained earlier might extend to the next set of five, making the hot hand even more pronounced.

There's one piece of the evidence that offers support for the "muscle memory" hypothesis. It turns out that the first two shots in each round were awful. The authors report that the first shot was made only 26 percent of the time, and the second shot only 39 percent. For the remaining twenty-three shots, the average success rate was 56 percent.

That "warm up" time is very consistent with a "muscle memory" hot hand.

-----

In fact, those first two shots were so miserable that the authors actually removed them from the dataset! If I understand the authors correctly, a player listed with 100 shots was analyzed for only 92 of those shots.

UPDATE: originally, I thought that rounds were stitched together, so removing those shots would increase observed streakiness from one round to the next. But Dr. Miller notes, in the comments, that they considered streaks within a single round only. In that case, as he notes, removing the first two shots has the effect of reducing "cold hand" streakiness, making the results more conservative.  

The removal of those shots, it seems to me, would be likely to overstate the findings a bit. The authors strung rounds together as if they were just one long series of attempts (even if they spanned different years; that seems a bit weird, that you'd say a player had a "hot hand" if he continued a 2004 streak in 2005, but never mind).

That means that when they string the last five shots of one round with the first five shots of the next, instead of something like

MHHHH MMHMH

they get 

MHHHH   HMH

which tends to create more streaks, since you're taking out shots that tend to be mostly misses, in the midst of a series of shots that tend to be mostly hits. ("M" represents a miss, as you probably gathered.)

I wonder if the significant effect the authors found would still have shown up without those omitted shots. I suspect it would have been, at least, significantly weaker. I may be wrong -- the authors showed streakiness both for hits and misses, so maybe the extra "MM" shots would have shown up in their "cold hand" numbers.

------

I bet you'd find a hot hand if you tried the equivalent contest yourself. Position a wastebasket somewhere in the room, a few feet away. Then, stay in one spot, and try to throw wads of paper into the basket. I'm guessing your first one will miss, and you'll adjust your shot, and then you'll get a bit better, and, eventually, you'll be sinking 80 to 90 percent of them. Which means, you have a "hot hand" -- once you get the hang of it, you'll be able to just repeat what you learned, which means hits will tend to follow hits.

Here's a more extreme analogy. Instead of throwing paper into a basket, you're shown a picture of a random member of the Kansas City Royals, and asked to guess his age exactly. After your guess, you're told how far you were off. And then you get another random player (which might be a repeat).

Your first time through the roster, you might get, say, 1/3 of them right. The second time through, you'll get at least 2/3 of them right -- the 1/3 from last time, and at least half the rest (now that you know how much you were off by, you only have to guess which direction). The third time through, you'll get 100%.

So, your list of attempts will look something like this (H for hit, M for miss):

MMMHMHMHMMHHHMMHHMHMMMHMHHHHMHHHMMHHHHHHHHMHHHHHHHH...

Which clearly demonstrates a hot hand.

And that's similar to what I think is happening here. 

------

The popular belief, among sportswriters and broadcasters, is that the hot hand - aka "momentum" or "streakiness" -- is real, that a team that has been successful should be expected to continue that way. But almost every study that has looked for such an effect has failed to find one.

That led to the coining of the term "hot hand fallacy" -- the belief that a momentum effect exists, when it does not. Hence the title of this paper: "Is it a Fallacy to Believe in the Hot Hand in the NBA Three Point Contest?"

So, does this study actually refute the hot hand fallacy? 

Well, it refutes it in its strongest form, which is the position that there NEVER exists a hot hand of ANY magnitude, in ANY situation. That's obviously wrong. You can prove it with the Kansas City Royals example, or ... well, you can prove it in your own life. If you score every word you misspelled as a miss, and the rest as a hit ... most of your misses are clustered early in life, when you were learning to read and write, so there's your hot hand right there.

The real "fallacy," as I see it, is not the idea that a hot hand exists at all, but the idea that it is a significant factor in predicting what's going to happen next. In most aspects of sports, the hot hand, when it does exist, is so small as to have almost no predictive value. 

Suppose a player has two kinds of days, equal and random -- "on," where he hits 60%, and "off" where he hits only 50%. That would give rise to a hot hand, obviously. But how big a hot hand? What should you predict as the chance of the player making his next shot?

Before the game, you'd guess 55% -- maybe he's on, or maybe he's off. But, now, he hits three straight shots. He has a hot hand! What do you expect now?

If my math is right, you should now expect him to shoot ... 56.3%. Not much different!

The "50/60 on/off" actually represents a huge variation in talent. The problem is that streaks are a weak indicator of whether the player is actually "on," versus whether he just had a lucky three shots. In real life, it's even weaker than a 1.3 percent indicator, because, for one thing, how do you know how long a player is "on" and how long he's "off"? I assumed a full game, but that's wildly unrealistic.

You can probably think of many reasons streakiness is a weak indicator. Here's just one more. 

The "56.3%" illustration was assuming that all shots were identical. In real life, if it's not a special case of a three-point contest ... well, when a player hits HHH, it might be evidence of a hot hand, but it also just could be that those shots were taken in easier conditions, that they were 60% shots instead of 50% shots because the defense didn't cover the shooter very well.

Real games are much more complicated and random than a three-point shooting contest. That's why I don't like the phrasing, that the authors of this NBA study found evidence of "THE hot hand effect". They found evidence of "A hot hand effect", one particular one that's large enough to show up in the contrived environment of a muscle-memory based All-Star novelty event. It doesn't necessarily translate to a regular NBA game, at least not unless you dilute it enough that it becomes irrelevant.

------

The "hot hand" issue reminds me of the "clutch hitting" issue. Both effects probably exist, but are so tiny that they're pretty much useless for any practical purposes. Academic studies fail to find statistically significant evidence, and imply that "absence of evidence" implies that no effect exists. We sabermetricians cheat a little bit, saving effort by saying there's "no effect" instead of "no effect big enough to measure."

So "no effect" becomes the consensus. Then, someone comes up with a finding that actually measures an effect -- this study for the hot hand, and "The Book" for clutch hitting. And those who never disbelieved in it jump on the news, and say, "Aha! See, I told you it exists!"  

But they still ignore effect size. 

People will still declare that their favorite hitter is certainly creating at least a win or two by driving in runs when it really counts. But now, they can add, "Because, clutch hitting exists, it's been proven!" In reality, there's still no way of knowing who the best clutch hitters are, and even if you could, you'd find their clutch contribution to be marginal.

And, now, I suspect, when the Yankees win five games in a row, the sportscasters will still say, "They have momentum! They're probably going to win tonight!" But now, they can add, "Because, the hot hand exists, it's been proven!" In reality, the effect is so attenuated that their "hotness" probably makes them a .501 expectation instead of .500 -- and, probably, even that one point is an exaggeration.  

My bet is: the "hot hand" narrative won't change, but now it will claim to have science on its side.

Is there evidence for a "hot hand" effect?

You're in the desert, exactly 10 miles south of home.  Instead of walking straight home, you decide to walk east one mile first.  How far away from home are you when you're done?

The answer: about 10.05 miles.  Your one-mile detour only added 0.05 miles to your return trip: one-half of one percent.

That's just a simple application of the Pythagorean theorem.  Draw a right triangle with sides 10 miles and 1 mile, and the hypotenuse will be 10.05 miles long.

------

That triangle thing is meant to explain why it might be difficult to find evidence for or against streakiness in baseball hitting expectations.

There's a belief that there are times when a player has a "hot hand", in the sense that he's somehow "dialed in" and is likely to perform better than his usual.  And, there might be periods when he's "cold," and should be expected to perform worse.  

Torii Hunter hit .370 in April, 2013.  Did he have a hot hand, or was he just lucky?  That's the question that we want to answer.  Maybe not specifically about Torii Hunter, but, in general ... when a player has a hot streak or a cold streak, is there something behind that?

The difficulty answering that is that there's a lot of luck involved, and it's hard to separate out the luck from the "hot hand" (if any).

If you assume each AB is independent of the previous one, then, over a month's worth -- say, 100 AB, which was Torii Hunter's total -- the SD of batting average, by luck alone, will be about 43 points.  That's massive ... it means that one time out of 20, a player will be 86 points off expected in a given month.

Now, suppose there's also a real "hot hand" effect, that's 1/10 the size of the luck effect.  Now, the SD of batting average, by luck and hot hands combined, will be one half of one percent higher, just like in the Pythagorean triangle example.  That's almost nothing -- .0002 in batting average.

Effectively, this level of hot hand is invisible, if it exists.

This finding might be good news to those who believe that players are often "on" or "off" (Morgan Ensberg is one of those believers).  But ... there are still things we *can* say about hot-hand effects, and those things won't be compatible with some of the ways people talk about streakiness.

-------

First, let's try to make an intelligent guess on how much hot-handedness can remain invisible.  As we saw, 10 percent of luck is impossible to see because the SD difference is only 0.5 percent.  If hot hand variation were 20 percent of luck, the SD would go up by two percent.  If hot hands were 30 percent of luck, the SD would go up by around 4.5 percent.  If it were 40 percent of luck, the SD would go up by around 7.7 percent. 

Actually, let me quote Bill James (subscription required), from his recent essay on streakiness that inspired me to write this:

" ... how much causal variation is it reasonable to think might be completely hidden in a mix of causal and random variation? 

"Well, if it was 50-50, it would be extremely easy ... [i]f you add a second level of variation equal to the first, it will create obviously non-random patterns.

"If it 70-30—in other words, if the causal variation was roughly half the size of the random variation—that, again, would be easy to distinguish from pure random variation.    Even if it was 90-10, we should be able to distinguish between that and pure random variation.   If it was 99-1, maybe we would have a hard time telling one from the other."

I think Bill is a little optimistic on the 90-10 ... you're still talking one-half of one percent.  On 70-30, though, I think he's right.  That would be around a 9 percent increase in the standard deviation, which I think we'd be able to find without too much difficulty.

For the sake of this argument, let's say that the limit we can observe is ... I dunno, let's say, 75-25.  That would mean the SD of hot hands was 33 percent of the SD of luck, which means the overall SD goes up by around 5.3 percent.  I think we could find that if we looked.  I think.

I may be wrong ... substitute your own number if you prefer.

------

It might be legitimate for a hot-hand advocate could say to us, "well, you don't know for sure that there's no talent streakiness, because you admit that your methods can't pick it up."

Well, maybe.  But if you make that argument, don't forget to also limit yourself to hot hand effects that are 33 percent of luck.  That's around 14 points (again, that's batting average over 100 AB).

That means only one player in 20 will be more than 28 points off his expected batting average for any given month.  Only one player in 400 will be more than 42 points "hot" or "cold" for the month.

Are you prepared to live with that?  When you find ten players hitting 100 points off their expected for the month of August, you'd have to say, "Well, it's virtually impossible that *any* of them were *that* hot ... that's 7 standard deviations, which never happens.  They may have been a bit hot, but they were mostly lucky."

Seriously, are you prepared to say that?

There's more.  With a 75-25 mix, the correlation between "hot hand" and performance would be a bit over 0.3.  That means, if you find a player hitting 2 SD better than expected, his "hot hand" expectation is +0.6 SDs.  That's around 8 points.

Seriously.  When a career .250 player hits .340 in May, you'd have to be saying, "wow, I bet that player has a hot hand.  I bet in May, he was really a .258 hitter!!!"

My guess is, most "hot hand" enthusiasts wouldn't bite that bullet.  They don't just want "hot hands" to exist, they want them to be a big part of the game.  When they see a player on a hot streak, they want to believe that player is *significantly better*, that he's a force to be reckoned with.  "He's 8 points hot" is just not a very good narrative.

But that's pretty much what it amounts to.  If 75-25 is the right ratio, then only around TEN PERCENT of a player's observed "hotness" would come from a hot hand.  The other 90 percent would still be luck.

-------

Now, one assumption in all those calculations is that "hot hands" are random and normally distributed.  Followers of streakiness would probably argue that's not the case.  Intuitively, it would seem that most players are just "themselves" most of the time, but, occasionally, someone will get exceptionally hot and really be 100 points better for a brief period.

For instance, maybe hothandedness manifests itself in outliers.  Maybe 38 out of 40 players are just normal.  But, the 39th player is cold and drops 60 points, while the 40th is hot, and gains those 60 points.   That still works out to the SD of around 14 points that we hypothesized as our limit.

If that's the case, then, the "check the SD method" would still fail to find anything.  

But ... now, we'd have other methods that would actually work.  We could count outliers.  If one player in 40 gains .060 every month, we should see a lot more exceptional months than we would otherwise.  

So, if that's how you think hot hands work, let us know.  Those kinds of hot hands *won't* be invisible.

------

And even if they exist at that level, it won't tell us much.  

Suppose a player has a month where he actually hits .060 better than normal.  Can you now say, "Ha!  See, that player had a hot hand!"

Nope.  Because, by the model, the chance of a player having a hot hand is 1 in 40.  But, the chance of a player having a +60 point month, just by luck, is, I think, something like 1 in 12.

A quick naive calculation is that, even in this extreme case, there's a 3-in-4 chance the player was just lucky!  (It's probably actually higher than that, if you do the proper calculation.)

------

So here's what I think we have:

1.  Moderate variations in talent might be impossible to disprove by the standard method.  For monthly "hot hands", that might be .014 points of batting average, as a guess.

2.  If those exist, they will rarely be higher than, say, 30 or 40 points.  So, when a .270 player hits .380 in August, it *can't* all be a hot hand.  It'll still be mostly luck.

3.  If hot hands manifest themselves in outliers, we should have no problem finding them.  The idea that players will regularly get "scorching hot" in talent ... well, I bet we could debunk that pretty easily.

4.  Even if hot hands do exist, unusual performances will be more driven by luck than by performance variations.  There will never be a case where you can look at a hot month, and have good reason to believe it wasn't just luck.

(Hat tip and more discussion: Tango)