A study on NBA home court advantage

Economist Tyler Cowen often links to NBA studies in his "Marginal Revolution" blog ... here's a recent one, from an August post. (Follow his link to download the study ... you can also find a press release by Googling the title.)

The study used a neural network to try to figure out what factors are most important for home (court) advantage (which I'll call "HCA"). The best fit model used twelve variables: two-point shots made, three-point shots made, and free throws made -- repeated for team at home, opposition on road, team on road, and opposition at home.

The authors write, 

"Networks that include shot attempts, shooting percentage, total points scored, field goals, attendance statistics, elevation and market size as predictors added no improvement in performance. ...

"Contrary to previous work, attendance, elevation and market size were not relevant to understanding home advantage, nor were shot attempts, shooting percentage, overall W-L%, and total points scored."

On reflection, it's not surprising that those other variables don't add anything ... the ones they used, shots made, are enough to actually compute points scored and allowed. Once you have that, what does it matter what the attendance was? If attendance matters at all, it would affect wins through points scored and allowed, not something independent of scoring. And "total points scored" weren't "relevant" because they were redundant, given shots made.

------

The study then proceeds to a "sensitivity analysis," where they increase the various factors, separately, to see what happens to HCA. It turns out that when you increase two-point shots made by 10 percent, you get three to four times the impact on HCA compared to when you increase three-point shots made by the same 10 percent.

The authors write,

"[This] suggests teams can maximize their advantage -- and hence their odds of winning -- by employing different shot selection strategies when home versus away. When playing at home, teams can maximize their advantage by shooting more 2P and forcing opponents to take more 2P shots. When playing away, teams can minimize an opponent's home advantage by shooting more 3P and forcing opponents to take more 3P shots."

Well, yes, but, at the same time, no. 

The reason increasing 2P by 10 percent leads to a bigger effect than increasing 3P by 10 percent is ... that 10 percent of 2P is a lot more points! Eyeball the graph of "late era" seasons the authors used (I assume it's the sixteen seasons ending with 2015-16). Per team-season, it looks like the average is maybe 2500 two-point shots made, but only 500 three-point shots.

Adding 10 percent more 2P is 250 shots for 500 points. Adding 10 percent more 3P is 50 shots for 150 points. 500 divided by 150 gives a factor of three-and-a-third -- almost exactly what the paper shows!

I'd argue that what the study discovered is that points seem to affect HCA and winning percentage equally, regardless of how they are scored. 

------

Even so, the argument in the paper doesn't work. By the authors' own choice of variables, HCA is increased by *making* 2P shots, not my *taking* 2P shots. Rephrasing the above quote, what the study really shows is,

"When playing at home, teams can maximize their advantage by concentrating on *making* more 2P and on forcing opponents to *miss* more 2P. That's assuming that it's just as easy to impact 2P percentages by 10 percent than to impact 3P percentages by 10 percent."

But we could have figured that out easily, just by noticing that 10 percent of 2P is more points than 10 percent of 3P.

------

The authors found that you increase your HCA more with a 10 percent increase in road three-pointers than by a 10 percent increase in road two-pointers. 

Sure. But that's because, with the 3P, you actually wind up scoring fewer road points. Which means you win fewer road games. Which makes your HCA larger, since winning fewer road games increases the difference between home and road. 

It's because the worse you do on the road, the bigger your home court advantage!

Needless to say, you don't really want to increase your HCA by tanking road games. The authors didn't notice that's what they were suggesting.

I think the issue is that the paper assumes that increasing your HCA is always a good thing. It's not. It's actually neutral. The object isn't to increase or decrease your HCA. It's  to *win more games*. You can do that by winning more games at home, increasing your home court advantage, or by winning more games on the road, decreasing your home court advantage.

It's one of those word biases we all have if we don't think too hard. "Increasing your advantage" sounds like something we should strive for. The problem is, in this context, the home "advantage" is relative to *your own performance* on the road. So it really isn't an "advantage," in the sense of something that makes you more likely to beat the other team. 

In fact, if you rotate "Home Court Advantage" 360 degrees and call it "Road Court Disadvantage," now it feels like you want to *decrease* it -- even though it's exactly the same number!

But HCA isn't something you should want to increase or decrease for its own sake. It's just a description of how your wins are distributed.

How Elo ratings overweight recent results

"Elo" is a rating system widely used to rate players in various games, most notably chess. Recently, FiveThirtyEight started using it to maintain real-time ratings for professional sports teams. Here's the Wikipedia page for the full explanation and formulas.

In Elo, everyone gets a number that represents their skill. The exact number only matters in the difference between you and the other players you're being compared to. If you're 1900 and your opponent is 1800, it's exactly the same as if you're 99,900 and your opponent is 99,800.

In chess, they start you off with a rating of 1200, because that happens to be the number they picked. You can interpret the 1200 either as "beginner," or "guy who hasn't played competitively yet so they don't know how good he is."  In the NBA system, FiveThirtyEight decided to start teams off with 1500, which represents a .500 team. 

A player's rating changes with every game he or she plays. The winner gets points added to his or her rating, and the loser gets the same number of points subtracted. It's like the two players have a bet, and the loser pays points to the winner.

How many points? That depends on two things: the "K factor," and the odds of winning.

The "K factor" is chosen by the organization that does the ratings. I think of it as double the number of points the loser pays for an evenly matched game. If K=20 (which FiveThirtyEight chose for the NBA), and the odds are even, the loser pays 10 points to the winner.

If it's not an even match, the loser pays the number of points by which he underperformed expectations. If the Warriors had a 90% chance of beating the Knicks, and they lost, they'd lose 18 points, which is 90% of K=20. If the Warriors win, they only gain 2 points, since they only exceeded expectations by 10 percentage points.

How does Elo calculate the odds? By the difference between the two players' ratings. The Elo formula is set so that a 400-point difference represents a 10:1 favorite. An 800 point favorite is 100:1 (10 for the first 400, multiplied by 10 for the second 400). A 200 point favorite is 3.16 to 1 (3.16 is the square root of 10). A 100 point favorite is 1.78 to 1 (the fourth root of 10), and so on.

In chess, the K factor varies depending on which chess federation is doing the ratings, and the level of skill of the players. For experienced non-masters, K seems to vary between 15 and 32 (says Wikipedia). 

------

Suppose A and B have equal ratings of 1600, and A beats B. With K=20, A's rating jumps to 1610, and B's rating falls to 1590. 

A and B are now separated by 20 points in the ratings, so A is deemed to have odds of 1.12:1 of beating B. (That's because the "400/20th" root of 10 is 1.12.)  That corresponds to an expected winning percentage of .529.

After lunch, they play again, and this time B beats A. Because B was the underdog, he gets more than 10 points -- 10.6 points (.529 times K=20), to be exact. And A loses the identical 10.6 points.

That means after the two games, A has a rating of 1599.4, and B has a rating of 1600.6.

------

That example shows one of the properties of Elo -- it weights recent performance higher than past performance. In their two games, A and B effectively tied, each going 1-1. But B winds up with a higher rating than A because his win was more recent.

Is that reasonable? In a way, it is. People's skill at chess changes over their lifetimes, and it would be weird to give the same weight to a game Garry Kasparov played when he was 8, as you would to a game Garry Kasparov played as World Champion.

But in the A and B case, it seems weird. A and B played both games the same day, and their respective skills couldn't have changed that much during the hour they took their lunch break. In this case, it would make more sense to weight the games equally.

Well, according to Wikipedia, that's might would actually happen. Instead of updating the ratings every game, the Federation would wait until the end of the tournament, and then compare each player to his or her overall expectation based on ratings going in. In this case, A and B would be expected to go 1-1 in two games, which they did, so their ratings wouldn't change at all.

But, if A and B's games were days or weeks apart, as part of different tournaments, the two games would be treated separately, and B might indeed wind up 1.2 points ahead of A.

------

Is that a good thing, giving a higher weight to recency? It depends how much higher a weight. 

People's skill does indeed change daily, based on mood, health, fatigue -- and, of course, longer-term trends in skill. In the four big North American team sports, it's generally true that players tend to improve in talent until a certain age (27 in baseball), then decline. And, of course, there are non-age-related talent changes, like injuries, or cases where players just got a lot better or a lot worse partway through their careers.

That's part of the reason we tend to evaluate players based on their most recent season. If a player hit 15 home runs last year, but 20 this year, we expect the 20 to be more indicative of what we can expect next season.

Still, I think Elo gives far too much weight to recent results, when applied to professional sports teams. 

Suppose you're near the end of the season, and you're looking at a team with a 40-40 record -- the Bulls, say. From that, you'd estimate their talent as average -- they're a .500 team.

Now, they win an even-money game, and they're 41-40, which is .506. How do you evaluate them now? You take the .506, and regress to the mean a tiny bit, and maybe estimate they're a .505 talent. (I'll call that the "traditional method," where you estimate talent by taking the W-L record and regressing to the mean.)

What would Elo say? Before the 81st game, the Bulls were probably rated at 1500. After the game, they've gained 10 points for the win, bringing them to 1510.

But, 1510 represents a .514 record, not .505. So, Elo gives that one game almost three times the weight that the traditional method does.

Could that be right? Well, you'd have to argue that maybe because of personnel changes, team talent changes so much from the beginning of the year to the end that the April games are worth three times as much as the average game. But ... well, that still doesn't seem right. 

-------

Techincal note: I should mention that FiveThirtyEight adds a factor to their Elo calculation -- they give more or fewer points to the winner of the game based on the score differential. If a favorite wins by 1 point, they'll get a lot fewer points than if they won by 15 points. Same for the underdog, except that the underdog always gets more points than the favorite for the same point differential -- which makes sense.

FiveThirtyEight doesn't say so explicitly, but I think they set the weighting factor so that the average margin of victory corresponds to the number points the regular Elo would award the winner.

Here's the explanation of their system.

-------

Elo starts with a player's rating, then updates it based on results. But, when it updates it, it has no idea how much evidence was behind the player's rating in the first place. If a team is at 1500, and then it wins an even game, it goes to 1510 regardless of whether it was at 1500 because it's an expansion team's first game, or because it was 40-40, or (in the case of chess) it's 1000-1000.

The traditional method, on the other hand, does know. If a team goes from 1-1 to 2-1, that's a move of .167 points (less after regressing to estimate talent, of course). If a team goes from 40-40 to 41-40, that's a move of only .005 points. 

Which makes sense; the more evidence you already have, the less your new evidence should move your prior. But if your prior moves the same way regardless of the previous evidence, you're seriously underweighting that previous evidence (which means you're overweighting the new evidence).

The chess Federations implicitly understand this, that you should give less weight to new results when you have more older results. That's why they vary the K-values based on who's playing. 

FIDE, for instance, weights new players at K=40, experienced players at K=20, and masters (who presumably have the most experience) at K=10.

------

As I said last post, I did a simulation. I created a team that was exactly average in talent, and assumed that FiveThirtyEight had accurately given them an average rating at the beginning of the year. I played out 1000 random seasons, and, on average, the team wound up at right where it started, just as you would expect.

Then, I modified the simulation as if FiveThirtyEight had underrated the team by 50 points, which would peg them as a .429 team. (They use their "CARM-Elo" player projection system for those pre-season ratings. I'm not saying that system is wrong, just checking what happens when a projection happens to be off.)

It turned out that, at the end of the 82-game season, Elo had indeed figured out the team was better than their initial rating, and had restored 45 of the 50 points. They were still underrated, but only by 5 points (.493) instead of 50 (.429). 

Effectively, the current season wiped out 90% of the original rating. Since the original rating was based on the previous seasons, that means that, to get the final rating, Elo effectively weighted this year at 90%, and the previous years at 10%. 

10% is close to 12.5%. I'll use that because it makes the calculation a bit easier. At 12.5%, which is one-eighth, it means the NBA season contains three "half lives" of about 27 games each. 

That is: after 27 games, the gap of 50 points is reduced by half, to 25. After another 27 games, it's down to 12. After a third 27 games, it's down to 6, which is 12.5% of where the gap started.

That means that to calculate a final rating, the thirds of seasons are effectively weighted in a ratio of 1:2:4. A game in April has four times the weight of a game in November. Last post, I argued why I think that's too high.

-------

Here's another way of illustrating how recency matters. 

I tweaked the simulation to do something a little different. Instead of creating 1,000 different seasons, I created only one season, but randomly reordered the games 1,000 times. The opponents and final scores were the same; only the sequence was different. 

Under the traditional method, the talent estimates would be the same, since all 1,000 teams had the same W-L record. But the Elo ratings varied, because of recency effects. They varied with an SD of about 26 points. That's about .037 in winning percentage, or 3 wins per 82 games.

If you consider the SD to be, in a sense, the "average" discrepancy, that means that, on average, Elo will misestimate a team's talent by 3 wins. That's for teams with the same actual record -- based only on the randomness of *when* they won or lost. 

And you can't say, "well, those three wins might be because talent changed over time."  Because, that's just the random part. Any actual change in talent is additional to that. 

--------

If all NBA games were pick-em, the SD of team luck in an 82-game season would be around 4.5 wins. Because there are lots of mismatches, which are more predictable, the actual SD is lower, maybe, say, 4.1 games. 

Elo ratings are fully affected by that 4.1 games of binomial luck, but also by another 3 games worth of luck for the random order in which games are won or lost. 

Why would you want to dilute the accuracy of your talent estimate by adding 3 wins worth of randomness to your SD? Only if you're gaining 3 wins worth of accuracy some other way. Like, for instance, if you're able to capture team changes in talent from the beginning of the season to the end. If teams vary in talent over time, like chess players, maybe weighting recent games more highly could give you a better estimate of a team's new level of skill.

Do teams vary in talent, from the beginning to the end of the year, by as much as 3 games (.037)?

Actually, 3 games is a bit of a red herring. You need more than 3 games of talent variance to make up for the 3 games of sequencing luck.

Because, suppose a team goes smoothly from a 40-win talent at the beginning of the year to a 43-win talent at the end of the year. That team will have won 41.5 games, not 40, so the discrepancy between estimate and talent won't be 3 games, but just one-and-a-half games.

As expected, Elo does improve on the 1.5 game discrepancy you get from the traditional method. I ran the simulation again, and found that Elo picked up about 90% of the talent difference rather than 50%. That means that Elo would peg the (eventual) 43-game talent at 42.7.

For a team that transitions from a 40- to a 43-game talent, the traditional method was off by 1.5 games. The Elo method was off by only 0.3 games. 

It looks like Elo is only a 1.2 game improvement over the traditional method, in its ability to spot changes in talent. But it "costs" a 3-game SD for extra randomness. So it doesn't seem like it's a good deal.

To compensate for the 3-game recency SD, you'd need the average within-season talent change to be much more than 3 games. You'd need it to be 7.5 games.

Do teams really change in talent, on average, by 7.5/82 games, over the course of a single season? Sure, some teams must, like they have injury problems to their star players. But on average? That doesn't seem plausible.

------

Besides, what's stopping you from adjusting teams on a case-by-case basis? If Stephen Curry gets hurt ... well, just adjust the Warriors down. If you think Curry is worth 15 games a season, just drop the Warriors' estimate by that much until he comes back.

It's when you try to do things by formula that you run into problems. If you expect Elo to automatically figure out that Curry is hurt, and adjust the Warriors accordingly ... well, sure, that'll happen. Eventually. As we saw, it will take 27 games, on average, until Elo adjusts just for half of Curry's value. And, when he comes back, it'll take 27 games until you get back only half of what Elo managed to adjust by. 

In our example, we assumed that talent changed constantly and slowly over the course of the season. That makes it very easy for Elo to track. But if you lose Curry suddenly, and get him back suddenly 27 games later ... then Elo isn't so good. If losing Curry is worth -.100 in winning percentage, Elo will start at .000 Curry's first game out, and only reach -.050 by Curry's 27th game out. Then, when he's back, Elo will take another 27 games just to bounce back from -.050 to -.025.

In other words, Elo will be significantly off for at least 54 games. Because Elo does weight recent games more heavily, it'll still be better than the traditional method. But neither method really distinguishes itself. When you have a large, visible shock to team talent, I don't see why you wouldn't just adjust for it based on fundamentals, instead of waiting a whole season for your formula to figure it out.

-------

Anyway, if you disagree with me, and believe that team talent does change significantly, in a smooth and gradual way, here's how you can prove it.

Run a regression to predict a team's last 12 games of the season, from their previous seven ten-game records (adjusted for home/road and opponent talent, if you can). 

You'll get seven coefficients. If the seventh group has a significantly higher coefficient than the first group, then you have evidence it needs to be weighted higher, and by how much.

If the weight for the last group turns out to be three or four times as high as the weight for the first group, then you have evidence that Elo might be doing it right after all.

I doubt that would happen. I could be wrong. 

Does previous playoff experience matter in the NBA?

Conventional wisdom says that playoff experience matters. All else being equal, players who have been in the post-season before are more capable of adapting to the playoff environment -- the pressure, the intensity, the refereeing, and so on.

FiveThirtyEight now has a study they say confirms this in the NBA:

"In the NBA postseason since 1980, the team with the higher initial Elo rating has won 74 percent of playoff series. But if a team has both a higher Elo rating and much more playoff experience, that win percentage shoots up to 86 percent. Conversely, teams with the higher Elo rating but much less playoff experience have won just 52 percent of playoff series. These differences are highly statistically and practically significant."

I don't dispute their results, but I disagree that they have truly found evidence that playoff experience matters.

------

There's a certain amount of random luck in the outcomes of games. NBA results have less luck than most other sports, but there's significant randomness nonetheless.

So, if you have two teams, each of which finish with identical records and identical Elo ratings, they're not necessarily equal in talent. One team is probably better than the other, but just had worse luck in the regular season. That's the team from which you would expect better playoff performance. 

But how do you tell them apart? If you have two teams, each of which finishes 55-27, with an Elo of (say) 1600, how can you tell which one is better?

One way is to look at their previous season records. If team A was .500 last year, while team B was .650, it's more likely that B is better. Sure, maybe not: it could be that team B lost a hall-of-famer in the off-season, while team A got a superstar back from injury. But, most of the time, that didn't happen, and you're going to find that team B is still the better team.

If you're looking at last season, the team with the better record is probably the team that got farther in the playoffs. And the team that got farther in the playoffs is probably the one whose players have more playoff experience.

So, when FiveThirtyEight notices that teams with playoff experience tend to outperform Elo expectations, it's not necessarily the actual playoff experience that's the cause. It could be that it's actually that the teams are better than their ratings -- a situation that correlates with playoff experience.

And, of course, "team being better" is a much more plausible explanation for good performance than "team has more playoff experience."

------

Here's a possible counterargument. 

The study doesn't just look at players' *last year's* playoff experience -- it looks at their *career* playoff experience. You'd think that would dilute the effect, somewhat. But, still. Teams tend to stay good or bad for a few years before you could say their talent has changed significantly. Also, players with a lot of playoff experience, even with other teams, are more likely to be good players, and good players tend to play for more talented teams (even if all that makes the team "more talented" comes from them).

------

Another counterargument might be: well, the previous season's performance is already baked into Elo. If a team did well last season, it starts out with a higher rating than a team that didn't. So, checking again what a team did last season shouldn't make any difference. It would be like checking which team played better on Mondays. That shouldn't matter, because the Elo's conclusion that the teams are equal has already used the results of Monday games. 

That's a strong argument, and it would hold up if Elo did, in fact, give last season the appropriate consideration. But I don't think it does. When it calculates the rating, Elo gives previous seasons a very low weighting.

I did a little simulation (which I'll describe next post), and found that, when two NBA teams start with different Elo ratings, but perform identically, half the difference is wiped out after about 27 games.

So, team A starts out with a rating of 1500 (projection of 41-41). Team B starts out with 1600 (52-30). After 27 games playing identically against identical opponents, the Elo difference drops from 100 points to 50. 

After a second 27 games, the difference gets cut in half again, and the teams are now only 25 points apart. After a third 27 games, the difference cuts in half again, to around 12 points. That takes us to 81 games, roughly an NBA season. 

So, at the beginning of the season, Elo thought team B was 100 points better than team A. Now, because both teams had equal seasons, Elo thinks B is only 12 points better than A.

And that's even after considering personnel changes between seasons. If the two ratings started out 100 points apart, their performance last season was actually 133 points apart, because FiveThirtyEight regresses a quarter the way to the mean during the off-season, to account for player aging and roster changes. 

133 points is about 15 games out of 82. So, last year B was 56-26, while A was 41-41. They now have identical 49-33 seasons, and Elo thinks B is only 1.4 games better than A.

In other words, after a combined 162 games, the system thinks A was less than two games luckier than B.

That seems like too much convergence. 

------

Under the FiveThirtyEight system, previous years' performance contributes only 12 percent of the rating; this year's performance is the remaining 88 percent.  And that's *in addition* to adjusting for personnel changes off-season. 

That's far lower than traditional sabermetric standards. For baseball player performance, Bill James (and Tango, and others) have traditionally put previous seasons at 50 percent. They use a "1/2/3" weighting. By contrast, the NBA is using "1/5/44".

Of course, the "1/2/3" is for players; for teams, it should be lower, because of personnel changes. Especially in the NBA, where personnel changes make a much bigger difference because a superstar has such a big impact. 

But, still, 12 percent is far too little weight to give to previous NBA seasons. That's why, when you want to know whose Elo ratings are unlucky, you actually do add valuable information about talent, by checking which teams played much better the last few years than they did this year. 

And that's why playoff experience seems to matter. It correlates highly with teams that did well in the past. 

------

I could be wrong; it shouldn't be too hard to test. 

1. If this hypothesis is right, then playoff experience won't just predict playoff success; it will also predict late regular season success, because the same logic holds. That might be tricky to test because some teams might not give their stars as many minutes in those April games. But you could still check using teams that are fighting for a playoff spot.

2. Instead of breaking Elo ties by looking at previous playoff experience, look instead at the teams' start-of-season rating. I bet you find an even larger effect. And I bet that after you adjust for that, the apparent effect of playoff experience will be much smaller.

3. For teams whose Elo ratings at the end of the season are close to their ratings at the beginning of the season, I predict that the apparent effect of playoff experience will be much smaller. That's because those teams will tend to have been less lucky or unlucky, so you won't need to look as hard at previous performance (playoff experience) to counterbalance the luck.

4. Instead of using Elo as an estimate of team talent entering the playoffs, estimate talent from Vegas odds in April regular-season games (or late-March games, if you're worried about teams who bench their stars in April). I predict you'll find much less of a "playoff experience" effect.



Hat Tip and thanks: GuyM, for link and e-mail discussion

Guy Molyneux and Joshua Miller debate the hot hand

Here's a good "hot hand" debate between Guy Molyneux and Joshua Miller, over at Andrew Gelman's blog.

A bit of background, if you like, before you go there.

-----

In 1985, Thomas Gilovich, Robert Vallone, and Amos Tversky published a study refuting the "hot hand" hypothesis, which is the assumption that after a player has recently performed exceptionally well, he is likely to be "hot" and continue to perform exceptionally well.

The Gilovich [et al] study showed three results:

1. NBA players were actually *worse* after recent field goal successes than after recent failures;

2. NBA players showed no significant correlation between their first free throw and second free throw; and

3. In an experiment set up by Gilovich, which involved long series of repeated shots by college basketball players, there was no significant improvement after a series of hits.

-----

Then, in 2015-2016, Joshua Miller and Adam Sanjurjo found a flaw in Gilovich's reasoning. 

The most intuitive way to describe the flaw is this:

Gilovich assumed that if a player shot (say) 50 percent over the full sequence of 100 shots, you'd expect him to shoot 50 percent after a hit, and 50 percent after a miss.

But this is clearly incorrect. If a player hit 50 out of 100, then, if he made his (or her) first shot, what's left is 49 out of 99. You wouldn't expect 50%, then, but only about 49.5%. And, similarly, you'd expect 50.5% after a miss.

By assuming 50%, the Gilovich study set the benchmark too high, and would call a player cold or neutral when he was actually neutral or hot.

(That's a special case of the flaw Miller and Sanjurjo found, which applies only to the "after one hit" case. For what happens after a streak of two or more consecutive hits, it's more complicated. Coincidentally, the flaw is actually identical to one that Steven Landsburg posted for a similar problem, which I wrote about back in 2010. See my post here, or check out the Miller paper linked to above.)

------

The Miller [and Sanjurjo] paper corrected the flaw, and found that in Gilovich's experiment, there was indeed a hot hand, and a large one. In the Gilovich paper, shooters and observers were allowed to bet on whether the next shot would be made. The hit rate was actually seven percentage points higher when they decided to bet high, compared to when they decided to bet low (for example, 60 percent compared to 53 percent).

That suggests that the true hot hand effect must be higher than that -- because, if seven percentage points was what the participants observed in advance, who knows what they didn't observe? Maybe they only started betting when a streak got long, so they missed out on the part of the "hot hand" effect at the beginning of the streak.

However, there was no evidence of a hot hand in the other two parts of the Gilovich paper. In one part, players seem to hit field goals *worse* after a hit than after a miss -- but, corrected for the flaw, it seems (to my eye) that the effect is around zero. And, the "second free throw after the first" doesn't feature the flaw, so the results stand.

------

In addition, in a separate paper, Miller and Sanjurjo analyzed the results of the NBA's three-point contest, and found a hot hand there, too. I wrote about that in two posts in 2015. 

-------

From that, Miller argues that the hot hand *does* exist, and we now have evidence for it, and we need to take it seriously, and it's not a cognitive error to believe the hot hand represents something real, rather than just random occurrences in random sequences. 

Moreover, he argues that teams and players might actually benefit from taking a "hot hand" into account when formulating strategy -- not in any specific way, but, rather, that, in theory, there could be a benefit to be found somewhere.

He also uses an "absence of evidence is not evidence of absence"-type argument, pointing out that if all you have is binary data, of hits and misses, there could be a substantial hot hand effect in real life, but one that you'd be unable to find in the data unless you had a very large sample. I consider that argument a parallel to Bill James' "Underestimating the Fog" argument for clutch hitting -- that the methods we're using are too weak to find it even if it were there.

------

And that's where Guy comes in. 

Here's that link again. Be sure to check the comments ... most of the real debate resides there, where Miller and Guy engage each other's arguments directly.

Pythagoras, log5, and the independence of game scores

Ted Turocy's comment from the previous post is worth repeating in full:

"My only note is that Phil's assuming implicitly in these examples that the two teams' "scoreboard scores" are statistically independent. Under that assumption, what he's written here is right (at least to first order). 

However, the "latent score" that underlies the Bradley-Terry-Luce-log5 model is not necessarily the "scoreboard score" that you see on the scoreboard (hence why I'm calling it the "scoreboard score"). It is possible for log5 still to work (or, work better than these calculations suggest) even if scoreboard scores are not extreme-value distributed, if the scoreboard scores aren't independent. 

But that is a fine point, because, in fact, the number of types of sport or game in which log5 works precisely is exactly zero. The number of them in which log5 works reasonably well seems to be quite large (depending on how much accuracy you demand)."

Ted makes three important points here.


1. Accuracy

I agree that log5 works reasonably well for most sports. I may have given the impression that I thought it was too wrong to be useful ... that's not the case. I do think, though, that it's too wrong to be useful to, say, the third decimal place.

In most cases, any inaccuracy in log5 gets lost in all the other inevitable biases involved in estimation. For instance, do we ever really know teams' talent against a .500 team, to plug into the formula? We don't. And the error in estimating them is likely to be at least as big as any inaccuracy in log5 itself, especially if we're dealing with middle-of-the-road teams.

Another commenter last post asked if there's anything better, or more accurate, than log5. Not that I know of. And it seems unlikely, because every sport has a differently shaped distribution of scores.

I think log5 is the best candidate because (1) it seems to work reasonably well for most sports, and (2) we have two cases where we know it works perfectly: retrospectively, and for sudden-death binary games. Those two cases suggest to me that there's some theoretical underpinning that makes it a good anchor. 

My point in all this log5 stuff is not to deny that it's useful. My point is, why does it seem to work so well, and what's the underlying theory? 

Speaking only for my own intuition ... I'm probably more confident in log5 now, than when I only had counterexamples like "height baseball."  That's because now, I have a certain gut understanding that on those occasions when log5 fails, it fails only for the most extreme cases.


2. Latent Scores

Ted points out that it's not actually the "scoreboard score" that needs to be logistically distributed for log5 to be theoretically correct. It can be a "latent score" instead.

What's a "latent score?"  It's an alternative measure of what happened in the game, but one that always preserves who won and who lost.

In baseball, the home team doesn't play the bottom of the ninth inning if it's ahead. Imagine changing the rules so that it *does* play that inning. In that case, if the score differential was logistically distributed before, it won't be logistically distributed now (since the new distribution will be wider -- teams will win by more runs than they used to, on average). 

But: if log5 worked before the change, it'll work after the change. Because, log5 only deals with talent in terms of probabilities of winning, and those aren't affected at all by whether the team in the lead plays the bottom of the ninth.

Something similar happens in basketball and football. In the NFL, if a team is up by 17 points with, say, four minutes to play, it won't try too hard to score more points. In effect, it will forfeit its "bottom of the ninth" by concentrating on running out the clock instead. That strategy might even result in the opposition scoring more points than they would otherwise -- but not to the extent that they win where they would have otherwise lost. So, maybe you can consider the "latent score" in football to be "what would have happened if the team in the lead had kept trying to run up the score."

In basketball, that kind of thing happens too -- running out the clock, playing "garbage time" bench players, and deliberate fouling. Those don't all work in the same direction, but they change the distribution of the score without (usually) affecting the outcome of the game.

Which brings us to:


3. Non-independence

In these examples, the "latent score" effect is the result of the fact that the two teams' scores aren't independent. A team's strategy, and therefore score, depends on the other team's score.

In baseball, when you know one team got shut out, your expectation for the other team's score should be lower -- because it's more likely to have been the home team and not have played the bottom of the ninth. In football, you'd expect blowouts to be less frequent than raw talent suggests, because a team that's safely (but barely safely) up late in the game is too busy managing the clock to occasionally score three quick touchdowns in the fourth quarter.

For both baseball and football, you'd expect the score differentials to be narrower than a logistic distribution, but for log5's accuracy to be unaffected by that score compression.

Can we measure that? Well, I'll try one possibility. It's not really that strong as evidence, because there are so many other factors involved than non-independence, but, I think, it's definitely suggestive. 

-----

Assume that the log5 assumptions are true for MLB -- that score differential is logistic, and team scores are independent. If that's true, what's the expected standard deviation of score differential over an MLB season?

It seems like there's not enough information to answer that question. And there isn't. We have to add one more thing: that the Pythagorean exponent that works best in baseball is 1.83. 

Now, we can actually figure out the SD of the difference in team runs per game. Or, we can come close.

Last post, I linked to a paper (pdf) that showed that, where log5 is accurate, you can compute the chance of a team winning by treating one "latent score differential" as a logistic with mean equal to the log odds ratio, and constant SD of 1.81. Then, you just compute the area under the curve on both sides of zero -- the left area is one team winning, and the right area is the other team winning.

For the distribution of "scoreboard score," we want the mean to be equal to the point spread, not the log odds ratio. To do that, we can multiply the mean by (point spread / log odds ratio). 

If we do that, then we need to also multiply the SD by the same amount. That keeps the shape of the distribution the same, which preserves the areas on both sides of zero. (Those areas are the two teams' respective win probabilities.)

Suppose one team is 0.1 runs better than the other, in talent, outscoring its opponent by an expected 4.6 to 4.5. With a Pythagorean exponent of 1.83, that's a winning percentage of .510, which is an odds ratio of 1.04, which is a log odds ratio of 0.04.

We want the mean to be 0.1, rather than 0.04. So we multiply the mean by 2.5. And, we also multiply the SD by that same 2.5. Now, the mean is 0.1, which we know is correct ... and the SD is now 4.5. 

So, for a given MLB game, the SD of (team A score - team B score) is, in theory, 4.5 runs.

That's the SD against the spread. We need another step if we want the SD for a full MLB season. We have to add in the variance of the spread. Suppose the average talent differential SD is 0.5 runs (at 10 runs per game, that means the favorite is .550). 4.5 squared plus 0.5 squared equals 4.6 squared.

So, in theory, we expect an SD of 4.6. Which is pretty close to real life! In 2009, the actual SD of score differential was 4.4. 

Why the difference? My gut says: (1) the difference between 4.6 and 4.4 is the extent of the "non-independence" of scores in baseball; (2) the bottom-of-the-ninth effect is the cause of most of the difference, since team strategy doesn't differ that much based on score; and (3) that the fact that 4.4 and 4.6 are so close together suggests that the "latent score" is close to logistic, and so log5 should work quite well for baseball. 

I don't have strong arguments for these three conclusions; I might be wrong.

-----

Now, let's look at the NFL. Suppose, again, that one team is 0.1 points better than the other, 26.1 points to 26. And assume that the best Pythagorean exponent for the NFL is 2.37, as Wikipedia says.

Instead of going through the calculation, I'll just show the formula that follows from the logic I described:

(In this formula, "spread" has to be equal to "expected favorite score" - "expected underdog score". I just used "spread" to make it easier to read.)

Plug in the numbers, and you get

SD(score diff) = 19.94

That's for a given game, so the 19.94 is the theoretical difference against the spread.

But, in real life, NFL game outcomes have an SD of only about 14 points against the spread.

Why the difference? Non-independence of scores, I'd argue again. If teams in the lead didn't deliberately run out the clock, they'd score a few more points. The difference here is 6 points. Could it be that if teams in the lead played more aggressively, they'd gain an extra 6 points on their opponents? Seems a bit high, but maybe not -- I haven't done any calculations, even the back-of-the-envelope variety.

My wild-ass guys says ... non-independence is maybe 4 points of the difference, and football scores being non-logistic (even if there were independence) is the other 2 points. But I don't really know.

One way you could check: treat the first half as if it were the entire game. Find the best Pythagorean exponent for half-game results, use the formula to predict the SD, and compare it to the actual first half SD. My guess: the two SDs would be closer than they are here.

------

For the NBA, the Pythagorean exponent is 13.91. Again assuming an 0.1 point favorite, the formula says:

SD(score diff) = 0.1 * (1.814) / [ 13.91 * log(100.1/100) ]
SD(score diff) = 13.05

The actual SD against the spread is about 11.5. So, basketball doesn't work out as well as baseball, but it does come closer than football. 

Why the discrepancy between 11.5 and 13.05? I'm guessing yet again, but, in this case, I think it's mostly the distribution, not the non-independence. 

In MLB and the NFL, there are few possessions, and runs/points come in bunches. Most possessions result in zero, so the SD per possession is high. Those things make for fat tails, so the distribution is closer to logistic.

In the NBA, there are about 100 possessions per team, and points are scored frequently. Scores are the sum of a large number of (nearly) identically distributed (nearly) independent variables with a low SD. That means that by Central Limit Theorem, you'd expect the differential to be closer to normal, with skinny tails, rather than logistic, with fat tails.

Again, I could be wrong. There is some non-independence going on. Late in the game, the team in the lead sacrifices points for clock, which makes the SD smaller. A team with a big lead plays its bench, which again makes the SD smaller. On the other hand, the trailing team will commit deliberate fouls, which usually results in the opposition padding its lead, which makes the SD larger. 

So, yes, there is non-independence going on, but I still suspect the size of the discrepancy would be roughly the same even if team scores were completely independent.

-------

You can use the formula to predict the pythagorean exponent for a given sport, when you only know the actual SD of game outcomes against the spread. You just switch the SD and exponent terms:

In the NFL case, if you sub in an SD of 14 under "Observed SD against spread", and then solve for "Pythagorean Exponent," it works out to 3.4. That's much higher than the 2.37 that's accepted as the exponent for the best Pythagorean estimator. 

But we knew that would happen, because this is just the mirror image of the discrepancy in SD that we found earlier for the NFL. Here, the actual exponent is about 70 percent of the theoretical one. There, the actual SD was about 70 percent of the theoretical one. 

--------

And, again: the discrepancy, the fact that you get 70 percent instead of 100 percent, has at least two causes:

1. The team scores aren't independent, in a way where the non-independence preserves the win probabilities; and

2. Even after correcting the "scoreboard score" for non-independence, and creating a "latent score," the distribution of score differential doesn't actually match the logistic distribution.

Number (1) doesn't affect how well log5 works. Number (2) *does* affect how well log5 works.

And that's why I suspect log5 might work better for football than for basketball, despite the fact that the discrepancy is higher. Because, in football, my gut says that most of the discrepancy is (1), but, for basketball, I suspect most of it is (2). 

When log5 does and doesn't work

Team A, with an overall winning percentage talent of .600, plays against a weaker Team B with an overall winning percentage of .450. What's the probability that team A wins? 

In the 1980s, Bill James created the "log5" method to answer that question. The formula is

P = (A - AB)/(A+B-2AB)

... where A is the talent level of team A winning (in this case, .600), and B is the talent level of team B (.450).

Plug in the numbers, and you get that team A has a .647 chance of winning against team B. 

That makes sense: A is .600 against average teams. Since opponent B is worse than average, A should be better than .600. 

Team B is .050 worse than average, so you'd kind of expect A to "inherit" those 50 points, to bring it to .650. And it does, almost. The final number is .647 instead of .650. The difference is because of diminishing returns -- those ".050 lost wins" are what B loses to *average* teams because it's bad. Because A is better than average, it would have got some of those .050 wins anyway because it's good, so B can't "lose them again" no matter how bad it is.

In baseball, the log5 formula has been proven to work very well.

------

There was some discussion of log5 lately on Tango's site (unrelated to this post, but very worthwhile), and that got me thinking. Specifically, it got me thinking: log5 CANNOT be right. It can be *almost* right, but it can never be *exactly* right.

In the baseball context, it can be very, very close, indistinguishable from perfect. But in other sports, or other contexts, it could be way wrong. 

Here's one example where it doesn't work at all.

Suppose that, instead of actually playing baseball games, teams just measured their players' average height, and the taller team wins. And, suppose there are 11 teams in the league, and there's a balanced 100-game season.

What happens? Well, the tallest team beats everyone, and goes 100-0. The second-tallest team beats everyone except the tallest, and winds up 90-10. The third-tallest goes 80-20. And so on, all the way down to 0-100.

Now: when a .600 team plays a .400 team, what happens? The log5 formula says it should win 69.2 percent of those games. But, of course, that's not right -- it will win 100 percent of those games, because it's always taller.

For height, the log5 method fails utterly.

------

What's the difference between real baseball and "height baseball" that makes log5 work in one case but not the other?

I'm not 100% sure of this, but I think it's due to a hidden, unspoken assumption in the log5 method. 

When we say, "Team A is a .600 talent," what does that mean? It could mean either of two things:

-- A1. Team A is expected to beat 60 percent of the opponents it plays.

-- A2. If Team A plays an average team, it is expected to win 60 percent of the time.

Those are not the same! And, for the log5 method to work, assumption A1 is irrelevant. It's assumption A2 that, crucially, must be true. 

In both real baseball and "height baseball," A1 is true. But that doesn't matter. What matters is A2. 

In real baseball, A2 is close enough. So log5 works.

In "height baseball," A2 is absolutely false. If Team A (.600) plays an average team (.500), it will win 100 percent of the time, not 60 percent! And that's why log5 doesn't work there.

-------

What it's really coming down to is our old friend, the question of talent vs. luck. In real baseball, for a single game, luck dwarfs talent. In "height baseball," there's no luck at all -- the winner is just the team with the most talent (height). 

Here are two possible reasons a sports team might have a .600 record:

-- B1: Team C is more talented than exactly 60 percent of its opponents

-- B2: Team C is more talented than average, by some unknown amount (which varies by sport) that leads to it winning exactly 60 percent of its games.

Again, these are not the same. And, in real life, all sports (except "height baseball") are some combination of the two. 

B1 refers completely to talent, but B2 refers mostly to luck. The more luck there is, in relation to talent, the better log5 works.

Baseball has a pretty high ratio of luck to talent -- on any given day, the worst team in baseball can beat the best team in baseball, and nobody bats an eye. But in the NBA, there's much less randomness -- if Philadelphia beats Golden State, it's a shocking upset. 

So, my prediction is: the less that luck is a factor in an outcome, the more log5 will underestimate the better team's chance of winning.

Specifically, I would predict: log5 should work better for MLB games than for NBA games.

--------

Maybe someone wants to do some heavy thinking and figure how to move this forward mathematically.  For now, here's how I started thinking about it.

In MLB, the SD of team talent seems to be about 9 games per season. That's 90 runs. Actually, it's less, because you have to regress to the mean. Let's call it 81 runs, or half a run per game. (I'm too lazy to actually calculate it.) Combining the team and opponent, multiply by the square root of two, to give an SD of around 0.7 runs.

The SD of luck, in a single game, is much higher. I think that if you computed the SD of a single team's 162 runs-scored-that-game, you'd get around 3. The SD of runs allowed is also around 3, so the SD of the difference would be around 4.2.

SD(MLB talent) = 0.7 runs
SD(MLB luck)   = 4.2 runs

Now, let's do the NBA. From basketball-reference.com, the SD of the SRS rating seems to be just under 5 points. That's based on outcomes, so it's too high to be an estimate of talent, and we need to regress to the mean. Let's arbitrarily reduce it to 4 points. Combining the two teams, we're up to 5.2 points.

What about the SD of luck? This site shows that, against the spread, the SD of score differential is around 11 points. So we have

SD(NBA talent) =  5.2 points
SD(NBA luck)   = 11.0 points

In an MLB game, luck is 6 times as important as talent. In an NBA game, luck is only 2 times as important as talent. 

But, how you apply that to fix log5, I haven't figured out yet. 

What I *do* think I know is that the MLB ratio of 6:1 is large enough that you don't notice that log5 is off. (I know that from studies that have tested it and found it works almost perfectly.) But I don't actually know whether the NBA ratio of 2:1 is also large enough. My gut says it's not -- I suspect that, for the NBA, in extreme cases, log5 will overestimate the underdog enough so that you'll notice. 

-------

Anyway, let me summarize what I speculate is true:

1. The log5 formula never works perfectly. Only as the luck/talent ratio goes to infinity, will log5 be theoretically perfect. (But, then, the predictions will always be .500 anyway.)  In all other cases, log5 will underestimate, to some extent, how much the better team will dominate.

2. For practical purposes, log5 works well when luck is large compared to talent. The 6:1 ratio for a given MLB game seems to be large enough for log5 to give good results.

3. When comparing sports, the more likely it is that the more-talented team beats the less-talented team, the worse log5 will perform. In other words: the bigger the Vegas odds on underdogs, the worse log5 will perform for that sport.

4. You can also estimate how well log5 will perform with a simple test. Take a team near the extremes of the performance scale (say, a .600/.400 team in MLB, or a .750/.250 team in the NBA), and see how it performed specifically against only those teams with talent close to .500.

If a .750 team has a .750 record against teams known to be average, log5 will work great. But if it plays .770 or .800 or .900 ball against teams known to be average, log5 will not work well. 

-------

All this has been mostly just thinking out loud. I could easily be wrong.