A study on NBA home court advantage

Economist Tyler Cowen often links to NBA studies in his "Marginal Revolution" blog ... here's a recent one, from an August post. (Follow his link to download the study ... you can also find a press release by Googling the title.)

The study used a neural network to try to figure out what factors are most important for home (court) advantage (which I'll call "HCA"). The best fit model used twelve variables: two-point shots made, three-point shots made, and free throws made -- repeated for team at home, opposition on road, team on road, and opposition at home.

The authors write, 

"Networks that include shot attempts, shooting percentage, total points scored, field goals, attendance statistics, elevation and market size as predictors added no improvement in performance. ...

"Contrary to previous work, attendance, elevation and market size were not relevant to understanding home advantage, nor were shot attempts, shooting percentage, overall W-L%, and total points scored."

On reflection, it's not surprising that those other variables don't add anything ... the ones they used, shots made, are enough to actually compute points scored and allowed. Once you have that, what does it matter what the attendance was? If attendance matters at all, it would affect wins through points scored and allowed, not something independent of scoring. And "total points scored" weren't "relevant" because they were redundant, given shots made.

------

The study then proceeds to a "sensitivity analysis," where they increase the various factors, separately, to see what happens to HCA. It turns out that when you increase two-point shots made by 10 percent, you get three to four times the impact on HCA compared to when you increase three-point shots made by the same 10 percent.

The authors write,

"[This] suggests teams can maximize their advantage -- and hence their odds of winning -- by employing different shot selection strategies when home versus away. When playing at home, teams can maximize their advantage by shooting more 2P and forcing opponents to take more 2P shots. When playing away, teams can minimize an opponent's home advantage by shooting more 3P and forcing opponents to take more 3P shots."

Well, yes, but, at the same time, no. 

The reason increasing 2P by 10 percent leads to a bigger effect than increasing 3P by 10 percent is ... that 10 percent of 2P is a lot more points! Eyeball the graph of "late era" seasons the authors used (I assume it's the sixteen seasons ending with 2015-16). Per team-season, it looks like the average is maybe 2500 two-point shots made, but only 500 three-point shots.

Adding 10 percent more 2P is 250 shots for 500 points. Adding 10 percent more 3P is 50 shots for 150 points. 500 divided by 150 gives a factor of three-and-a-third -- almost exactly what the paper shows!

I'd argue that what the study discovered is that points seem to affect HCA and winning percentage equally, regardless of how they are scored. 

------

Even so, the argument in the paper doesn't work. By the authors' own choice of variables, HCA is increased by *making* 2P shots, not my *taking* 2P shots. Rephrasing the above quote, what the study really shows is,

"When playing at home, teams can maximize their advantage by concentrating on *making* more 2P and on forcing opponents to *miss* more 2P. That's assuming that it's just as easy to impact 2P percentages by 10 percent than to impact 3P percentages by 10 percent."

But we could have figured that out easily, just by noticing that 10 percent of 2P is more points than 10 percent of 3P.

------

The authors found that you increase your HCA more with a 10 percent increase in road three-pointers than by a 10 percent increase in road two-pointers. 

Sure. But that's because, with the 3P, you actually wind up scoring fewer road points. Which means you win fewer road games. Which makes your HCA larger, since winning fewer road games increases the difference between home and road. 

It's because the worse you do on the road, the bigger your home court advantage!

Needless to say, you don't really want to increase your HCA by tanking road games. The authors didn't notice that's what they were suggesting.

I think the issue is that the paper assumes that increasing your HCA is always a good thing. It's not. It's actually neutral. The object isn't to increase or decrease your HCA. It's  to *win more games*. You can do that by winning more games at home, increasing your home court advantage, or by winning more games on the road, decreasing your home court advantage.

It's one of those word biases we all have if we don't think too hard. "Increasing your advantage" sounds like something we should strive for. The problem is, in this context, the home "advantage" is relative to *your own performance* on the road. So it really isn't an "advantage," in the sense of something that makes you more likely to beat the other team. 

In fact, if you rotate "Home Court Advantage" 360 degrees and call it "Road Court Disadvantage," now it feels like you want to *decrease* it -- even though it's exactly the same number!

But HCA isn't something you should want to increase or decrease for its own sake. It's just a description of how your wins are distributed.

How Elo ratings overweight recent results

"Elo" is a rating system widely used to rate players in various games, most notably chess. Recently, FiveThirtyEight started using it to maintain real-time ratings for professional sports teams. Here's the Wikipedia page for the full explanation and formulas.

In Elo, everyone gets a number that represents their skill. The exact number only matters in the difference between you and the other players you're being compared to. If you're 1900 and your opponent is 1800, it's exactly the same as if you're 99,900 and your opponent is 99,800.

In chess, they start you off with a rating of 1200, because that happens to be the number they picked. You can interpret the 1200 either as "beginner," or "guy who hasn't played competitively yet so they don't know how good he is."  In the NBA system, FiveThirtyEight decided to start teams off with 1500, which represents a .500 team. 

A player's rating changes with every game he or she plays. The winner gets points added to his or her rating, and the loser gets the same number of points subtracted. It's like the two players have a bet, and the loser pays points to the winner.

How many points? That depends on two things: the "K factor," and the odds of winning.

The "K factor" is chosen by the organization that does the ratings. I think of it as double the number of points the loser pays for an evenly matched game. If K=20 (which FiveThirtyEight chose for the NBA), and the odds are even, the loser pays 10 points to the winner.

If it's not an even match, the loser pays the number of points by which he underperformed expectations. If the Warriors had a 90% chance of beating the Knicks, and they lost, they'd lose 18 points, which is 90% of K=20. If the Warriors win, they only gain 2 points, since they only exceeded expectations by 10 percentage points.

How does Elo calculate the odds? By the difference between the two players' ratings. The Elo formula is set so that a 400-point difference represents a 10:1 favorite. An 800 point favorite is 100:1 (10 for the first 400, multiplied by 10 for the second 400). A 200 point favorite is 3.16 to 1 (3.16 is the square root of 10). A 100 point favorite is 1.78 to 1 (the fourth root of 10), and so on.

In chess, the K factor varies depending on which chess federation is doing the ratings, and the level of skill of the players. For experienced non-masters, K seems to vary between 15 and 32 (says Wikipedia). 

------

Suppose A and B have equal ratings of 1600, and A beats B. With K=20, A's rating jumps to 1610, and B's rating falls to 1590. 

A and B are now separated by 20 points in the ratings, so A is deemed to have odds of 1.12:1 of beating B. (That's because the "400/20th" root of 10 is 1.12.)  That corresponds to an expected winning percentage of .529.

After lunch, they play again, and this time B beats A. Because B was the underdog, he gets more than 10 points -- 10.6 points (.529 times K=20), to be exact. And A loses the identical 10.6 points.

That means after the two games, A has a rating of 1599.4, and B has a rating of 1600.6.

------

That example shows one of the properties of Elo -- it weights recent performance higher than past performance. In their two games, A and B effectively tied, each going 1-1. But B winds up with a higher rating than A because his win was more recent.

Is that reasonable? In a way, it is. People's skill at chess changes over their lifetimes, and it would be weird to give the same weight to a game Garry Kasparov played when he was 8, as you would to a game Garry Kasparov played as World Champion.

But in the A and B case, it seems weird. A and B played both games the same day, and their respective skills couldn't have changed that much during the hour they took their lunch break. In this case, it would make more sense to weight the games equally.

Well, according to Wikipedia, that's might would actually happen. Instead of updating the ratings every game, the Federation would wait until the end of the tournament, and then compare each player to his or her overall expectation based on ratings going in. In this case, A and B would be expected to go 1-1 in two games, which they did, so their ratings wouldn't change at all.

But, if A and B's games were days or weeks apart, as part of different tournaments, the two games would be treated separately, and B might indeed wind up 1.2 points ahead of A.

------

Is that a good thing, giving a higher weight to recency? It depends how much higher a weight. 

People's skill does indeed change daily, based on mood, health, fatigue -- and, of course, longer-term trends in skill. In the four big North American team sports, it's generally true that players tend to improve in talent until a certain age (27 in baseball), then decline. And, of course, there are non-age-related talent changes, like injuries, or cases where players just got a lot better or a lot worse partway through their careers.

That's part of the reason we tend to evaluate players based on their most recent season. If a player hit 15 home runs last year, but 20 this year, we expect the 20 to be more indicative of what we can expect next season.

Still, I think Elo gives far too much weight to recent results, when applied to professional sports teams. 

Suppose you're near the end of the season, and you're looking at a team with a 40-40 record -- the Bulls, say. From that, you'd estimate their talent as average -- they're a .500 team.

Now, they win an even-money game, and they're 41-40, which is .506. How do you evaluate them now? You take the .506, and regress to the mean a tiny bit, and maybe estimate they're a .505 talent. (I'll call that the "traditional method," where you estimate talent by taking the W-L record and regressing to the mean.)

What would Elo say? Before the 81st game, the Bulls were probably rated at 1500. After the game, they've gained 10 points for the win, bringing them to 1510.

But, 1510 represents a .514 record, not .505. So, Elo gives that one game almost three times the weight that the traditional method does.

Could that be right? Well, you'd have to argue that maybe because of personnel changes, team talent changes so much from the beginning of the year to the end that the April games are worth three times as much as the average game. But ... well, that still doesn't seem right. 

-------

Techincal note: I should mention that FiveThirtyEight adds a factor to their Elo calculation -- they give more or fewer points to the winner of the game based on the score differential. If a favorite wins by 1 point, they'll get a lot fewer points than if they won by 15 points. Same for the underdog, except that the underdog always gets more points than the favorite for the same point differential -- which makes sense.

FiveThirtyEight doesn't say so explicitly, but I think they set the weighting factor so that the average margin of victory corresponds to the number points the regular Elo would award the winner.

Here's the explanation of their system.

-------

Elo starts with a player's rating, then updates it based on results. But, when it updates it, it has no idea how much evidence was behind the player's rating in the first place. If a team is at 1500, and then it wins an even game, it goes to 1510 regardless of whether it was at 1500 because it's an expansion team's first game, or because it was 40-40, or (in the case of chess) it's 1000-1000.

The traditional method, on the other hand, does know. If a team goes from 1-1 to 2-1, that's a move of .167 points (less after regressing to estimate talent, of course). If a team goes from 40-40 to 41-40, that's a move of only .005 points. 

Which makes sense; the more evidence you already have, the less your new evidence should move your prior. But if your prior moves the same way regardless of the previous evidence, you're seriously underweighting that previous evidence (which means you're overweighting the new evidence).

The chess Federations implicitly understand this, that you should give less weight to new results when you have more older results. That's why they vary the K-values based on who's playing. 

FIDE, for instance, weights new players at K=40, experienced players at K=20, and masters (who presumably have the most experience) at K=10.

------

As I said last post, I did a simulation. I created a team that was exactly average in talent, and assumed that FiveThirtyEight had accurately given them an average rating at the beginning of the year. I played out 1000 random seasons, and, on average, the team wound up at right where it started, just as you would expect.

Then, I modified the simulation as if FiveThirtyEight had underrated the team by 50 points, which would peg them as a .429 team. (They use their "CARM-Elo" player projection system for those pre-season ratings. I'm not saying that system is wrong, just checking what happens when a projection happens to be off.)

It turned out that, at the end of the 82-game season, Elo had indeed figured out the team was better than their initial rating, and had restored 45 of the 50 points. They were still underrated, but only by 5 points (.493) instead of 50 (.429). 

Effectively, the current season wiped out 90% of the original rating. Since the original rating was based on the previous seasons, that means that, to get the final rating, Elo effectively weighted this year at 90%, and the previous years at 10%. 

10% is close to 12.5%. I'll use that because it makes the calculation a bit easier. At 12.5%, which is one-eighth, it means the NBA season contains three "half lives" of about 27 games each. 

That is: after 27 games, the gap of 50 points is reduced by half, to 25. After another 27 games, it's down to 12. After a third 27 games, it's down to 6, which is 12.5% of where the gap started.

That means that to calculate a final rating, the thirds of seasons are effectively weighted in a ratio of 1:2:4. A game in April has four times the weight of a game in November. Last post, I argued why I think that's too high.

-------

Here's another way of illustrating how recency matters. 

I tweaked the simulation to do something a little different. Instead of creating 1,000 different seasons, I created only one season, but randomly reordered the games 1,000 times. The opponents and final scores were the same; only the sequence was different. 

Under the traditional method, the talent estimates would be the same, since all 1,000 teams had the same W-L record. But the Elo ratings varied, because of recency effects. They varied with an SD of about 26 points. That's about .037 in winning percentage, or 3 wins per 82 games.

If you consider the SD to be, in a sense, the "average" discrepancy, that means that, on average, Elo will misestimate a team's talent by 3 wins. That's for teams with the same actual record -- based only on the randomness of *when* they won or lost. 

And you can't say, "well, those three wins might be because talent changed over time."  Because, that's just the random part. Any actual change in talent is additional to that. 

--------

If all NBA games were pick-em, the SD of team luck in an 82-game season would be around 4.5 wins. Because there are lots of mismatches, which are more predictable, the actual SD is lower, maybe, say, 4.1 games. 

Elo ratings are fully affected by that 4.1 games of binomial luck, but also by another 3 games worth of luck for the random order in which games are won or lost. 

Why would you want to dilute the accuracy of your talent estimate by adding 3 wins worth of randomness to your SD? Only if you're gaining 3 wins worth of accuracy some other way. Like, for instance, if you're able to capture team changes in talent from the beginning of the season to the end. If teams vary in talent over time, like chess players, maybe weighting recent games more highly could give you a better estimate of a team's new level of skill.

Do teams vary in talent, from the beginning to the end of the year, by as much as 3 games (.037)?

Actually, 3 games is a bit of a red herring. You need more than 3 games of talent variance to make up for the 3 games of sequencing luck.

Because, suppose a team goes smoothly from a 40-win talent at the beginning of the year to a 43-win talent at the end of the year. That team will have won 41.5 games, not 40, so the discrepancy between estimate and talent won't be 3 games, but just one-and-a-half games.

As expected, Elo does improve on the 1.5 game discrepancy you get from the traditional method. I ran the simulation again, and found that Elo picked up about 90% of the talent difference rather than 50%. That means that Elo would peg the (eventual) 43-game talent at 42.7.

For a team that transitions from a 40- to a 43-game talent, the traditional method was off by 1.5 games. The Elo method was off by only 0.3 games. 

It looks like Elo is only a 1.2 game improvement over the traditional method, in its ability to spot changes in talent. But it "costs" a 3-game SD for extra randomness. So it doesn't seem like it's a good deal.

To compensate for the 3-game recency SD, you'd need the average within-season talent change to be much more than 3 games. You'd need it to be 7.5 games.

Do teams really change in talent, on average, by 7.5/82 games, over the course of a single season? Sure, some teams must, like they have injury problems to their star players. But on average? That doesn't seem plausible.

------

Besides, what's stopping you from adjusting teams on a case-by-case basis? If Stephen Curry gets hurt ... well, just adjust the Warriors down. If you think Curry is worth 15 games a season, just drop the Warriors' estimate by that much until he comes back.

It's when you try to do things by formula that you run into problems. If you expect Elo to automatically figure out that Curry is hurt, and adjust the Warriors accordingly ... well, sure, that'll happen. Eventually. As we saw, it will take 27 games, on average, until Elo adjusts just for half of Curry's value. And, when he comes back, it'll take 27 games until you get back only half of what Elo managed to adjust by. 

In our example, we assumed that talent changed constantly and slowly over the course of the season. That makes it very easy for Elo to track. But if you lose Curry suddenly, and get him back suddenly 27 games later ... then Elo isn't so good. If losing Curry is worth -.100 in winning percentage, Elo will start at .000 Curry's first game out, and only reach -.050 by Curry's 27th game out. Then, when he's back, Elo will take another 27 games just to bounce back from -.050 to -.025.

In other words, Elo will be significantly off for at least 54 games. Because Elo does weight recent games more heavily, it'll still be better than the traditional method. But neither method really distinguishes itself. When you have a large, visible shock to team talent, I don't see why you wouldn't just adjust for it based on fundamentals, instead of waiting a whole season for your formula to figure it out.

-------

Anyway, if you disagree with me, and believe that team talent does change significantly, in a smooth and gradual way, here's how you can prove it.

Run a regression to predict a team's last 12 games of the season, from their previous seven ten-game records (adjusted for home/road and opponent talent, if you can). 

You'll get seven coefficients. If the seventh group has a significantly higher coefficient than the first group, then you have evidence it needs to be weighted higher, and by how much.

If the weight for the last group turns out to be three or four times as high as the weight for the first group, then you have evidence that Elo might be doing it right after all.

I doubt that would happen. I could be wrong. 

Does previous playoff experience matter in the NBA?

Conventional wisdom says that playoff experience matters. All else being equal, players who have been in the post-season before are more capable of adapting to the playoff environment -- the pressure, the intensity, the refereeing, and so on.

FiveThirtyEight now has a study they say confirms this in the NBA:

"In the NBA postseason since 1980, the team with the higher initial Elo rating has won 74 percent of playoff series. But if a team has both a higher Elo rating and much more playoff experience, that win percentage shoots up to 86 percent. Conversely, teams with the higher Elo rating but much less playoff experience have won just 52 percent of playoff series. These differences are highly statistically and practically significant."

I don't dispute their results, but I disagree that they have truly found evidence that playoff experience matters.

------

There's a certain amount of random luck in the outcomes of games. NBA results have less luck than most other sports, but there's significant randomness nonetheless.

So, if you have two teams, each of which finish with identical records and identical Elo ratings, they're not necessarily equal in talent. One team is probably better than the other, but just had worse luck in the regular season. That's the team from which you would expect better playoff performance. 

But how do you tell them apart? If you have two teams, each of which finishes 55-27, with an Elo of (say) 1600, how can you tell which one is better?

One way is to look at their previous season records. If team A was .500 last year, while team B was .650, it's more likely that B is better. Sure, maybe not: it could be that team B lost a hall-of-famer in the off-season, while team A got a superstar back from injury. But, most of the time, that didn't happen, and you're going to find that team B is still the better team.

If you're looking at last season, the team with the better record is probably the team that got farther in the playoffs. And the team that got farther in the playoffs is probably the one whose players have more playoff experience.

So, when FiveThirtyEight notices that teams with playoff experience tend to outperform Elo expectations, it's not necessarily the actual playoff experience that's the cause. It could be that it's actually that the teams are better than their ratings -- a situation that correlates with playoff experience.

And, of course, "team being better" is a much more plausible explanation for good performance than "team has more playoff experience."

------

Here's a possible counterargument. 

The study doesn't just look at players' *last year's* playoff experience -- it looks at their *career* playoff experience. You'd think that would dilute the effect, somewhat. But, still. Teams tend to stay good or bad for a few years before you could say their talent has changed significantly. Also, players with a lot of playoff experience, even with other teams, are more likely to be good players, and good players tend to play for more talented teams (even if all that makes the team "more talented" comes from them).

------

Another counterargument might be: well, the previous season's performance is already baked into Elo. If a team did well last season, it starts out with a higher rating than a team that didn't. So, checking again what a team did last season shouldn't make any difference. It would be like checking which team played better on Mondays. That shouldn't matter, because the Elo's conclusion that the teams are equal has already used the results of Monday games. 

That's a strong argument, and it would hold up if Elo did, in fact, give last season the appropriate consideration. But I don't think it does. When it calculates the rating, Elo gives previous seasons a very low weighting.

I did a little simulation (which I'll describe next post), and found that, when two NBA teams start with different Elo ratings, but perform identically, half the difference is wiped out after about 27 games.

So, team A starts out with a rating of 1500 (projection of 41-41). Team B starts out with 1600 (52-30). After 27 games playing identically against identical opponents, the Elo difference drops from 100 points to 50. 

After a second 27 games, the difference gets cut in half again, and the teams are now only 25 points apart. After a third 27 games, the difference cuts in half again, to around 12 points. That takes us to 81 games, roughly an NBA season. 

So, at the beginning of the season, Elo thought team B was 100 points better than team A. Now, because both teams had equal seasons, Elo thinks B is only 12 points better than A.

And that's even after considering personnel changes between seasons. If the two ratings started out 100 points apart, their performance last season was actually 133 points apart, because FiveThirtyEight regresses a quarter the way to the mean during the off-season, to account for player aging and roster changes. 

133 points is about 15 games out of 82. So, last year B was 56-26, while A was 41-41. They now have identical 49-33 seasons, and Elo thinks B is only 1.4 games better than A.

In other words, after a combined 162 games, the system thinks A was less than two games luckier than B.

That seems like too much convergence. 

------

Under the FiveThirtyEight system, previous years' performance contributes only 12 percent of the rating; this year's performance is the remaining 88 percent.  And that's *in addition* to adjusting for personnel changes off-season. 

That's far lower than traditional sabermetric standards. For baseball player performance, Bill James (and Tango, and others) have traditionally put previous seasons at 50 percent. They use a "1/2/3" weighting. By contrast, the NBA is using "1/5/44".

Of course, the "1/2/3" is for players; for teams, it should be lower, because of personnel changes. Especially in the NBA, where personnel changes make a much bigger difference because a superstar has such a big impact. 

But, still, 12 percent is far too little weight to give to previous NBA seasons. That's why, when you want to know whose Elo ratings are unlucky, you actually do add valuable information about talent, by checking which teams played much better the last few years than they did this year. 

And that's why playoff experience seems to matter. It correlates highly with teams that did well in the past. 

------

I could be wrong; it shouldn't be too hard to test. 

1. If this hypothesis is right, then playoff experience won't just predict playoff success; it will also predict late regular season success, because the same logic holds. That might be tricky to test because some teams might not give their stars as many minutes in those April games. But you could still check using teams that are fighting for a playoff spot.

2. Instead of breaking Elo ties by looking at previous playoff experience, look instead at the teams' start-of-season rating. I bet you find an even larger effect. And I bet that after you adjust for that, the apparent effect of playoff experience will be much smaller.

3. For teams whose Elo ratings at the end of the season are close to their ratings at the beginning of the season, I predict that the apparent effect of playoff experience will be much smaller. That's because those teams will tend to have been less lucky or unlucky, so you won't need to look as hard at previous performance (playoff experience) to counterbalance the luck.

4. Instead of using Elo as an estimate of team talent entering the playoffs, estimate talent from Vegas odds in April regular-season games (or late-March games, if you're worried about teams who bench their stars in April). I predict you'll find much less of a "playoff experience" effect.



Hat Tip and thanks: GuyM, for link and e-mail discussion

log5 estimates are biased when we use the wrong measure of "talent"

The "log5" method tries to predict a team's chance of winning a game based on its talent and that of its opponent. The basic formula, for teams A and B, is  

P = (A - AB)/(A+B-2AB)

A few months ago, I wrote that there's no theoretical reason for the formula to always work. In fact, there's an obvious counterexample where it doesn't work. Consider "height baseball," where the taller team always wins. Suppose team A is .700, because it's taller than 70 percent of its opponents, while team B is .400, being taller than only 40 percent of its opponents. The formula predicts team A will win 77.8 percent of games against B, but, of course, it will win 100 percent.

So why doesn't log5 work? I think I've found one reason, which I'll explain in this post. 

(There's a second reason -- which is actually a first reason, since it came back in 2011. In a blog post, Tango showed another example, using sprinter times, of how the odds ratio method (on which log5 is based) doesn't work, and Kincaid explained why in the comments. When I started writing this post, I originally thought mine was the same argument, just explained differently. But it's not. My argument actually doesn't apply to Tango's example ... I'll try to explain the Tango/Kincaid logic in a future post.)

------

Suppose you have team A, an .800 talent, playing team D, a .500 talent. What is the probability team A wins?

It seems that the answer should be ... well, .800. If team A is .800 against the league, which averages .500, then you'd think it should be .800 against a bona fide average team. And, the log5 method confirms the inutition -- plug in the numbers, and you do, indeed, get .800.

But that can't be right. I think it has to be the case that the .800 team plays *better* than .800 against the league-average team, and that it's easy to see why without any fancy math.

It actually doesn't depend on any technicality about what it actually means to be an .800 team. 

For instance, it's not because, if a team is .800 against the rest of the league, it must be *worse* than .800 in general, since it doesn't have to play itself. Even if you fix that problem, team A will have to be better than .800 against team D.

It's not because of home/road issues either, or the difference between observed .800 and talent .800 ... adjust for those, and the result still holds.

Let me restate the question in more detail, to try to eliminate some of those technicalities: 

----

In a league with no home field advantage, there are seven teams, A through G. 

If team A played a balanced schedule against all of them -- including itself (or a clone of itself) -- you would expect it to finish with an .800 record. So, in that respect, team A "has .800 talent".

By the same definition, teams B through G, respectively, have .700 talent, .600, .500 ... all the way down to .200 talent.

When team A (.800) plays team D (.500), what's the probability A wins?

----

The answer: not .800.

----

Let's create a little spreadsheet of team A's performance against all seven teams. It looks like this:

 matchup         probability
---------------------------
.800 vs .800
.800 vs .700
.800 vs .600
.800 vs .500
.800 vs .400
.800 vs .300
.800 vs .200

Now, let's fill in the log5 estimate for each one of those matchups:

 matchup           log5
---------------------------
.800 vs .800       .500
.800 vs .700       .727
.800 vs .600       .631
.800 vs .500       .800
.800 vs .400       .857
.800 vs .300       .903
.800 vs .200       .941

Those look quite reasonable, except that ... they don't average out to .800! They average out only to .766.

 matchup           log5
---------------------------
.800 vs .800       .500
.800 vs .700       .631
.800 vs .600       .727
.800 vs .500       .800
.800 vs .400       .857
.800 vs .300       .903
.800 vs .200       .941
---------------------------
 Average           .766

There's no trick here. This is a real, valid counterexample, one that shows that log5 doesn't actually work. And there's nothing special about our choice of .800. The average would always wind up too low, except for a team that's exactly .500.

Suppose we abandon the log5 estimates, then, and just try to fill in probabilities that seem reasonable. Can we do that, while insisting that the middle number stay .800? 

We have to hold the first number at .500, since, when a team plays a clone of itself, it must win 50 percent of its games, by definition. So we start with a chart that looks like this:

 matchup       probability
---------------------------
.800 vs .800        .500
.800 vs .700
.800 vs .600
.800 vs .500        .800
.800 vs .400
.800 vs .300
.800 vs .200
---------------------------
 overall avg        .800

From here, how do we fill in the second and third lines? One obvious way, that seems not too unreasonable, is just to stick in ".600" and ".700". 

 matchup       probability
---------------------------
.800 vs .800        .500
.800 vs .700        .600
.800 vs .600        .700
.800 vs .500        .800
.800 vs .400
.800 vs .300
.800 vs .200
---------------------------
 overall avg        .800

Having done that, it seems reasonable to just continue the pattern:

 matchup       probability
---------------------------
.800 vs .800        .500
.800 vs .700        .600
.800 vs .600        .700
.800 vs .500        .800
.800 vs .400        .900
.800 vs .300       1.000 
.800 vs .200       1.100
---------------------------
 overall avg        .800

That does, indeed, keep the average at .800. But it's obviously wrong -- it makes no sense to estimate that team A beats the .200 team 110% of the time.

So, is there another way we can fill this in, while keeping the .500 and .800 estimates, so that it all makes sense? No, I don't think that's possible. 

Right now, the first three lines of the chart average .600, which is .200 points below the .800 average we're shooting for. Therefore, the bottom three lines must average .200 points *above* .800. In other words, the bottom three lines have to average 1.000! Clearly, that can't be done.

So, we have to decrease the second and third lines. Maybe we change them to, say, .650 and .750. If we do that, then the first three lines average only .167 points below .800. Now, the bottom has to average "only" .967. Which, again, doesn't pass the sniff test.

Try if you want, but I'm pretty sure that you're not going to find anything that seems like a plausible breakdown. The only way to get something that looks reasonable, I think, requires the middle line to be something higher than .800.

--------

How much higher? From the original log5 chart, we see that

(a) team A was .766 overall, but
(b) team A played .800 ball against the .500 team.

If a .766 team goes .800 against an average team, maybe we can extrapolate that an .800 team would go, say, .840 against an average team.

Plugging .840 into the middle slot, and filling in the rest in some plausible fashion to average .800, maybe the chart would look something like this:

 matchup       probability
---------------------------
.800 vs .800        .500
.800 vs .700        .690
.800 vs .600        .770
.800 vs .500        .840
.800 vs .400        .900
.800 vs .300        .940
.800 vs .200        .960
---------------------------
 overall avg        .800

That's just a guess, of course. But, no matter what the true values are, the point remains: the middle entry must be significantly higher than .800.

And, another consequence: all the outcome probabilities, other than between equal teams, are *more extreme* than log5 suggests. The log5 formula is too conservative, always underestimating the favorite's chances of winning, when there is a favorite.

--------

So, log5 doesn't actually work. But, I think, there's an easy way to tweak it so that it DOES work. 

And that is: instead of using the log5 formula with the respective teams' expected talent against the league, we use their expected record against a .500 team. 

In our league, a team that finished .800 overall beats an average team 84% of the time, not 80%. Which means, for this new definition of log5, it's not an .800 talent, it's an .840 talent.

Let's reserve the word "talent" for its usual meaning, the expected record against the league, and use the made-up word "5talent" to mean talent against a .500 team. In our seven-team league, a team with a talent of .800 has a "5talent" of .840.

What's the 5talent of the rest of the teams? We can guess. If an .800 talent is an .840 5talent, maybe a .700 team is .720, a .600 team is .610, and so on. 

Repeating the log5 calculation using 5talent instead of talent, we get:

5talent matchup     log5
------------------------------
.840 vs .840        .500
.840 vs .720        .671
.840 vs .610        .771
.840 vs .500        .840
.840 vs .390        .891
.840 vs .280        .931
.840 vs .160        .965
------------------------------
 Overall avg        .796

Not bad! Under our estimates, a team that's an .840 5talent works out to a .796 talent. You could easily tweak the assumptions to get the average to .800 exactly, if you wanted to.

--------

Why does this happen, that log5 doesn't work if you use league performance? 

Because the win probabilities are based on the odds ratio. 

The log5 method works like this: suppose you have an .800 team against a .400 team. The .800 team has average 4:1 odds of winning. The .400 team has average 2:3 odds of winning. Divide 4/1 by 2/3, and you get 6/1. So the .800 team has 6:1 odds of beating the .400 team. That works out to an .857 winning percentage.

(I used odds ratios instead of the "usual" log5 formula, but it's exactly the same thing. If you do some algebra on the odds ratio calculation, you can actually derive the log5 formula at the top of this post.

When I calculate log5 probabilities, I actually use the odds ratio method, because the method is easy to remember, and I don't have to memorize the formula.)

The log5 formula is based on *multiplication* of *odds ratios*. But a team's overall average record, the one we normally talk about, is based on *addition* of *probabilities*. Those are two different sets of two different things. 

We calculated the bottom line of the chart, the overall winning percentage, as the arithmetic mean of the win probabilities. But, the odds ratio method doesn't know about arithmetic means and win probabilities. It knows only about geometric means of odds ratios. 

And, as it turns out, if we calculate the average as the geometric mean of the odds ratios ... well, then everything works! The geometric mean of the odds ratios against all teams is the same as the odds ratio against the average team.

Going back to the chart, and going back to the .800 talent, I'll convert the probabilities to odds ratios. (The odds ratio is the probability of winning divided by the probability of losing.)

talent matchup     log5     odds ratio
------------------------------------------
.800 vs .800       .500        1.00
.800 vs .700       .631        1.71
.800 vs .600       .727        2.67
.800 vs .500       .800        4.00
.800 vs .400       .857        6.00
.800 vs .300       .903        9.33
.800 vs .200       .941       16.00
------------------------------------------
 arithmetic mean   .766        
  geometric mean               4.00 (.800)

     
The .766 arithmetic mean doesn't equal the .800 talent, but the 4.00 geometric mean of the odds ratios *does* equal the 4.00 odds ratio talent. 

In other words, a team that's a 4.00 odds ratio talent against the league overall is also a 4.00 talent against the league average team. 

-------

OK, I cheated a bit. The reason the geometric mean works out perfectly is that, in our league, the team talents are symmetrically distributed around .500. 

If the talents aren't symmetrical, it doesn't work out perfectly -- I found the geometric mean comes out a little too high. However: it's a lot closer than the arithmetic mean works out to be. And, most leagues are symmetrical enough that it wouldn't be an issue in real life. There aren't many leagues with, say, 20 teams with .480 talent, but one team at .900. 

(Also... I wonder if you use the 5talents in the chart instead of the talents, if the geometric mean might not work out perfectly in that case, even for non-symmetrical leagues. That's just a gut feeling, and I need to think about it more.)

-------

I think that what makes the arithmetic mean give such a serious underestimate is that our league has large extremes of team talent, spanning a range from .200 to .800. The farther the numbers are from .500, the more the geometric mean of the odds ratio differs from the arithmetic mean of the probabilities. 

One measure of "extremes" is the standard deviation. In our hypothetical seven-team league, the SD of talent is .200. In actual Major League Baseball, the SD of talent is only around .055.

So, let's simulate the MLB spread with an example where, instead of teams varying from .800 to .200, they only vary from .565 to .435. Here's the log5 chart:

 talent matchup    log5     odds ratio
------------------------------------------
.565 vs .565       .500      1
.565 vs .500       .565      1.2989
.565 vs .435       .628      1.6870
------------------------------------------
arithmetic mean    .564
 geometric mean              1.2989 (.565)

This league has a talent SD of .053, similar to MLB. And, with that smaller SD, log5 is a very good fit. Now, a team with a 5talent of .565 has a talent of .564 -- so close that it makes no material difference. 

It looks like the log5 method is very, very sensitive to the dispersion of talent in the league -- bumpng the SD from .053 to .200 -- only a factor of 4 -- made the log5 discrepancy jump from 1 point all the way to 40 points.

So that explains why, even though log5 produces conservatively biased estimates when we use the "usual" definition of talent, it nonetheless works so well for baseball.  It's because in MLB, the spread in talent is reasonably small.

------

We can repeat this for other sports. Ten years ago, Tom Tango found the SD of talent to be .134 in the NBA, and .143 in the NFL.

 SD of talent
-------------------------------
.200 hypothetical 7-team league
.143 NFL
.134 NBA
.055 MLB
.053 hypothetical 3-team league
-------------------------------

We know that log5 is pretty far off for the 7-team league, and pretty good for MLB. The two leagues in between -- the NFL and NBA -- are right in the middle. 

If we do a five-team league, with talents ranging from .700 to .300, the SD is .141, which is pretty close to the NBA and NFL. Here's the calculation for the .700 team:

talent matchup     log5     odds ratio
------------------------------------------
.700 vs .700       .500        1.00
.700 vs .600       .609        1.56
.700 vs .500       .700        2.33
.700 vs .400       .778        3.50
.700 vs .300       .845        5.44
------------------------------------------
 arithmetic mean   .686        
  geometric mean               2.33(.700)


We can estimate that in the NFL or NBA, a .686 talent has a "5talent" of .700, meaning that it'll beat a .500 team 70% of the time. 

That means that if you're estimating a .686 talent against a .500 talent, your estimate is going to be .014 too conservative. That's approximately what it'll be for any pair of teams about that far apart. It'll be more accurate for opponents closer in talent, and less accurate for talent mismatches, at least to the point of diminishing returns (if log5 says your team is .990, it's not possible that it really should be 1.005).

Does that bother you, that when you use log5 on teams with significantly different talent, your estimate is off by as much as .014? If it doesn't, just go ahead and keep using log5. 

But for any study that's looking for small effects ... well, to me, it seems to me that .014 points is probably as big as the effect you're looking for. If you don't correct for it, you're underestimating the favorite by, what, maybe a quarter as much as home field advantage?

If you do some kind of "record after a time zone change on a hot streak" study, and you find an effect of .014 points ... since hot streaks mean good teams, could it just be that all you did was rediscover that log5 is biased too conservatively when you use the wrong definition of talent?  



UPDATE, 8/25/16: Changed title of post and reworded a few sentences to emphasize that the "bias" in log5 is not intrinsic to the formula itself, but occurs when we improperly use "talent" instead of "5talent." Hat tip: Ted Turocy. See longer explanation here.  

Noll-Scully doesn't measure anything real

The most-used measure of competitive balance in sports is the "Noll-Scully" measure. To calculate it, you figure the standard deviation (SD) of the winning percentage of all the teams in the league. Then, you divide by what the SD would be if all teams were of equal talent, and the results were all due to luck.

The bigger the number, the less parity in the league.

For a typical, recent baseball season, you'll find the SD of team winning percentage is around .068 (that's 11 wins out of 162 games). By the binomial approximation to normal, the SD due to luck is .039 (6.4 out of 162). So, the Noll-Scully measure works out to .068/.039, which is around 1.74.

In other words: the spread of team winning percentage in baseball is 1.74 times as high as if every team were equal in talent.

------

Both "The Wages of Wins" (TWOW), and a paper on competitive balance I just read recently (which I hope to post about soon), independently use Noll-Scully to compare different sports. Well, not just them, but a lot of academic research on the subject.

The Wages of Wins (page 70 of the second edition) runs this chart:

2.84 NBA
1.81 AL (MLB)
1.67 NL (MLB)
1.71 NHL
1.48 NFL

The authors follow up by speculating on why the NBA's figure is so high, why the league is so unbalanced. They discuss their "short supply of tall people" hypothesis, as well as other issues.

But one thing they don't talk about is the length of the season. In fact, their book (and almost every other academic paper I've seen on the subject) claims that Noll-Scully controls for season length. 

Their logic goes something like this: (a) The Noll-Scully measure is actually a multiple of the theoretical SD of luck. (b) That theoretical SD *does* depend on season length. (c) Therefore, you're comparing the league to what it would be with the same season length, which means you're controlling for it.

But ... that's not right. Yes, dividing by the theoretical SD *does* control for season length, but not completely.

------

Let's go back to the MLB case. We had

.068 observed SD
.039 theoretical luck SD
-------------------------
1.74 Noll-Scully ratio

Using the fact that SDs follow a pythagorean relationship, it follows that

observed SD squared = theoretical luck SD squared + talent SD squared

So

.068 squared = .039 luck squared + talent squared

Solving, we find that the SD of talent = .056. Let's write that this way:

.039 theoretical luck SD
.056 talent SD
------------------------
.068 observed SD
---------------------------------------
1.74 Noll-Scully (.068 divided by .039)

Now, a hypothetical. Suppose MLB had decided to play a season four times as long: 648 games instead of 162. If that happened, the theoretical luck SD would drop in half (we'd divide by the square root of 4). So, the luck SD would be .020. 

The talent SD would remain constant at .056. The new observed SD would be the square root of (.020 squared plus .056 squared), which works out to .059:

.020 theoretical luck SD
.056 talent SD
-------------------------
.059 observed SD
---------------------------------------
2.95 Noll-Scully (.059 divided by .020)

Under this scenario, the Noll-Scully increases from 1.74 to 2.95. But nothing has changed about the game of baseball, or the short supply of home run hitters, or the relative stinginess of owners, or the populations of the cities where the teams play. All that changed was the season length.

--------

My only point here, for now, is that Noll-Scully does NOT properly control for season length. Any discussion of why one sport has a higher Noll-Scully than another *must* include a consideration of the length of the season. Generally, the longer the season, the higher the Noll-Scully. (Try a Noll-Scully calculation early in the season, like today, and you'll get a very low number. That's because after only 15 games, luck is huge, so talent is small compared to luck.)

It's not like there's no alternative. We just showed one! Instead of Noll-Scully, why not just calculate the "talent SD" as above? That estimate *is* constant for season length, and it's still a measure of what academic authors are looking for. 

Tango did this in a famous post in 2006. He got

.060 MLB
.058 NHL
.134 NBA

If you repeat Tango's logic for different season lengths, you'll get the same numbers.  Well, you'll get different results because of random variation ... but they should average somewhere close to those figures.

---------

Now, you could argue ... well, sometimes you *do* want to control for season length. Perhaps one of the reasons the best teams dominate the standings is because NBA management wanted it that way ... so they chose a longer, 82-game season, in order to create some space between the Warriors and the other teams. Furthermore, maybe the NFL deliberately chose 16 games partly to give the weaker teams a chance.

Sure, that's fine. But you don't want to use Noll-Scully there either, because Noll-Scully still *partially* adjusts for season length, by using "luck multiple" as its unit. Either you want to consider season length, or you don't, right? Why would you only *partially* want to adjust for season length? And why that particular part?

If you want to consider season length, just use the actual SD of the standings. If you don't, then use the estimated SD of talent, from the pythagorean calculation. 

Either way, Noll-Scully doesn't measure anything anybody really wants.