Thursday, September 22, 2016

When and why log5 doesn't work

Six years ago, Tom Tango described a hypothetical sprinting tournament where log5 didn't seem to give accurate results. I think I have an understanding, finally, of why it doesn't work, thanks to 

(a) Ted Turocy, 

(b) a paper from Kristi Tsukida and Maya R. Gupta (.pdf; see section 3.4, keeping in mind that "Bradley-Terry model" basically means "log5"), and 

(c) this excellent post by John Cook. 

(You don't have to follow the links now; I'll give them again later, in context.)

-----

It turns out that the log5 formula makes a certain assumption about the sport, an assumption that makes the log5 formula work out perfectly. That assumption is: that the set of score differentials follows a logistic distribution

What's the logistic distribution? It's a lot like the normal distribution, a bell-shaped curve. They can be so similar in shape that I'd have trouble telling them apart by eye. But, the logistic distribution has fatter tails relative to the "bell". In other words: the logistic distribution predicts rare events, like teams beating expectations by large amounts, will happen more often than the normal distribution would predict. And it predicts that certain common events, like close games, will happen less often.

The log5 formula works perfectly when scores are distributed logistically. That's been proven mathematically. But, where scores aren't actually logistic, the formula will fail, in proportion to how real life varies from the particular logistic distribution the log5 formula assumes.

That's why the formula didn't work in the sprinting example. Tango explicitly made the results normally distributed. Then, he found that log5 started to break down when the competitors became seriously mismatched. That is: log5 started to fail in the tail of the distribution, exactly where normal and logistic differ the most. 

-------

Here's a rudimentary basketball game. Two teams each get 100 possessions, and have a certain probability (talent) of scoring on each possession. Defense doesn't matter, and all baskets are two points.

Suppose you have A, a 55 percent team (expected score: 110) and B, a 45 percent team (expected score, 90). We expect each team's score to be normally distributed, with an SD of an SD of almost exactly 10 points for each team's individual score.*

(* This is just the normal approximation to binomial. To get the SD, calculate .45 multiplied by (1-.45) divided by 100. Take the square root. Multiply by 2 since each basket is 2 points. Then, multiply by 100 for the number of possessions. You get 9.95.)

Since the two teams are independent, the SD of the score difference is the square root of 2 times as big as the individual standard deviations. So, the SD of the differential is 14 points.

By talent, A is a 20-point favorite over underdog B. For B to win, it must beat the spread by 20 points. 20 divided by 14 equals 1.42 SD. Under the normal distribution, the probability of getting a result greater than 1.42 SD is 0.0778. 

That means B has a 7.78 percent chance of winning, and A a 92.22 chance. The odds ratio for A is 92.22 divided by 7.78, which is 11.85.

So, that's the answer. Now, how well does log5 approximate it? To figure that out, we need to figure the talent of A and B against a .500 team.

Let's say Team C is that .500 team. Against C, team A has an advantage of 0.71 SD. The normal distribution table says that's a winning percentage of .7611, which is an odds ratio of 3.186. Similarly, team B is 0.71 SD worse than C, so it wins with probability (1 - .7611), or odds ratio 1 / 3.186.

Using log5, what's the estimated odds ratio of A over B? It's 3.186 squared, or 10.15. That works out to a win probability of only .910, an underestimate of the .922 true probability.

Log5 estimate:  .910, odds ratio 10.15
Correct answer: .922, odds ratio 11.85

To recap what we just did: we started with the correct, theoretically-derived probabilities of A beating C, and of B beating C. When we plugged those exact probabilities into log5, we got the wrong answer.

Why the wrong answer? Because of the log5 formula's implicit assumption: that the distribution of the score difference between A and B is logistic, rather than normal.

What specific logistic distribution does log5 expect? The one where the mean is the log of the odds ratio and the SD is about 1.8. (Logistic distributions are normally described by a "shape parameter, which is the SD divided by 1.8 (pi divided by the square root of 3, to be exact). So, actually, the log5 formula assumes a logistic distribution with a shape parameter of exactly 1.)

So, in this case, the log5 formula treats the score distribution as if it's logistic, with a mean of 2.3 (the log of the log5-calculated odds ratio of 10.15) and an SD of 1.8 (shape parameter 1). 

We can rescale that to be more intuitive, more basketball-like, and still get the same probability answer. We just multiply the mean, SD, and shape parameter by a constant. That's like taking a normal curve for height denominated in feet, and multiplying the mean and SD by 12 to convert to inches. The proportion of people under 5 feet under the first curve works out to the same as the proportion of people under 60 inches in the second curve.

In this case, we want the mean to be 20 (basketball points) instead of 2.3. So, we can multiply both the mean and the scale by 20/2.3. That gives us a new logistic distribution with mean 20 and SD 15.6.

That's reasonably close to the actual normal curve we know is correct:

Normal:   mean 20, SD 14
Logistic: mean 20, SD 15.6

It's reasonably close, but still incorrect. In this case, log5 overestimates the underdog in two different ways. First, it assumes a logistic distribution instead of normal, which means fatter tails. Second, it assumes a higher SD than actually occurs, which again means fatter tails and more extreme performances.

------

Here, I'll give you some stolen visuals.  I'm swiping this diagram from that great post John Cook post I mentioned, which compares and contrasts the two distributions. Here's a comparison of a normal and logistic distribution with the same SD:





The dotted red line is the normal distribution, the solid blue line is the logistic, and both have an SD of about 1.8.  It looks like the logistic tails start getting fatter than the normal tails at around 4.3 or something, which is around two-and-a-half SD from the mean.

But, for our purposes, it's the area under the curve that we care about, the CDF. Here's that comparison, shamefully stolen from the Tsukida/Gupta .pdf I linked to earlier:




From here, you can see that from infinity to around, I dunno, maybe 1.7 or so, the area under the logistic curve is larger than the area under the normal curve.

And again, this is when the curves have equal SDs.  In this basketball example, the log5 assumption has a higher SD than the actual normal, by 15.6 points to 14. So the overestimate of the underdog is even higher, and probably starts earlier than 1.7 SD.

------

Just to make sure my logic was correct, I ran a simulation for this exact basketball game. I created seven teams, with expectations running from 110 points down to 90 points. I ran a huge season, where each team played each other team 200,000 times. 

Then, I observed the simulated record of champion team A (the 110 point team) against team C (the team of average talent, the 100-point team). And I observed the simulated record of basement team B (the 90 point team) against team C. 

Here's what I got:

A versus C: 151878- 48122 (.7594), odds ratio 3.156
B versus C:  47599-152401 (.2380), odds ratio 0.312

To estimate A versus B using log5, we just divide 3.156 by 0.312:

A versus B: 181990-18010 (.9099), odds ratio 10.105

But, the actual results from the simulated season were:

A versus B: 184395-15605 (.9220), odds ratio 11.816

In the simulation, log5 understimated the favorite almost exactly the same way as it did in theory:

Simulation
--------------------------------------
Log5 estimate:  .909, odds ratio 10.11
Correct answer: .922, odds ratio 11.82

Theory
--------------------------------------
Log5 estimate:  .910, odds ratio 10.15
Correct answer: .922, odds ratio 11.85

-------

So that's what I think is going on. If the distribution of the difference in team scores doesn't follow the logistic distribution well enough, you'll get log5 working poorly for those talent differentials for which the curves match the worst. In the case where score differentials are normal, like this basketball simulation, the worst match is when you have a heavy favorite.

For other sports, it's an empirical question. Take the actual score distribution for that particular sport, and fit it to the scaled logistic distribution assumed by log5. Where the curves differ, I suspect, will tell you where log5 projections will be least accurate, and in which direction.



Labels: , , ,

Wednesday, September 14, 2016

Another case where log5 works perfectly

Here's another case where log5 works perfectly: sudden death foul shooting.

Two players each take a foul shot. The player who makes his shot wins, if the other misses. If both players sink the shot, or both players miss, they each take another one. This continues until there's a winner.

Suppose player A is an 80% shooter, and player B is a 40% shooter. The chance that A makes and B misses is .800 multiplied by (1 - .400), which works out to .480. The chance that B makes and A misses is (1 - .800) multiplied by .400, which is .080. 

So, A beats B 480 times for each 80 times that A loses to B. So, A's odds of winning are 6:1.

And that's exactly what log5 predicts. A's make ratio is 4:1, and B's make ratio is 2:3. Divide 4/1 by 2/3 and you get 6/1, which is the right answer.

------

Actually, that's a bit of a cheat. The log5 formula isn't based on chances of making a foul shot; it's based on the chance of beating a .500 player.

We can fix that problem. Consider player C, who happens to hit exactly 50% of foul shots. I won't do the calculaton again, but you can easily figure out that player A beats player C 80% of the time, but player B beats player C only 40% of the time.

So, player A is an .800 player against a .500 player, and player B is a .400 player against a .500 player. 

We're not quite done yet. Player C was defined as one who hits 50% of foul shots, not one that wins 50% of games. They're not the same thing. We can fix that by just assuming the league average player is both a .500 player and a 50% shooter. That seems arbitrary, but I'm just trying to come up with an example of when log5 works, so arbitrary is fine.

But, actually, we don't need that assumption. 

I've been saying all along that for talent, you need to use the expected odds ratio against a .500 team. But, actually, you don't need to be that specific. You can actually use the expected odds ratio against *any* other team (as long as it's the same other team for both sides).

So, it doesn't matter if player C is a .500 player, or a .600 player, or a .979 player. If you know A beats him X% of the time, and B beats him Y% of the time, you can just use X and Y in the log5 formula and it'll still work.

(Why does that work? Because the odds ratio against any given player is always a fixed multiple of the odds ratio against any other given player. So it doesn't matter whether you calculate A's odds ratio over B as a/b, or xa/xb -- it comes out the same regardless of x.)

That means that the log5 calculation using .800 (A's record against C) and .400 (B's record against C) is valid. The log5 formula works perfectly for sudden-death foul shooting.

------

What if we change the game a little, by extending it to two tries instead of one? This time, each player takes two shots, and whoever makes more wins the game. (Again, if it's a tie, repeat the game with two more shots each.)

If I've done my calculations right ... log5 does NOT work for this new game.

Assuming shots are independent, these are the probabilities of the three players hitting 2, 1, and 0 shots:

            0-for-2  1-for-2  2-for-2
---------------------------------------
A (.800)      .04      .32      .64
B (.400)      .36      .48      .16
C (.500)      .25      .50      .25

From that, we can calculate the exact probability that A beats B on the first two shots. It works out to 65.28%:

A wins 2-0   .64 * .36 = .2304
A wins 2-1   .64 * .48 = .3072
A wins 1-0   .32 * .36 = .1152
------------------------------
Total                    .6528

The chance that B beats A works out to 7.68 percent:

B wins 2-0   .16 * .04 = .0064
B wins 2-1   .16 * .32 = .0512
B wins 1-0   .48 * .04 = .0192
------------------------------
Total                    .0768

For every 6,528 games that A wins, B wins only 768 games. That's an odds ratio of exactly 8.5:1.

Now, bring C into the picture. I won't repeat all the calculations, but instead of eight-and-a-half, log5 gives an estimate of eight-and-three-elevenths:

Odds ratio of A over C: 56:11
Odds ratio of B over C: 24:39
-------------------------------------------
Odds ratio of A over B: 2184:264 = 8.2727:1

So, in this case, log5 fails:

log5 estimate of A over B: 8.2727:1 
Correct odds  of A over B: 8.5000:1 

The log5 estimate works out to a winning percentage of .892. The correct win probability is .895. It's close, but it's still wrong.

------

So, why doesn't log5 work here, and in Tango's case? 

Because: it's a known result (hat tip: Ted Turocy) that for log5 to work, scores have to follow a certain, specific distribution. In most sports, they don't. How well log5 works depends on how well the real-life distribution of scores follows the assumed, theoretical distribution.

I'll get to that (finally) next post.



(Previous log5 posts: One, and Two)

(Updated 9/15 to remove incorrect reference to "height baseball.")

Labels: ,

Saturday, September 03, 2016

A case where log5 works perfectly

Suppose there's a coin-flipping league where every team has the same talent. After the season is over, you notice one team in the standings at .800, and another is at .400. Those records include the two teams facing each other at least once.

What is the probability, in retrospect, that the .800 team beat the .400 team in a particular game where they met? The log5 formula says you figure it out like this:

.800 is a ratio of 4 wins per loss
.400 is a ratio of 2/3 wins per loss

4 divided by 2/3 equals 6 

6 wins per loss is 6 wins per 7 games, which is .857.

(You can use the traditional form of the log5 formula if you want, to get the same .857.)

And it turns out that, in this case, the log5 formula DOES work. It works perfectly. The probability is indeed .857, and you can prove that.

I'll work out this particular example. Call the two teams A (.800) and B (.400). Suppose there were only 5 games in the season, so that A went 4-1 and B went 2-3. 

Suppose the two teams only met one time. What's the chance A won that game?

Start with the the case where A beat B. If that happened, A would have to have gone 3-1 in its other four games, and B would have to have gone 2-2. 

There are four permutations where A goes 3-1 (WWWL, WWLW, WLWW, LWWW), and six ways for B to go 2-2 (WWLL, WLWL, WLLW, LWWL, LWLW, LLWW).

That means there are 24 (6 x 4) ways to draw up the season when A beats B.

Now, suppose that B beat A. That means A went 4-0 otherwise, and B went 1-3. 

There is only one way for A to go unbeaten (WWWW), and only four ways to arrange B's 1-3 (WLLL, LWLL, LLWL, LLLW). 

That means that there are 4 (1 x 4) ways to draw up the season when B beats A.

Since this is coin flipping, all the cases have equal probability of happening. So, A beats B 24 times for every 4 times that B beats A. 

That's a ratio of 6:1, which is 6/7, which is .857 -- exactly as log5 predicts.

-------

It's not that hard to go from this example to a proof. Just replace the raw numbers by variables for number of games total (n), number of games A wins (a), and number of games B wins (b). When you count permutations, you'll wind up with factorial terms, and when you divide the A permutations by the B permutations, the factorials will cancel out, and you'll be left with

p = (a/(n-a)) / (b/(n-b))

Which is exactly the log5 formula.

I don't know much about the history of log5, but some of you do. Was this part of the genesis of log5, that it could be proven to work retrospectively, so when it seemed to work pretty decently as a forecast, it became the standard?

-------

But wait a minute -- last month, I argued that log5 couldn't possibly work when you used season records. If you recall, I posted this chart:

 matchup           log5
---------------------------
.800 vs .800       .500
.800 vs .700       .631
.800 vs .600       .727
.800 vs .500       .800
.800 vs .400       .857
.800 vs .300       .903
.800 vs .200       .941
---------------------------
 Average           .766

This says that an .800 team, playing against the league as log5 would predict, would actually play at a .766 pace. That's a contradiction -- it should be .800 -- so log5 must be wrong!

One difference between then and now is that, before, we had the .800 team playing against a clone of itself. That's not true here, so let's redo the chart without the first line: 

 matchup           log5
---------------------------
.800 vs .700       .631
.800 vs .600       .727
.800 vs .500       .800
.800 vs .400       .857
.800 vs .300       .903
.800 vs .200       .941
---------------------------
 Average           .810

Well, the average still isn't .800, so we still have a problem.

So, what's going on? Is my logic wrong here, or is my logic wrong there? Does log5 work, or doesn't it?

This bugged me for a while, until I sorted it out in my head. I think both conclusions are correct. The log5 formula actually *does* work in this case, and it actually *does not* work in the other case, for exactly the reasons described. 

But what about these charts that show the contradiction? They apply there, but they don't apply here. 

The difference is: when .800 is the *talent* of the team, it's constant, and you can use it on every line of the chart. But, when you use the *retrospectively observed performance*, it changes with every game. So you can't use .800 in every line of the chart.

Suppose the (eventual) 4-1 team wins the first game. In that case, it's only an (eventual) 3-1 team after that. That means its retrospectively observed performance next game isn't .800, it's .750. That means you have to draw up the chart like this:

  matchup           log5
---------------------------
 .800 vs .700       .631
 .750 vs .600       .667
 ...

If it loses the first game, it's 1.000 after, and you draw up the chart like this:

  matchup           log5
---------------------------
 .800 vs .700       .631
1.000 vs .600      1.000
...

So the chart has to be different every time, based on what actually happens in the games. 

I believe that if you were to do every possible permutation of the season, weighted by log5 probability, and average the averages, you would indeed wind up with .800. 

-------

Now, the proof that validates the retrospective use of log5 only works because we assumed that games are decided by coin flips. If that weren't the case, then all the permutations wouldn't prospectively have an equal chance of happening, and the logic would fall apart. 

But would the *result* still hold? If you don't know A's talent or B's talent, but they still go 4-1 and 2-3, respectively, does the 6 out of 7 still hold?

I don't think it does. Again, imagine "height baseball," where the taller team always wins. It could be that A is the second-tallest team out of 6, and B is the fourth-tallest. That would be consistent with the 4-1 and 2-3 records (imagine a round-robin season).  But A would have a 100% chance of beating B, not 85.7%. 

So this is a special case. Whether log5 works here because there's something special about the 50%, or whether it's because all teams are the same, or whether it's just that the average record against all teams happens to equal the record against the average team ... I don't know.

But still. To me, there are no coincidences in math, just relationships that look coincidental until we see the connection. Maybe when I understand log5 better, it'll be self-evident why it works here. 

As I said, some of you guys reading this are much more familiar with the intricacies of log5 than I am. Is this a known result? Am I reinventing a wheel?



Labels: ,