Finding a true talent level for an outcome distribution

What is the distribution of team talent in major league baseball? For instance, how many teams are good enough to actually win 100 games in a season?

You can’t just go by actual won-loss records, because any team that actually wins 100 games has probably done so aided by a bit of luck. To oversimplify, a team that wins 100 games might be a 95-game team that got lucky, or a 105-game team that got unlucky. There are many, many more 95-game teams than 105-game teams, and so the average team that wins 100 games is probably closer to 95 games than to 105 games.

In general, the distribution of talent is much narrower than the distribution of actual results. One way to see this is to consider the extreme case – suppose every team had the same (.500) talent, as if every game outcome was determined by flipping a coin. In that case, the talent distribution is the narrowest it can possibly be, but the season outcomes are normally distributed with standard deviation about 6.

So, how can we determine the talent distribution, given that we only know the outcome distribution? Tangotiger has a method. He points out that

var(outcome) = var(talent + luck) = var(talent) + var(luck) + 2 cov(talent, luck)

Since luck is random, it doesn’t correlate to talent, and the covariance term is zero. So

var(outcome) = var(talent) + var(luck)

We can observe var(outcome) from actual W-L records, and we can figure out var(luck) from the binomial distribution. And so we can easily figure out var(talent). Here’s a quote from Tom’s post on the method in more detail. He’s figuring out var(talent) for the NFL:

Here is one way to figure out the var(true) for any league.

Step 1 - Take a sufficiently large number of teams (preferably all with the same number of games).

Step 2 - Figure out each team’s winning percentage.

Step 3 - Figure out the standard deviation of that winning percentage. I just did it quick, and I took the last few years in the NFL, and the SD is .19, which makes var(observed) = .19^2

Step 4 - Figure out the random standard deviation. That’s easy: sqrt(.5*.5/16) 16 is the number of games for each team.

So, var(random) = .125^2

Solve for: var(obs) = var(true) + var(rand)

var(true), in this case, is .143^2

So the standard deviation of talent in the NFL is .143.

Tango tells us the SD of talent in MLB is about 0.060. If we assume that MLB teams are normally distributed with mean .500 and SD .060, then 99.5 wins (.614) is 1.90 standard deviations from the mean.

Looking up 1.90 in a cumulative normal distribution table tells us that 2.9% of teams have the talent to win 100 games or more. That’s 1 in 34, or about one team per season.

But in any case, the point of this post is not the specific value, but rather, Tangotiger’s method. It’s simple, it’s easy to calculate, it’s theoretically sound, and it’s extremely useful in all kinds of situations.