How much has parity increased in MLB?

The MLB standings were tighter than usual in 2014. No team won, or lost, more than 98 games. That's only happened a couple of times in baseball history.

You can measure the spread in the standings by calculating the standard deviation (SD) of team wins. Normally, it's around 11. Two years ago, it was 12.0. Last year, it was down substantially, to 9.4.

Historically, 9.4 is not an unprecedented low. In 1984, the SD was 9.0; that's the most parity of any season since the sixties. More recently, the 2007 season came in at 9.3, with a team range of 96 wins to 96 losses.

But this time, people are noticing. A couple of weeks ago, Ben Lindbergh showed that this year's preseason forecasts have been more conservative than usual, suggesting that the pundits think last year's compressed standings reflect increased parity of talent. They've also noted another anomaly: in 2014, payroll seemed to be less important than usual in predicting team success. These days, the rich teams don't seem to be spending as much, and the others seem to be spending a little more.

So, have we actually entered into an era of higher parity, where we should learn to expect tighter playoff races, more competitive balance, and fewer 100-win and 100-loss teams? 

My best guess is ... maybe just a little bit. I don't think the instant, single-season drop from 12 games to 9.4 games could possibly be the result of real changes. I think it was mostly luck. 

-----

Here's the usual statistical theory. You can break down the observed spread in the standings into talent and luck, like this: 

SD(observed)^2 = SD(talent)^2 + SD(luck)^2

Statistical theory tells us that SD(luck) equals 6.4 games, for a 162-game season. With SD(observed) equal to 12.0 for 2013, and 9.4 for 2014, we can solve the equation twice, and get

2013: 10.2 games estimated SD of talent 
2014:  7.0 games estimated SD of talent

That's a huge one-season drop, from 10.2 to 7.0 ... too big, I think, to really happen in a single offseason. 

Being generous, suppose that between the 2013 and 2014 seasons, teams changed a third of their personnel. That's a very large amount turnover. Would even that be enough to cause the drop?

Nope. At least, not if that one-third of redistributed "talent wealth" was spread equally among teams. In that case, the SD of the "new" one-third of talent would be zero. But the SD of the remaining two-thirds of team talent would be 8.3 (the 2013 figure of 10.2, multiplied by the square root of 2/3).

That 8.3 is still higher than our 7.0 estimate for 2014! So, for the SD of talent to drop that much, we'd need the one-third of talent to be redistributed, not equally, but preferentially to the bad teams. 

Is that plausible? To how large an extent would that need to happen?

We have a situation like this: 

2014 talent = original two thirds of 2013 talent 
            + new one third of 2013 talent 
            + redistribution favoring the worse teams

Statistical theory says the relationship between the SDs is this:

SD(2014 talent) squared = 

SD(2013 talent teams kept)^2 +
SD(2013 talent teams gained)^2 + 
SD(2013 talent teams kept) * SD(2013 talent teams gained) * correlation between kept and gained * 2

It's the same equation as before, but with an extra term (shown in bold). That term shows up because we're assuming a non-zero correlation between talent kept and talent gained -- that the more "talent kept," the less your "talent gained". When we just did "talent" and "luck", we were assuming there was no correlation, so we didn't need that extra part. (We could have left it in, but it would have worked out to zero anyway.)

The equation is easy to fill in. We saw that SD(2014 talent) was estimated at 9.4. We saw that SD(talent teams kept) was 8.3. And we can estimate that SD(talent teams gained) is 12.0 times the square root of 1/3, which is 5.9.

If you solve, you get 

Correlation between kept and gained = -0.57

That's a very strong correlation we need, in order for this to work out. The -0.57 means that, on average, if a team's "kept" players were, say, 5th best in MLB (that is, 10 teams above average), its "gained" players must have been 9th worst in MLB (5.7 teams below average). 

That's not just the good teams getting worse by signing players that aren't as good as the above-average players they lost -- it's the good teams signing players that are legitimately mediocre. And, vice-versa. At -0.56, the worst teams in baseball would have had to have replaced one-third of their lineup, and those new players would have to have been collectively as good as those typically found on a 90-win team.

Did that actually happen? It's possible ... but it's something that you'd easily be able to have seen at the time. I think we can say that nobody noticed -- going into last season, it didn't seem like any of the mainstream projections were more conservative than normal. (Well, with the exception of FanGraphs. Maybe they saw some of this actually happen? Or maybe they just used a conservative methodology.)

One thing that WAS noticed before 2014 is that the 51-111 Houston Astros had improved substantially. So that's at least something.

And, for what it's worth: the probability of randomly getting a correlation coefficient as extreme as 0.57, in either direction, is 0.001 -- that is, one in a thousand. On that basis, I think we can reject the hypothesis that team talent grew tighter just randomly.

(Technical note: all these calculations have assumed that every team lost *exactly* one-third of its talent, and that those one-thirds were kept whole and distributed to other teams. If you were to use more realistic assumptions, the chances would improve a little bit. I'm not going to bother, because, as we'll see, there are other possibilities that are more likely anyway.)

------

So, if it isn't the case that the spread in talent narrowed ... what else could it be? 

Here's one possibility: instead of SD(talent) dropping in 2014, SD(luck) dropped. We were holding binomial luck constant at 6.4 games, but that's just the average. It varies randomly from season to season, perhaps substantially. 

It's even possible -- though only infinitesimally likely -- that, in 2014, every team played exactly to its talent, and SD(luck) was actually zero!

Except that ... again, that wouldn't have been enough. Even with zero luck, and talent 10.3, we would have observed an SD of 10.3. But we didn't. We observed only 9.4. 

So, maybe we have another "poor get richer" story, where, in 2014, the bad teams happened to have good luck, and the good teams happened to have bad luck.

We can check that, in part, by looking at the 2014 Pythagorean projections. Did the bad teams beat Pythagoras more than the good teams did?

Not really. Well, there is one obvious case -- Oakland. The 2014 A's were the best team in MLB in run differential, but won only 88 games instead of 99 because of 11 games of bad "cluster luck". 

Eleven games is unusually large. But, the rest of the top half of MLB had a combined eighteen games of *good* luck, which seems like it would roughly cancel things out.

Still ... Pythagoras is only a part of overall luck, so there's still lots of opportunity for the "good teams having bad luck" to have manifested itself. 

-----

Let's do what we did before, and see what the correlation would have to be between talent and luck, in order to get the SD down to 9.4. The relationship, again, is:

SD(2014 standings)^2 = 

SD(2014 talent)^2 + 
SD(2014 luck)^2   + 
SD(2014 talent) * SD(2014 luck) * correlation between 2014 talent and 2014 luck * 2 

Leaving SD(2014 talent) at the 2013 estimate of 10.2, and leaving SD(2014 luck) at 6.4, we get

Correlation between 2014 talent and luck = -0.39

The chance of a correlation that big (either direction) happening just by random luck is 0.033 -- about 1 in 30. That seems like a big enough chance that it's plausible that's what actually happened. 

Sure, 1 in 30 seems low, and is statistically significantly unlikely in the classical "1 in 20" sense. But that doesn't matter. We're being Bayesian here. We know something unlikely happened, and so the reason it happened is probably also something unlikely. And the 1/30 estimate for "bad teams randomly got lucky" is a lot more plausible than the 1/1000 estimate for "bad teams randomly got good talent."  It might also be more plausible than "bad teams deliberately got good talent," considering that nobody noticed any unusual changes in talent at the time.

------

Having got this far, I have to backtrack and point out that these odds and correlations are actually too extreme. We've been assuming that all the unusualness happened after 2013 -- either in the offseason, or in the 2014 season. But 2013 might have also been lucky/unlucky itself, in the opposite direction.

Actually, it probably was. As I said, the historical SD of actual team wins is around 11. In the 2013 season, it was 12. We would have done better by comparing the "too equal" 2014 to the historical norm, rather than to a "too unequal" 2013. 

For instance, we toyed with the idea that there was less luck than normal in 2014. Maybe there was also more luck than normal in 2013. 

Instead of 6.4 both years, what if SD(luck) had actually been 8.4 in 2013, and 4.4 in 2014?

In that case, our estimates would be

SD(2013 talent) = 8.6
SD(2014 talent) = 7.6

That would be just a change of 1.0 wins in competitive balance, much more plausible than our previous estimate of a 3.2 win swing (10.2 to 7.0).

------

Still: no matter which of all these assumptions and calculations you decide you like, it seems like most of the difference must be luck. It might be luck in terms of the bad teams winding up with the good players for random reasons, or it might be that 2013 had the good teams having good luck, or it might be that 2014 had the good teams having bad luck.

Whichever kind of luck it is, you should expect a bounceback to historical norms -- a regression to the mean -- in 2015. 

The only way you can argue for 2015 being like 2014, is if you think the entire move from historical norms was due to changes in the distribution of talent between teams, due to economic forces rather than temporary random ones. 

But, again, if that's the case, show us! Personnel changes between 2013 and 2014 are public information. If they did favor the bad teams, show us the evidence, with estimates. I mean that seriously ... I haven't checked at all, and it's possible that it's obvious, in retrospect, that something real was going on.

------

Here's one piece of evidence that might be relevant -- betting odds. In 2014, the SD of Bovada's "over/under" team predictions was 7.16. This year, it's only 6.03.*

(* Bovada's talent spread is tighter than what we expect the true distribution to be, because some of team talent is as yet unpredictable -- injuries, trades, prospects, etc.)

Some of that might be a reaction to bettor expectations, but probably not much. I'm comfortable assuming that Bovada thinks the talent distribution has compressed by around one win.*

Maybe, then, we should expect a talent SD of 8.0 wins, rather than the historical norm of 9.0. That's more reasonable than expecting the 2013 value of 10.2, or the 2014 value of 7.0. 

If the SD of talent is 8, and the SD of luck is 6.4 as usual, that means we should expect the SD of this year's standings to be 10.2. That seems reasonable. 

------

Anyway, this is all kind of confusing. Let me try to summarize everything more understandably.

------

The distribution of team wins was much tighter in 2014 than in 2013. As I see it, there are six different factors that could have contributed to the increase in standings parity:

-- 1. Player movement from 2013 to 2014 brought better players to the worse teams (to a larger extent than normal), due to changes in the economics of MLB.

-- 2. Player movement from 2013 to 2014 brought better players to the worse teams (to a larger extent than normal), due to "random" reasons -- for instance, prospects maturing and improving faster for the worse teams.

-- 3. There was more randomness than usual in 2013, which caused us to overestimate disparities in team talent.

-- 4. There was less randomness than usual in 2014, which caused us to underestimate disparities in team talent.

-- 5. Randomness in 2013 favored the better teams, which caused us to overestimate disparities in team talent.

-- 6. Randomness in 2014 favored the worse teams, which caused us to underestimate disparities in team talent.

Of these six possibilities, only #1 would suggest that the increase in parity is real, and should be expected to repeat in 2015. 

#3 through to #6 suggest that 2013 was a random aberration, and would suggest that 2015 would be more like the historical norm (SD of 11 games) rather than like 2013 (SD of 12 games). 

Finally, #2 is a hybrid -- a one-time random "shock to the system," but with hangover effects into future seasons. If, for instance, the bad teams just happened to have great prospects arrive in 2014, those players will continue to perform well into 2015 and beyond. Eventually, the economics of the game will push everything back to equilibrium, but that won't happen immediately, so much of the 2014 increase in parity could remain.

------

Here's my "gut" breakdown of the contribution each of those six factors:

25% -- #1, changes in talent for economic reasons
 5% -- #2, random changes in talent
10% -- #3, "too much" luck in 2013
20% -- #4, "too little" luck in 2014
10% -- #5, luck favoring the good teams in 2013
30% -- #6, luck favoring the bad teams in 2014

Caveats: (1) This is just my gut; (2) the percentages don't have any actual meaning; and (3) I could easily be wrong.

If you don't care about the reasons, just the bottom line, that breakdown won't mean anything to you. 

As I said, my gut for the bottom line is that it seems reasonable to expect 2015 to end with a standings SD of 10.2 wins ... based on the changes in the Vegas odds.

But ... if there were an over/under on that 10.2, my gut would say to take the over. Even after all these arguments -- which I do think make sense -- I still have this nagging worry that I might just be getting fooled by randomness.