Challenge: design a study that measures player improvement

If everything doubled in size last night, how could we prove it happened? We can’t do it by measuring things, because the size of rulers, and therefore of inches, would also have doubled. We could check the speed of light, but if the laws of nature changed at the same time, to reflect the doubling, we’d be out of luck. This is an old philosophical riddle, to which one answer is that it makes no sense to say that everything doubled, because if there’s absolutely no way in which the universe can be seen to differ after the doubling, the universe is actually exactly the same as before.

The baseball equivalent is, if all players got twice as good in the last 100 years, how could we prove that happened? I posted on this a couple of weeks ago (here), and my point then was that the Cramer method (which compares players last year to the same players this year to see if the rest of the league got better) doesn’t work.

As I argued, one thing you can do is notice that unlike the “doubling in size” case, where the laws of physics also changed, in baseball, the laws of nature are the same as ever. You can look at pitch speeds, and ball distances, and so on. You can use physics as the unchanging ruler to check performance against.

But can you design an experiment, like Cramer tried to do, that will find an answer without looking to physics? I can’t find the reference, but I’m pretty sure Bill James once speculated that there’s no way to do it. I think I agree.

I think the changes in major-league skill are so closely tied in with the changes in players as they age, that you can’t disentangle one from the other. For instance, Bill James’ famous 1982 study showed (I’m oversimplifying) that hitters lost 7% of their value between age 27 and 28. Which is true. But did they also lose 7% of their ability? Not necessarily. They might have lost only 6% of their ability, but the league improved by 1% under their feet, so they lost a total of 7% of their value. Or maybe they lost 5% and the league improved by 2%. Or some other combination. Right now, we don’t know which is correct.

Can anyone think of a study, using only Retrosheet data (no physics), that would allow us to reach any conclusions about the rate of improvement in major league baseball? You don’t have to actually do the study, just describe it. It can be as complicated as you want. And it can return any valid conclusions at all – even a wide confidence interval is OK, as long as it’s justified by the logic of the study and the evidence.

I’m betting it can’t be done.