Can we tell simulation from real life?
I was a participant on the "randomness" panel at the Sloan Conference last month. One of the questions was, "How can fans get a feel for how much luck there is in sports?"
My answer went something like this: Play simulation games, like APBA or Strat-O-Matic for baseball. You'll find that, one game, team A will win 11-1, and, the next, they might lose 8-2 to the same opponents. Even with exactly the same talent, as determined by the game, the results will vary widely just because of random variation.
What I wanted to add at the time, but trailed off because I lost my train of thought, was: if you're skeptical, you might think that those games are "over"-random, given that they use dice rolls and all. But ... it turns out that random APBA outcomes are very, very close to real-life outcomes. For instance, I'd bet that pairs of "11-1 then 2-8" games are almost exactly as common in baseball history as they would be in APBA-simulated baseball history.
Now, I have no actual evidence for that, but I think it's true. Still, I got to thinking ... what are the ways where real life and APBA *would* be different? That is, suppose I handed you a bunch of actual game box scores, and a bunch of APBA box scores. Would you be able to tell which pile was which?
We need to add some assumptions. Let's suppose that the simulation is as "perfect" as sabermetric knowledge permits -- that is, it uses proper log5, the best park effects, the best guess at how DIPS should work, the proper understanding that batters hit better with runners being held on first base, and so on. Let's suppose, too, that we clone the team's managers, and let them make game decisions the same way as real life (when to change pitchers, put in a pinch hitter, call for a hit-and-run, etc.).
And, let's also assume that we're going to weed out games with the really weird things, the ones that no simulation could be smart enough to to include with the right probabilities, like Derek Jeter's famous "flip" throw home, or the time the ball bounced off Jose Canseco's head for a home run. Or, if you prefer, assume that the simulation IS smart enough, if that doesn't bend your brain too much.
Really, what we're trying to do here is assume that the simulation has every probability perfect: it's just that the outcomes are independent and randomly determined, by dice rolls based on the probabilities, instead of by actual play of the game by flesh-and-blood humans.
If we did all that, could anyone tell the difference?
My gut answer: it would be hard. There are some things we could look for. Injuries, for instance, mean that batters would be a tiny bit "streaky", in that bad performance would be clustered more than randomly, during those times when the player is hurt. You might find that, in real life, rookies start out well and peter out, as opponents figure out their weaknesses, whereas in APBA, the cards are fixed.
But, overall, I think even the most knowledgable experts would have trouble telling the pile of real box scores from the pile of simulated box scores.
Think about this in concrete terms, of what you, personally would do. Suppose I took one of those computer games, Pursue the Pennant, or something, and simulated a bunch of games from the 1978 schedule. And I print off the box scores, and put them alongside the real ones, and I hand them to you in person.
Assuming you don' t actually remember a lot of details from 1978 games -- like actual scores, or player performances -- what would you do to figure out which was which?