How extreme are simulation game results?

Last post, I tried to figure out the theoretical breakdown of luck in team records.  In standard errors, I got:

31.9 runs from career years batting
31.9 runs from career years pitching
23.9 runs from event clustering
23.9 runs from opposing team's event clustering 
39.1 runs from Pythagoras

A couple of posts before that, I had done the same thing, but for my "luck" study.  There, the "career year" luck estimates were higher.  (The clustering and Pythagoras estimates were roughly the same, but that's because I used pretty much the same method).

42.7 runs from career years batting
48.5 runs from career years pitching
23.8 runs from event clustering
25.8 runs from opposing team's event clustering 
39.1 runs from Pythagoras

There are good reasons for my "career year" estimates being farther off -- mostly, because I had to interpret changes in talent (aging, injuries, learning to hit a breaking ball) as luck.  (There are also selective sampling issues.)

Anyway, part of the reason I did all this was because of a comment from Ted Turocy in my post on simulation games:

"Do you also assume that the player "cards" have been suitably regressed to the mean? If not -- which is the case with all standard season sets/disks -- then the simulation will tend to have more extreme totals on the player leaderboards."

I hadn't thought of that.  Ted is right ... most simulations don't regress to the mean.  If Mark McGwire hit 70 home runs, the game will be calibrated so that McGwire's expectation is 70.  Which means, around half the time, he'll actually hit *more* than 70.  In fact, the binomial SD for McGwire's 1998 is around 8 homers, so that, a good proportion of simulated seasons, he might hit 80 or more!

The *team* totals will be more extreme too, then, and so will the team standings.  So I wondered: how much more extreme?  I seem to recall, some time ago, APBA putting out a promotional flyer with the results of a full simulated season (sent in by a customer), and showing how similar it was to the actual standings.  You'd think, though, without regressing to the mean, you'd have too much of a spread, though.

Well, now we can figure it out.

From 1973 to 2001 (omitting strike seasons), the SD of team records (normalized to 162 games) was almost exactly 11 wins.  The theoretical SD of luck is 6.36 wins, which means the SD from talent is almost exactly 9 wins.  (9 squared plus 6.36 squared equals 11 squared.)  

So, that's 90 runs.

But, now, the simulation is increasing that, by taking the "career years" luck, and making it part of the player's "talent" (which is what the card represents).  For the team as a whole, it works out to 31.9 runs pitching, and 31.9 runs hitting.

So the SD of talent is now 101 runs -- the square root of (90 squared plus 31.9 squared plus 31.9 squared).

Which means the SD of observed W-L records is now 119 runs, or 11.9 wins -- the square root of (101 squared + 63.6 squared).

11.0 wins -- real life
11.9 wins -- APBA

Not much difference -- around a single win.   

So, if you figure the most extreme team in a year is maybe 2 SD from the mean ... the top and bottom teams should be two wins more extreme than real life.  (They won't necessarily be the *same* teams.)

That's not a big deal ... you probably wouldn't even notice.  Real life often bumps the extremes well beyond that.

The real-life SD of wins is 11.0, but it varies quite a bit by season.   Here are the 26 season SDs:

10.3,  9.9, 11.7, 11.8, 14.4, 12.3, 12.9, 11.6, 10.5, 9.8, 8.9, 12.6, 10.3, 9.8, 12.0, 10.0, 9.1, 9.7, 10.2, 12.2, 10.0, 9.6, 13.5, 12.5, 10.0, 13.0, 14.8, 13.4, 13.5, 10.8, 10.1, 9.3, 11.1, 11.4, 11.0, 11.4

11.9 fits right in.  The extremes are much higher than that.  In 1984, the SD of wins was less than 9.  In 2002, it was 14.75.  I've never seen anyone mention either, that there was so much parity in 1984 but so little in 2002.  It seems that we don't even notice *big* changes.

The overall SD of the SDs was more than 1.5 ... so a change from 11.0 to 11.9 is only 0.6 SDs from the previous mean.  For statistical significance, you'd need to more triple that ... which means you'd need roughly ten times as many seasons.  It would take 360 years of Major League Baseball to get a 50 percent chance of statistical significance for a single unregressed APBA season.

It's a smaller effect than I had been thinking it would be.

------

In any case, that just applies to team records.  You'll be able to notice the spread more easily in individual results.

For a season of 600 AB, the binomial standard deviation of batting average is 18.7 points.  

In 2012, Buster Posey led the majors in batting average, at .336.  In a simulation, he'd have a 50% chance of beating that.

Miguel Cabrera, who hit .330, would have probably a 40% chance.  Andrew McCutchen, half an SD behind, would have a 30% chance.  Mike Trout, also around 30%.  Adrian Beltre (.321), 20%.  So far, that adds up to 170%, or an average of 1.7 simulated hitters beating .336.  And that's only after looking at five players!

So, it's virtually assured that the simulated leader will outhit the actual leader.  That's probably true in any of the major statistics where players get similar numbers of opportunities.

------

Bottom line: in an APBA season, you'll notice players are more extreme than real life -- but you probably won't notice that teams are.

Can we tell simulation from real life?

I was a participant on the "randomness" panel at the Sloan Conference last month.  One of the questions was, "How can fans get a feel for how much luck there is in sports?"

My answer went something like this: Play simulation games, like APBA or Strat-O-Matic for baseball.  You'll find that, one game, team A will win 11-1, and, the next, they might lose 8-2 to the same opponents.  Even with exactly the same talent, as determined by the game, the results will vary widely just because of random variation.

What I wanted to add at the time, but trailed off because I lost my train of thought, was: if you're skeptical, you might think that those games are "over"-random, given that they use dice rolls and all.  But ... it turns out that random APBA outcomes are very, very close to real-life outcomes.  For instance, I'd bet that pairs of "11-1 then 2-8" games are almost exactly as common in baseball history as they would be in APBA-simulated baseball history.

Now, I have no actual evidence for that, but I think it's true.  Still, I got to thinking ... what are the ways where real life and APBA *would* be different?  That is, suppose I handed you a bunch of actual game box scores, and a bunch of APBA box scores.  Would you be able to tell which pile was which?

We need to add some assumptions.  Let's suppose that the simulation is as "perfect" as sabermetric knowledge permits -- that is, it uses proper log5, the best park effects, the best guess at how DIPS should work, the proper understanding that batters hit better with runners being held on first base, and so on.  Let's suppose, too, that we clone the team's managers, and let them make game decisions the same way as real life (when to change pitchers, put in a pinch hitter, call for a hit-and-run, etc.).

And, let's also assume that we're going to weed out games with the really weird things, the ones that no simulation could be smart enough to to include with the right probabilities, like Derek Jeter's famous "flip" throw home, or the time the ball bounced off Jose Canseco's head for a home run.   Or, if you prefer, assume that the simulation IS smart enough, if that doesn't bend your brain too much.

Really, what we're trying to do here is assume that the simulation has every probability perfect: it's just that the outcomes are independent and randomly determined, by dice rolls based on the probabilities, instead of by actual play of the game by flesh-and-blood humans.

If we did all that, could anyone tell the difference?  

My gut answer: it would be hard.  There are some things we could look for.  Injuries, for instance, mean that batters would be a tiny bit "streaky", in that bad performance would be clustered more than randomly, during those times when the player is hurt.  You might find that, in real life, rookies start out well and peter out, as opponents figure out their weaknesses, whereas in APBA, the cards are fixed.  

But, overall, I think even the most knowledgable experts would have trouble telling the pile of real box scores from the pile of simulated box scores.

Think about this in concrete terms, of what you, personally would do.  Suppose I took one of those computer games, Pursue the Pennant, or something, and simulated a bunch of games from the 1978 schedule.  And I print off the box scores, and put them alongside the real ones, and I hand them to you in person.

Assuming you don' t actually remember a lot of details from 1978 games -- like actual scores, or player performances -- what would you do to figure out which was which?

NHL and luck: simulation results

Last post, I calculated that in an 82-game NHL season, luck is almost as important as talent.  Talent accounted for 53 percent of team variance in the standings, while luck accounted for the other 47 percent.

That makes it seem like the standings are hugely unpredictable.  But, that's not really the case.  Fifty-three percent is still enough for the best teams to come to the top, for the most part.

I ran my NHL season simulator a bunch of times, using the 2010-11 season.  I'll show you one of the seasons that came out with typical randomness ... in this case, 48 percent luck. 

Teams are ranked in order of simulated points.  Assumed talent (goal differential, regressed 22 percent to the mean) is in parentheses.  ("Assumed talent" is based only on the raw stats.  I'm pretty sure the Canucks aren't really a true +61 -- based on their seasons before and after -- but I'm not making any team-specific adjustments for that.)

OK, here's the Western Conference:

113 Nashville Predators (+22)
109 Vancouver Canucks (+61)
107 St. Louis Blues (+8)
104 Chicago Blackhawks (+25)
 99 Anaheim Ducks (-3)
 98 Phoenix Coyotes (+9)
 96 Los Angeles Kings (+12)
 94 Detroit Red Wings (+13)
-----------------------------
 91 San Jose Sharks (+27)
 86 Calgary Flames (+11)
 84 Dallas Stars (-4)
 77 Colorado Avalanche (-51)
 74 Edmonton Oilers (-53)
 69 Minnesota Wild  (-21)
 57 Columbus Blue Jackets (-31)
 
And here's the Eastern:

112 Boston Bruins (+46)
108 New York Rangers (+22)
 99 Buffalo Sabres (+9)
 97 Philadelphia Flyers (+33)
 95 Washington Capitals (+19)
 95 New York Islanders (-26)
 92 Tampa Bay Lightning (+3)
 90 Montreal Canadiens (+5)
---------------------------
 89 Florida Panthers (-25)
 87 Carolina Hurricanes (-2)
 86 Ottawa Senators (-41)
 86 New Jersey Devils (-31)
 82 Toronto Maple Leafs (-23)
 81 Pittsburgh Penguins (+25)
 77 Atlanta Thrashers (-38)

Generally, the good teams did well and the bad teams did poorly.  But, there were a few exceptions that contributed to the 47 percent luck. 

In the East, the Islanders and Penguins basically swapped positions.  Pittsburgh had a horrible simulated year, finishing next to last despite actually being the fourth best team.  Other than that ... the East finished very much as expected: all the plus teams making the playoffs, and all the minus teams missing out.

In the West, the Sharks missed the playoffs despite being the second-best team.  Effectively, they switched with the Ducks, who wound up in fifth place despite being worse than average.  Also of note: the Blues wound up a few spots north of where they should have been.

Other than those five teams ... nothing really that notable.  But you can imagine how the big stories of the year would be the Penguins' collapse and the Islanders resurgence.

--------

That's pretty typical for what you get with around 50 percent luck.  If you want to get a better feel, here's a few more.  I won't give you the whole record, but I'll just tell you what happened.  (If you want some more sets of random season standings in full, e-mail me and I'll run a bunch off for you.) 

1.  The simulation showed 45% luck.  The Anaheim Ducks (-3) finished fourth, but the Red Wings (+13) finished tenth and out of the playoffs.  In the east, Tampa Bay (+3), Montreal (+5), and Buffalo (+9) took the first three spots.  The best teams took four of the next five, except that the Leafs (-23) took the eighth spot.  Washington (+19) finished ninth.

2.  Luck was 44%.  The Kings (+12) missed the playoffs in favor of the Wild (-21), who finished seventh in the West.  In the East, two bad teams, the Leafs (-23) and Panthers (-25), finished fifth and sixth, respectively, while Pittsburgh (+25) missed the playoffs by two points.

3.  Luck was 47%.  In the West, nothing too serious ... Anaheim (-3) and Minnesota (-21) took the last two playoff spots from the Flames (+11) and Kings (+12), who were right behind them.  But in the East ... the top three were the Sabres (+9), Hurricanes (-2) and Habs (+5).  The Islanders (-26) tied with Boston (+46) and Philadelphia (+33), for 5/6/7th.  The Capitals (+19) missed the playoffs by a point, and the Penguins (+25) by nine points. 

4.  This one was 52% luck ... but, strangely, it wasn't that big a deal.  In the West all the plus teams made the playoffs except the Sharks (+27).  In the East, six of the top seven made the playoffs (Buffalo, +9, was the odd man out); the surprises were the Devils (-31) and Islanders (-26) snatching seventh and eighth.  Toronto (-23) also surprised, missing the playoffs by only two points.  I guess what happened here was that the randomness was mostly *within* the playoff and non-playoff groups, instead of *between* them. 

--------

Those were all typical seasons, with around 45% luck.  But some seasons had more, and some seasons had less.  Here's a summary of all one hundred.  Explanation follows the chart.


Team (talent)     Pts    Pos  1st/POff/Best
------------------------------------------
Vancouver (61)    113    1.6   64  99   1
Chicago (25)      100    4.9   11  88   1
San Jose (27)      99    5.3    7  83   1
Nashville (22)     98    5.6    6  82   1
Detroit (13)       97    6.1    6  74   1
Los Angeles (13)   96    6.5    1  71   1
Calgary (11)       95    6.7    1  73   1
Phoenix (9)        93    7.7    0  58   2
St. Louis (8)      93    7.3    3  60   1
Dallas (-4)        90    9.1    0  38   2
Anaheim (-3)       90    9.0    1  35   1
Minnesota (-21)    84   11.1    0  21   3
Columbus (-31)     81   11.8    0  15   3
Edmonton (-53)     74   13.4    0   3   6
Colorado (-51)     73   13.9    0   0  10

Boston (46)       108    2.7   40  98   1
Philadelphia (33) 102    4.2   14  93   1
Pittsburgh (25)   101    4.4   13  89   1
Washington (19)   100    4.9    4  94   1
Rangers (22)      100    5.1   12  83   1
Tampa Bay (3)      94    7.0    8  65   1
Buffalo (9)        94    7.3    2  65   1
Montreal (5)       93    7.4    2  64   1
Carolina (-2)      91    8.2    3  53   1
Florida (-25)      85   10.5    0  25   2
Islanders (-26)    85   10.5    2  24   1
Toronto (-23)      84   10.5    0  27   2
New Jersey (-31)   80   12.0    0   9   5
Atlanta (-38)      79   12.5    0   5   5
Ottawa (-41)       77   12.8    0   6   5


Reading across the bottom line of the chart as an example: we assumed the Ottawa Senators had a talent level corresponding to a regulation goal differential of -41.  On average, that team wound up with 77 points in the simulated standings.  It finished, on average, 12.8th in its conference.  It finished first zero times in 100 chances, and made the playoffs six times.  The highest position it ever reached was fifth place.

The chart looks pretty reasonable, overall; the outcomes corresponded to the talent.  But, for single seasons, we see that strange things happened.  The New York Islanders, at -26, finished first twice, just by luck.  The Washington Capitals, a good team, should have finished first a lot more than 4 times.  And it looks like Colorado had miserable luck; they never finished higher than tenth, while Edmonton, which was actually two goals worse, made the playoffs three times and finished sixth at least once.

Also, it looks like the Canucks, by far the best team in the league, missed the playoffs once just by random chance. 

(By the way, I ran the simulation again, just to make sure it wasn't a programming error; the Washington and Colorado outliers disappeared.  Also, the Canucks made the playoffs all 100 times.)

You can get a feel for the number of anomalies in the average season, just by adding up some numbers.  Of course, it depends how you define "anomaly".  Suppose we say an anomaly occurs in a conference when:

-- one of the best three teams misses the playoffs.
-- one of the worst three teams makes the playoffs.
-- the first place team is not in the top five in talent.

Those frequencies, respectively, were 50, 38, and 23.  So we have 111 anomalies total, 1.11 per season.  That's an average; some seasons have more than one, some seasons have none.  But, typically, there will be one big regular-season surprise that's completely due to luck.

---------

There were still a couple of things I wanted to check.  One is, in one hundred seasons, what's the weirdest, most luck-filled one look like?  Another one is, what's the typical talent difference is between, say, sixth place and tenth -- that is, the five teams fighting for the last three playoff spots?  My gut says there won't be much ... but I haven't checked yet. 

I'll do those in future posts.

A rudimentary NFL season simulation

Following a post by Tango a couple of weeks ago on the playoff systems of the various sports, I thought I'd try writing a simulation. This is an update of that work in progress. Actually, I've only started on the NFL, and I haven't even done playoffs yet, just the regular season. But I thought I'd at least share what I've got so far.

In the simulation, each of the 32 teams was assigned a "true talent," from a normal distribution with mean .500 and standard deviation .143. No team was allowed to have talent higher than .900 or lower than .100; if they did, they were moved to .900 or .100. Then, all 32 teams were moved the same amount (arithmetically) in the same direction to get the overall talent to average exactly .500. (I think this method actually reduces the expected SD below .143, but I didn't bother fixing that.)

The 16-game schedule is random, instead of unbalanced (with the restriction that a team can't face any another team more than twice). There are no tie games. There is no home field advantage (although that would be easy to add in). The chance of winning each game is determined by the log5 method. There are no ties in games. Ties in the standings (division or wild card) are broken randomly.

As I said, I stopped there for now; haven't done playoffs yet. That's the next step, along with home field advantage.

Anyway, here are some results. Each result is out of 100,000 seasons. Every result came from a different run of the simulation. Results varied a fair bit per run, but I think everything is reasonably typical.

--------

I checked for all teams out of 3,200,000 (32 teams, 100,000 seasons) that finished more than 8 games above or below their talent. That's hard to do, obviously. Also, the worse or better you are, the harder it is. It's (relatively) easier for an 8-8 team to go 16-0 than for a 3-13 team to go 11-5. Amplifying that is the fact that there are a lot more 8-8 teams than 3-13 teams. However, offsetting that, a little bit, is the fact that the 3-13 team can also go 12-4 or 13-3 or better.

In any case ... there were 43 cases where a team differed from its talent by 8 games or more. Of those, 26 were teams that outperformed, and 17 were teams that underperformed.

The biggest differential was in season 98,534, where the Broncos a team that had talent of 4.56 wins (out of 16), but went 14-2, for a differential of 9.44 games. That was the only team with a differential of 9 or more. Part of the reason it did so well was that it faced inferior opponents. You'd expect any given team's opponents to average 8.00 games of talent. But in that season, the Broncos' opponents' talent was only 7.45 games. Not a huge difference, but still.

Actually, when it comes to extreme events, a small difference in opponents makes a big difference in probability. Of the 43 teams in the sample, 38 of them had records that went in the direction "aided" by the opposition (in the sense that the underperforming teams played better-than-expected opponents, and vice versa). That's 38-5 in favor.

The worst team in the sample was the season 63,924 Jets, a 3.03 team that went 12-4 (playing 7.11-win opponents). The best team in the sample was a Bucs team that was expected to win 12.04 games, but instead went 4-12 (playing 8.58-win opponents).

--------

I also took a look at teams that went 0-16.

Those results probably aren't as realistic, because they're heavily dependent on the shape of the tail of the talent distribution ... and we really don't know what that is. Recall that we chose a normal distribution that gets truncated at .100 (1.6 wins). Both those choices -- normal, and truncated -- are arbitrary and probably not close enough to real life. (Also, teams could drop below .100 in talent from the adjustment that sets all league-years to .500.)

In addition, the other shortcuts in the simulation probably skew the results too. The mainstream results are probably right, but the extremes are extremely sensitive to some of the assumptions.

With those caveats: there were 5,663 of those 0-16 teams out of 3.2 million, and their average talent was .181, which is just under 3-13. I suspect the talent of actual flesh-and-blood 0-16 teams is higher than that, but I really don't know.

16-0 should be exactly symmetrical, so I won't show that separately.

--------

I checked for four-way ties where every team has the same record. That happened 878 times out of 800,000, or about once every century.

There were five seasons out of 100,000 where two divisions had a four-way tie. Actually, that might be a little high ... the test runs had only 1 or 2 such seasons.

---------

Anyway, before I start on the playoffs, and repeating this for other sports leagues, I'm looking for feedback on what I've got so far. Any suggestions?

And, if you want me to run the sim and check for something in particular, let me know in the comments. It's real easy to add a couple of lines of code to check for something specific.

A Diamond Mind simulation as baseball strategy research

A science column from Alan Schwarz a couple of weeks ago investigates the effects of various baseball strategies, using a simulation.

To check out batting orders, Schwarz got Luke Kraemer at Diamond Mind to simulate two sets of 100 seasons of the 2008 Yankees. In one set, A-Rod batted fourth; in the other set, he batted ninth. The difference was 42 runs; the regular Yanks scored 789 runs, while the A-Rod-at-the-bottom-of-the-order Yanks scored only 747.

Schwarz doesn't tell us how he checked intentional walks, but finds that they are a bad strategy, costing five runs per season. That's not a very useful result; there are times when the IBB makes more sense, and times when it makes less sense. Which did Diamond Mind simulate?

Stolen Bases: Diamond Mind took the 2008 Rays and the 2008 A's, and reversed their respective propensities to steal ("switched their mind-sets," is what the article says). The A's dropped by 20 runs, but the Rays *improved* by 47 runs, "suggesting that perhaps the Rays were running too often in real life."

As it turns out, the real Tampa Bay team stole 142 bases and were caught only 50 times, for a 74% success rate; that should put them well in the black, compared to the rule of thumb that you need to be successful 67% of the time to break even. So I'm at a loss to explain the 47 run difference.

The only thing I can think of is a sample size issue. I think the SD of a team's runs scored in a single game is about 3. So the SD of a season's worth of runs is 3 times the square root of 162, or about 38 runs. The SD of the average of 100 season's worth is one-tenth of that, or about 4 runs. The difference between two 100-season averages is the square root of 2 times that, or about 5.4 runs.

But 47 runs is almost 9 standard deviations. So I'm still not sure what's going on.

Finally, the sacrifice bunt. When the simulation forced the bunt-avoiding Red Sox (27 SH in 2008, compared to the league-average 34) to do it more often, they lost 19 runs. But when they got the bunt-loving Mets (73, league average 66) to do it less, the result was also a loss – 15 runs. Schwarz concludes that the Mets' real-life bunting was better than the Red Sox, that they chose to bunt in more favorable situations. But, weren't both these numbers based on the simulation? If so, the real-life situations should make no difference.

If the comparisons, however, *were* based on real life, then we have sample size issues based on the real-life sample, which is only 162 games, with an SD of about 38 runs. Maybe the 2008 Mets and Red Sox scored more or fewer than the simulation because of luck? We should be able to tell by looking at Runs Created – but, for some reason, almost all teams undershot their RC estimate in 2008 (and their Base Runs estimate too, at least for the versions I tried).

Anyway, while I like the simulation method, I wish the results had been presented more clearly. As it stands, I'll stick to "The Book"'s conclusions on these issues of baseball strategy.

P.S. Here's what Tony LaRussa thinks of these results:

“There’s way too much importance given to what you can produce from a machine,” he said. “These are human beings, and I don’t think any computer is going to model that close to what we deal with at this level.”

Hat Tip: Daniel Hamermesh at Freakonomics