Monday, February 18, 2013

More luck in outcomes doesn't imply less skilled players

A couple of weeks ago, Michael Mauboussin posted about how there's so much luck in the NHL's shortened season. (Mauboussin is the author of "The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing".) 

Most of the comments to the post didn't quite get it.  In particular, one commenter wrote, disbelievingly, 

"Did you watch the Rangers/Bruins game last night?  Luck?"

That, I think, is a pretty common response -- and not unreasonable.  NHL hockey players are among the best in the world.  hey've practiced thousands of hours, and any serious weakness in their game is instantly exploited by their opponents. How can we say that hockey is mostly luck?

Well, we don't. We're not actually arguing that hockey has more luck than skill. We're arguing something different: that the *outcome* represents luck more than skill. Or, more specifically, we're saying that the standings are more the product of *differences in luck between teams* than *differences in skill between teams*.

Which is what Mauboussin replied to the commenter:

"Here's the point: the article doesn't say that hockey players are not skillful (they are). It says that the skills of the players on opposing teams offset one another, leaving more to luck." 

The misunderstanding, it seems, stems from some readers not realizing we're talking about relative skill ... rather, they think we're talking about actual levels of player talent. We're not. If you found a peewee hockey league where the relative distribution of talent was the same as in the NHL, the luck/skill analysis would be exactly the same. The NHL season would be 60% luck, and the peewee season would also be 60% luck. The statement says nothing about how skilled the players are in a real-world sense, and it says nothing about how much skill is actually required to play the game.

Maybe we should making that more explicit. Instead of, "The standings are 60% luck," or "Hockey is 60% luck," we can say, "The standings are determined 60% by which teams are luckier than their opponents, and 40% by which teams are more talented than their opponents."


Another part of the problem, perhaps, is that, in real life, we're not all that used to luck being a big factor. We rarely observe such small differences in skill as exist in most sports. In the NBA, even the worst team has a 10% chance to beat the best team. But I'm pretty sure that, at work, if you pick five random employees, and play them against the five best basketball players on staff, they'd win, like, zero percent of the time.  

In my recreational ball hockey league, my team is pretty bad. When we play the best team in the league, we *know* we're going to lose. No 10 percent, no 5 percent chance. We're going to get beat. Probably 19-3, or something. Maybe, if we're lucky, it'll be 13-5, or 14-7, or some such.  

In the real world, differences in talent are large. In professional sports, they're small.  

As Mauboussin says, in a related interview,

"This leads to one of the points that I think is most counter to intuition. As skill increases, it tends to become more uniform across the population. Provided that the contribution of luck remains stable, you get a case where increases in skill lead to luck being a bigger contributor to outcomes. That’s the paradox of skill."

We're not used to that. In our personal worlds, people are very different.  In school, the SAT-taking senior that's good at math is going to outscore the one who isn't, at least 99 percent of the time. The best programmer at work will finish faster than the worst programmer at work, 99.9 percent of the time.  And so on.  

In real life, when we see one person perform better than another, we can be fairly sure that person is actually better at the task.  

We can't do that as readily in professional sports. There, skill is much more uniform. We're not used to that, the idea that you can't tell the better team just by watching. We're not used to luck being such a big factor.



At Thursday, February 21, 2013 10:16:00 AM, Anonymous Anonymous said...

Great follow-up post. But I have a question. The researcher states
NHL is ~ 60% luck(this is similar to my research) However, in a previous post you suggested that at 73 games
skill = luck which implies in a 82 game sched. sill is app 52% ? any thoughts? Am I confused?thx


At Thursday, February 21, 2013 10:26:00 AM, Blogger Phil Birnbaum said...

Mauboussin says 60% "seems" to be right, but it's not. The way I read it, anyway ...

At Thursday, February 21, 2013 10:27:00 AM, Anonymous Anonymous said...

Phil; Here is a reason that suggests
your calculation of 53% skill may be inaccurate: It comes from B. Burke of NFL adv. stats:
If skill is 53% than the best team wins 53% of time & 1/2 of 47% which means best team wins ~77% of time.
This is way to high? why ? Because the best algorithms I have created or seen on NHL only reach ~63% for predicting NHL reg. gms. D(part1)Dan

At Thursday, February 21, 2013 10:33:00 AM, Anonymous Anonymous said...

Working backward best team wins 35% + 30 luck = ~ 65.
So by my calculations luck is ~ 65% higher than Mauboussin.
10% difference between actual prediction model & theoretical prediction model is quite large?

At Thursday, February 21, 2013 12:12:00 PM, Blogger Phil Birnbaum said...

The "X% luck" figure does not necessarily mean the best team wins X + 1/2 (100-x) percent of the time, as you assume. It has to do with the standings differences only, not individual games.

At Thursday, February 21, 2013 1:52:00 PM, Anonymous Anonymous said...

Thanks for yr response.
As I mentioned I got it from B. Burke's great nfladvstats site.

here is the link where he justifies the rational behind the equation I use?

At Thursday, February 21, 2013 5:33:00 PM, Blogger Phil Birnbaum said...

Right, his number means something different from the numbers in this post.


Post a Comment

<< Home