Thursday, August 29, 2013

How good are the best poker players?

Last post, I mentioned a study by Steven Levitt and Thomas Miles that argued for skill in poker.  A couple of people later sent me ungated links to that study (.pdf) ... when I read it, I was surprised how small the differences seem to be between players.

Levitt and Miles (call them LM) found that more skilled players (as judged by various sets of poker writers) made a 30%+ return in tournaments, while non-ranked players lost money, to the tune of negative 15 percent.  

But, looking more closely at the study, that seems to be a very small difference. As it turns out, the skilled players were only between 4 and 29 percent more likely than the average player to make the final table.

That seems tiny.  In team sports, if you pick the best team at the beginning of the year, you'd think they've got to be at least twice as likely to make the finals, right?  In baseball, a random team has a 1 in 15 chance to make the World Series.  A *good* team, on the other hand ... well, you'd think it would be 1 in 5 or something, right?  Three times as high?

And in *individual* sports ... look at golf, say.  If there are 100 players in a typical PGA tournament, a random player has a 10 percent chance of finishing in the top ten.  In his career, Tiger Woods finished in the top ten around 63 percent of the time.  That's not 29 percent -- it's 6,200 percent.

(It's not that great an analogy, of course, but even if I tried to make the comparison more reasonable, the difference would still be huge.)


Also ... I'm not completely sure that the differences in the LM study are statistically significant.  In fact, the study doesn't even talk about statistical significance, which is kind of strange, for an academic study.

And it's not easy to figure out, because the authors don't give us enough information.  What we'd want to see is, the mean and SD of profit per tournament for the skilled players, and the man and SD of profit for the rest.  We don't get that.  Instead, we get the mean and SD *per player*, rather than per player-tournament.  That doesn't help, because some players enter ten times as many tournaments than others, so we can't separate talent from opportunity.

Also, we get the mean and SD for buy-in (entry fee) separately from the mean and SD for winnings, so we don't even know the mean and SD for profit or for return on investment!

I find that kind of strange.


The authors' data show there was an average 1,152 players per each of the 56 tournaments (using Table 3, which leaves out the higher-stakes "Main Event").  Guessing that each final table is 9 players, that's 504 overall seats at the final tables combined.  Skilled players were 12.6 percent of the entries, so you'd expect them to account for about 64 of those seats.  

If they were 20 percent more likely to get there, that's only an extra 13 seats.

So, what LM found is that skilled poker players take a seat from less-skilled poker players one out of every four tournaments, roughly.

That doesn't seem like a lot, does it?


Is there really that much luck?  In the comments to the last post, Brian Burke wrote,

"I think once a player has learned the basics, he's par with even the best players in terms of skill. Once all players are equal in terms of skill, luck dominates the outcome. Even if players differed significantly in terms of skill, it might take thousands of hands to determine who is the better player.
"I think of poker like a complex version of rock, paper, scissors. Strictly speaking, RPS a game of skill. The skill is this: don't be predictable. Once everyone realizes the basic skill, now it's effectively a game of pure luck."

The evidence so far supports that, at least a little bit.  It could certainly be that the small effects LM found are just the (equally-skilled) good players taking turns beating up on the crappy ones.


Or, maybe the sample the authors used -- the 2010 World Series of Poker (WSOP) -- was an outlier, or too small a sample.  Here's some evidence, perhaps, that suggests that might be the case.

The "Main Event" of the WSOP is the biggest one.  There's a $10,000 buy-in, and, this year, there were 6,352 entries.

In July, they played enough poker to eliminate 6,343 of those entries, leaving only nine.  Those nine players will play their final table in November.

Right now, you can place a bet on which of those players will win.  If they were close to equal in skill, the odds would be almost the same.  Are they?

Well, it's hard to tell ... the players all enter the final round with different amounts of chips.  The higher bankrolls have better odds.  Here are the odds I found, with the bankrolls in brackets:

J.C. Tran (38,000,000)            9/5  
Amir Lehavot (29,700,000)         9/2  
Marc McLaughlin (26,525,000)      5/1  
Jay Farber (25,975,000)           7/1  
Ryan Riess (25,875,000)           6/1  
Sylvain Loosli (19,600,000)       8/1  
Michiel Brummelhuis (11,275,000) 13/1  
Mark Newhouse (7,350,000)        15/1  
David Benefield (6,375,000)      15/1  

I don't know how to convert chip counts into odds, so most of this data doesn't help me.  But there's one exception: Ryan Riess is favored over Jay Farber, 6/1 vs. 7/1, even though Farber has (a tiny bit) more chips.  

Does that suggest Riess is significantly better?  Maybe.  Or, it could be that the bookmaker expects more money on Riess.  Or, it could have something to do with seating position (I don't know much about poker).  But ... it's something.

If you look at the overall odds, the implied probabilties add up to about 128%, which means you'd have to bet $128 to win $100 (the difference is the bookmaker's take).  That's pretty high, so even if the Farber/Reiss difference isn't justified, it's still not an exploitable profit opportunity.

What if we look at the pre-tournament odds instead?  I found this page, which shows odds on the top 20 players.  Here are 1-3 and 18-20:

Daniel Negreanu (CAN)   40/1
Phil Ivey (USA)         50/1
Phil Hellmuth (USA)     50/1
Patrik Antonius (FIN)  100/1
Pius Heinz (GER)       100/1
Russell Thomas (USA)   100/1

Those odds are much, much higher than the LM study would suggest.  100/1 suggests these players are at least 6,000% more likely to win, rather than 20% as implied by the study.  (Actually, the 20% was for the final table.  Generously assuming that the top final-table player has twice the average chance of winning, that still only makes a 40% increased chance of being the overall champion.  Forty is still less than six thousand.)


Ah!  After I wrote that, I found something much better.  Here are a bookie's odds on Daniel Negreanu winning at least one of the WSOP's 57 tournaments:

Yes: 11/10
No:   8/13

If the average tournament has 1100 players, and Negreanu enters all 57 of them -- with only one buy-in for each -- he'd have roughly a 57/1100 chance of a win if he were average.  If his chance is 40 percent above average, that goes up to roughly 80/1100, or 13:1 against.

But, here, you can lay 13/8 instead of 13/1, which suggests that the best players are *eight times* more likely to win than average.

Did I do that right?


So, what's happening with the original study?   It could be that the poker analysis industry isn't very good at figuring out who the best players are.  It could be that the VERY best poker players are much better than just the "best."  It could be that 2010 was an outlier. 

I'm thinking it's a combination of all three.  But, I don't really have any expertise here.  I know some of you reading this are serious poker players ... do you guys know what's going on?

Labels: ,


At Thursday, August 29, 2013 3:06:00 PM, Anonymous Daniel said...

Daniel Negreanu played in tournaments with varying sizes, including a few with under 200 entrants. Using the 1100 person average underestimates his chances of winning greatly (i.e. if he were to play in two events with fields of two and 2198 he would be far more likely to win a bracelet than if he played in two 1100 person events).

As for the pre-tournament betting odds, I would take them with a grain of salt if the field isn't included as an option. It's a sucker bet to get people to go for famous names.

At Thursday, August 29, 2013 3:46:00 PM, Blogger Phil Birnbaum said...

Good point on the event size. The "eight times" may really be more like four or six, depending on the distribution of participants.

You probably want... the harmonic mean? if that's, say, 800 instead of 1100, that would be six-something times average,

At Friday, August 30, 2013 12:10:00 AM, Anonymous TB said...

Another point in favor of skill in poker is to simply point out that, unlike chess, computer AI is not yet able to beat the game at the highest levels of competition. You might argue that many more resources have gone into developing chess AI, and it's probably true. Yet several teams have been working on the poker AI problem for many years now, and arguably there is a larger financial incentive to develop good poker AI than there is for chess. A summary of the current state of affairs can be found here:

At this point computers seem to be able to hold their own against the best humans at heads-up limit hold 'em, but note that this is the simplest form of poker that is regularly played.

At Friday, August 30, 2013 12:37:00 PM, Anonymous Alex said...

I'm working through this as I type it, so let's see how it turns out. One analogy might be football, but it's important to note that the poker paper looks across a number of tournaments. So let's pretend that each NFL season is a tournament.

There are about 30 NFL teams (more now but fewer in earlier years, so we'll use 30 to make the math easier). If they were all equal, each team would have a 1 in 30 chance of winning the Super Bowl, which we'll say is equivalent to making a final table (it's actually much more likely, since a poker tournament has far more than 30 entrants). They've played 47 Super Bowls, so each team should have won about 1.5 times.

But let's say that three teams (about the top 10% of the league) are actually 'top teams' and have an extra 20% chance to win. That means that those three teams, in aggregate, should have won more like 1.9 Super Bowls each for a total of about 6. What we've actually seen is that the top three teams (Steelers, Cowboys, and 49ers) have won 16 Super Bowls. If I did my math correctly, that suggests those franchises are actually 219% more likely to win instead of the 20% we see in the poker group. Some portion of that is dumb luck; there's obviously some chance the Steelers are actually an average franchise that was lucky, but it's what we've observed.

An analogue to the Super Bowl example would be to look at Phil Hellmuth. He's won 13 bracelets, the first one in 1989. If he entered 30 tournaments every year in those 24 years, he would've played 720 tournaments; 13 wins is a 1.8% win rate (let alone making final tables). There's no way he could be expected to win 1.8% of the time as an average player, and that percentage has to be much greater than the Steelers' equivalent.

Of course, we have to consider some differences between poker and football. One I mentioned already is the sheer number of players. Even if a professional poker player is clearly better than the average, he has to get through hundreds of those average players. If the Steelers are clearly better than other franchises at football (perhaps the 49ers are a better current example), they only really have to get past maybe 12 teams. All those extra opportunities to lose has to lower the chances for a poker player. Another factor would be the initial spread between players/teams. A big difference in the NFL would be a double-digit favorite. Those teams might be expected to win, say, 80-85% of the time. If you took a professional poker player and put him against a poor player for whatever the poker tournament equivalent of a single NFL game is, would you expect him to win that often? I'm not sure that I would; that number seems awfully high.

At Friday, August 30, 2013 5:24:00 PM, Blogger Phil Birnbaum said...

Thanks, Alex! It sounds like 2010 was just not a typical WSOP ... fewer Hellmuths than usual.

At Friday, August 30, 2013 9:31:00 PM, Anonymous Alex said...

To be fair Hellmuth is like the Steelers of poker. He's the top example. If you just look at the Steelers, you'd say top football teams are 400% better than average (or whatever the number is), but if you say the top 3 teams then you have more like 200%, and so on. If you take the top 10% of poker bracelet winners, you'll get something lower than Hellmuth but probably higher than 20%. It looks like LM used 720 poker players as the pros, so you can imagine the number coming down to 20%, especially if some of those guys were lucky to be on the list in the first place.

At Friday, August 30, 2013 10:42:00 PM, Blogger Phil Birnbaum said...

Right, I was mentally trying to figure out if the selectively sampled best record (Hellmuth) could be typical of the highest random result ("first order statistic," I think it's called.) My gut said, "nope".

At Sunday, September 01, 2013 1:35:00 PM, Blogger j holz said...

It's already been mentioned, but you can't extract much information from a one-way betting market. I'm sure the sportsbook would happily let you lay 4-1 on every home team in tomorrow's MLB games, but that doesn't imply that the home-field advantage in baseball is anywhere near 80%-20%.

One obvious challenge with this study is that the "top" 250 players make up a huge percentage of the entries in many of the events--and those tournaments are also the ones with the highest entry fees. If I am reading the paper correctly, these will be more heavily weighted, and the 9% house fee to play the tournament will put a big dent into their collective ROI. (There is a popular saying in poker that it's no good being the tenth best player in the world if the other nine are sitting at your table.)

Look at the link and note the extreme negative correlation between buy-in amount and number of players (excluding the Main Event):

On the Hellmuth-Steelers front: both poker players and football teams can go through cycles where they are dominant for awhile, then struggle against a new generation of opponents. All of the current top players (as I define the word) are happy to let Phil Hellmuth sit in their game, because he is a consistent loser against high-level competition.

At Sunday, September 01, 2013 2:04:00 PM, Blogger Phil Birnbaum said...

If you have some idea what the vigorish is, you can get a range for the true odds. As I wrote, at the final table, the implied probabilities add up to 128%. So, overall, the odds can't be more than 28% too high.

If you know that poker one-sided odds add up to 200% (say), you can at least say that the true odds are "probably" double what's offered.

At Thursday, September 05, 2013 12:18:00 AM, Anonymous Anonymous said...

that assumes that the vig is applied equally to all teams, which it is almost certainly not.

At Friday, September 06, 2013 10:14:00 PM, Anonymous Alan McIntire said...

To convert chip balance to odds of winning, just divide total chips,
190k.75 million, by your chip balance.
They're all bad bets. The one that comes closest to fair is
Sylvain Loosli with actual odds of
190.75/19.6 = 9.73 so odds against winning are 8.73 to 1

At Tuesday, October 01, 2013 6:03:00 PM, Anonymous Anonymous said...

Check this new link from Pinnacle Sports regarding WSOP betting:

At Tuesday, October 08, 2013 6:23:00 PM, Blogger Brian Cavers said...

On second note, Alan, your method is probably fine for first place. ICM is used more for equity based on the payout structure


Post a Comment

<< Home