Monday, February 12, 2007

A "value added" Super Bowl study

Economists love gambling markets data almost as much as sabermetricians love Retrosheet data. And so it's not that suprising that less than VII days after Super Bowl XLI, an academic economist has run an analysis of the betting market during the big game.

It's by Keith Jacks Gamble, and it's called "
An Analysis of the Super Bowl Using Price Changes on an Online Prediction Exchange." It's funny that he calls it "online prediction" instead of "online gambling," especially considering his name.

Anyway, I don't want to sound like I'm making fun of it, because it's a really nice paper. Gamble followed the market's estimate of the teams' winning probabilities during the game, and checked how those changed after specific big plays. For instance, the Bears' touchdown on the opening kickoff reduced the Colts' chances of winning – according to the market -- from 68% to 57.25%. And after the Colts threw an interception on their first drive, the Colts' chances dropped even farther, down to 49.25%.

(One thing I wish is that Gamble had told us more about the bid-ask spread. Gamble took the market's probability estimate as the mean of the spread; this is probably decent enough if the spread is narrow, but not so much if the spread is wide. That's because market efficiency puts the "correct" probability somewhere between the bid and ask, but not necessarily the middle. That is, in an efficient market, you don't see a 20-dollar-bill to have a bid-ask of $18/$19 – because that would enable easy profits to be made buying them up at $19. But $15/$20 is possible in a rational market, as are $20/$25 and $19.99/$20.01. And only in that last case does averaging the spread lead to the "right" estimate.

So if Gamble got his 68% from a bid/ask of, say, 65%-71%, there's more reason to doubt it than if it came from 67.5%/68.5%.)

There's no guarantee, of course, that the market is correctly calculating the probabilities – but economic conventional wisdom says it should be close. And it does seem pretty reasonable. The Colts were favored by 7 points over 60 minutes. After the Bears' touchdown, the seven points were erased – but the Colts now would expect one extra possession over the rest of the game. That extra possession is worth something – maybe two or three points? – so a probability of 57.25% seems about right.

After the subsequent interception that gave possession back to the Bears, they were up by 7. If the Colts normally would outscore the Bears by 7 in 60 minutes, their advantage might be down to only 6.5 with the time remaining in the game – giving a half-point advantage to the Bears. And now the number of possessions should be about even (since the Bears had the ball now, but the Colts would receive the kickoff in the second half). So you'd expect the Colts to be at about a half-point disadvantage now, which is reasonably consistent with the observed 49.25%.

(It isn't enough to consider possessions and time alone – there's still the issue of field position. It's possible that if you take field position into account (discussed a bit
here), you might come up with 49.25% exactly.)

Gamble then computes the game's MVP by adding up all the probability changes of the plays he was involved in – similar to the "
value added" method for baseball. I'm not absolutely thrilled by this method, because of the difficulty of assigning the individual plays to the individual players, and because using the specific probabilities tends to reward "clutchness," which is sensitive to the time and situation.

But, in any case, Kelvin Hayden was the most valuable player by this method, adding 18 percent of a win on the strength of his one interception/touchdown return; the top Bear was Muhsin Muhammad, at 12.25. (But illustrating one of the weaknesses of the method – the Hayden interception was on a pass meant for Muhammad. If you attribute any of that interception's impact to Muhammad, he becomes much less valuable.)

Rex Grossman was the biggest negative impacter on both teams, with –36.5. Peyton Manning was a quite-respectable 9.75.

(Hat tip:
Marginal Revolution.)

Labels: , ,


At Monday, February 12, 2007 10:59:00 PM, Anonymous Anonymous said...

It's interesting to compare the win probabilities of the Super Bowl in this paper with those computed by ProTrade on See also here and here.

Some of the disparity can be explained by the fact that we are comparing a real-time gambling market with a model based on historical averages, but I also believe it's much more difficult to determine accurate fine-grained win probability estimates on a play-by-play level in football than it is in baseball.

At Monday, February 12, 2007 11:18:00 PM, Blogger Phil Birnbaum said...

Thanks, Jim ... I hadn't seen that before.

The main difference seems to be that the market takes into account that the Colts were better than the Bears, while the ProTrade analysis assumes they were equal. (In the second link, ProTrade says the Hester TD return raised the Bears' chances from 50% to 70%, but the line had the Bears' original chance at much less than 50%.)

But it's still interesting that the Tradesports market had the effect of the TD at only 11%, while ProTrade had the effect at 20%.

At Wednesday, February 14, 2007 10:08:00 AM, Blogger Tangotiger said...

It would have to be 0.20.

It's easy enough to check... find all games where the point spread was exactly 7 points. For those games, figure out how often the favorite won the game. My guess is that it would be .700.

In the Super Bowl, we had the incredible luxury of a team spotting another team 7 points (with only a few meaningless seconds ticked away).

I can't access it from work, but maybe someone here can try it:

That's Andy's site. Pick a couple of NFL teams where the spread would have been around 7 points. Try a few until you get to something approaching 7. Report back the winning percentage for the favorite.

At Wednesday, February 14, 2007 10:10:00 AM, Blogger Tangotiger said...

By the way, the points to win converter I use is: w = .03*PTS

So, a team favored to win by 3 means they will win .590 times.

At Wednesday, February 14, 2007 10:24:00 AM, Blogger Phil Birnbaum said...


I found one game where the Bears were a 51.5% chance to beat Seattle by 8.5 or more, but 78.5% to beat them outright. That's 23% for 8-and-a-half points, which roughly corresponds to your metric.

But I think you're missing that the situation after the Chicago touchdown wasn't the same as a pick-em game. That's because at that point, Indianapolis was guaranteed one extra possession. If that possession is worth an average 3 points, that accounts perfectly for the difference between TradeSports' 11% and ProTrade's 20%.

At Wednesday, February 14, 2007 2:57:00 PM, Blogger Tangotiger said...

I don't see how one possession can possibly be worth 3 points. How many possessions are there in football? 15? Plus the possession isn't from some random spot on the field, but closer to their own 20 or 25. A possession is worth only 1 or 2 points.

However, I'm still skeptical on that, since after every touchdown, the opposing team will get the possession (i.e., a TD is 7 points minus whatever you want to assign for the "turnover").

This would imply then that after every major score, the win prob should change by .20, but changes by only .15 because of the loss of possession.

At Wednesday, February 14, 2007 3:15:00 PM, Blogger Phil Birnbaum said...

You're right -- 3 points is too high. If each team gets 15 possessions, and scores 20 points, that's only about 1.3 points. Adjust for the fact that the Colts were better than the Bears, and maybe you're up to 1.5 points. Which is exactly what you said.

So at .03 wins per point, that's about .045. So the Colts should have been about 54.5%.

The market had them at 57.25%.

Where does the difference come from? Probably my estimates are somewhat off -- real points-per-game and possessions-per-game data would help. Any other reasons?

(I don't think the fact that teams lose possession after scoring wouldn't affect this analysis.)

At Thursday, February 15, 2007 4:22:00 PM, Anonymous Anonymous said...

two thoughts on this:

1. the Romer paper (go for it on 4th down more often) has the value of receiving a kickoff at 0.62 points, so the value of indy's extra possession is even less a factor on the % prob win (maybe just 2%)

2. i think the difference is a matter of endgame coaching strategy: meaning that the incentive for indy to win by 8 is a lot lower than the incentive for indy to win by 1 after being down 7 points to start the game. That extra 5% probably comes from those situations where in the original case, late in the 4th quarter indy would sit on a 5 pt lead as opposed to the case of a 2 point deficit when they would kick the field goal

but still, the 57% seems a tad high, i would've been a seller

At Thursday, February 15, 2007 5:47:00 PM, Blogger Phil Birnbaum said...

1. You're right. I wonder why I keep thinking it's worth more? It just *feels* like it should be.

2. I think you're right again that late-game strategy is the difference. Never thought of that.

As for whether 57% is high, it would be interesting to find all games where the 7-pt underdog scored the first touchdown, and see how often they won. It still seems reasonable to me, but that's because I'm still fixating on that nearly-worthless extra possession.

At Thursday, February 15, 2007 10:54:00 PM, Anonymous Anonymous said...

FYI, the average NFL game has about 24 possessions, or 12 per team. The average number of points scored per possession is about 1.75.

At Friday, February 16, 2007 1:22:00 PM, Anonymous Anonymous said...

one thing to keep in mind in this case is that avg starting field position after a kickoff is own 26 or something like that. avg starting field position on all possessions is better than that, owing to the difference between the 0.62 # and Jim's # of 1.75...

The paper in general is very interesting -- a novel way to use win share concept in a more fluid sport. I wonder if there is a way to refine this for linemen? (as a tool for coaches, for example)

At Wednesday, March 07, 2007 11:43:00 AM, Anonymous Anonymous said...

Hi all,

Thank you for the nice comments. I've written a expanded draft that touches on many of them. It's available here: Superbowl Analysis.


Post a Comment

<< Home