An NFL field goal "choking" study

In a comment to last week's post on choking in basketball, commenter Jim A. posted a link to this analysis of choking in football. It comes from a 1998 issue of "Chance" magazine, a publication of the American Statistical Association.

The paper comes to the conclusion that field-goal kickers do indeed choke under pressure.

Authors Christopher R. Bilder and Thomas M. Loughlin looked at every place kick (field goal or extra point) in the 1995 NFL season. They ran a (logit) regression to predict the probability of making the field goal, based on a bunch of criteria, like distance, altitude, wind, and so on. They designated as "clutch" all those attempts that, if successful, would have resulted in a change of lead.

I assume that kicks starting or resulting in a tie count as "change of lead" -- if so, then clutch kicks are those where the kicking team is behind by 0 to 3 points.

The authors narrowed their model down by eliminating variables that didn't appear to explain the results much. The final model had only four variables:

-- clutch
-- whether it was an extra point or a field goal
-- distance
-- distance * (dummy variable for wind > 15mph)

It turned out that clutch kicks were significantly less successful than non-clutch kicks, by an odds factor of 0.72. If, in a non-clutch situation, your odds of making a field goal were 5.45:1 (which works out to 84.5%, the overall 2008 NFL average), then, to get your clutch odds, you multiply 5.45 by 0.72. And so your corresponding odds in a clutch situation would be 3.93:1 (80%).

It's a small drop -- less than five percentage points overall -- but statistically significant nonetheless.

----

Now, to ask the usual question (albeit one the paper doesn't ask): could there be something going on other than choking? Some possibilities:

1. All attempts the study consideres "clutch" are, by definition, made by a team that's either tied or behind in the score. Wouldn't that be selection bias, since the "clutch" sample would be disproportionately comprised of teams who are, overall, a bit worse than average? Worse teams would have worse field goal kickers, which might explain the dropoff.

The paper ignores that possibility, explicitly assuming that all FG kickers are alike:

"One difference there is no difference between placekickers is that NFL-caliber placekickers are often thought of as "interchangeable parts" by teams. NFL teams regularly allow their free-agent placekickers, who are demanding more money, to leave for other teams because other placekickers are available."

That makes no sense: free-agent quarterbacks leave for other teams too, but that doesn't mean all are equal. Besides, if all placekickers were the same, those "other teams" wouldn't sign them either.

So I wonder if what's really going on is that the kickers in "clutch" situations are simply not as good as the kickers in other situations. The discrepancy seems pretty large, though, so I wonder if that effect would be enough to explain the five percentage point difference.

2. One of the other factors the authors considered was time left on the clock. It turned out to be significant, originally, but, for some reason, it was left out of the final regression.

But clutch kicks would be more likely to occur with less time on the clock. Behind by 3 points with two seconds remaining, a team would try the field goal. Behind by 5 points with two seconds remaining, the team would try for a touchdown instead.

Why does that matter? Maybe because, if there's lots of time on the clock and the team isn't forced to kick, they might not try it if conditions are unfavorable (into the wind, for instance). But with time running out, they'd have to give it a shot even if conditions were less favorable. So time-constrained kicks would have a lower success rate for reasons other than "choking".

3. The assumption in the regression is that all the coefficients are multiplicative. Perhaps that's not completely correct.

In low-wind conditions, the regression found that every yard closer to the goalposts changes your success odds to 108% of their original. And clutch changes your odds to 72% of the original. So, according to the model, going one yard closer but in a clutch situation should change your odds to 108% of 72%, or 78%.

But what if that's not the case? What if multiplying isn't strictly correct? Suppose that "clutch" makes the holder more likely to fumble the snap, by a fixed amount, and there's also an effect on the kicker that's proportional to the final probability. In that case, multiplying the two effects wouldn't be strictly correct -- only an approximation. And, therefore, the regression would give biased estimates for the coefficients. If the "distance" coefficient is biased too high, but "clutch" kicks happen to be for longer distances, that would explain a higher-than-expected failure rate.

4. The paper included kicks for extra points (PATs), which are made some 99% of the time. And there were lots of PAT attempts in the sample, even more than field goal attempts. At first I thought those could confuse the results. If there were no clutch factor, you'd expect exactly one clutch PAT to be missed. What if, by random chance, there were two instead? That would imply a large odds ratio factor for the PATs, based only on one extra miss, which wouldn't be statistically significant at all.

Could that screw up the overall results? I did a little check, and I don't think it could. I think the near-100% conversion rate for PATs is pretty much ignored by the logistic regression. But I'm not totally certain of that, so I thought I'd mention it here anyway.

5. The authors found that the odds of making a PAT were very much higher than the odds of making a field goal of exactly the same distance -- an odds ratio of 3.52. That means that if the odds of making a PAT are 100:1, the odds of making the same field goal are only 28:1.

What could be causing that difference? It could be a problem with the model, or it could be that there is indeed something different about a PAT attempt.

What could be different about a PAT attempt? Well, perhaps for an FG attempt, both teams are trying harder to avoid taking a penalty. For the defensive team, a penalty on fourth down could give the kicking team enough yards for a first down, which could turn the FG into a TD. For the offensive team, a penalty might move them out of field goal range completely. Those situations don't apply when kicking a PAT.

In clutch situations, the incentives would be different still. Suppose it's a tie game with one second left on the clock, and a 25-yard attempt coming. An offensive 10-yard penalty would hurt a fair bit: it would turn a 90% kick into an 80% kick, say. A defensive penalty wouldn't hurt as much, though: it might only turn the 90% kick into a 95% kick.

Normally, a defensive penalty hurts more than an offensive penalty: it could create a first down, rather than just a more difficult kick. But in late-game situations, an offensive penalty hurts more than a defensive penalty: it lowers the success rate by more than a defensive penalty raises it.

Therefore, in a clutch situation, could it be that FGs are intrinsically more difficult, just because the offense has to play more conservatively, but the defense can play more aggressively?

Evaluating field-goal kickers

In the third quarter of the Super Bowl, the Colts, up by a point, sent in kicker Matt Stover to attempt a 51-yard field goal on fourth-and-11. He missed.

Was the field goal attempt bad strategy? As you would guess, there was a fair amount of second guessing. Carl Bialik, the Wall Street Journal's "Numbers Guy," pointed out one of the more bizarre criticism -- that Stover shouldn't have been sent out because he was too old.

That's bizarre. Jamie Moyer was 46 last year, four years older than Stover. He got blown out a few times, but, after a bad game, you didn't hear journalists and bloggers argue that he shouldn't have pitched because he was too old. Everyone understands that pitchers sometimes have bad games, and it's silly to harp on a single performance. Moreover, age is irrelevant -- what counts is performance. A bad pitcher or bad kicker is just a bad pitcher or bad kicker, whether he's 25 or 45.

My feeling is that the reaction to Stover's miss, the second-guessing and search for causes, is partly a result of the culture of NFL football. In hockey, a player fans on a goalmouth pass, or misses an empty net, and it's unfortunate, but it's part of the game, and there'll be other opportunities later. In baseball, strikeouts happen all the time, even (perhaps especially) to the best players, and they're considered routine.

But in my (limited) experience of watching football, it's different. A dropped pass or a missed block is taken very, very seriously; there's a presumption that every player is expected to make every play, and that every miss isn't just part of the game, but a failure of the player to do his job. A shortstop bobbles a grounder, and life goes on. A receiver lets a pass bounce of his chest, and the announcers are all over him for his lack of concentration and letting his team down.

If you look at the statistics, I'd bet that missed catches are much more common than errors on ground balls. But in baseball, it's just understood that even the best infielders are going to wind up with a few errors over the course of a season. In the NFL, at least in the heat of the moment, it seems like the only acceptable level of performance is 100%.

And that just seems wrong to me. League-wide, a 51-yard field goal attempt is successful only about 55% of the time. Admittedly, Stover has been worse than that, but still, when you try something that succeeds only about half the time, and then it winds up failing, all that second-guessing isn't really called for. Kicking a 51-yard field goal has about the same success rate as making two consecutive foul throws. Has any player in the history of the NBA missed a foul, and then criticized for missing it because he was too old?

I just don't get it. Is it just an NFL perfectionism thing?

-----

In any case, another thing that strikes me about the NFL is how kickers are evaluated on such small sample sizes. A 30-yard field goal is successful about 90 percent of the time. A kicker misses a couple of those, and suddenly he's a goat, and his job is in jeopardy. Why is that? The chance of a kicker missing two out of three such kicks is about 1 in 37. That's about the same chance of a .300 hitter going 0-for-10. If Albert Pujols did that, everyone would know, from his record, that he's just having a little slump. But if a field goal kicker misses a couple, where's the perspective? It seems like, fairly often, a high-profile miss and a guy loses his job.

I suppose it's possible to think up a situation where that makes sense. If a kicker can go along for years and years, then suddenly lose his abilities overnight, then it might indeed make sense to worry when he uncharacteristically misses a couple. You could easily think up a mathematical model where the team's best strategy is to fire a any kicker who misses two out of three easy kicks. But that doesn't seem realistic to me, even though it's theoretically possible.

Maybe it's that we're not keeping the right statistics. If one kicker is 90%, and another is 85%, well, maybe the first guy kicked for shorter distances, which is why his success rate is higher. If the second kicker went 2-for-4 from 55 yards, that's actually an excellent performance, but it reduces his overall rate. So the success rate may mean very little, especially considering that kickers have fairly similar success rates and that the sample sizes are small.

Why not start by keeping a record of how the kicker does relative to average? If a player makes a 30-yarder, one that normally gets made 90% of the time, he beat the average by 0.1 kicks. If he misses, he's below average by 0.9. Add up all his kicks over all his seasons, and you'd get a reasonably accurate picture of how good he is, at least if he has a reasonably long career. What's Stover's record in that regard? I don't know.

But even so, there are more basic stats we can look at. I can go to his Pro Football Reference page, and see that Stover had a pretty good season, it looks like. He went 9 for 11. He missed one between 30 and 39 yards, and he missed his only attempt of 50+ yards. From 30 yards or less, he's missed only one kick in his last ten seasons. That seems like a reasonable record ... there's nothing there that suggests that there's something wrong when he misses a single 51-yard try in the Super Bowl.

And what I don't understand is why teams don't have a better idea how good a kicker is. If there are only 35 kicks a season on which to evaluate him ... well, how about evaluating him in practice? Get him to kick a few every week, and keep track of how he does. Sure, practice conditions aren't the same as game conditions, but they're better than nothing, aren't they? If you have one guy who makes 75% of 50-yarders on an empty field on a weekday afternoon, and another guy who only makes 50%, isn't that still useful information when you're trying to decide who you're going to sign for next season?

I don't know anything about actually playing the game of football, so this could be completely wrong. But wouldn't it be a good idea to audition kickers by making them kick and seeing how well they do? You could try them at various distances, with good holds and bad holds, with good blockers and bad blockers. I don't know how long it takes to recover after a kick, but if it takes, say, five minutes, you could still get a guy to kick a season's worth of balls in an afternoon. If you think your guy is losing it, can't you just test him out? Why wait until he misses an important kick, then decide he's washed up on that very little evidence?

And here's another idea: is there any predictive value to where the kick goes, rather than just whether it's good or not? I'd think that a boot right down the middle is better than one that just sneaks inside the goal post. If you had tapes of all the kicks, you could give every guy an accuracy score, and see if that has any correlation to future field-goal percentages.

There's gotta be more information out there than "he went 31 for 35 last year." And if you don't have it, it seems like it would be pretty cheap to get.