An NFL field goal "choking" study

In a comment to last week's post on choking in basketball, commenter Jim A. posted a link to this analysis of choking in football. It comes from a 1998 issue of "Chance" magazine, a publication of the American Statistical Association.

The paper comes to the conclusion that field-goal kickers do indeed choke under pressure.

Authors Christopher R. Bilder and Thomas M. Loughlin looked at every place kick (field goal or extra point) in the 1995 NFL season. They ran a (logit) regression to predict the probability of making the field goal, based on a bunch of criteria, like distance, altitude, wind, and so on. They designated as "clutch" all those attempts that, if successful, would have resulted in a change of lead.

I assume that kicks starting or resulting in a tie count as "change of lead" -- if so, then clutch kicks are those where the kicking team is behind by 0 to 3 points.

The authors narrowed their model down by eliminating variables that didn't appear to explain the results much. The final model had only four variables:

-- clutch
-- whether it was an extra point or a field goal
-- distance
-- distance * (dummy variable for wind > 15mph)

It turned out that clutch kicks were significantly less successful than non-clutch kicks, by an odds factor of 0.72. If, in a non-clutch situation, your odds of making a field goal were 5.45:1 (which works out to 84.5%, the overall 2008 NFL average), then, to get your clutch odds, you multiply 5.45 by 0.72. And so your corresponding odds in a clutch situation would be 3.93:1 (80%).

It's a small drop -- less than five percentage points overall -- but statistically significant nonetheless.

----

Now, to ask the usual question (albeit one the paper doesn't ask): could there be something going on other than choking? Some possibilities:

1. All attempts the study consideres "clutch" are, by definition, made by a team that's either tied or behind in the score. Wouldn't that be selection bias, since the "clutch" sample would be disproportionately comprised of teams who are, overall, a bit worse than average? Worse teams would have worse field goal kickers, which might explain the dropoff.

The paper ignores that possibility, explicitly assuming that all FG kickers are alike:

"One difference there is no difference between placekickers is that NFL-caliber placekickers are often thought of as "interchangeable parts" by teams. NFL teams regularly allow their free-agent placekickers, who are demanding more money, to leave for other teams because other placekickers are available."

That makes no sense: free-agent quarterbacks leave for other teams too, but that doesn't mean all are equal. Besides, if all placekickers were the same, those "other teams" wouldn't sign them either.

So I wonder if what's really going on is that the kickers in "clutch" situations are simply not as good as the kickers in other situations. The discrepancy seems pretty large, though, so I wonder if that effect would be enough to explain the five percentage point difference.

2. One of the other factors the authors considered was time left on the clock. It turned out to be significant, originally, but, for some reason, it was left out of the final regression.

But clutch kicks would be more likely to occur with less time on the clock. Behind by 3 points with two seconds remaining, a team would try the field goal. Behind by 5 points with two seconds remaining, the team would try for a touchdown instead.

Why does that matter? Maybe because, if there's lots of time on the clock and the team isn't forced to kick, they might not try it if conditions are unfavorable (into the wind, for instance). But with time running out, they'd have to give it a shot even if conditions were less favorable. So time-constrained kicks would have a lower success rate for reasons other than "choking".

3. The assumption in the regression is that all the coefficients are multiplicative. Perhaps that's not completely correct.

In low-wind conditions, the regression found that every yard closer to the goalposts changes your success odds to 108% of their original. And clutch changes your odds to 72% of the original. So, according to the model, going one yard closer but in a clutch situation should change your odds to 108% of 72%, or 78%.

But what if that's not the case? What if multiplying isn't strictly correct? Suppose that "clutch" makes the holder more likely to fumble the snap, by a fixed amount, and there's also an effect on the kicker that's proportional to the final probability. In that case, multiplying the two effects wouldn't be strictly correct -- only an approximation. And, therefore, the regression would give biased estimates for the coefficients. If the "distance" coefficient is biased too high, but "clutch" kicks happen to be for longer distances, that would explain a higher-than-expected failure rate.

4. The paper included kicks for extra points (PATs), which are made some 99% of the time. And there were lots of PAT attempts in the sample, even more than field goal attempts. At first I thought those could confuse the results. If there were no clutch factor, you'd expect exactly one clutch PAT to be missed. What if, by random chance, there were two instead? That would imply a large odds ratio factor for the PATs, based only on one extra miss, which wouldn't be statistically significant at all.

Could that screw up the overall results? I did a little check, and I don't think it could. I think the near-100% conversion rate for PATs is pretty much ignored by the logistic regression. But I'm not totally certain of that, so I thought I'd mention it here anyway.

5. The authors found that the odds of making a PAT were very much higher than the odds of making a field goal of exactly the same distance -- an odds ratio of 3.52. That means that if the odds of making a PAT are 100:1, the odds of making the same field goal are only 28:1.

What could be causing that difference? It could be a problem with the model, or it could be that there is indeed something different about a PAT attempt.

What could be different about a PAT attempt? Well, perhaps for an FG attempt, both teams are trying harder to avoid taking a penalty. For the defensive team, a penalty on fourth down could give the kicking team enough yards for a first down, which could turn the FG into a TD. For the offensive team, a penalty might move them out of field goal range completely. Those situations don't apply when kicking a PAT.

In clutch situations, the incentives would be different still. Suppose it's a tie game with one second left on the clock, and a 25-yard attempt coming. An offensive 10-yard penalty would hurt a fair bit: it would turn a 90% kick into an 80% kick, say. A defensive penalty wouldn't hurt as much, though: it might only turn the 90% kick into a 95% kick.

Normally, a defensive penalty hurts more than an offensive penalty: it could create a first down, rather than just a more difficult kick. But in late-game situations, an offensive penalty hurts more than a defensive penalty: it lowers the success rate by more than a defensive penalty raises it.

Therefore, in a clutch situation, could it be that FGs are intrinsically more difficult, just because the offense has to play more conservatively, but the defense can play more aggressively?