NFL coaching decisions cost 0.73 wins per team

By making bad decisions on fourth down, NFL coaches are sacrificing almost three-quarters of a win per season.  That's from Matt Meiselman, who crunched some numbers with Brian Burke and posted on Brian's site.  

In 2012, the Cleveland Browns were the "worst", sacrificing a probabilistic 1.02 wins by making 42 "wrong" decisions.  The Packers were the least "worst", giving up only around half an expected win.  I would have expected New England to represent well in this measure, since Bill Belichick has often been touted as a sabermetrically-savvy coach, but the Patriots were only a bit better than average, at 0.6.

Those numbers are based on expectations for an average team.  It's quite likely that they overstate the cost, if the probabilities vary a lot based on quality of team.  My suspicion is that the quality effect is pretty small, because the spread of "wrongness" is so narrow.  In fact, the spread suggests to me that coaches are generally following the same "book" of conventional wisdom, with individual differences being pretty minor.  

The article implies that the losses are due to coaches generally being risk-averse, but doesn't give the numbers.  Is *every* bad decision caused by playing it too safe?  95 percent?  50 percent?  I don't know the answer.  My gut says ... I dunno, I'll guess 92 percent of cases are when the coach should have gone for it and didn't, instead of when he shouldn't have and did.  Matt/Brian, if you're reading this, am I close?  

-------

I'm shocked at how high the numbers are.  Losing 0.73 wins is huge, considering that the difference between a playoff team and an average team is only, what, two games out of sixteen?  

I'd bet that's by far the biggest in-game coaching factor in any major sport (leaving out the decision of who plays).  In baseball, it's the equivalent of 4.6 games per 162, which is about the same percentage of distance to the playoffs.  But I can't see that MLB managers would have anything near that much influence.

-------

At the Sloan convention, there was a lot of talk about how analytics people can increase their influence ... like, what to do or say to get coaches and management to listen to us numbers geeks.  

But, in this case, I think there's an easier path.  Any time there's a fourth-down decision, the TV broadcast could put the probabilities on the screen.  Like, for instance, "teams that go for it should be expected to win 48% of the time, while teams that punt should win only 30% of the time."  That's simple enough for viewers to understand ... which means, fans will be second-guessing the coach based on the numbers, rather than random feelings.  It would still be fun to discuss ... the ESPN guys could argue about why the percentages don't apply in this particular case, because the offense is poor, or the defense has momentum, or whatever.

In any case, it would change the nature of the second-guessing.  Right now, a coach may attract 1 pound of criticism when he plays it by the book, and 5 pounds when he goes for it.  With the probabilities on the screen, maybe the 1:5 ratio will immediately change to 1:3 or something, and then, over time, as the stats gain acceptance, all the way to 1:1.  Then, you've reached the tipping point where the coaches' incentives change.  Now, they take more flak, and sacrifice more job security, when they *don't* go by the percentages.  It wouldn't take long, I suspect, for things to change after that.

Mid-season coaching changes are correlated with disappointing records

The new issue of JQAS is out. Its second article, by three Swedish researchers, is called "Coach Succession and Team Performance: The Impact of Ability and Timing – Swedish Ice Hockey Data." The idea is that you can use a regression to try to figure out whether a coaching change helps or hurts the team. Alas, I think the authors are confusing cause and effect.

Their regression predicts this year's winning percentage based on a bunch of factors: last year's winning percentage, the coach's career winning record, the number of games coached, whether there was a coaching change, and whether that coaching change came in mid-season.

They find that mid-season coaching changes are associated with bad records, and conclude that it takes time for the team to adapt to the new coach. But isn't it more likely that the bad record led to the coaching change, rather than the other way around? Teams don't fire their coach when he's having a good season.

Actually, it's possible that the current season's record is only for the new coach; but, still, you'd expect a bad record in those cases. The previous coach was fired when the team underperformed, and (even considering regression to the mean) the underperformance is partly intrinsic to the team's talent. So the performance should still be similarly sub-par with the new coach, even if the coaching change didn't make any difference at all.

Another result finds that within teams, there's a high correlation between the coach's previous record and the current record – for all six teams studied, it was between .72 and .77. I didn't really expect it to be that high. You could assume that teams go through stages of badness and goodness, and each is likely to be associated with one particular coach. But I didn't think that would bring the correlation up as high as it did. Then again, there are only 21 seasons in the study for each team.

How do coaches improve their teams?

How much do coaches improve player performance? According to Dave Berri, of "The Wages of Wins," not much at all. According to a study coauthored by Berri (as quoted in this Ryan McCarthy article in Slate), only eight out of 19 coaches studied had any "statistically discernible" effect on performance. And only the difference between the very best and very worst coaches was statistically significant.

I haven't seen the numbers, because the study is unpublished. However, at a 5% significance level, you'd expect about one out of the nineteen coaches to be significantly different from zero, so the fact that the top coach was significantly better than the bottom coach is to be expected.

Anyway, the study quotes a Dan Oliver study as coming to a different conclusion. That analysis is in Oliver's book "Basketball on Paper"; I, um, haven't got to that chapter yet, but I really should do that soon. McCarthy says that Oliver shows

Coaches like Phil Jackson can be worth up to an additional 12 wins per year.

I gotta read that chapter: 12 wins sounds like a huge number, and I'm now very curious.

In any case, my gut leans to Berri's finding, that coaches don't affect a player's performance all that much. But, then again, I've always thought that there's no way a baseball manager could consistently cause a team to beat its Runs Created estimate, but some managers do appear to be able to do that. So I may be wrong.

My thinking is that coaches and managers *can* make a big difference, but that comes about in the choice of who to play. I remember back in the early 1980s, when Tony Fernandez, by all accounts ready for the major leagues, sat in the minors while the Blue Jays stuck with the mediocre Alfredo Griffin. Glancing at the Baseball Reference pages for those two guys, we find that, in 1984, Griffin had –3.2 batting wins, while Fernandez had –0.6. So if manager Bobby Cox had played Tony and benched Alfredo, the Jays would have won almost three extra games – which is huge. (It wouldn't have affected that particular pennant race – the Jays won 89 games in 1984, but that was the year the Tigers ran away with it at 104-58.)

I'm sure you can think of other examples ... I remember Bill James once writing that Dick Williams had a perverse fascination with Rodney Scott, keeping him on the roster when he obviously wasn't of major-league caliber. And in his Managers book, James also called Cito Gaston "inert," because he used his bench so little. However, Cito did win two consecutive World Series. And when you think about it, why would you bench Roberto Alomar, even for a few games a year, to play Luis Sojo? Unless you think the days off would *really* help Alomar, you're just throwing away runs.

What I'd really like to see is a study that somehow figures out what managers made the best use of the talent they had available. I'm sure every manager has, at some point, played a mediocre established player instead of an unknown but more-talented one. But which ones do it the most? Who has a penchant for leaving Tony Fernandezes languishing while Alfredo Griffins get playing time? Who doesn't bother giving 150 plate appearances to some crappy 35-year-old first baseman with no power? Who knows which players are good enough to be given a shot?

It would be a tough study to do: you'd have to estimate what all the bench players were capable of, figure out who was in the minors and how good they were, and compare all those possibilities to the guys who actually played. But I'd bet that if you found a way to do that, you'd discover some managers are legitimately worth an extra game or two a year, just from their personnel decisions.

Hat tip: Carl Bialik