Perhaps fighting penalties *don't* help NHL teams win

Earlier today, I wrote about a story in today's National Post that quoted a study saying major penalties help a team win. Alas, a look at the study's results show that isn't true.

Here's the paper again. It's by John Herald Heyne, Aju Fenn, and Stacey Brook, and it's called "NHL Team Production."

The authors try to estimate team standings points by running a regression on a number of offensive and defensive variables. I'll list them: goals allowed, assists, faceoffs won, faceoffs lost, penalty minutes, major penalties, even-strength goals, power-play goals, short-handed goals, shooting percentage, saves, and season.

The method suffers from the same problem that I previously wrote about here (in reviewing a paper also co-written by one of the same authors): namely, that team performance is determined directly by goals scored and goals allowed (assuming timing is random, as is normally assumed for baseball), and the other variables are expected to impact on goals, not on wins directly. For instance, if a team scores 300 goals and allows 280, why would you expect its winning percentage to depend on how many assists it accumulated in scoring those 300 goals? Or why would it matter how many faceoffs it lost on the way to giving up those 280 opposition scores?

Also, the study includes several sets of variables that measure almost the same thing. For instance, it includes team plus/minus. That statistic is exactly equivalent to even-strength goal differential multiplied by the average number of skaters on the ice (say, 4.6, to take 4-on-4s into account). Because of that, it measures almost the same thing as even strength goal differential.

In his "Win Probabilities" study (see page 3), Alan Ryder shows that historically, each NHL goal has been worth .1457 wins. If the difference between a win and a loss equals two points, a goal is .2904 points. But from 2000-2004, the seasons the new study covers, teams who lost in overtime got one standings point. That happened in about 12% of losses. And so a win is worth only 1.88 points more than a loss, not 2.00 points. Adjusting for that turns the .2904 into .2730.

That is, Ryder's work shows that

Points = league average + .2730 (GF – GA)

How does this paper's results compare to these? Let's concentrate, for now, only on those variables that have to do with goals.

The paper gives the following coefficients:

-0.2047 goals allowed
+0.0935 assists for
+0.0889 even strength goals for
+0.2192 power play goals for
+0.2942 shorthanded goals for
+0.0279 plus/minus

Now, according to this website, there is about 1.55 assists per goal. From a quick check of nhl.com, I estimate there are 4.6 skaters on the ice for the average even-strength goal. Also, a lazy estimate is that 60% of goals are scored even strength, 37% on the power play, and 3% shorthanded. (Can't manage to get a permalink – go to nhl.com and do "Stats," then choose "report view – goals for" under "team comparison reports.")

From all that, we can do a bit of algebra and reduce the above to goals scored and allowed. For instance, .0935(assists) = .0935 * 1.55 (goals) = .145(goals). After all the simplification, if I've done it right, the six coefficients above collapse down to:

+0.2591 goals for
-0.2817 goals allowed

which compares well to the previous Ryder numbers:

+0.2730 goals for
-0.2730 goals allowed

So far, the study has only duplicated Ryder's results, using a more complicated method.

And so the significance of other variables puzzles me. After looking at goals for and goals allowed, we'd expect none of the other variables to affect winning. After all, if you lose 4-3, it shouldn't matter if you took ten penalties or none – any power play goals against are already accounted for in the score.

In that case, why is the coefficient for penalties significant? Or the coefficient for faceoffs won and lost? Or for major penalties? Or for shooting percentage and saves?

I'm at a loss. The only thing I can think of is this: after accounting for goals for and against, all that's left is timing of goals. These other variables may have to do with timing. For instance, more overtime games equals more points, even for identical goal differentials. The more overtime games, the more minutes; the more minutes, the more faceoffs. So that's one way faceoffs could affect points. However, the same would be true for faceoffs lost, but that coefficient goes the other way! So that theory is out.

Another theory is that not every team gets 1.55 assists per goal. Some teams might get only, say, 1.45 assists, because of their style of play. That would tend to underestimate their points. In that case, other determinants of the team's quality would tend to fill the gap. That would be faceoff performance and shooting percentage for the offence, and faceoff and goalie performance for the defense. That could be the answer, but it doesn't have to be.

Regardless, most of the authors' conclusions aren't justified by the data. For instance, they write that

"PIM implies that a team … is playing a man down, which is a huge disadvantage. Therefore, the more penalties a team takes, the harder it is going to be for them to win games."

This cannot be the correct explanation, for reasons already stated. In the regression, the coefficient for PIM implies that total team points go down with PIM, but *only when all other variables are held constant*. Since goals against and shorthanded goals for are two of those other variables, and those have to be held the same, the regression actually implies that a team will gather fewer standings points *when a penalty is killed without any goals being scored*. This is much harder to explain. The same is true for faceoffs, and save percentage, and all the other variables.

Also, as the National Post article highlighted, the authors say that the more major penalties, the better the team's position in the standings. They conclude that major penalties spark the team.

But I don't think that's true – they simply misinterpret the results of the regression again. The regression shows that major penalties result in more points *keeping all other variables constant*. PIM is one of those variables. To keep total penalty minutes constant while increasing major penalties by one, you have to eliminate 2.5 additional minor penalties. That's a trade where you gain a penalty that is likely offset by an opponent's fighting penalty, and you lose five minutes of being shorthanded. Obviously, that's a good thing, and that's why the coefficient is negative.

To see the effect of an additional major alone, you want to both (a) add one major, and (b) add five minutes to the PIM total. If you use the coefficients to compute the sum of both changes, the effect is now very close to zero, and probably not even statistically signficant. Thus, there is no evidence at all that major penalties help the team.

I suspect most of the the other results in the paper are also artifacts, due to the use of proxy variables for goals for and against. I'd bet that if you just used plain old Goals For and Goals Allowed, instead of all those other proxies like assists and PPG and plus/minus, you'd get all the other variables suddenly becoming a lot less significant.

The bottom line is that, unfortunately, because of the way the study was structured, none of the results is convincing – and the featured conclusion, the one about major penalties, is not confirmed by the evidence at all.