Why EV Shot Quality Matters

A couple of months ago the Contratian Goaltender did a study that looked at goalies who switched teams and compared them with goalies who remained on the same team. He concluded that the effect of shot quality in the NHL was, on average, very small. More recently, Vic Ferrari expanded on this work with a similar study of his own and presented a similar case: shot quality does not have a big impact on save percentage at even strength.

On the one hand, these studies make sense. As much as teams will try not to give up good scoring opportunities, it's ultimately the player with the puck that will choose whether or not to shoot. If the chance of a goal is not sufficient, the player simply won't shoot the puck. The result ends up being that good defence tends to manifest itself in fewer shots reaching the goalie instead of many shots of inferior quality.

On the other hand, I think about what would happen if indeed I were a player in the NHL. It's not at all hard for me to imagine the results. A lot of losing my man. A lot of missed checks. A lot of getting beat one on one. Basically, a lot of scoring chances relative to total shots with me on the ice. Now, no one in the NHL is as bad at playing defense as I am (unless Rob Schremp makes it to the NHL) but it does seem rather intuitive that some teams would be significantly better at limiting the chance to shot ratio than others.

Even if the above isn't really the case I would still expect certain goaltenders to get lucky while others got unlucky. Even if, on average, shot quality is consistent, there ought to be one or two goalies that get particularly lucky or unlucky in terms of the number of scoring chances per shot faced. This is something that Vic pointed out when he looked at individual skaters scoring chance to Corsi ratio during the season. While the majority of the players had very similar Corsi and scoring chance rates, a few didn't. At the time of the study Marc Pouliot was particularly lucky and Eric Cole was particularly unlucky. I would expect to see something similar with regard to goaltenders.

I think that the work done recently by CG and Vic has shown that there is a lot less "Scott Reynolds on the Ice = High Quality Chances" in the NHL. But I don't think that they've demonstrated that shot quality does not exist in the NHL today. Even if shot quality is primarily luck (as it seems to be), that does not mean it doesn't exist or that it doesn't need to be taken into consideration when evaluating the past performance or predicting the future performance of a particular goaltender.

This brings me to one of the methods currently available for measuring shot quality (at even strength). Gabriel Desjardins is currently using a system that predicts goaltender shot quality based primarily on shot distance. Now, I think he's primarily measuring a lot of luck along with some terrible or great defence by specific teams. As such, I think it's quite likely that a lot of his work won't pass the traditional "smell test." In other words, because he's mostly measuring luck, New Jersey might allow very high quality shots while Tampa Bay might allow very low quality shots. As such, I decided on a different kind of smell test. I looked at the numbers for goalies who faced at least 250 shots in both the AHL and NHL in the same season. Because Desjardins' numbers only go back two seasons, I was only able to use the last two seasons worth of data. There are some obvious limitations here. Firstly, the AHL only publishes overall save percentage, so I won't be able to look at strictly EV results which would be preferable. Secondly, there are only 15 different goalies that meet the criteria so the average may be a little bit off of what it should be going from one league to the other. Thirdly, any observed differnece from one league to another may be the result of luck at the AHL level or at the NHL level or both. Nonetheless, and with those limitations in mind, these are the results:

In order for Desjardins' method to pass this (rather crude) smell test, I would expect that goalies who outperformed expectations at the NHL level to be ranked as facing "low" shot quality while goalies who underperformed at the NHL level to be described as facing "high" shot quality. My categories for shot quality are low (expected EVSv% is .915 or better), average (.915-.905) and high (.905 or worse). This is how various players performed:

The results here are not particularly encouraging but, as I said at the outset, there were a ton of factors other than EV shot distance that may have caused the discrepancies between AHL and NHL save percentages (goalies getting hot or cold for a period of time, the number of PP shots faced at the NHL or AHL level, elements of shot quality other than shot distance). One of the biggest problems with the way shot quality is measured now is the use of shot distance to begin with. I think the real key to shot quality analysis will be having accurate scoring chance data kept. If that happens, shot quality could probably be effectively measure using a ratio of scoring chances to shots on goal or simply bypassing the middle man and using a scoring chance percentage to evaluate goaltenders.