Monday, July 27, 2009

Why EV Shot Quality Matters

A couple of months ago the Contratian Goaltender did a study that looked at goalies who switched teams and compared them with goalies who remained on the same team. He concluded that the effect of shot quality in the NHL was, on average, very small. More recently, Vic Ferrari expanded on this work with a similar study of his own and presented a similar case: shot quality does not have a big impact on save percentage at even strength.

On the one hand, these studies make sense. As much as teams will try not to give up good scoring opportunities, it's ultimately the player with the puck that will choose whether or not to shoot. If the chance of a goal is not sufficient, the player simply won't shoot the puck. The result ends up being that good defence tends to manifest itself in fewer shots reaching the goalie instead of many shots of inferior quality.

On the other hand, I think about what would happen if indeed I were a player in the NHL. It's not at all hard for me to imagine the results. A lot of losing my man. A lot of missed checks. A lot of getting beat one on one. Basically, a lot of scoring chances relative to total shots with me on the ice. Now, no one in the NHL is as bad at playing defense as I am (unless Rob Schremp makes it to the NHL) but it does seem rather intuitive that some teams would be significantly better at limiting the chance to shot ratio than others.

Even if the above isn't really the case I would still expect certain goaltenders to get lucky while others got unlucky. Even if, on average, shot quality is consistent, there ought to be one or two goalies that get particularly lucky or unlucky in terms of the number of scoring chances per shot faced. This is something that Vic pointed out when he looked at individual skaters scoring chance to Corsi ratio during the season. While the majority of the players had very similar Corsi and scoring chance rates, a few didn't. At the time of the study Marc Pouliot was particularly lucky and Eric Cole was particularly unlucky. I would expect to see something similar with regard to goaltenders.

I think that the work done recently by CG and Vic has shown that there is a lot less "Scott Reynolds on the Ice = High Quality Chances" in the NHL. But I don't think that they've demonstrated that shot quality does not exist in the NHL today. Even if shot quality is primarily luck (as it seems to be), that does not mean it doesn't exist or that it doesn't need to be taken into consideration when evaluating the past performance or predicting the future performance of a particular goaltender.

This brings me to one of the methods currently available for measuring shot quality (at even strength). Gabriel Desjardins is currently using a system that predicts goaltender shot quality based primarily on shot distance. Now, I think he's primarily measuring a lot of luck along with some terrible or great defence by specific teams. As such, I think it's quite likely that a lot of his work won't pass the traditional "smell test." In other words, because he's mostly measuring luck, New Jersey might allow very high quality shots while Tampa Bay might allow very low quality shots. As such, I decided on a different kind of smell test. I looked at the numbers for goalies who faced at least 250 shots in both the AHL and NHL in the same season. Because Desjardins' numbers only go back two seasons, I was only able to use the last two seasons worth of data. There are some obvious limitations here. Firstly, the AHL only publishes overall save percentage, so I won't be able to look at strictly EV results which would be preferable. Secondly, there are only 15 different goalies that meet the criteria so the average may be a little bit off of what it should be going from one league to the other. Thirdly, any observed differnece from one league to another may be the result of luck at the AHL level or at the NHL level or both. Nonetheless, and with those limitations in mind, these are the results:

In order for Desjardins' method to pass this (rather crude) smell test, I would expect that goalies who outperformed expectations at the NHL level to be ranked as facing "low" shot quality while goalies who underperformed at the NHL level to be described as facing "high" shot quality. My categories for shot quality are low (expected EVSv% is .915 or better), average (.915-.905) and high (.905 or worse). This is how various players performed:

The results here are not particularly encouraging but, as I said at the outset, there were a ton of factors other than EV shot distance that may have caused the discrepancies between AHL and NHL save percentages (goalies getting hot or cold for a period of time, the number of PP shots faced at the NHL or AHL level, elements of shot quality other than shot distance). One of the biggest problems with the way shot quality is measured now is the use of shot distance to begin with. I think the real key to shot quality analysis will be having accurate scoring chance data kept. If that happens, shot quality could probably be effectively measure using a ratio of scoring chances to shots on goal or simply bypassing the middle man and using a scoring chance percentage to evaluate goaltenders.


JLikens said...

Good post -- That's a creative way to test the model.

I did some analysis on shot quality data -- specifically, that contained in Alan Ryder's work at hockey analytics -- when I first started blogging back in the fall.

While both shot quality for and shot quality against correlated with shooting and save percentage (respectively), the correlations were not large (r=~0.4).

Interestingly, the metric was fairly repeatable across seasons at the team level (r=~0.6), even with the arena recording bias accounted for.

In light of the recent studies that have cast doubt on the importance of shot quality at EV (this one included), I think that the focus needs to shift to special teams factors in terms of explaining team-to-team variation in shot quality.

Personally, I think that penalty differential (which is repeatable at the team level) is one big factor.

Also, there's some evidence that shooting percentage on special teams is less subject to randomness than at EV (for example, PP S% is more reliable than EV S%, and Vic has shown that team factors are relevant with respect to PK SV%).

Scott Reynolds said...

Thanks JLikens. I agree that special teams is a very important place to look with regard to differences in shot quality. Teams that take fewer penalties should, of course, face easier shots.

With that said, I still think that EV shot quality exists, only that it largely results from luck and not skill. So even if we expect the shots to scoring chances ratio at EV to be very similar from team to team, it would still be useful to identify which goaltenders, in any particular season, had an unusually high or unusually low number of scoring chances to shots. And it's bound to happen. I'm just not sure that the current metrics are doing it that well. If we were able to count more scoring chance numbers, we could test this idea going forward.

Coach pb9617 said...

Hey Patrick,

Check your email...