Wednesday, March 18, 2009

Player Evaluation at the Deadline

The trade deadline has come and gone and there has been much analysis of each individual trade around the internet. I've decided to add my voice to the fray on a more macro scale, namely, to look at one way the teams that made these trades may have been evaluating players. A popular topic around the Oilogosphere is the dichotomy between "saw him good" scouting and scounting via evidence provided by statistical analysis. Jonathan Willis recently did a post on the Edmonton Oilers recent aquisition of Patrick O'Sullivan and speculated that one reason that the Oilers may have liked him is his particularly good play against the Oilers. O'Sullivan's boxcar numbers (goals, assists and points) were much better against the Oilers than his performance against the rest of the league. Is this significant? In an attempt to answer that question I looked at all of the NHL players who were traded at the deadline (with the exception of Erik Cole since his circumstances are quite unique) and compared their performance against the team that had acquired them with their performance against the rest of the league over the last two NHL seasons. I believe this is the same method that Jonathan used earlier. The following table is organized by the difference in the player's points per game between these two groups:

What can be concluded from this? First of all, a couple of caveats: these are all small sample sizes so teams shouldn't really be using these games to decide on the merits of a given player. Similarly, it may be difficult for us, as observers, to judge how much this data is getting used. In addition, not all of these players were obtained to play in an offensive role so these statistics probably aren't helpful for judging whether or not San Jose picked up Travis Moen because he played well against them.

With those caveats in mind these statistics look pretty random to me. On the whole I think it's doubtful that NHL teams as a whole prioritize these games in their overall evaluation. They may play a part, but I doubt that it's a dominant role. A good team to illustrate this is Phoenix who obtained a couple of players near the top of this list (Lombardi and Prust) and a couple of players near the bottom (Upshall and Dawes). Anaheim did this as well with their aquisitions of Wisniewski and Christensen. Other teams that made significant aquisitions fall anywhere from the bottom of the list (Moore, Morris), to the middle (Jokinen, Antropov) and of course to the top (O'Sullivan, Eminger). It's perhaps true that certain teams (like Edmonton or Florida) do depend heavily on how a player has performed against them in the past but I think it's more likely that O'Sullivan and Eminger's overperformance against their new teams is just the luck of the draw (some are bound to overperform and some underperform, especially given the small sample size).

This does not, of course, answer how much value is placed on statistical analysis and how much on first-hand scouting reports but it does hopefully provide some insight into how teams are evaluating players.


Jonathan Willis said...

Nice post Scott! I had planned to do something similar but hadn't gotten around to it just yet.

Just a counter-point: I took those players and put them in a spreadsheet and eliminated guys who had <3 games against their new team. It makes quite a difference.

Okay, a second counter-point: I wonder if plus/minus would alter things?

Okay, okay, a third counter-point: I don't imagine all NHL teams employ this particular method, so perhaps what I should do is look at the record of GM's year-over-year and see if any of them consistently do.

Scott said...

With regard to the first point, if you just take guys with 3 games, 6 outperformed and 11 underperformed their exepected performance so I don't think that makes a huge difference (although admittedly, most of the guys that underperformed didn't do so significantly).

Yeah, looking at plus/minus might be helpful. I thought about it halfway through but I didn't really want to go back and do all those guys over again so I just left it out, but it would probably be instructive to check on.

The third point is the really interesting one I think. We could check on teams like Florida, Edmonton and maybe even Phoenix to see if they consistently buy on overperformers.

Cory said...

I think that the head to head matchups are a big factor in the bias, Scott.

For me personally, I see Legwand play against the Oiler's best players usually, takes key draws, kills penalties, etc. Then when I look at his scoring history I just assume that he tallied those numbers playing that type of ice time against everyone else in the league too.

If it turns out that he hasn't ... then I'm overvaluing the guy.

Scott said...

I think that you're right when it comes to fans' perceptions Cory. A lot of fans don't watch many games involving exclusively other teams and consequently have their perceptions coloured by a player's performance against their favourite team (I know this is true of me). That said, NHL management teams employ pro scouts and, I would think, actively try to gauge a player's league-wide performance. I think they're much less prone to this kind of mistake.

I'll probably take a closer look at the Oilers aquisitions over the last few years to see if there is a pattern but league-wide there doesn't seem to be a trend of teams aquiring a disproportionate amount of players that have succeeded against them in the recent past. At least, it doesn't seem to have been the case at the trade deadline this year.