Monday, May 4, 2009

See Them Good or Not At All

Immediately after the trade deadline Jonathan Willis pointed out the fact that the recently acquired Patrick O'Sullivan had performed much better against the Oilers than he had against the rest of the league over the past two seasons. Willis suggested that this might reflect one of the reasons O'Sullivan had been acquired, namely, management had "seen him good" as the phrase goes. Shortly thereafter I looked at all of the trades that happened at the deadline using the same methodology (goals, assists and points over a two-year period, including all playoff games). It looked to me like these trades showed a seemingly random assortment of results for individual players against the teams that had acquired them. Some had performed better against their newly acquired team than the rest of the league, some about the same and still others performed worse. This may show that the results are random or it may simply be the result of the fact that some teams rely on "saw him good" more than others. In order to get a better idea of how much the Oilers rely on this method I've used the same methodology to look at every Oilers acquisition since the lockout. This chart organizes them by the difference in points per game:

The first thing to note right off the top is that several of the Oilers acquisitions are not on the chart (Allan Rourke, Ryan Potulny, Denis Grebeshkov, Joni Pitkanen, Toby Petersen, Ladislav Smid, Robert Nilsson and Jan Hejda). That's because those players did not play any NHL games against the Oilers in the two seasons preceding their acquisition. If we include players who played three or fewer games against the Oilers along with these eight we already have 18 of the 27 player acquisitions. This is pretty good evidence that even if the Oilers organization is using a "saw him good" method of scouting, it's certainly not dependent on games against the Oilers.

It's important to noth at the outset that Even with the "larger" samples we're still only working with between five and thirteen games so there's room for quite a bit of luck to seep into the results. Having said that, of the remaining 9 players 6 do significantly better in terms of PPG and 3 are about the same. Since most of these players are offensive players (Lupul, Pronger, Spacek, Sykora, O'Sullivan, Visnovsky) I'm not sure how instructive it is to look at the +/- column, but in this case 5 outperform, one runs at about the same pace and three do worse against the Oilers.

The results, then, seem to offer a bit of a mixed bag. Two thirds of the players who were acquired since the lockout came without much of any recent track record against the Oilers, but those that do have a track record have a successful one. In the case of Dustin Penner in particular, it's hard not to think that his 7 points in 10 games along with a tidy +5 weren't influential. Furthermore, many of the Oilers main acquisitions have this somewhat positive track record (Pronger, Lupul, Penner, Visnovsky)

Conclusions? (1) This is not the main criteria for player evaluation as some major acquisitions do not have much of a track record at all (Cole, Pitkanen, Souray, Samsonov). (2) The Oilers interest does seem to be influenced by how a player plays against them to some degree. (3) Sanderson was a salary dump and Samsonov had a reputation which leaves Curtis Glencross as a bit of an outlier.


Jonathan Willis said...

Nice breakdown, Scott. I imagine that if this sort of thing were done league-wide there'd be similar results for each team; results that show a modestly better performance across the whole against the team in question vs. everyone else, but it probably isn't the driving factor so much as a subconcious motivator.

Then again, I could just be out to lunch ;)

Scott said...

The subconscious motivator seems about right to me. One of the biggest arguments against a bigger role is the fact that Tyler Arnason still isn't an Oiler ;) I might take a shot at a few other teams who haven't changed managers recently to see how the Oilers compare.

The Falconer said...

I like what you've done here. One thought--GMs might be influenced by other games they have seen that player play in. For example, when the Atlanta Thrashers acquired Steve McCarthy from Vancouver two years ago the Thrashers GM gushed about a game he watched McCarthy play in juniors in his draft year. He had always wanted him.

Of course it turned out to be a small size problem. He just happened to see McCarthy on one of best days and he simply wasn't that good all the time.

But I suspect you intuition is correct. GMs like the rest of us to put too much weight on evidence we have personally observed rather than give equal weight to all the evidence.

Scott said...

I do agree that general managers and scouts will be strongly influenced by other games they've seen as well. This seemed like a good way of cataloguing at least some of that bias without getting into too much guessing. I think the degree to which "saw him good" matters probably varies from individual to individual. If I look at a few more teams it will be interesting to see which guys seem to rely the most on "saw him good" and which guys the least.