How goalies age, and the survivorship bias problem

This morning, Steve Burtch published a study arguing that goalies do not show any appreciable decline with age. The key plot was this one, in which the x-axis is the goalie’s age and the y-axis is how many standard deviations above or below league average he was (for goalies with at least 30 games played):

It’s clearly true that the observed performance of old goalies isn’t appreciably worse than the observed performance of young goalies. The problem is that word "observed" — we don’t get to observe the performance of all goalies at all ages. JaredL previously looked at this issue and showed how few goalies continue to play heavy minutes into their late 30’s. This creates what is called a survivorship bias.

Let’s take Steve’s claim at face value, that goalies do not decline appreciably with age. Let’s further assume that the best goalies are the ones most likely to have long careers. Those top-tier goalies would have excellent save percentages in their late 20’s, and (per our first assumption that they don’t decline with age) they would continue to post strong numbers in their late 30’s.

As the lesser goalies’ careers come to an end (per our second assumption) and only the premier goalies are left playing, we would actually expect to see a rise in the average save percentage — the only goalies left at age 37 would be our future Hall-of-Famers still performing at a very high level. Of course, that’s not what we see; average save percentage is basically flat with age, as both Steve and Jared showed.

So that means one of our assumptions must be wrong. Either a goalie’s likelihood of having a long career is uncorrelated with how good he is or goalies do decline with age.

Steve’s approach can suffer from survivorship bias because he allows the list of goalies being compared to change with age — instead of asking whether 34-year-old goalies are worse than younger goalies as a whole, he should be asking whether they are worse than they were earlier in their careers.

His evaluation of performance at age 35 (to pick a year at random) includes 29 goaltenders. Of those, five are already in the Hall of Fame, and some fraction of Brodeur/Joseph/Kolzig/Vokoun/Thomas/Hasek will eventually join them (along with, perhaps, Osgood). The group of 25-year-old goaltenders is nowhere near as strong — we don’t expect 30% of those guys to make the Hall of Fame. As a result, comparing the elite group’s performance to that much broader cross-section is misleading; if the premier goalies only posted run-of-the-mill numbers in their later years, then this clearly represents a decline in their performance.

To illustrate the effect this has on his approach, I have reproduced his chart, but with a few changes:

The resulting plot looks like this:

Here we can plainly see that the set of goalies who play past the age of 35 is clearly an above-average group (look at how many of the below-average dots in the 20’s are blue) and that those goalies are indeed declining with age (note the steep slope of the regression line for those goalies). It seems clear which of our two assumptions is wrong.

Unfortunately, what I have done here in sorting goalies into two groups is the kind of threshold-based analysis that I have previously noted can be risky. So as one final check, let’s perform the same analysis, but this time moving the threshold from 35 to 33 to make sure we don’t get a markedly different result. Here’s how that plot looks:

Lowering the cutoff brings in some lesser goalies, so we see more red dots below 0 early in their career and we see the trend line for the group dropping below average at an earlier age.

But the overall result is exactly the same: goalies do decline with age, and the goalies who have long careers are the ones who are better than average to begin with and so can withstand more decline and remain viable NHL options.

Obviously this data can’t say whether the decline is because the older goalies suffer more minor injuries or lose quickness or reflexes or heart or whatever. Apparently Steve’s goal is to somehow sort out what causes the decline, and perhaps he will find something interesting there; I can’t say until I see that data.

But I wholeheartedly reject his current conclusion that "the correlation between goaltender performance and age is nil. Goaltender performance may not be random, but it’s NOT related to the age of the goalie playing the position at the NHL level" — once you account for survivorship bias, it seems very clear that goalie performance does decline with age.