Are Quality Starts A Repeatable Skill?

There aren’t nearly as many statistics available to evaluate goalies as there are to evaluate forwards and defencemen.  That’s true whether you’re talking about the standard boxcar stats that newspapers and NHL.com have reported for years, or whether you’re referring to the newer breed of numbers often called “analytics” or “advanced stats”.  But I would be remiss if I did not acknowledge that there are a few more ways to look at goalies than there were a few years ago.  A couple of weeks ago I talked about dividing saves into “danger zones” based on their distance from the net.  This time I’m going to take a look at Quality Starts.

Quality Starts were created back in 2009 at the Hockey Prospectus web site.  Here’s how Rob Vollman defines them in the 2013 edition of Hockey Abstract:

The idea of Quality Starts was to improve on judging goalies by their win-loss record.  There are lots of things that a goalie can’t control, like how many goals his team scores or how many shots they allow, but he does have a good degree of control over what percent of shots are stopped.  Therefore, the reasoning goes, what you want to judge a goalie on is whether he played well enough to give his team a chance to win.  

I see Quality Starts pop up reasonably often, and Hockey Reference added them to the list of goalie stats it tracks a year or two ago.  One thing that’s always bugged me about the stat is that we don’t know if it’s actually representative of a real skill.  Do goalies actually have any ability to control the distribution of their SV%?  I found it hard to imagine that they could.  It reminds me of hearing people say that, sure, Grant Fuhr allowed a lot of goals, but he made the big saves when the game was on the line.  But surely, goalies can’t actually control which games they make more saves in, can they?  If they can, why don’t they just choose to stop more pucks all the time?

The Leafs recently acquired a goalie who looks pretty good from the standpoint of Quality Starts.  Last year, among the 42 goalies with at least 30 games played, Frederik Andersen was 4th in QS%.  And over the past three seasons, among the 34 goalies who played at least 100 games, Andersen ranks 9th, nestled in between Braden Holtby and Sergei Bobrovsky.  Not bad.  If QS% represents a real, repeatable skill, the Leafs have acquired one of the league’s most consistent starters.

But is it a repeatable skill?

The first thing I decided to take a look at was whether QS% is a repeatable skill that carries over from season-to-season.  If a goalie has a good year or two by QS%, should we expect them to continue to put up good numbers?

I collected QS% for all goalies who played at least 40 games from 2012-13 to 2013-14, and at least 40 games in the next two seasons as well.  Here’s how well QS% in that first pair of years predicts QS% in the second pair:

There is an effect, but it’s very small.  Indeed, it would be quite amazing if there was no effect, since some goalies have consistently higher SV% than other goalies, and that alone should produce some kind of impact.  But it’s quite small.  There’s a very limited ability for goalies to maintain a strong rate of Quality Starts over time.

Here’s another way to think about it.  Let’s replace QS% in the first two years with SV% and see how well we can predict QS% in the next two:

It turns out that past SV% is better at predicting future QS% than past QS% is at predicting future QS%.  That may seem counter-intuitive, but I think it makes perfect sense.  Over large samples, SV% is somewhat of a repeatable skill.  So if SV% in large samples is a real skill, it makes sense that it does a better job of predicting future Quality Starts, because how good a goalie is at getting Quality Starts is mostly a result of how high his SV% is, as you’ll soon see.

Let’s take a look at a scatter plot of SV% vs QS% for all goalies who played at least 80 games in the three seasons from 2013-14 to 2015-16:

As you can see, the rate at which a goalie finishes with a Quality Start lines up extremely closely with his overall save %.  It’s not just that goalies who have a higher SV% have more Quality Starts, which is obviously what ought to happen.  It’s that the increase is nearly linear.  The ratio of a goalie’s games that are Quality Starts rises almost entirely in line with his SV%.  This is exactly what we would expect to see If the distribution of SV% from game-to-game is essentially random.

There’s still some unexplained variance, though.  An R2 of 0.75 is quite high, but it’s not a perfect fit.  What if we increase our sample size a bit?  Let’s bump up to 4 years, but this time let’s only look at goalies who have played quite a lot of games – at least 150.

As we increase our sample size we’re seeing that the rate of quality starts comes even closer in line with a goalie’s SV%.  The less room there is for randomness in the results, the more we’re seeing that goalies don’t really have any ability to control the particular games in which they play well, or the consistency with which they do it.

Speaking of goalie consistency, Carolina Hurricanes hockey analyst Eric Tulsky wrote about this topic a few years ago.  Eric wanted to test whether the oft-repeated idea that Ilya Bryzgalov was a “streaky” goalie was borne out by the data.  He ran a simple simulation based on Bryzgalov’s career SV% and shot distribution to see whether Bryzgalov really was any more streaky than a model picking saves based on a random distribution weighted by SV%.  You can read more about his full method in the link above.

What did Tulsky find?  Bryzgalov turned out to be no more streaky than chance would predict.  Nor was Marc-Andre Fleury.  Or Henrik Lundqvist, or Pekka Rinne, or Jaroslav Halak, or Carey Price.  Every goalie Eric looked at had a pattern of SV% that was distributed nearly identically to a random model based on that goalie’s career SV%.  None of the goalies was any more “consistent” than a random distribution of games by SV%.

This lines up with the results I’ve demonstrated above.  Goalies don’t appear to have any ability to be more consistent than you’d expect any other goalie with a similar SV% to be.  Quality Starts are not a repeatable skill.  There is no such thing as a goalie who gives his team a chance to win more often than any other goalie with a similar SV% would be expected to do.