Are Quality Starts A Repeatable Skill?

There aren’t nearly as many statistics available to evaluate goalies as there are to evaluate forwards and defencemen.  That’s true whether you’re talking about the standard boxcar stats that newspapers and NHL.com have reported for years, or whether you’re referring to the newer breed of numbers often called “analytics” or “advanced stats”.  But I would be remiss if I did not acknowledge that there are a few more ways to look at goalies than there were a few years ago.  A couple of weeks ago I talked about dividing saves into “danger zones” based on their distance from the net.  This time I’m going to take a look at Quality Starts.

Quality Starts were created back in 2009 at the Hockey Prospectus web site.  Here’s how Rob Vollman defines them in the 2013 edition of Hockey Abstract:

If the starting goalie stopped at least a league average number of shots, which was typically 91.3% prior to 2009–10 and 91.7% since then, or if he played at least as well as a replacement-level goalie (88.5%) while allowing two goals or fewer, we defined that as a Quality Start.

The idea of Quality Starts was to improve on judging goalies by their win-loss record.  There are lots of things that a goalie can’t control, like how many goals his team scores or how many shots they allow, but he does have a good degree of control over what percent of shots are stopped.  Therefore, the reasoning goes, what you want to judge a goalie on is whether he played well enough to give his team a chance to win.  

I see Quality Starts pop up reasonably often, and Hockey Reference added them to the list of goalie stats it tracks a year or two ago.  One thing that’s always bugged me about the stat is that we don’t know if it’s actually representative of a real skill.  Do goalies actually have any ability to control the distribution of their SV%?  I found it hard to imagine that they could.  It reminds me of hearing people say that, sure, Grant Fuhr allowed a lot of goals, but he made the big saves when the game was on the line.  But surely, goalies can’t actually control which games they make more saves in, can they?  If they can, why don’t they just choose to stop more pucks all the time?

The Leafs recently acquired a goalie who looks pretty good from the standpoint of Quality Starts.  Last year, among the 42 goalies with at least 30 games played, Frederik Andersen was 4th in QS%.  And over the past three seasons, among the 34 goalies who played at least 100 games, Andersen ranks 9th, nestled in between Braden Holtby and Sergei Bobrovsky.  Not bad.  If QS% represents a real, repeatable skill, the Leafs have acquired one of the league’s most consistent starters.

But is it a repeatable skill?

REPEATABILITY

The first thing I decided to take a look at was whether QS% is a repeatable skill that carries over from season-to-season.  If a goalie has a good year or two by QS%, should we expect them to continue to put up good numbers?

I collected QS% for all goalies who played at least 40 games from 2012-13 to 2013-14, and at least 40 games in the next two seasons as well.  Here’s how well QS% in that first pair of years predicts QS% in the second pair:

qs2qs

There is an effect, but it’s very small.  Indeed, it would be quite amazing if there was no effect, since some goalies have consistently higher SV% than other goalies, and that alone should produce some kind of impact.  But it’s quite small.  There’s a very limited ability for goalies to maintain a strong rate of Quality Starts over time.

Here’s another way to think about it.  Let’s replace QS% in the first two years with SV% and see how well we can predict QS% in the next two:

sv2qs

It turns out that past SV% is better at predicting future QS% than past QS% is at predicting future QS%.  That may seem counter-intuitive, but I think it makes perfect sense.  Over large samples, SV% is somewhat of a repeatable skill.  So if SV% in large samples is a real skill, it makes sense that it does a better job of predicting future Quality Starts, because how good a goalie is at getting Quality Starts is mostly a result of how high his SV% is, as you’ll soon see.

Let’s take a look at a scatter plot of SV% vs QS% for all goalies who played at least 80 games in the three seasons from 2013-14 to 2015-16:

3yrqs

As you can see, the rate at which a goalie finishes with a Quality Start lines up extremely closely with his overall save %.  It’s not just that goalies who have a higher SV% have more Quality Starts, which is obviously what ought to happen.  It’s that the increase is nearly linear.  The ratio of a goalie’s games that are Quality Starts rises almost entirely in line with his SV%.  This is exactly what we would expect to see If the distribution of SV% from game-to-game is essentially random.

There’s still some unexplained variance, though.  An R2 of 0.75 is quite high, but it’s not a perfect fit.  What if we increase our sample size a bit?  Let’s bump up to 4 years, but this time let’s only look at goalies who have played quite a lot of games – at least 150.

sv2qs4yr

As we increase our sample size we’re seeing that the rate of quality starts comes even closer in line with a goalie’s SV%.  The less room there is for randomness in the results, the more we’re seeing that goalies don’t really have any ability to control the particular games in which they play well, or the consistency with which they do it.

Speaking of goalie consistency, Carolina Hurricanes hockey analyst Eric Tulsky wrote about this topic a few years ago.  Eric wanted to test whether the oft-repeated idea that Ilya Bryzgalov was a “streaky” goalie was borne out by the data.  He ran a simple simulation based on Bryzgalov’s career SV% and shot distribution to see whether Bryzgalov really was any more streaky than a model picking saves based on a random distribution weighted by SV%.  You can read more about his full method in the link above.

What did Tulsky find?  Bryzgalov turned out to be no more streaky than chance would predict.  Nor was Marc-Andre Fleury.  Or Henrik Lundqvist, or Pekka Rinne, or Jaroslav Halak, or Carey Price.  Every goalie Eric looked at had a pattern of SV% that was distributed nearly identically to a random model based on that goalie’s career SV%.  None of the goalies was any more “consistent” than a random distribution of games by SV%.

This lines up with the results I’ve demonstrated above.  Goalies don’t appear to have any ability to be more consistent than you’d expect any other goalie with a similar SV% to be.  Quality Starts are not a repeatable skill.  There is no such thing as a goalie who gives his team a chance to win more often than any other goalie with a similar SV% would be expected to do.

          • HockeyKeeperKit

            To expand, don’t you think a team like Anaheim would want their goalie back at a $2.5 million dollar contract? Maybe not with Bernier and Gibson already paid and their current Lindholm cap crunch, but a team like LA would sure as hell make a claim given their current issues.

          • Ron K

            Another marvelous post by miss Kitty. Andersen’s contract is $5 million per for 5 years Kitty. Here you were ready to send him off for a conditioning stint and he’s been just great since you said that. For being a goalie yourself you certainly don’t know much about them or how to handle them when things aren’t going well. Just get rid of him and get another eh Kitty? lol
            Stick to your fantasy leagues and stats Kitty……..

          • Ron K

            This IS what you said Kitty. “To expand, don’t you think a team like Anaheim would want their goalie back at a $2.5 million dollar contract?”
            Andersen is signed at $5 million per for 5 years Kitty. You really need to stay away from your computer when you’re strung out on drugs like this. I can’t believe you typed “Not even what I said”.
            You have some serious, serious issues Kitty. Please get some help with your mental issues. I’m really tired of having to correct you all the time with your mindless blabber and diatribe trying to lie your way out of everything you say. You need professional help Kitty……….

          • HockeyKeeperKit

            The original comment recommended waivers as an option (which I strongly recommended against). If we sent the Andersen to waivers and he cleared, we’d never stand a chance of recalling him as a team could claim him at half the price.

            Re-Entry Waivers 101: “…the team who claims the player is only responsible for half of the player’s salary and cap hit for the duration of the contract, and the team who waived the player picks up the other half.”

            $5 / 2 = $2.5 (math isn’t your strong suit after all…)

            I’m also getting tired of re-re-correcting you Ronny. You deemed me a doctor after all so I should really start charging…

          • Ron K

            Oh……So you’re assuming that everyone should know that you were referring to a previous comment about putting Andersen on waivers which was posted 4 comments before yours. Just another fine example of your mindless approach to communicating with others.
            A doctor? Of deceit and deception perhaps. Doctor Dikhead OK? You babble like a baby Kitty. Get some professional help before those drugs you’re taking fry your brain completely.

          • HockeyKeeperKit

            Bryan repeated his initial question and I replied, so yes, I would expect that “MOST” people to follow that progression. You apparently have the mental capacity/retention of a goldfish, however, and can’t recall more than one comment at a time…

            That’s also kind of the concept of individual threads. See, the indentation implies that comments are replies to original comments.

            * 3 comments before mine (math…)

          • Ron K

            Nope. You never clearly defined the statements in your comment. To assume that I should read 4 or 5 other comments to figure what you are talking about is nothing short of pure arrogance on your part. That’s something that’s consistent with your attitude on most of your posts which got you in trouble with me in the first place. So, I wouldn’t go down that road again if I were you.

          • Ron K

            It’s not the internet that scares me Kitty, it’s people like you that frighten me to death. I can’t believe they let you circulate among the public. That’s a frightening thought.

  • tealeaves

    The superiority of SV% makes sense being the sample size of saves is greater then the sample size of stars. This is analogous to using goals versus shot attempt data to evaluate a player or teams. Both can work, though shot data gets you an answer before goals can converge to something repeatable and useful.

  • Brad M.

    “There is no such thing as a goalie who gives his team a chance to win more often than any other goalie with a similar SV% would be expected to do. “

    Who, in his/her right mind, is disputing this? Who has ever disputed this?

    If two goalies stop a similar percentage of pucks shot at them, of course they are giving their teams a similar “chance to win.”

  • Rob Vollman

    I’m glad this comes up from time to time, because you’re not the only one who gets tripped up by this.

    Even though your exact point was made in the same chapter from which you quoted the definition (although not in the same detail), as was Tulsky’s study, not everybody has that book, or is familiar with the various studies.

    I’ve never seen anyone actually use QS in a predictive fashion on air or in print, but sometimes casual discussions on message boards or social media can drift that way.

    I think the problem is how easy it is to confuse descriptive and predictive stats. Just because something does a good job describing what happened, doesn’t mean that it is suitable for predicting what will happen in the future. Also, that doesn’t mean that the statistic is flawed or useless.

    As for Andersen, if you want some useful stats that could have helped predict his early struggles, set aside quality starts and look his high goal support in Anaheim, and the low average shooting percentages of the opponents he was assigned to face. In essence, he was a textbook example of a goalie whose numbers may have been boosted by outside factors.