Analytics mailbag: Save percentages, PDO, and repeatability

9 years ago

A week or so ago I opened this space up to questions about analytics, in an effort to make these numbers and terms that we use often on the hockey blogosphere more accessible to the average fan.

My initial plan was to have a post or two with all the questions and answers, but it’s more practical to dedicate a post to each question and get as much information as possible in the reply. I won’t publish names or emails with questions and will edit questions to remove all personal details, so even if you are a mainstreamer with a anti-stats reputation to maintain, nobody will ever know. Our first question is on the subject of save percentages and PDO.

Is PDO really a useful statistic? On a team wide scale, I feel like its a flawed stat. A team with a top ten goalie is much, much more likely to sustain a PDO considerably higher than a team with a bottom-ten starter. Even shooting rates seem to be somewhat sustainable for skilled teams, although much more variable so that argument is much less significant than the goaltending side. Wouldn’t, mostly due to goal tending but slightly shooting talent too, PDO not really measure luck but instead to some degree skill? I can see on an individual player level how it could be more effective.
Thoughts? Do you think PDO is truly a good indicator of luck?

First, it’s worthwhile to explain what “PDO” actually is.

PDO doesn’t actually stand for anything. It takes its name from an Internet commenter who first threw out the concept years ago on an obscure Edmonton Oilers blog:

Hows this for ugly? Lets pretend there was a stat called “blind luck.” Said stat was simply adding SH% (on-ice shooting percentage) and SV% (on-ice save percentage) together. I know there’s a way to check what this number should generally be, but I hate math so lets just say 100% for shits and giggles.
Oiler players who had over 101%:
Nilsson (103.9), GlenX (103.8), Cogliano (103.4), Stortini (102.8), Horcoff (101.6), Rourke (101.4), Moreau (101.4), Gilbert (101.1), Greene (101.1).
And Oilers who had under 99%:
Smid (98.7), Brodziak (98.5), Roy (98.5), Tarnstrom (98.00), Stoll (97.6), Visnovsky (97.3), Sanderson (96.9), Reasoner (96.8), Pouliot (96.7), Thoresen (95.5), Jacques (87.1).
You’ll notice the first group tended to get extensions while the second ground tended to get shipped out of town.

Shooting percentage is goals for divided by shots for. On-ice shooting percentage counts all the goals and shots (usually at five-on-five) for everybody on a player’s team. Conversely, on-ice save percentage only counts the goalie’s save percentage when said player is on the ice. PDO is the addition of these two numbers. Since every shot must result in a goal or a save, the combined PDO for every player in the league is still 100%.

Still with me?

You can apply this to teams as well, where it is very, very useful. Stats guys like to say that a team with a PDO of 103% is likely winning games at an unsustainable rate, and a team with a PDO of 97% is losing games at an unsustainable rate. This is because goals are pretty random events and can happen for a variety of lucky reasons: a fluke bounce, a blown coverage by an otherwise steady defenceman, a deflection off a defender’s skate, and so on. In a perfect world, say the stats guys, teams will have 100% PDOs after many many many many games. A team’s PDO is constantly working its way towards the average, and the difference between teams is the difference in the number of shots they take and allow, and not shooting or save percentages.

(That’s a bit liberal with what statisticians think, but let’s work under that assumption for clarity’s sake.)

Our questioner says that this is “flawed”, and that teams with better goaltenders are more likely to maintain higher PDOs. That makes sense, in theory. After all, better goaltenders can be expected to maintain higher save percentages, and also, skilled teams should theoretically have higher shot percentages, right?

Well, kind of. It’s a bit complicated.

ExtraSkater.com has data from two 82-game seasons. Those are 2011-12 and 2013-14. For every team, there is a game log, where you can sort stats from a myriad of different categories. I have taken every team’s five-on-five statistics for goals for and against, and shots for and against, and broken them up into Games 1-41 and Games 42-82, so we effectively have a first half and a second half. (My spreadsheet can be found here on Google Spreadsheets)

If our questioner is correct, the teams from the first half will have notably higher save percentages and PDOs in the second half than average. (This analysis has been done before, but it’s always good to stay current with the data)

Our highest five save percentages in the first half of the season:

2012 Bruins: 0.946
2014 Kings: 0.943
2014 Bruins: 0.939
2012 Sharks: 0.939
2012 Rangers: 0.938

In the second half, those teams had the following save percentages: 0.901, 0.926, 0.941, 0.920, and 0.915. Combined, they had a save percentage of 0.920, a couple of percentage points below the NHL average over the full two-season time period! Only the 2014 Bruins managed to maintain an elevated save percentage for the full season.

This is why we use PDO, in a way. While it makes sense that higher save percentage = better goalie = continued high save percentage, the data doesn’t actually bear that out.

To further illustrate this, I grouped the 60 team seasons (2 seasons x 30 teams) into five “buckets” of 12 teams each. The first bucket is the best 12 teams in save percentage, the second bucket is teams ranked 13th to 24th, and so on.

Here is how those teams fared in the first half versus the second half:

	First Half Save %	Second Half Save %
Bucket 1	0.938	0.922
Bucket 2	0.926	0.928
Bucket 3	0.921	0.925
Bucket 4	0.915	0.920
Bucket 5	0.911	0.919

Average	0.922	0.923

Of the five buckets, only Bucket 2 deviated more than a percentage point further away from the average. There was hardly a noticeable difference between Bucket 1 and Bucket 4 in the second half, despite a 23-point percentage difference in the first half of the season.

This is why the word “regression” is commonly used. Generally, over a long stretch of time, differences between goalies will show in the percentages. But over a half season, or even a full season, there just isn’t enough time to differentiate between players. Any notable spikes that bring players far from the average are probably due to short-term luck. In the end, they come back to earth over the longer span of games.

Let’s try that same experiment using “PDO” as a whole.

	First Half PDO	Second Half PDO
Bucket 1	102.2%	100.0%
Bucket 2	100.6%	99.6%
Bucket 3	99.9%	100.8%
Bucket 4	99.0%	100.1%
Bucket 5	98.2%	99.5%

Woe be the 2014 New York Rangers, who had a 97.6% PDO in the first half of the season, and three points out of the final playoff spot. I wrote some kinder things about them than most hockey writers did back in December, and they’ve rocketed up and are one game away from the Stanley Cup Finals.

Meanwhile, the 2012 Boston Bruins, after starting out the season on fire coming off their first Stanley Cup win since 1972, hit the proverbial wall after a strong first half and fell in the first round of the playoffs. Their PDO fell from 104.2% (largely off their strong goaltending) to 98.2% in the second half (largely off their poor goaltending).

The market application here is that it’s difficult to predict percentages in the short term, and when I say “difficult”, I mean “impossible”. Over a three-year period, Boston’s PDO is the highest in the league: at 101.6%. So yes, while it’s plausible that teams with better goalies can maintain higher save percentages, they still have to be within a workable range. Be suspicious of teams that have PDOs outside anywhere from 98.5% to 101.5%, like the 2013 and first-half 2014 Toronto Maple Leafs, since there’s a very good chance their success (or failures) are not repeatable at the same rate.

To our anonymous asker, I hope this answers your question. I also hope and pray that nobody who is in a hockey pool with me learns to effectively make moves with PDO.

If you have a statistical-related question, don’t hesitate to send an email to camcharron(at)gmail.com.

Analytics mailbag: Save percentages, PDO, and repeatability

Check out these posts...