Brendan Shanahan asked Kyle Dubas to explain PDO to MLSE board

If you want to know how the game has changed, this anecdote should do it.

Recently Toronto Maple Leafs president Brendan Shanahan, an old-school power forward during his 1500-plus game NHL career, asked assistant general manager Kyle Dubas to explain the concept of PDO to Maple Leafs ownership, Dubas told James Mirtle of the Globe and Mail. 

We don’t know what case Dubas and Shanahan were making to ownership at the time, but based on what we know of recent meetings between the MLSE board and Maple Leafs management, imagining a statistic named after an internet commenter (Brian King) being used to justify dramatically remaking a flawed Maple Leafs team doesn’t seem farfetched. Certainly it’s less farfetched than a former theLeafsNation.com site editor weaselling his way into the club’s front office… 

What exactly does PDO tell us about recent Maple Leafs history? And what else did Dubas reveal at this week’s Sloan Conference. Read past the jump! 

Let’s go back to the start of Brian Burke’s tenure. Since the 2008-09 season, in the middle of which Burke was named general manager of the Maple Leafs, this club has gone through several permutations. The core of this team though – from Phil Kessel, to Dion Phaneuf, to Tyler Bozak, to Jake Gardiner, to Joffrey Lupul – was constructed largely under Burke’s watch. 

The team Burke took over was awful, though perhaps they weren’t as bad as they appeared. Yeah it’s not a great sign that the 2008-09 Maple Leafs’ three leading scorers were Alexei Ponikarovsky, Jason Blake and Matt Stajan, but the club was actually only about average by Corsi For percentage. 

The issue was mostly in net, where a washed up Curtis Joseph, an inconveniently hot Martin Gerber, and the dreaded Vesa Toskala combined to provide the Leafs with .903 save percentage-quality goaltending at even-strength – the worst in the league. 

Over the next year Burke would remake the team, bringing in some unrestricted free agent defenders like Francois Beauchemin and Mike Komisarek, a vast number of college and unsigned free agents (including Bozak, and hey, remember Christian Hanson) and acquiring Phil Kessel in an ill-fated – but secretly kind of reasonable – trade with the Boston Bruins. During Burke’s first full season he’d trade seemingly half of his team to Calgary for Dion Phaneuf. 

From a puck possession stand point the team began to improve. Burke eventually managed to piece together a really solid second line in Mikhail Grabovski, Nikolay Kulemin and Clarke MacArthur. The Maple Leafs still had top of the roster problems, but they were fast, and showed signs of improved two-way form. 

The issue in net would fester though, until the tail end of the 2010-11 season when James Reimer showed up and stopped everything thrown at him for half of a season. 

In 2011-12 the Maple Leafs started the year strong and even looked like a bona fide playoff threat. The problem? As it has so often been in Toronto, was in goal. 

James Reimer tried to play through a concussion that season and wasn’t himself, Jonas Gustavsson was himself and that was an even bigger problem, and the club collapsed down the stretch. At least Burke managed to turn Luke Schenn into James van Riemsdyk at the draft, but the season was largely without a silver lining otherwise.

Towards the tail end of the goaltending induced tailspin, Burke and Nonis made their biggest mistake, in my view. They hired Randy Carlyle. 

Ron Wilson isn’t my favourite hockey coach, but I always thought he deployed Toronto’s roster in a smart way. He knew he couldn’t control the middle of the ice with the centremen and defenders that he had available, and so the Maple Leafs forechecked aggressively and played at a fast pace to counteract their deficiencies. They never had the goaltenders to back up that style, but I tend to think that a lack of puck-stopping talent was the major factor that cost Wilson his job.

Carlyle’s impact on the Maple Leafs’ puck possession numbers was instant and dramatic. And not at all positive.

Here’s a graph from war-on-ice.com of the Maple Leafs’ rolling 10-game shot attempt differential since the 2009-10 season. I’ve annotated it, as you can see: 

(Courtesy: war-on-ice.com)

And you can see the numbers start to peak back up following Carlyle’s dismissal this season…

Carlyle was the wrong fit for the Maple Leafs roster he inherited. Though his possum defensive schemes took advantage of Kessel’s against the grain scoring ability, it neutered the effectiveness of the fast, attacking depth players who Wilson leaned so heavily on. Players like Gardiner, Grabovski, MacArthur. 

Toronto began to get out-shot and out-shot dramatically.

Oh and just because I don’t want to hear any ‘Carlyle kept opposing offensive attacks to the outside’ arguments in the comments, the rolling 10-game scoring chance differential graph looks essentially identical to the shot attempt one:

(Courtesy: war-on-ice.com)

Interestingly, during his presentation at the Sloan Conference this weekend, Dubas dropped a graph featuring very similar shot attempt differential numbers, and related it to a coaching change:

These numbers matter anyway, but even if you’re skeptical, they should matter to you if you’re a Maple Leafs fan because they sure seem to matter to the team you’re rooting for.

PDO doesn’t stand for anything, but it was revealed at the Sloan Conference that Maple Leafs brass like to call it percentage-driven outcome as a joke (at least I think it’s a joke). It’s a decent joke, assuming it is one, and a worthwhile acronym, since yeah, that’s precisely what PDO purports to capture.

The metric is really simple, actually, it’s just the sum total of a team’s on-ice 5-on-5 shooting percentage and save percentage. So save percentage, on average, accounts for roughly 92 percent of PDO. 

The NHL game has changed significantly over a generation, and perhaps the biggest change is the quality of the league’s goaltending. As recently as the early 80s, but really into the mid-90s, goalies couldn’t really cover up the bottom of their crease. They didn’t play along the ice and shooters could slide a puck into the net. 

Now you almost always have to lift the puck to score, and good luck finding space and time to do that against contemporary NHL-caliber talent deploying modern NHL-caliber defensive systems. 

With the improvement in NHL goaltending, what we’ve observed in recent years is that percentages in the NHL are relatively fixed. PDO, for example, has been found to roughly regress to 100 (so a .920 on-ice save percentage and eight percent on-ice shooting percentage when combined would be considered a ‘normal’ PDO). If a player or team was above 100, they were likely to come back down to earth. If they were below, they were likely to start benefitting from the bounces a bit going forward.

Sometimes a player – like Nazem Kadri for example – will go on a run in which their team is converting on 13 percent of all 5-on-5 shots when they’re on the ice. That run can last for 50 games, or even a full season. The superlatives will flow. 

Over time though, the bounces tend to even up and come in the wash. What can happen with a player, can happen with a team. 

I know passionate consumers of hockey content are dealing with some ‘luck’ fatigue in reading contemporary hockey analysis, but it’s really important to remember that the quality of NHL players is so high these days that the number of players who can really influence the percentages – or shot quality – in a meaningful, lasting way is tiny. 

Of the 526 players who logged at least 3,000 5-on-5 minutes between 2007-2014, only 25 skaters – 4.7 percent of regular NHLers – managed an on-ice shooting percentage above 9.5 percent (which would be considered 0.5 percent above ‘normal’). For the most part these are the players you’d expect – your Sidney Crosby’s, your Steven Stamkos’ etc.. 

Only 32 skaters – or six percent percent of regular NHLers – managed an on-ice shooting percentage below 6.5 (which would be considered .5 percent below ‘normal’). Again this list includes your Stephane Veilleuxs and your Travis Moens, as you’d expect.

Anyway let’s bring this back to the Maple Leafs and Carlyle. Under Carlyle the Leafs began to get outshot by a historic rate. Though some didn’t want to see it, the Maple Leafs regularly looked lost without the puck. 

On the other hand, the goals came at an unsustainable rate, and the goaltending was superb. Both James Reimer and Jonathan Bernier had stellar runs of puck-stopping success, which made the club look more imposing than it was. 

The Maple Leafs made the playoffs during the lockout shortened 2013 season. They clearly thought they were way ahead of schedule, and that the numbers were bogus. Current Maple Leafs brass now refers to that fateful 2013 playoff series against the Bruins as “the worst thing that ever happened to this team,” according to Bruce Arthur of the Toronto Star.

So what happened to the Maple Leafs in 2013 and how does it relate to Dubas making a presentation about PDO to the folks at Maple Leaf Sports and Entertainment? What happened is that the club believed their own eyes. They fell for their own bull-shit. 

Useful players like Mikhail Grabovski were jettisoned, while passengers like Tyler Bozak were retained at a high cost. Out went an under-rated (and underpaid) top-six winger in Clarke MacArthur, in came the anchor that is David Clarkson. And on and on. 

“What analytics taught me is your eyes and your mind are lying sons of bitches in the worst absolute way,” Dubas said Saturday, summing up the bias issue perfectly.

When the Maple Leafs came undone at the tail end of the 2013-14 season, it felt inevitable. Actually it was a statistically unlikely event, but some basic principles apply here.

The main one is that when you can’t control the flow of games, you’re left without a safety net for when the hockey god’s – those fickle lords of ice and puck – decide to take a piss on you. Over the course of 82 games, that’ll happen, you’ll run into three crazy hot goalies in a row. Guys will start gripping their sticks too tight. The game won’t be fun. 

A good team though will at least control those games. Hopefully on their streak of bad luck, whether it’s partly injuries or fatigue or they’re playing guilty in Newark after a fun night out in Manhattan, whatever, they’ll pull out a 2-1 shootout win, or force overtime in the third, and they’ll manage to take three points out of an unlucky five-game stretch. A bad team – like the Maple Leafs in 2013-14 – are more likely to lose all five. 

Here’s how the Maple Leafs have shot, percentage-wise, from 2009-10 to now (10-game rolling average, on-ice shooting percentage):

(Courtesy: war-on-ice.com)

The Leafs are probably an above average finishing team, and have maintained a high shooting clip for years. All of their peaks are followed by a valley though, which is the pattern we’d expect if shooting percentage were in fact an ephemeral thing rather than a talent generated one.

In 2013 the valley didn’t have enough time to cost them because of the lockout shortened season, but it cost them in 2014. It cost them again this season (poor Peter Horachek).

Here’s the larger half of PDO, goaltending, (10-game rolling average, on-ice save percentage):

(Courtesy: war-on-ice.com)

You can see in 2011 when Reimer took over, and in 2012 when the goaltending cost Ron Wilson his job. You can see how Bernier’s hot start to the 2014 season tailed off, and again, you can see how once their goaltending normalized, this current iteration of the Maple Leafs started to look vulnerable this past season.

(Courtesy addfunny.com)

That the Maple Leafs have finally realized that they’ve been trending in the wrong direction for several years – with a short, luck-based run of decent results – is a good thing. It’s too bad that it took almost three years to get to this point, but it’s something. You can’t fix a problem you’re ignoring.

That Dubas and Shanahan have attempted to explain PDO to Maple Leafs ownership suggests to me that the much-hyped scorched earth Maple Leafs rebuild might actually come to pass. After all, there’s no way you can look at the influence PDO has had on this club’s recent history and conclude anything aside from – ‘we’re very, very far away from contending’. 

This weekend Dubas characterized opposing teams as being far ahead of the Maple Leafs – both on the ice and in terms of applying hockey’s underlying numbers to decision making in a helpful way. Presumably that playing field will be levelled somewhat with the advent of player tracking technology. 

NHL clubs are about to be awash in new data, and while those expecting shot attempt differential and PDO to go the way of the Dodo are likely to be disappointed, surely there will be some extraordinary inferences and insight to be ferreted out of that data set. 

One thing about the composition of the Maple Leafs’ analytics and research team that perhaps hasn’t received enough attention is that the Maple Leafs went out and hired people who had different skill sets from the highest profile advanced statistical hockey analysts. Darryl Metcalf and Rob Pettapiece didn’t exactly do the analytical heavy lifting that, say, Tyler Dellow, Tim Barnes, and Eric Tulsky accomplished over the years (though Cam Charron fits that mould somewhat). 

No, what Metcalf and Pettapiece do is build databases, and organize data in a way that makes it intuitive for the user, or the layman, to sort and prioritize information. 

There may be an enormous advantage to be gained from applying insights gleaned as a result of the reams of new data that we’re all expecting in the not-so-distant future. On this score, it would seem to me that the Maple Leafs have set themselves up relatively well. 

Now it’s about getting everyone on the same page.

“From the president down to the manager to the coach and scouts,” Dubas said of integrating analytical thinking through the Maple Leafs organization. “You’re going to have a lot more success than if you have one person on your staff alone saying this is important.

“If you have a coach that isn’t really buying into it and you’re bringing in personnel that are analytics-friendly and you have a coach that isn’t, it’s not going to work, because they have to work together. They can’t work separately.”