The Extra 2% Continued: A Response to A Response

Earlier this week I wrote a piece about improvements that can be made at the margins of a hockey team. The basic premise was that fourth lines and third pairs move the needle much more than most think and based on my model, a switch from Ben Smith, Roman Polak, and Matt Hunwick to Peter Holland, Martin Marincin, and Frank Corrado is the difference between 89 and 93 points, at least by my model.

Today, David Johnson aka @hockeyanalysis on Twitter offered a different perspective to the piece, citing the Leafs defensive issues as the team’s biggest problem and showing that removing the three players I pointed out as issues would be a further detriment to the team’s defensive woes. That is 100 percent an argument I can get behind, after all, the role of those three is to play defence so they should be better at it.

But it is something I already discussed in the initial piece:

Traditionally speaking, we all know what role a fourth line and bottom pairing play. They don’t usually do much and their job is basically “don’t get scored on, we don’t care about offence with you guys out there, just shut the other team down.” And while they may be effective at slowing the game down, they’re completely ineffective because they’re only focusing on one area of the ice. The goal it to out-score the other team, both ends of the ice are valuable. To focus on one doesn’t make much sense. You hear the complaint about offensive-minded players needing to find a “200 foot game” but rarely will you hear that about some fourth line pluggers or shutdown d-men.

So while those three may be effective at playing defence, the game isn’t decided by how many goals the team allows, it’s decided by goal differential, and the three in question have very little offensive talent to speak of that would make their defensive acumen worth it.

At the start of his piece, Johnson cites each player’s Corsi percentage this year (and that the players in question were all at the bottom) perhaps with under the guise that Corsi was the only reason they should spend some time in the press box. Interestingly enough, when it’s time to talk about actual goals he only uses GA60 which is a little deceiving. 

When you look at goals percentage, you get Smith and Polak at the bottom, with William Nylander being the only player lower. Funny enough, Polak’s goals percentage is actually lower than his Corsi. Hunwick is near the top, but it’s on the strength of an unsustainably-high-for-a-defenseman on-ice shooting percentage. All the players in question rank highly by on-ice save percentage, and perhaps they do have an effect on that, but with a sample of just 20 games, that’s not something worth reading into right now.

And that’s a big issue with Johnson’s critique. It is incredibly disingenuous to use goal metrics in a 20 game sample as anything remotely meaningful. I have time for larger samples of goal analysis, but 20 games? In 20 games Matt Martin has been on the ice for all of five goals against. Five. One extra bounce and well look at that his GA60 jumps up by 0.4 goals. Goals are incredibly noisy due to their rarity and it’s the reason most people turn to Corsi, there are more events to analyze. It stops people from drawing any meaningful conclusions from five goals in 20 games and forces them to look at the bigger picture. If the same trend continues over multiple seasons, perhaps there may be something there, but I’m not buying 20 games.

Johnson then goes into GA60 rates for Leafs d-men during their time with Mike Babcock, which shows Frank Corrado with the highest rate at 3.24. Terrible right? Well, Corrado has only played 40 games with the Leafs. In that time his goalies have stopped just 89.4 percent of his shots against, three percent below the league average rate. Is part of that Corrado’s fault? Probably. It wouldn’t surprise me at all if the shots Corrado allowed were higher quality than most d-men because he isn’t very good, but I do doubt whether it’s to that large of an effect. There’s simply way too much variance over such a small timeframe to be sure, and it’s probably what’s hindering his chances of getting back into the lineup.

At the opposite end of the spectrum is Roman Polak’s sparkling 1.97 GA60 which looks pretty great on the surface. This year his GA60 RelTM is -0.58 while last year it was -0.69. Clearly, he suppresses goals against right? Well, what happened to the seven years before that where he was at 0.76, 0.24, 0.17, 0.51, 0.21, 0.14, 0.29? Did he just suddenly realize how to negate shot quality all of a sudden and make his goalies save shots better than before (if so kudos to Babcock because we got a 360 makeover on our hands). I’m not sure, maybe there’s an argument for it, but I don’t see it.

And whether his GA60 is sparkling of late or not, he’s had a negative effect on goals percentage in seven of his 10 seasons. Does it really matter if goals against are down, if the team is a net negative with him on the ice?

You can do the same kind of analysis for all the players available and what you’ll get is pretty much a mixed bag. Hunwick has five bad years by goals percentage and four good ones, this season happened to land heads, err I mean, happened to be a good one. Ben Smith hasn’t played much in the NHL so I’m really not going to bother analyzing goals for him, but he’s been up and down too.

The point is that using goals for analysis, especially at this sample size, is incredibly unreliable. And if you’re going to go that route, I’m not sure why you’d ignore the other 50 percent of the equation.

That’s why most people make models to begin with: to account for what’s accountable and be aware of what isn’t. When I wrote my original piece on it, about 25 percent of it was spent on my own perceived weaknesses of the model. There’s very little you can tell me about what’s wrong with my model that I do not already know. I have never once claimed it to be perfect, in fact, I literally wrote that it is not in that original piece. No model is perfect (and you will never find a model maker who thinks so), especially in a sport as dynamic as hockey and especially one created by a guy whose last math class was grade 12 calculus.

But at the same time, this isn’t rocket science. We have a pretty good idea of what things matter and we have a pretty good grasp of things we should be accounting for. My model is a very simple one. Very simple. It assigns a linear weight to things we all believe to be important. Points and shots are there. On-ice Corsi too. But so are on-ice goals. I didn’t forget about them when I made my model. They’re just a small part – and I know how noisy they can be – but they’re there because they do matter. 

That’s just the basics of it. For my team level analysis (ie, how I got the four-point difference in the initial piece) I use a weighted three-year average of every individual metric inside my model, with each year weighted based on a multi-variate regression of the prior three seasons. Each stat is then regressed to the mean based on its variance (ie, goals regress more than shots on goal). It’s a lot more complex, but at the heart, it’s basically measuring how good we can reasonably expect each player to be, at each particular metric in the model, based on how good he’s been in the past and how large of a sample we have on him. 

So to say “your model says this is what’s likely to happen (and no it’s not an “absolute” certainty), but have you considered that these guys have the lowest goals against” is a pretty condescending statement to make considering that it’s something already being considered. 

And that’s where we come full circle. Goals against are being considered. But for these particular players they do not outweigh the rest of their negative qualities.

Now there are a few inevitable errors to all of this and blind spots. I won’t fight them and I’m happy Johnson decided to critique my work here. When I first posted I was expecting more of it, but didn’t get much in terms of what could be missing, hopefully, because I covered my bases in the original post. Critiques are a good thing. They provide alternate perspectives from an outsider who won’t be blinded by the attachment that comes from creating something. They give you chances to learn, grow, and improve.

On the surface, I think Johnson has a valid argument. I personally think defensive skill is largely underrated in my model. I see players like Marc-Edouard Vlasic and Niklas Hjalmarsson and Chris Tanev get rated lower than some might suspect. It’s difficult to assess defensive value using the data that is currently available.

At the same time, though, I do think that defensive acumen is a bit overstated and a lot of what we see there is related to systems and goaltenders. While it may be off on certain players, I think as a whole, when all the players are combined into one team rating it is a pretty accurate assessment of that team’s talent. If it wasn’t I would probably be very poor betting with it, but I am not. The model would probably also struggle to pick which team is more likely to win on most nights, but it doesn’t.

I don’t think it’s the best model in existence (I would give that honour to DTM’s variant of WAR, personally), but I am reasonably confident in it and what it’s shown so far. Maybe it is a little too harsh on Hunwick, Smith, and Polak, all rated as below replacement level. Maybe it is too kind to Marincin, Holland, and Corrado, all rated as bottom line or bottom-pairing guys. But based on the separation between each group, I’m much more confident that one group is better than the other. And no, that’s not absolute. I’m willing to keep an open mind that these players are better than my model says they are, and perhaps better data will one day show that. But I don’t think on-ice goal statistics are the answer.

You may be 100 percent certain that my model isn’t perfect, and in fact, I would 100 percent agree with you. But I am 100 percent certain it is a more accurate representation of player quality than goal differential.

  • tealeaves

    Has anyone shown that an extra 2% is important? why not 1% or better yet an extra 3 or 5%
    This 2% sounds an awful lot like asking someone to give 102 or 110%

    • HockeyKeeperKit

      It’s based on a book title about baseball statistics. The main point in the last article was swapping the 3-for-3 players conveniently boiled down to about 2 more wins on a season if I recall, or 4 more points in his model. Does it seem like much at the present? Not really. But 4 extra points on the season can be huge when you’re fighting for a playoff spot.

  • Harte of a Lion

    Dom, as a fan I appreciate the time and effort you put in analyzing all of the players to come up with the 2% however we still have to play the games. I’m not sure why Babcock has no faith in Peter Holland but those are the facts. Also, no matter how poorly Polak and Hunwick have played, the team does needs a veteran presence even if I wished they were other veterans. I believe Frankie Corrado or even one or more of my favourite Marlies like Justin Holl, Rinat Valiev or Victor Löov would be upgrades on the back end, however certain decisions/players being played is due to the upcoming expansion draft.

    • LukeDaDrifter

      Hi Buddy, Not too many comments by you here lately. I miss your reasoned positive perspective. It balances out the negativity from the armchair GM’s who are trying to accelerate the rebuild. I know you follow the Marlies closely. Because of band width limitations over here, I have only been able to watch some highlights. Do you have season tickets again this year? How do the new look Marlies appear to you? I see Nielson is getting lots of goals and assists. Dermott I see is getting assists. How are they looking defensively? Do you think they are close? How is their skating? Do they have NHL strength and quickness?

      • Harte of a Lion

        I’ve been very sick from the chemo however on the good days, I’m still in for my 2 cents US which is approximately 2.6 cents Canadian. Have given away most of my Marlies tickets. They are still a great team however they have struggled on the road somewhat and replacing Brennan’s production from the point “by committee” is not yet working. Nielsen looks like a 3rd round steal but needs to improve his 5×5 play. Valiev isn’t putting up huge points but drives the play every shift. Löov is Löov and has been a steady presence on the ice and deserves a chance with Valiev on the big team. Depending what the Leafs do in free agency to improve their defence, Löov and Valiev could replace Polak and Hunwick next season.
        Leipsic who continues to leads the AHL in points has grown his game by improving his speed and edge work while working diligently on his coordination (deflections) and his already NHL caliber shot. Kapanen looks like a new player this year and is almost a point /game while equaling his 2015/16 goal total in 27 less games. Rychel has had terrible puck luck but his motor never stops and you can feel every shift he is about to break out. So many prospects, so little time to write about them LOL.
        There are certainly some holes on this years Marlies however without replacing Brennan or Nylander, that’s to be expected. Even with Laich, Michalec and Greening, they are still one of the youngest teams in the AHL.
        Luke, Glad someone reads and appreciates my contributions. Your in the Philippines? The other side of the world. Go Leafs!

        • LukeDaDrifter

          Thanks for the in depth update on the Marlies. Sorry to here about your health issues. I know it can’t be fun. Yes I am in Philippines. They do have a summer and winter here but only a real expert could tell the difference.With the daylight savings change in Canada, hockey normally comes on a 8 in the morning. What I was really impressed with Nielson, in preseason and on Marlies highlights, was his ability to get a shot through from the blueline.With so many good shot-blockers these days, that is a lot harder than it looks. I would have to agree with your thoughts on Bracco. Adam Brooks in the WHL is averaging just over 2 points per game as well. With all these promising prospects it makes you wonder how it will all play out in the end. It makes you wonder if we end up trading a young prospect that we regret for years later, as we have seen in the past. On the other hand there is a lot less job vacancy’s this year already.

  • LukeDaDrifter

    This of course leads to some big questions. Why are Polak, Hunwick and Smith in the lineup, instead of Holland, Corrado and Marincin? Did the Leafs make a huge mistake hiring the guy considered the best coach in hockey (Babcock), making him the highest paid coach, at 7 million with term? Are we now finding out it was all just smoke and mirrors? Is Babcock not really good at evaluating hockey players?. Is Babcock poor at putting forward lines and defensive pairing together? Could we have saved a lot of money and improved our winning percentage by putting an analytics guys behind the bench? With Corrado’s 5 on 5 corsi number one on the Leafs D-men and Rielly’s 5 on 5 corsi sixth, shouldn’t Rielly be sitting in the press box and Corrado playing on the top pairing?

    • HockeyKeeperKit

      You really have to stop comparing low depth-chart players with top depth-chart players to try to prove a point. You did a similar thing before comparing someone else to Gretzky et al.. Reilly is on the top line for his offense. We all know that, including you. His defense is improving. We are simply comparing the bottom pairing guys right now so stop with the sensationalist comparisons.

      • LukeDaDrifter

        Ask Babcock. He believes Reilly is also the Leafs best defender. He recently explained why Rielly is not seeing much power play time. After the other team kills the penalty Babcock knows they will ice their best line. He wants Reilly available to defend against them. He can’t do that if Rielly needs a rest after just coming off the ice. For me that explained why it is not uncommon to see a team after killing a penalty comes right back and scores. I always though it was because the team got a lift from killing the penalty. It appears there is a lot more to it than I ever thought.

        It was Jeff’s stat chart I was using, which he published here and again the other day on twitter. Rielly was also on that chart so why wouldn’t I use him. Either the chart has value or it doesn’t. Why is it ok for Jeff to use that chart to show how good Corrado is but I can’t use it to show how unreliable Corsi stats are? Why is it when the analytic numbers are completely opposite to common sense there are always a so many good excuses to explain away results? Here we are saying it was the goalies fault. Other times I hear it was puck luck or look who he was playing with etc.

        • HockeyKeeperKit

          https://uploads.disquscdn.c

          I can’t really speak for Jeff’s post as I never saw it. Because of it, I looked up the data myself and Rielly is top 3 (attached) while Polak and Hunwick are dead last. If Rielly is being used more defensively than that’s to be expected. Polak and Hunwick’s are probably being used moreso off defensive starts so I’m also not really surprised they low but they are drastically lower than the rest.

          Rielly and Polak/Hunwick (or even Corrado) are being used differently or have at least different skill sets. Their numbers aren’t really comparable. The Rielly-Zaitsev pairing appear to be used even more in the defensive zone that Polak and Hunwick. While Polak/Hunwick do have more defensive starts than offensive, their lean isn’t as dramatic as Rielly/Zaitsev.

          Not to get too off topic on stats, but the Rielly/Zaitsev and Carrick/Gardiner combos are far more stable pairings than Polak and Hunwick; not in the sense of overall play, simply in the fact that they play together far more (last 10 games, those two pairs played together about 34% and 18% of the time). Polak and Hunwick have shifted around a lot more so it is likely either of them that would get subbed out. You could probably throw Marincin in there as he doesn’t have a consistent pairing. Again, I don’t want Corrado in all the time, just more often.

        • HockeyKeeperKit

          The article explained why those numbers would be so low. Bad goalie S% was the main factor as was the team being largely rookies following the trade deadline in a race to get 1st overall. Is Corrado an all-star? No. Is he marginally better than Polak or Hunwick. Possibly. Lets find out for sure? Play him every third game. Every back-to-back (yet I’m sure he’ll get the blame for a tired team). Give the guy a chance.

    • Kanuunankuula

      Because Babcock is good, but not perfect. This seems to elude a lot of folks here. You can believe he is the best coach in hockey and still think his personnel choices are lacking.

      But we do have a GM. He does not have to agree with the coach and his personnel usage. He even has the power to negate them to some degree. I think it’s beginning to reflect rather poorly on that side that they’re not stepping in.