The fatal flaw of analytics

Whenever a hockey player doesn’t talk to the media, a common refrain from reporters is that “the media is how fans connect to players”.

That is true, but not in the way the reporters think. The reality is that the media that connects us to hockey players is the television and radio—we’re dependent on those mediums to glean as much information as we can get about a hockey game without ever being there. Television is the best, because it’s visual. Radio is second best, because you can follow the progression of the game in real-time. There’s a big drop-off from there. I dislike following a game via a boxscore because there’s no drama involved, and even worse is reading the game recap the next day in the local newspaper, because even a 700-word recap can gleam over 95 per cent of the game as if it never happened.

It’s not so much a player’s post-game quotes that fans want to be privy to, but to actions. We are just months removed from two of my favourite moments as a sports fan: Marie-Philip Poulin’s heroics in the gold medal game of the women’s Olympic hockey tournament, and Percy Harvin’s kick return for a touchdown in the Superbowl. I consumed both events on a television, have a vivid recollection of my mental state during those events, but have, curiously, no recollection of what either athlete said after their respective games. It’s funny how that works. Two weeks before the Superbowl, I was lucky enough to be in Seattle for the NFC Championship game, a game best-remembered for Richard Sherman’s post-game interview. To the 60 thousand of us in attendance, however, or celebrating in Seattle after the win, we were too busy partying to have any care as to what Sherman said. The event did not shape any aspect of the way the game was played on the field, but reporters made so much of it—reporters who bank on interesting post-game quotes to write their stories.

I don’t mean this post to read like a study in media criticism, but there seems to be a concern in Toronto that Brendan Shanahan won’t use the Toronto Maple Leafs’ analytics budget the way we’d hope for it to be spent.

Per Mirtle:

In a previous post, I made the point that it didn’t matter whether an educated fan consulted ExtraSkater or watched games. As long as he had an opportunity to consume the actions of players, he’s going to be no more wrong than a guy like Shanahan or Dave Nonis when it comes to projecting performances. That’s not meant to be a slight on Shanahan or Nonis, but more a testament to how the technology has evolved to allow the die-hard fans to watch any game they choose. You can work hard enough and have the most-informed opinion on any player in the world.

There’s a misconception that analytics ends with Corsi and Fenwick. Those two numbers, which indicate puck possession, don’t tell you much about particular players, but can be a starting point for gathering more information.

Over the last two seasons, Tyler Bozak has a Corsi percentage of 27.5% when he’s not on the ice with Phil Kessel. The end point isn’t “Bozak is terrible without Kessel” (although it would be a pretty good guess) but if you want to think analytically, you’d want to consult the video or break down the data even further to find out why the Leafs get out-shot 72-28 when Bozak is on the ice without his regular winger. Is he earning more defensive zone starts? Is he being asked to dump the puck in more? Is he caught in the offensive zone because he’s waiting for passes that are unfortunately not coming from players named Phil Kessel?

There’s a lot to gather in, and a lot of things one could research that don’t end with Corsi or Fenwick. Ideally, you’d break down situations and find out what lineup decisions, strategies and player combinations are the most successful, and then further break down why. To what extent does chance play a factor? To what extent does the coach’s breakout tactics play a factor?

You can figure out all those things, but ultimately it doesn’t matter if you don’t have enough resolve or conviction to implement a strategy based on the information you’ve discovered. In Mirtle’s same interview with Shanahan, it seems clear that he agrees Randy Carlyle’s defensive zone strategy did not work a year ago and that the Leafs were too late to implement changes.

It’s fair enough for Shanahan to say “we think Randy is a good coach and he’s committed to being less aggressive in the defensive zone, more aggressive in the neutral zone and more lenient with our talented defencemen”.

Shanahan didn’t say that, though. He said this:

And it reeks of yet another executive valuing the results over the process.

Carlyle has been a coach in the National Hockey League for nine seasons. He is 1/2 in “winning Stanley Cups” when he has gone into a season without Stanley Cup-winning experience, and yet is somehow 0/7 as a winning coach. A more damning statistic appears to the left. It seems that since winning a Stanley Cup, Carlyle no longer has the ability to win a Stanley Cup.

But the lone 2007 Cup is the crutch and the excuse. The Leafs could spend a billion dollars trying to find out why the team gets outshot 72-28 when Bozak is on the ice but not Kessel, but that money is better off not being spent if nothing gets implemented to change the fact. Fans that are banking on the Leafs spending their analytics budget are really banking on the Leafs making tough decisions, such as not retaining Randy Carlyle, an easily-marketable coach because he has Stanley Cup-winning experience.

The critical difference between fans and management is that fans have no obligation to put the needs of themselves and their job security and networks over the actual needs of the organization. In hockey, the research generally indicates that the more aggressive plays with higher reward, such as skating the puck in or out of the zone rather than dumping it and chasing after it. That’s not dissimilar from other sports like football or baseball, when the math overwhelmingly shows teams punt too much (giving up possession) or bunt too much (risking giving up a precious out for a single base). In tennis, the research shows that players are too conservative on their second service, for fear of a double fault (even though a weaker second serve generally means a much easier return shot for the opponent and a huge advantage for the non-serving player).

Ultimately, a risky decision that fails will be judged more harshly than a conservative decision that fails, since the riskier decision, especially if analytics are involved, doesn’t have results or an easy narrative to back up on. I’m not certain if the Maple Leafs fight at a disadvantage with this kind of stuff, but it’s been clear over the last few seasons that management is willing to make moves that make the team look better in the short-term and ignoring longer term consequences.

Whether Shanahan spends the analytics budget or not doesn’t change that fact. The Maple Leafs still need somebody in place with the trust of ownership to make the tough decisions even if a section of the fan base disagrees. Every major Maple Leafs move since the James van Riemsdyk trade has been predictable and none have fared particularly well (check the comments on this post, compared to the present day beliefs in Toronto about Joffrey Lupul). I don’t expect the Carlyle extension to work out long term, but right now the excuses for keeping Carlyle were easier to make than the excuses for firing him, and that’s all that matters to the front office at this point.