Hockey’s Structure and Incentive Problem

 

 

Yesterday I discussed some of the entrenched misperceptions about statistical analysis in hockey and why they are misguided. Today, in a post called "Hockey’s Counting Problem", Cam Charron looked at some of the significant obstacles standing in the way of effective, evidence-based analysis truly getting a foothold in the NHL’s upper offices.

Two challenges he touches on that I want to discuss in greater depth are top-down vs bottom-up processes and the clash between long-term decision making and short-term incentives.

Cam touches on why "advanced stats" have proliferated in blogs, comment sections and messagboards over the last decade or so: the benefits of an open, non-centralized structure are iterative, self-correcting processes. Meaning, the spontaneous order of a bunch of independent minds conducting experiments and making arguments results in a sort of "wisdom of crowds" effect, where evidence and truth eventually win out. This bottom-up process means things that can skew decision making, like authority, hierarchy and adherence to convention are lessened*.

Hockey teams, in contrast, tend to be rigidly top-down, hierarchical organizations. In addition, the league is very much an "old boys club" consisting largely of same-names and cultural warriors who are committed to defending the status quo. As result, there are entrenched, institutional barriers to innovation, both within teams and throughout the entire league. 

Of course, there are obvious reasons teams no doubt strive to improve their player evaluation and scouting – everyone is trying to get the upper hand, after all. The problem is the assumptions of authority figures aren’t as easily challenged in this sort of setting: like water, information doesn’t readily flow uphill, especially if it runs counter to the authority of the guy in charge.

Teams are "bunkered" as well, meaning they don’t tend to share data or breakthroughs with each other. Think of it like a collection of prisoners separated by foot-thick concrete walls with no windows. Each guy is trying to find a way to break out of jail and each guy has certain, unique bits of information. If they could share their data and collectively generate an escape plan, their chances of breaking out drastically increase. Instead, each prisoner wiles away his hours gazing at the walls, foiled by the unknown unknowns. 

As a result, there’s limited data sharing and feedback on top of a greater risk of convention, authority and groupthink obscuring efforts to advance knowledge.

As Cam notes above, the other reason hockey analysis has evolved much more rapidly amongst "outsiders" like bloggers is because it’s easier (that is, less costly) to be publicly wrong versus your average coach or GM. In hockey, the vast majority of "new" analysis is long-term, broadbrush stuff – looking to tease apart the skill signal from the vast, competing noise can take a lot of time/shots/games etc. Amateurs and fans engaging in this analysis outside of the incentives and pressures of actually running a team have the benefit of time and endless iterations to eventually get things right. Guys in positions of power in the NHL? Not so much. 

Unfortunately, the informational and predictive value of a single hockey game is incredibly small relative to the value of a game in the eyes of fans/players and decision makers. For example, it can take between 3000-4000 shots to get a feel for a goalies true talent level, meaning an average 30-shot game is about 0.01 (1%) of the info you need to make a truly informed decision about a puck stopper. In contrast, NHL executives and coaches are judged on individual wins and losses as well as clusters of success or failure that can have very little to do with the actual quality of their roster or their decisions. I’ve argued previously that coaches are often sacrificed before the percentages; that is, fired because of natural variance. 

NHL decision makers are almost always under pressure to win now – and if they aren’t winning now, to figure out why they’re losing as quickly as possible. They are frequently dealing with only small snippets of useful information, often awash in a sea of competing signals and unpredictable or uncontrollable factors like schedule strength, officiating and injuries, to say nothing of the natural ebb and flow of human performance. The catch-22, then, is that the concern for the short-term can override the need to be patient and sort out true talent signals from the dead-ends and blind-alleys. And sometimes even patience isn’t enough – managers and bench bosses will occasionally have to make decisions about players based on less than a full season of data.

There aren’t any obvious or easy solutions to the issues posited above. Different clubs will have different corporate cultures, so openness to new ideas will vary to some degree depending on the people and management in place (although everyone will continue to suffer from the "bunkering" effect, or the inability to share, check on and improve findings). Encouraging experimentation and the challenging of even long-standing assumptions inside an organization would probably help.

In addition, GM’s will always have to battle the urge to overly weight and react to short-term results given how much time/data it takes to truly separate the wheat from the chaff in the NHL. It would help to have stable, patient ownership as well.