Forecasting James Reimer and Jonathan Bernier

If you’re a sports analyst, one of the main problems is confronting the issue of survivorship bias.

In sports, results drive most of the business, even though the results from the previous year don’t always impact the results in the next year. Very few teams across sports are willing to be patient and stick to a process that will lead them into making good bets. “Buy low” is generally accepted in the stock market but not in sports. The Toronto Maple Leafs this offseason have bought high on Tyler Bozak and David Clarkson, rather than bought low on Clarke MacArthur, Mikhail Grabovski and Nikolai Kulemin.

Say you have two players with a “true talent” of 20 goals per season. The first player got some bad puck luck and scored just 15, while the second got some puck puck, a few extra chances, and scored 25. More often than not, when choosing between the two players, a team will go for the 25-goal scorer, even though the 15-goal scorer would be cheaper and he’s just as likely to produce at the same level in the future.

Despite scoring just 14 goals to David Clarkson’s 45 over the last two seasons, I firmly believe that Nikolai Kulemin is just as good as the slightly older, much more expensive unrestricted free agent the Maple Leafs picked up. Unfortunately, there is no way to test the theory unless you have thousands of multiple universes, and half of them play Kulemin in Clarkson’s spot in the lineup and the other half play Clarkson in that spot. We’d have to average out the totals of the universe to see who was right or wrong. For better or for worse, we have just the one universe, and its randomness, though teams make worse decisions because of it, makes sports more exciting.

I can sit here and say that I’d take Kulemin over Clarkson, but I’m not under a lot of pressure to bring notable players to Toronto and sell tickets and jerseys and convince Vinny from Woodbridge that the team is doing its best to perform on the ice. Working on the margins doesn’t necessarily work for a team like Toronto where every move is scrutinized in hindsight. I prefer to work with foresight.

Which brings us to Jonathan Bernier.

Bernier is an interesting case because he blends “buying low” with “buying high”. The Leafs spent quite a bit on Bernier: a pick, a depth winger, a backup goaltender and $500k in salary cap space, plus the contract they gave him, which works out to a $2.9-million cap hit.

That’s the price you pay for a former first round pick when you need to sell to your fanbase that you’re looking to fix any lingering goaltender issues.

But then there’s the “buying low” aspect. Bernier never made it as a starter in the NHL, at least not yet. Among active goalies, he is 61st in minutes played, behind noted gems such as Al Montoya, Jeff Drouin-Deslauriers, and — most notably — James Reimer. His career save percentage of .912 and even strength save percentage of .916 are not particularly impressive numbers. The average backup goaltender would have made 3.4 fewer saves over Bernier’s career than Bernier did, and when you consider it’s been seven years since he was the 11th overall pick, his NHL resume is none too impressive.

Comparatively, James Reimer has been the Leafs’ starter for three separate half seasons. Making his debut in 2011, Reimer dragged the Leafs out of the depths of the Eastern Conference standings with a .921 save percentage and a 20-10-5 record. On New Year’s Eve 2010, the Leafs were 13-19-4 and in 13th place. The next day, Reimer took over as the de-facto starter, and at the end of the season, Toronto was 37-34-11 and in 10th spot. Concussion and injuries took him over in the 2011-2012 season, but he responded with a .924 save percentage in 2013 and led the Maple Leafs to their first playoff appearance since Ed Belfour was the Leafs goalie. Belfour has not only retired, but he’s also got himself a Hall of Fame plaque.

Compare the two career records:

With more shots faced, and performing better relative to the average backup (by combining the even strength save rates of goaltenders that were not in the Top 30 in shots faced in any given year), Reimer appears to be the leader.

But should we be entirely convinced he’ll have the better career? Think about regression to the mean:

Reimer possibly represents an extreme case. Since he’s performed well at even strength since coming to the NHL, I’d have him as my opening day starter.

That said, can we use comparable players to see whether Bernier or Reimer will perform better over the next three seasons? Well, we can try.

The first step was downloading all of the NHL.com goaltender data (I looked at even strength only, since I’m a believer in total save percentages being influenced by a strong penalty kill) and sorting it per season, determining the save percentage of an average backup, and filling in birthdates for players as necessary. While the NHL season officially listed a 24-year-old as a player that turned 24 during the hockey season, for simplicity I used the junior hockey style, marking down a 24-year-old as a player that turned 24 during 2012.

With help of the delightful Rob Pettapiece in helping me figure out Microsoft Excel’s Pivot Tables, I was able to sort career-to-date statistics of goaltenders at age 24, and how they did beyond that. Survivorship bias is, unfortunately, huge when evaluating goaltenders. If Reimer were the first round pick in 2006 and Bernier were the third round pick, I would have no doubt that Bernier would not even be on the Maple Leafs’ radar.

It’s difficult to find comparable players, however, particularly in Reimer’s case. Limiting my goaltenders that were born between 1974 and 1985 so I would have full data prior to 1998 when the NHL began tabulating EV save statistics, and so that I could cover three seasons for goalies that were 24 three years ago, there are not a lot of goalies who came close to Reimer’s 28.0 Saves above Average Backup (SAR). He would have been 10th on our list.

Let’s define the comparables to include all players within .005 of save percentage at even strength up to age 24, and see how they combine to perform at ages 25, 26 and 27:

There are 24 goalies age 24 and before belonging in Reimer’s sample, and 27 in Bernier’s which were culled to 18 and 22, respectively. Of those goalies that continued, they performed at a similar level. The Reimer group’s overall save percentage should be considered slightly higher since they tended to play in eras when even strength save percentages were slightly lower, but overall, the difference is pretty minimal, especially when you consider that certain goalies who (perhaps unluckily) played themselves out of a job in the Bernier group may have had more success in their future years if given the chance.

This is not a perfect way of doing it at all, but there are no real “good” ways of forecasting goaltenders. They get into the league, they try to make saves, and if they don’t make enough of them they don’t last and never get the chance to regress back to the mean.

So perhaps looking at Jonathan Bernier’s save percentage and criticizing the move isn’t the right way to go. Bernier is probably a pretty good goalie. Where the move deserves criticism is that the Leafs gave up a goalie with a similar track record in EV SV% (.9162 to .9171) to bring over a goalie that was a year younger and has made just 439 more even strength saves at the NHL level. Bernier also costs $2,787,500.00 more when you figure in the salary cap retention of the deal, plus a depth winger and a pick. 

In other words, if you’re going to buy low on a goaltender with a spotty career record under the belief that he’ll regress towards the mean, you may as well actually buy low on him.