August 17, 2010

Expanded Horizons

Mariano Rivera, Outlier

by Tommy Bennett

the archives are now free. All Baseball Prospectus Premium and Fantasy articles more than a year old are now free as a thank you to the entire Internet for making our work possible. Not a subscriber? Get exclusive content like this delivered hot to your inbox every weekday. Click here for more information on Baseball Prospectus subscriptions or use the buttons to the right to subscribe and get instant access to the best baseball content on the web.

Subscribe for $4.95 per month Recurring subscription - cancel anytime. Subscribe for $39.95 for one year a 33% savings over the monthly price! Purchase a $39.95 gift subscription a 33% savings over the monthly price! Already a subscriber? Click here and use the blue login bar to log in.

Analysis Begins with Questions

One of the great things about baseball analysis is that you can take a relatively simple question—like “who is the best relief pitcher in baseball?”—and break it down into dozens of sub-questions. Put another way, we have to answer questions about methodology before we can answer the substantive question at hand.

An ideal quantitative methodology for determining the best relief pitcher in baseball would rely on information known to be predictive of future success, including peripheral stats like strikeouts and walks. But it’s also true that, because even peripheral stats can bounce around a fair bit over a limited number of innings, we want to know something about outcomes. Outcomes give us a measure of performance that doesn’t entirely overlap with peripherals. By using win/run expectancy to value relievers, we are not simply duplicating the information provided by peripheral stats while adding noise to the mixture. On the contrary, we are valuing relievers on a different axis: the crucible of high leverage.

But there’s another, and I believe overlooked, reason to focus on outcomes. To see why, we have to start with a very particular sort of answer to the question posed at the outset of this column (“who is the best relief pitcher in baseball?”). So let’s borrow some methodology from an article I wrote back in March to look at some of the best relievers not yet slapped with the closer tag. We’ll take WXRL and SIERA for all pitchers who have pitched solely in relief. For each pitcher, we’ll calculate how many standard deviations they are away from the mean in each category. Then we’ll add them together. For example, a pitcher who was one standard deviation better than the mean in both SIERA and WXRL would get a score of two.

So if you do that and figure out the top 10 scores for the current season, you get a list that looks like the following:

Name	SIERA	WXRL
Brian Wilson	2.47	4.78
Heath Bell	2.85	4.71
Carlos Marmol	1.96	3.46
Joakim Soria	3.02	4.65
Hong-Chih Kuo	2.38	3.61
Matt Thornton	2.18	3.25
Joaquin Benoit	1.29	2.13
Daniel Bard	3.07	3.85
Mike Adams	2.67	3.00
Luke Gregerson	2.13	2.31

This is a decent list. The kind of list you can live with a for a while, thinking to yourself that there are some well-established guys (Bell, Marmol, Soria) as well as some targets of man-crushes (Thornton, Benoit, Adams). But then you notice the glaring omission. See it yet?

One Size Fits Almost All

The problem with this list, and with any list like it created in the last 10 years really, is that you could just throw it out. Who needs a list like this when the answer to the question “who is the best relief pitcher in baseball?” is so simple? You could just give the same answer—Mariano Rivera—every year and be right in a very real sense every year. It’s not that Rivera fares especially poorly by this metric. He actually comes in 13^th out of nearly 200 qualified pitchers. It’s that by not putting him closer to the top, the list fails to pass the smell test—it appears wrong on its face. Instead of a definitive ranking, we get a neat-o list with little meaningful difference between the individual rankings. That’s disappointing.

Part of the reason why Rivera doesn’t rank higher is because his SIERA is not especially fantastic this year. At 3.01, Rivera clocks in at 31^st among relievers with at least 20 innings. By more traditional metrics, of course, Rivera is having a superlative year: 1.06 ERA, 36/7 K/BB in 42 1/3 IP. That isn’t enough for SIERA, however. His relatively low K rate hurts him, and he gets no credit for the exceptional job he’s done at preventing home runs (just the one this year, thank you very much). He’s also having a career year in terms of preventing hits: just 23 all season. So what gives?

Let’s start with the hit rates. Rivera has been consistently excellent at preventing hits. His career BABIP is just .263, and he has allowed fewer than seven hits per nine innings. To illustrate the point, let’s compare Rivera’s seasonal hit rates to the distribution of hit rates for all relievers from 2007 through this year.

Now, I suppose it’s possible that if we could clone Rivera and run his career over again, we’d end up with a spread closer to the mean for all relievers. But I didn’t single out Rivera’s hit rates because they were low, I singled them out because all of his other pitching attributes have been so dominant for so long. That means our selection bias sensors can calm down for just a second, and we can marvel at what is a pretty darn impressive ability to limit hits.

Of course, any pitcher who strikes out enough batters will have a similar sort of distribution, since hits per nine innings is influenced by the total amount of contact made by opposing batters. To illustrate this point, let’s compare Rivera’s chart above to a similar chart for another highly successful reliever: Billy Wagner.

Wagner’s career .265 BABIP is just barely behind Rivera’s mark, and his average H/9 of 6.0 is actually lower than Rivera’s, as reflected in the chart above. What we can learn from these pitchers is that the shorthand that pitchers don’t control outcomes on balls in play breaks down at the extreme represented by elite relief pitchers. These pitchers—who basically only pitch with maximum effort in higher than average leverage situations—can in fact show the ability to limit hits more than starting pitchers. This is true even above and beyond the ~.005 split in BABIP between relievers and starters. That means estimators that model reliever performance on the assumption that hit rates revert heavily back to the mean may overlook this particular skill. But it also means Rivera isn’t unique—in other words, that can’t be what keeps him from the top of the chart above, since plenty of relievers exhibit the same ability.

Now we should ask whether there is any other skill that Rivera has that we might be overlooking. Sure, he has posted above average ground ball rates his entire career, but that is definitely reflected in his SIERA (which includes ground ball rates as well as various ground ball interaction terms). What about home runs? Rivera has allowed just 61 home runs in his career—one that has spanned 1,132 innings. Since 1996, he hasn’t allowed more than seven home runs in a single season, and from 1996-2009, he has allowed three home runs or fewer in eight different seasons.

In fact, if we look at the distribution of Rivera’s seasonal HR/9 rates versus all relievers from 2007 through this season, we notice just how exceptional Rivera has been at limiting home runs.

Now if we compare that with a similar chart for Wagner, we’ll see a big difference.

Now, some of that is due to the differences in ground ball rates Rivera and Wagner have put up over the course of their careers. But whatever your favorite ERA predictor/post-dictor/third-world dictator may be, most assume in one way or another that limiting fly balls—not home runs—is the relevant skill. And while that appears to be true for someone like Wagner, who has posted rates of home runs per fly ball that hover around league average, it is decidedly not the case with a rate environment extremophile like Rivera.

Question of the Day

It appears as though our best estimators do a good job for 99 percent of all players, but break down at extremes. A pitcher like Rivera, who is extreme in almost every way possible, simply doesn’t rate properly if you use the same metrics used to measure other guys. The alternative view is that Rivera really isn’t quite as good as he’s made out to be. Is anyone willing to make that argument?