December 16, 2015

Baseball Therapy

Have We Been Underpricing Relievers?

by Russell A. Carleton

the archives are now free. All Baseball Prospectus Premium and Fantasy articles more than a year old are now free as a thank you to the entire Internet for making our work possible. Not a subscriber? Get exclusive content like this delivered hot to your inbox every weekday. Click here for more information on Baseball Prospectus subscriptions or use the buttons to the right to subscribe and get instant access to the best baseball content on the web.

Subscribe for $4.95 per month Recurring subscription - cancel anytime. Subscribe for $39.95 for one year a 33% savings over the monthly price! Purchase a $39.95 gift subscription a 33% savings over the monthly price! Already a subscriber? Click here and use the blue login bar to log in.

Last year, the must-have item of the Winter Meetings shopping season was a catcher with mad framing skills. In a short period of time, Hank Conger, Ryan Hannigan, Miguel Montero, and everyone who owned a chest protector on the Padres roster changed teams. This year, tastes have changed. Now, the new hip thing that all the cool teams have is a crazy good closer. More to the point, a second crazy good closer to pitch in the role once known as “the eight- inning guy.” It’s not enough to have one shoe any more. You need two.

Since Wade Davis got Wilmer Flores looking in the 12^th inning of Game 5, Craig Kimbrel has been traded for a quartet of young prospects. Apparently not satisfied with that, the Red Sox also nabbed Mariners wunderkind set-up man Carson Smith. Not to be outdone, the Phillies got a quintet of kids for their closer, Ken Giles. The Dodgers had agreed to trade for Aroldis Chapman to pair with Kenley Jansen, although allegations of Chapman being involved in a domestic violence incident scuttled the deal. And the Yankees are reportedly dangling Andrew Miller as possible trade bait.

What’s interesting is that while deals for high-end relievers have always happened, the knee-jerk reaction that a smart team should never spend resources to chase something as fleeting as relievers, and especially closers, seems to have been skipped this year. Yes, the save stat is still silly. Yes, all of those relievers will pitch only 65 innings this year, and it’s still hard to trust their numbers in such small sample sizes. Yes, there will be some small number of relievers that put up microscopic ERA numbers in a small sample and will score a huge contract based on that next year.

A few weeks ago, in reviewing the Kimbrel trade, I suggested that the issue with paying for high-end relievers isn’t that it doesn’t make sense in terms of expected value, but that it’s a high volatility investment. Signing (or trading for) a very good closer makes perfect sense, even if you have to put some assets into it. I think a lot of the closer-hate came from a misunderstanding of what those truths mentioned above really meant. Saves are silly because they reward only the guy who was the last pitcher and ignores the other two guys who pitched scoreless innings in the seventh and eighth. It’s not that the closer doesn’t deserve an “attaboy.” He did his job, but the fact that he pitched in the ninth and racked up saves doesn’t necessarily mean that he’s better than the guy who did most of his work in the eighth. Teams shouldn’t pay for past saves; they should pay for the ability to get outs in tough situations. Saves might be an indicator of that talent, but not always. But while saves are not necessarily a good indicator of pitcher talent, I think people forgot to look at how incredibly valuable the act of “saving” a game is.

Fast forward to last week on Effectively Wild, where Ben and Sam wondered aloud whether the market was going crazy or if this was all a rational thing to do. Maybe it made sense to grab one elite reliever, but what about the teams that were grabbing two? It was Sam who spoke the profound words that the flip side of realizing the ninth inning is just like the eighth inning is realizing the eighth inning is just like the ninth. If we’re going to say that if you can pitch the eighth, you can pitch the ninth, then it also makes sense that if you’re going to get someone to lock down the ninth, you might as well get someone to lock down the eighth.

Sam also brought up the idea that one reason for opening up the vault for a reliever was the idea that relievers decline faster than starters. All big contracts take a risk on the back end, often paying for years when a player won’t be worth his salary to buy the upfront years where he will be a bargain, but are relievers more likely to turn more quickly into pumpkins?

What to make of the market for relievers? Have we been undervaluing them all along?

Warning! Gory Mathematical Details Ahead!

In reviewing the Kimbrel deal, I tried to make the case that WAR is a bad way to evaluate relievers. The problem with WAR in this case is that in our attempt to strip the context out of everything, we neglect that relievers, particularly closers, are used in a very specific context. Closers pitch at times when the leverage is high. By definition, if they don’t do their job well, it means much more in the win-loss column (the one that actually counts) than just a randomly selected inning. And yes, WAR calculations do attempt to account for that, but I think the best way to show the difference is to reference this chart of the top pitchers by WAR and this one showing the top pitchers by win probability added. The “Best WAR” chart is all starters. The WPA chart is a healthy mix of both starters and relievers. When your role is defined by being in high leverage situations, it makes more sense to use the metric that accounts for leverage.

The flip side of that is that even a “decent” closer is going to rack up a lot of WPA, not because it’s a sign of him being good, but because he’s going to pitch in a lot of high leverage situations. Still, since he is pitching in those spots, we want him to be as good as he can be.

We see that a reliever pitching in high leverage spots and who has a good (or lucky… or both) year can add value on par with that of a starting pitcher. So, why is it that, even with today’s inflated prices for relievers, we haven’t seen the Zack Greinke contract being given to a top reliever? Or even the Johnny Cueto one? And realistically, does it make sense for a team to have two elite relievers, or is that a waste of resources.

In reviewing the Kimbrel trade, I found that the “average” team faces roughly 40 ninth-inning save situations per year. “Save” situations in the eighth inning (the pitching team is winning at the start of the inning by three or fewer) actually happen slightly more often, with the average team getting about 44 of them over the course of a year. Better teams will face more of them because they will be more likely to have a lead. We also know that in the modern game of baseball, it’s rare that a starter is actually out there for the eighth inning anymore, so it’s likely that the manager will be in his bullpen at the time. Why not have one of the best relievers in the game to hold down the fort then too?

In ninth inning situations, the difference between a completed save and a blown one is fairly easy to figure on the win probability scale, especially in the bottom of the ninth. If the visiting team’s closer blows it and gives up both the tie and lead, his team loses. If he only allows the other team to tie, he’s basically taken the game to a 50/50 shot. (The exact win probability will depend on whether the closer is actually pitching for the home or road team.) But as shorthand, we assume that an extra blown save in the ninth inning is worth about .75 wins to a team.

In the eighth inning, using data from 2005-2014, a home team entering the top of the eighth inning up by one had a 77.87 percent chance of winning. If they kept the lead, their win expectancy went to 89.04 percent, but if they gave up one run, it drops to 64.58 percent and if they give up two runs (now they are down a run!), it falls to 29.56 percent. For the home team an extra “blown save” in the eighth inning is worth at least .41 wins. For the visitors, getting into the top of the ninth with a one-run lead has a win probability of 84.59 percent. If the eighth-inning guy had allowed his team to get tied up going into the top of the ninth, his team now has a win probability of 46.96 percent. If he allows the home team to take even a one-run lead, the visitors are now looking at a win probability of 17.74 percent, making an extra blown eighth-inning “save” for a visiting team worth an average of about .52 wins. The average extra “eighth-inning save” is worth at least .46 wins. Let’s round it to half a win. That’s the difference between preserving the lead and giving it up.

We saw last week that a good year for a closer was a save conversion rate of about 90 percent while a bad one might be around 80 percent. We see that ninth-inning saves are more valuable in terms of win probability added, but half a win is nothing to sneeze at. If there are 44 eighth-inning “save” chances for an average team (probably a few more for a good one), and having a good guy in the eighth-inning role can save 10 percent (4.4 of them) from becoming blown saves, that’s more than 2.2 wins worth of value.

To put it another way, if I could administer a potion to a closer and have it guarantee that the reliever would protect 90, rather than 80 percent of close leads in the eighth inning, that would be worth 2.2 wins, and that vial of potion should be priced accordingly. The problem is that we know that randomness makes it far from a guarantee that things will work that way. Now, it’s a matter of how confident you are in the potion.

With the Kimbrel trade, I put the value of moving from an 80 percent closer to a 90 percent closer at 3 wins, although with the caveat that because of sample size and randomness issues, a guy who would be a 90 percent closer given a billion chances might only notch an 80 percent rate over the course of 162 games. Still, we see that if a team could come up with some amount of predictive certainty that they had a reliever who was more on the 90 percent side, even if they were only 33 percent sure, they would have a good case to make that he’s worth a win or so more than the average closer. Still, that win is worth “only” $7 million, but it makes sense that high-end closers would be making that much more than a standard issue closer salary, which usually checks in around $5 million-$6 million, for those who were obtained in free agency. So, somewhere in the $10 million-$15 million range would be about right. Coincidentally, that’s around where top closer salaries tend to fall. And having second eighth-inning reliever in the $8 million-$12 million range could probably be justified too for some teams. So, the salary scale for relievers seems to be about right. Whether we’re handing the salaries out to the right people is another question, but we’re at least dealing with rational numbers.

And while a player’s seasonal save conversion rate might have a lot of variance in it, we’re not completely in the dark figuring out who the good ones and who the bad ones are. In fact, while a typical closer pitches to only 250-300 batters per year, that’s enough to get a good read on his strikeout and walk and ground ball rates. We know that those are fairly consistent over time, so why not use those as proxies? Enter the question mentioned by Mr. Miller of whether relievers actually decline at a rate faster than starters. That’s plausible, but is it true?

To look, I used data from 2010-2015 and separated out consistent starters (95 percent of their appearances in a season came as starters and they threw more than 150 IP, and the same was true the following year) from consistent relievers (95 percent of their appearances were in relief and they threw more than 50 IP in two consecutive years). I looked at their strikeout, walk, and ground ball rates in consecutive years. I put together a mixed design ANOVA (for the initiated, time[2] x group[2]) looking at each stat. In this sort of analysis, the most important thing to look at is the interaction between time and group.

Outcome	Starters, year 1	Starters, year 2	Relievers, year 1	Relievers, year 2
GB%	45.3%	45.2%	45.4%	45.5%
BB%	7.0%	6.8%	8.3%	8.1%
K%	20.1%	20.2%	24.3%	23.6%

The interaction term for GB% and BB% was not significant, but the results for strikeouts were. Starters, on average, pretty much kept their strikeout rates level. Relievers, though they strike more people out in general, lost 0.7 percentage points on their strikeouts over time. When I played around a little bit breaking the data out by age, it became clear that the strikeout finding was being driven mostly by “older” relievers (31 and up).

What’s interesting is that I looked at two more measures which should have some decent reliability: fastball velocity, and horizontal break on sliders.

Outcome	Starters, year 1	Starters, year 2	Relievers, year 1	Relievers, year 2
FB velo (mph)	91.2	91.0	92.8	92.5
SL X-break (in)	2.42	2.43	3.01	3.06

It doesn’t seem that relievers are losing velocity or movement on their breaking pitches at a greater rate than starters (both interactions were non-significant), so it doesn’t seem to be a loss of stuff. So, it doesn’t seem that relievers physically deteriorate faster than starters, but their results do.

(Side note: For the super-duper-initiated, you might be thinking, if he used a bunch of year-matched pairs, doesn’t that violate the independence of observation assumption. In other words, A.J. Burnett’s 2010 compared to his 2011 numbers gets a line. So does his 2011 compared to his 2012. Yes. You are right. And I actually went to the trouble of fitting a proper growth curve model. And it said the same basic thing.)

It’s not clear why relievers see a drag on their strikeout rate. One possibility is that we know that relievers often get banished to the pen because they lack a proper third pitch. In the bullpen, they might face only four hitters, so they don’t have to worry too much about mixing pitches and they can get by on stuff. But over time, “the book” on a pitcher gets out and even if a pitcher only has two pitches, if a batter is just kinda guessing, there’s at least a 50-50 chance he’s at least guessing the correct one. Then, it’s just a matter of seeing if he can time it up. Relievers may also be guys who specifically get by more on “stuff” and so when they lose a little bit, there goes their one weird trick to get guys out.

But the message is clear. Relievers become a liability to decay more quickly as they get older. So, when signing a free agent, a starter is more likely to hold his value than is a reliever over a few years. We’d expect to see shorter duration contracts for relievers… and that’s exactly what we get.

The Market is Wise

The idea of having a “second closer” or of spending resources to chase after an elite reliever just to have him pitch the eighth doesn’t seem to make sense at first, and running the numbers, having that sort of firepower in the eighth inning isn’t quite as valuable as having it in the ninth. That’s not the question that a baseball team has to answer, though. The question is how much value having that ace reliever in the eighth provides toward winning a ballgame, and if the answer is “a lot,” then how much is being charged for that privilege, and is that the best deal out there?

Given that there’s been a bias against paying anyone but the one guy who will pitch in the ninth for you any sort of money, there seems to be a nice little market inefficiency. I’ve made the case before that it’s a market inefficiency that has some big error bars around it. Relievers do still have to deal with the small sample size issue, and paying $10 million per year or a couple prospects for a flashy eighth inning guy who puts up a 4.50 ERA is never fun. But from a rational point of view, it makes sense to do.

What surprised me is that it seems that the types of contracts that are given to relievers are entirely rational in terms of their scale, both in dollars and years. Maybe we’re still curing teams of the over-emphasis on previous save totals, but the market seems to be in the correct place for relievers. And perhaps we’ll finally see eighth inning guys get their due. And bigger paychecks.

Russell A. Carleton is an author of Baseball Prospectus. Follow @pizzacutter4
Click here to see Russell's other articles. You can contact Russell by clicking here