This article originally appeared in By The Numbers, SABR's Statistical Analysis subcommittee's newsletter, Volume 5, Number 4, Dec. 1993. The only change made to the original article is the regeneration of Figure 1 with new parks and values (the original numbers were lost), and the corresponding modification of the text describing that figure. One significant change has been made to the method for calculating the SN stats since this article was published; that change is documented in the summary.

In recent years, we've seen the development and growing use of two measurements designed to evaluate starting pitchers on a game-by-game basis: Quality Starts and Game Score. Both measures are attempting in some way to look at the quality of each outing the starter has, rather than looking at the average or cumulative performance over the course of year like ERA does. But both measures have their weaknesses as total measures of pitching performance.

The arguments against Quality Starts are well known. Detractors point out that the worst qualifying outing -- 6 innings and 3 earned runs -- is not "quality" at all. A related objection is that Quality Starts makes no attempt to quantify the degree of quality a start has -- 6 innings, 3 runs is the same as 8 innings, 2 runs which is the same as a 9-inning shutout.

Partly in answer to these objections, Bill James developed the Game Score, which combines a starter's box score numbers (IP, H, ER, R, BB, K) using weights, where the weights are assigned such that the league average score is around 50, the best imaginable score is around 100, and the worst imaginable score (by someone outside the state of Colorado) is around 0. Game Score is acknowledged as an interesting measure of "game domination" by a starter, but it has weaknesses as a total measure of starter quality (i.e., his contribution to team victories): it's too dependent on strikeouts, possibly too dependent on hits and walks (after all, the number of runs given up is really the only thing that matters), and it isn't park-adjusted.

Despite the weaknesses of these two measures, looking at a pitcher's starts game-by-game is still a good idea. Looking at each start's contribution to winning, rather than cumulative run-prevention over the course of a year (ERA or Pitching Runs), can help us answer questions like: Given equal ERAs, do some pitchers pitch in a way that will tend to win more games than other pitchers? In particular, is it better for a starter to be flaky -- either very good or very bad on a given day -- or consistently average? Does the park have a smaller influence on the value of the start when the start is very good or very bad?

So here's what we'd like out of a stat measuring the quality of a start:

• it should depend only on numbers appearing in a box score.
• it should be independent of a pitcher's support, both from his team's offense and from his team's relievers.
• it should be park-adjusted.
• the resulting measurement should be in terms of some kind of meaningful unit, such as games or runs, rather than being a unitless index (and, ideally, it should be obvious to any baseball fan what a good or bad score in those units is).
• most importantly, it should reflect the contribution that a start had toward winning the game.

I've developed a couple of measurements that meet these five requirements. (Actually, the ideal stat would also be very easy to compute, but hey, 5 out of 6 isn't bad, right?). Support-Neutral Wins and Support-Neutral Losses (SNW and SNL) measure the expected number of wins and losses a pitcher would have with his outings, if he got average support from his offense and his bullpen. Support-Neutral Value Added (SNVA) measures the total number of games that an average team would win given the pitcher's starts, over the number of games they'd win with a league average starter. All of these stats are computed using only the number of innings pitched, number of runs given up, and the park the game was pitched in. SNVA may be a slightly more accurate measure of a starter's actual value compared to league average, but the SNW/L record has the advantage of being flexible and more understandable. Both of them, in my opinion, constitute an improvement over Thorn and Palmer's Pitching Runs as a total measure of starter worth.

Support-Neutral Wins is calculated by determining the probability that a pitcher would get the win for each start he has, and then summing up the individual probabilities over all of his starts. The sum gives you the number of wins a pitcher could expect to get for an average team, given his performances. A "performance" here consists only of the number of innings pitched, the number of runs (not earned runs) given up, the park in which the game was played, and whether the pitcher was at home or on the road -- SNW assumes that these are the only things which influence whether the pitcher wins or loses.

The rest of this section describes the formulas that are used to calculate SNW; readers who aren't interested in the specific methods of calculation are welcome to skim or skip to the next section.

To calculate the probability that a pitcher wins the game, we just need to look at the definition of a win: A starting pitcher wins the game if his team has the lead when he's taken out of the game, and they never relinquish that lead. So, for a given outing by the starter, the probability that he gets the win is just the probability that his team will take the lead (score more runs than the starter gives up) by the time he's removed times the probability that they'll hold that lead until the game is over.

To put this into a formula, we just need to determine and add up the probabilities of all the different ways his team can take and hold a lead:

where

The above formula is actually a simplification of the formula I use in my software to calculate SNW (I'll refer to the formula in my software as the "real" SNW formula). In order to make it easier to explain, I made a few assumptions to get the formula above. First, that formula assumes that the starter comes out of the game after pitching a full inning (i.e., he pitches no extra thirds of an inning). The formula is complicated somewhat when thirds of an inning are taken into account, but the same general idea applies: his team must be leading when he comes out, and his team must hold the lead for the extra thirds in the inning he leaves, plus all the rest of the remaining innings. The real SNW formula does take thirds of an inning into account.

Second, the above formula doesn't explicitly take the park into account. To take park effects into account, we need to make SNW, PScore, and PHold be functions of the park in which the game is played. A hitter's park should inflate the probabilities that an average team will score a high number of runs, and a pitcher's park should do the opposite. The real SNW formula does take park into account. I talk a little more about my handling of park effects in the Appendix.

Third, the above formula doesn't take into account whether the starter is pitching at home or on the road. Maybe contrary to intuition, this does make a difference. Consider a starter who leaves after pitching the 7th inning: if he's at home, he's pitched the top of the 7th, so he gets credit for the runs his team scored in the first 6 innings, plus the runs they score in the bottom of the 7th; if he's on the road, however, he pitched the bottom of the 7th, so he gets credit for the runs his team scored in the first 7 innings, plus the runs they score in the top of the 8th. So, all other things being equal, it's easier for pitchers to get wins (and harder for them to get losses) when they pitch on the road. The formula above is for a pitcher pitching at home, and the road formula is slightly different. The real SNW formula does take home/road status into account.

Finally, the above formula doesn't quite reflect the full definition of a pitcher's win -- a starter can't get the win unless he goes 5 innings or more. Presumably, this extra condition was put into the win rule to reduce the number of undeserving starters getting lucky wins. But when you're assigning fractions of a win, rather than 1 win or 0 wins, there's no possibility of getting lucky. So, the real SNWL formula does not take the five-inning condition into account, although, for the purposes of comparison, I do calculate an expected win (E(W)) number which is equal to 0 if the pitcher goes less than 5 innings and equal to SNW otherwise.

Let's finish off the formula above. PScore is easy to find recursively, provided you know an average team's single-inning scoring distribution, PInningScore:

where

PHold is a little more complicated, since you have to see to it that the pitcher's team never relinquishes the lead. Still, it's not too hard to reduce it to the following (below, "tr" stands for the number of runs the pitcher's team scores in an inning, and "or" stands for the number the opposing team scores in an inning):

The only remaining unknown is the single-inning scoring distribution, PInningScore. But that's readily available from linescores of past games. The scoring distribution (separate distributions for each league) I'm using right now was taken from a few weeks of linescores in USA TODAY from late-April and early-May of 1992. I'll probably be able to get a more accurate distribution someday, but I'm sure that this one is close enough.

The SNL value for a single start is calculated analogously to SNW.

SNW and SNL gives us a nice way of getting a "fair" W/L record for a starter, which can then be used to compare to his actual W/L record, or a replacement-level winning percentage, etc. (see the Results section). But these numbers calculate how likely it is that the pitcher will win or lose the game -- i.e., get the "W" or "L" next to his name in the box score. A related but slightly different notion is the likelihood that the team will win when a pitcher takes the mound. In measuring the starter's contribution to team victories, we'd like to evaluate how much the outing by the starter changes the team's chance of winning from what it was at the beginning of the game (which I'll assume to be 50%). This is what SNVA is designed to measure.

Not surprisingly, the formula for SNVA looks pretty similar to the formula for SNW:

where

Also not surprisingly, PATWin looks a lot like PHold:

What SNVA gives us (when summed over all a pitcher's starts) is the number of games in the standings he's worth to his team above the average starter. Of course, this is exactly the same unit (games above the average player) that all of Total Baseball's [1] measurements are in. So it'll be interesting to compare SNVA to Thorn and Palmer's Adjusted Pitching Runs¹ to see how well they correlate and also where the differences lie.

Table 1: Top 20 AL Starters in 1992, ranked by SNW-SNL

Table 2: Top 20 NL Starters in 1992, ranked by SNW-SNL

On the flip-side, Tables 3 and 4 show the worst 10 starting pitchers in 1992 for each league.² Not surprisingly, many of these guys showed up in different uniforms in 1993, several on expansion teams.

Table 3: Bottom 10 AL Starters in 1992, ranked by SNW-SNL

Table 4: Bottom 10 NL Starters in 1992, ranked by SNW-SNL

This method also allows you to evaluate the level of luck a pitcher experienced in his W/L record -- i.e. it allows you to look at how much a pitcher's actual W/L record differs from his expected W/L record given the way he pitched. Tables 5 through 8 show the luckiest and unluckiest starters in each league in 1992. No one should be surprised that Jack Morris, who compiled a 21-6 record despite a 4+ ERA, was far and away the luckiest starter in either league last year. SNW/L evaluation shows that you'd expect his 1992 performance to produce a 13-13 mark if he had gotten average support. Equally unsurprising is the result that Jim Abbott was the unluckiest pitcher in either league. The Angels gave him enough support only for a miserable 7-15 record, while his pitching actually merited something closer to 13-9.

APR evaluates Perez as being 14.5 runs -- about one-and-a-half games -- better than Leibrandt. However, SNVA shows that, when the pitchers' performance is evaluated game-by-game, Leibrandt was actually a little better than Perez.

The key to this discrepancy between the two measurements is found in the amount of consistency the two pitchers exhibited in their starts. Perez was a model of consistency last year; he rarely got bombed, but he also was rarely dominating. Leibrandt, on the other hand, was one of the least consistent pitchers in the majors. And that is the most surprising result I've seen so far from these SN stats: run-prevention stats such as ERA and APR tend to undervalue flaky pitchers, and overvalue consistent ones, at least when you consider them pitching for an average team. Tables 10 through 13 show the "flakiest" (most inconsistent) and "steadiest" (most consistent) pitchers in the leagues last year, as evaluated by the variance of the SNVA of their individual starts. You can see from those tables that APR pretty consistently underestimates a pitcher's value when the pitcher is flaky, and pretty consistently overestimates his value when he's steady. 9 of the 10 flakiest pitchers in both the NL and AL were underestimated by APR, and 8 of the 10 steadiest in the NL and 10 of the 10 steadiest in the AL were overestimated by APR. And the pitchers for whom there were really large discrepancies between APR and SNVA -- Leibrandt, Kyle Abbott, Gooden, Hammond, Sutcliffe, Perez, Kamieniecki, McDowell -- all showed up near the top of the predicted list.

The reason for this undervaluing is that APR counts all runs as equal, while in fact all runs do not contribute an equal amount toward winning/losing a game. In particular, Bill James did a study that showed that runs scored by a team after they've already scored 5 in a game do not contribute the same amount toward the probability of winning than those first 5 runs did [3]. So, pitchers who give up more than 5 runs in a couple of games will be undervalued by ERA and APR, because those really crummy outings probably weren't quite as crummy as ERA and APR would have you believe.

As an example of this, consider a David Wells outing from 1992: he gave up 13 runs in 4+ innings. APR just subtracts his 13 runs from the number of runs a league average pitcher would have given up in those same 4 innings (about 2), and concludes that Wells was worth about -11 runs, or -1.1 games, in that start. Did Wells really cost the Blue Jays more than a game in the standings with that awful start? Of course not. He guaranteed them a loss, of course, but they had some chance of losing the game to begin with anyway -- about a 50% chance if you make the simplifying assumption that they're an average team. SNVA gives a far more reasonable value for Wells's start: it was worth about -0.5 games. That's as much as a single start can cost you. Wells didn't have the requisite 15 starts to show up in Table 12, but you can see from his record in Table 3 how much he was underestimated by APR.

One other question I've been looking at is how the value of starts is influenced by park effects. Figure 1 shows the SNVA for a 9-inning complete game in both Atlanta's Fulton County Stadium (the NL's most extreme hitters' park in 1992) and the San Francisco's Candlestick Park (the NL's most extreme pitchers' park in 1992). We can see from the figure that the effect of the park on the value of the start is far less at the two extremes of start quality than it is for middle-of-the-road starts. The difference between Fulton County and Candlestick for the value of a 9-inning, 4-run start is almost four times as large as the difference between Fulton County and Candlestick for the value of a shutout.

Figure 1: SNVA for Fulton County Stadium (top line) and Candlestick
Park (bottom line), given that the starter pitched 9 innings

This would imply that methods of park adjustments which simply multiply a pitcher's "raw" value by a park factor might be over- or underestimating the park's actual effect on his value. Since the park's effect on very good or very bad starts is much less than on average starts, a reasonable hypothesis would be that very good or very bad pitchers deserve less of a boost (or less diminishment) to their rating than current park adjusment methods give them.

However, the preliminary investigation of this hypothesis I have done on real starting pitchers (with 1992 data) has failed to find much support for it. I'd still like to do some more work on this issue.

Here are a few of the problems with these measurements:

• They assume that scoring distributions of an inning are independent from the distributions of surrounding innings.
• They (like most other measures of pitching) don't account for situational pitching. A pitcher who gets a big lead is likely to start throwing all fastballs, and he may give up a few meaningless runs that he wouldn't have given up without the big lead. I'm not too worried about this, because I don't think those big-lead situations are common enough for anybody to make much of a difference.
• They don't account for differences in the ways pitchers are used by their managers. Some pitchers get left in the game to get pounded, some are routinely yanked early, etc. Note however that SN stats do a better job than other methods of mitigating the manager's effect. If Cito Gaston leaves David Wells in the game to give up 13 runs, SNVA produces a rating which is not much different than if Gaston had yanked Wells after giving up "only" 7 or 8 runs.
• They don't account for the defense playing behind the pitcher. Suffice it to say that this is a very hard problem.

   

I've presented Support-Neutral Wins, Losses, and Value Added, three park- and league-adjusted measurements of the value of individual starts, and of starting pitchers. I feel these are a valuable addition to existing measurement methods, both because they can provide a measurement of pitcher worth in units which are familiar to all baseball fans (pitcher wins and losses) and because they seem to be a slightly more accurate measure of the true value of a start than existing methods.

Special thanks to Greg Spira, whose discussion sparked many of the ideas presented here. Thanks to David Tate and others on the Internet newsgroup rec.sport.baseball, who provided valuable feedback on the method. And thanks to my wife, Cindy, for reading this paper and giving me many useful suggestions.

One possible way of incorporating park effect numbers into these measurements would be to take whatever final value the above formulas produce (SNW, SNL, or SNVA) and multiply it by some park effect constant for the pitcher's home park. This is essentially the approach Thorn and Palmer use in Total Baseball. But the method of calculating the Support-Neutral stats allows a potentially more informative use of park effects. Since park effects (as printed in Elias, e.g.) reflect how a park inflates or deflates average scoring ability, it makes sense to have the "average team" playing behind the pitcher effected by the park, and then calculate the likelihood that the pitcher's outing plus this park-adjusted average team will lead to a win. So for any game, the PInningScore (league average scoring) distribution is adjusted to reflect the park's effect on run scoring. The resulting number then reflects the park's effect on winning rather than cumulative run scoring/prevention.

The question then becomes: how do you translate a single park effect percentage like the ones in Elias (the only source of park effects I have) into an adjusted PInningScore distribution? There are an infinite number of ways to do this. The way I'm doing it now is to change the probability of scoring 0 runs by one factor, and change the probability of scoring i runs for i>1 by another factor, such that the total number of expected runs scored in an inning is increased/reduced by the Elias number. For example, if the Astrodome decreases scoring by 10%, I increase PInningScore(0) for the Astrodome by one factor, and decrease PInningScore(i) for i>1 by another factor, such that the expected single-inning score reflected by PInningScore is reduced by 10% from the park-neutral scoring distribution. If that isn't clear (and I'm sure it isn't), I should say that I don't think it makes much difference the exact method used.

¹ Adjusted Pitching Runs is the basic metric which Thorn and Palmer (the authors of Total Baseball) use to evaluate pitchers. APR is the number of runs prevented by a pitcher that a league average pitcher would've given up. The APR that I'm using in this paper differs from Thorn and Palmer's statistic in two ways: 1) I'm using runs where Thorn and Palmer use earned runs, and 2) the method of park adjustment I use is a simplification of the one used in Total Baseball. It is included here for comparison with SNVA.

² Actually, it's probably inaccurate to use the word "worst" here, since the method of ranking the pitchers -- ranking them according to SNW-SNL -- sets the baseline for comparison at league average (anyone below .500 gets a negative rating). Of course, it's quite possible for a below-average pitcher to still be valuable to his team. A better method of producing this list might have been to compare a pitcher's SN record to a lower baseline, e.g., a .450 pitcher. This would have left pitchers like Hershiser and McCaskill, who pitched a lot of innings at somewhat below-league-average performance, off of the lists in favor of other pitchers who pitched fewer innings but at further-below-average performance.