June 8, 2011

Between The Numbers

Another look at ZiPS

by Colin Wyers

Yesterday I claimed that the rest-of-season projections at Fangraphs are wrong. Again, I am well aware this is a statement from interest, as I made this claim as I was announcing our own rest-of-season PECOTA projections.  Some people were skeptical of my illustrations, so I’ll go into the weeds here and explain why I think the methodology is incorrect, as opposed to simply disagreeable. First, I’ll lay out the methodology in neutral terms, then I will provide my own commentary.

What I’ve done is recreate at least a portion of the methodology used to compute rest-of-season ZiPS, as based on this spreadsheet, particularly, the portion having to do with HR. (If you want to look at the methods yourself, just download that spreadsheet, and copy the batter worksheet into a blank sheet so you can unhide the locked columns in the sheet.) First, you need to take the number of games a player’s team has played, and divide by 162. This is called G% in the original spreadsheet. For clarity I will call this G_RT.

Now you take that, along with a player’s preseason projected HR (which I’ll abbreviate ProjHR) and his season to date HR. To figure his projected rest-of-season HR according to ZiPS, you do this:

The first term of this is simply a weighted average, where a player’s preseason and season-to-date HR totals are averaged based upon weights of 11 and a value that floats along with team games. After 60 games, the weight is about 3, after 120 games the weight is about 6, after a full season it is of course 8. Note that these are total home runs, not home run rates. The second term simply prorates the weighted average out to the remaining games of the season.

The method in the ZiPS rest-of-season spreadsheet uses park adjusted data in this step, and then converts to a player’s particular park; I did not have access to the specific park factors being used. And as a shortcut, I used a constant value of G_RT for all players, rather than compute the correct value for each individual player.

And yet, even without those steps, I get a very good agreement with the values at Fangraphs – using the same data as I did for yesterday’s post, I come up with 23.09 projected HR for Bautista; the listed total with that data was 23. If I do this for all players, I get an average absolute error of .29; if I round the projected HR totals I did to whole numbers (as is done for the values I downloaded from Fangraphs), I get an average absolute error of .19. (If I limit myself to players with at least 100 projected AB rest-of-season, I get an average error of .19 as well.) So even with excluding the park factors, I get a very good agreement with the values currently being published; if Fangraphs has changed the methodology originally used in the spreadsheet I have, it’s been an extremely modest change.

Everything is computed by that same formula: AB, H, 2B, 3B, BB, SO, etc. And once you’ve done that, you have rest-of-season ZiPS.

Now, my thoughts: First, the weight for a player’s previous seasons is based on two factors--a constant of 11, and a player’s projected playing time. There is going to be a modest correlation between a player’s projected playing time and his previous playing time, of course, so in this sense I suppose there is some accounting for the reliability of a past projection, but it seems to occur almost entirely by accident. And for young players who are expected to hold down a job for a whole season (your Jason Heyward types, for instance) it treats those forecasts exactly the same as it does those for an experienced veteran with the same expected playing time.

Secondly, it treats all events the same way. If you take two players with the same hit totals and at-bats, one with a lot of HR and one with a lot of singles, the rest-of-season forecast at Fangraphs will treat those exactly the same when it comes to projecting their batting average going foward. The rest-of-season forecast also does not distinguish between a player whose low batting average is driven by a low BABIP, versus one whose low BABIP is driven by a high strikeout rate. And it uses the same arbitrary 11 and 8 weights for HR as it does for triples.

Writing at ESPN today (how serendipitious!) Dan Szymborski, the creator of ZiPS and the in-season projection methodology, says:

Not all bad seasons are created alike, though. For example, changes in walk and strikeout rate are far more likely to be retained going forward than changes in batting average on balls in play (BABIP). A homer outage for a home run hitter is a greater cause of concern than a singles hitter having a single outage.

And he’s absolutely right; it’s just that the rest-of-season ZiPS projections don’t know those things. It treats changes in walks rates just the same as changes in hit rates.

The pitcher projections are essentially the same as the batter projections. There is a reduced set of stats to project, as one might expect,  but ERA, has some peculiarities of its own. A component ERA is used in place of standard ERA for projecting future performance, which would be comforting except:

1. The component ERA involved includes H/9 as one of its terms (along with SO/9, BB/9 and HR/9); basic DIPS theory strongly suggests that accuracy would be improved by using an ERA estimator that does not include hits on balls in play. (The use of H/9 instead of something like BABIP also substantially shrinks the coefficient of the strikeout, meaning that the greater impact of strikeouts compared to outs on BIP for predictive purposes is being largely ignored.)
2. The component ERA is computed twice, once before and once after park-adjusting the four inputs. After computing the park-adjusted component ERA, rest of season ZiPS then adds the difference between the unadjusted component ERA and observed ERA to the adjusted component ERA. That probably destroys any benefit you gain from using component ERA instead of just using observed ERA, especially since you’re using a version of component ERA that includes hits allowed.
3. The same 11/8 ratio we saw for batting stats is used for the pitching projections as well, including ERA.

Tthe pitching methodology means that a pitcher whose peripherals suggest he’s pitching in line with his preseason projection can still see his numbers change significantly because of ball in play outcomes, or bases empty/men on splits, or relievers allowing a greater or lesser number of inherited runners to score.

Am I suggesting you shouldn’t use the rest-of-season ZiPS projections? Of course I am; but that is clearly a biased statement. So let me suggest instead, when looking at a player’s rest of season ZiPS forecast, ask yourself how much of the change is due to the player’s ability being better measured, and how much is due to the peculiarities in how the rest-of-season adjustments are being done.

Colin Wyers is an author of Baseball Prospectus. Follow @cwyers
Click here to see Colin's other articles. You can contact Colin by clicking here