Reintroducing PECOTA

They're Here!

by Colin Wyers

Printer-friendly

Contact Author

And here we are–the release of the 2011 PECOTAs.

While I have your attention, I’d like to say a few words about the production of the PECOTAs this year. I guess I don’t have to tell you that I’m filling some pretty big shoes here–Nate Silver is probably the most famous sabermetrician not named Bill James, and PECOTA is where Nate made his biggest mark in our community. And so I’m building on his work–and work by people like Clay Davenport and Gary Huckabay, too. I am, as they say, standing on the shoulders of giants. (In big shoes, apparently–this is what I get for mixing my metaphors.) So I owe them, and others I’ve probably neglected to mention, my deepest thanks.

But even with all that, I wouldn’t have gotten this far without a lot of help. I couldn’t have accomplished what I’ve done without the help of everyone here at Baseball Prospectus, who have really given me all the support I could ask for. But I want to thank a few people especially–it’s a fine line to walk, as if I list too few I risk upsetting someone unfairly left off, and if I list too many, you won’t finish reading the list. So with that in mind–very special thanks to Rob McQuown, Mike Fast, Ben Lindbergh, and Steven Goldman. Gentlemen, take a bow, and everyone, please, give them a round of applause.

Now, then, on to business.

This is the first release of PECOTA, and as such will continue to undergo revisions through the remainder of the offseason. The program we use to generate the PECOTAs is continually evolving, and when we discover new ways to improve the forecasts, we'll make those changes and pass the updated forecasts on to you. We’ll also be updating periodically to keep up with players who switch teams.

In addition, we have several PECOTA features yet to roll out out–first will be the Depth Charts, which combine the PECOTA forecasts with estimates of a player’s role and playing time. Those should be ready a week from now, and will be available in two forms: the team Depth Chart pages, and the Player Forecast Manager (which is receiving some upgrades as well).

After that, we’ll be publishing the PECOTA cards, featuring perks like the percentiles and the ten-year forecasts. We’ll update you more as we get closer to that point.

We’re also revamping the cards to use the new Wins Above Replacement Player model we’ve developed. PECOTA has already been adapted to use the new WARP, so WARP baselines have shifted a bit from what you’re used to seeing. The biggest change is among relief pitchers, who take a major hit. Please keep this in mind as you review these forecasts. We know that many of you are relying on these forecasts for your fantasy teams, and we thought that it was better to get the forecasts out now rather than wait for when the entire site was ready to transition to new WARP.

Now, some of you may be asking, “How good are the PECOTAs this year?” Of course, we won’t know the answer for another eight months or so. But we can come up with an educated guess, if we make the assumption that there’s nothing special about predicting the 2011 season, and that a system that works over previous seasons will work in succeeding seasons.

In the course of producing the PECOTAs, we generate forecasts for every player who played from 1950 through 2010. These aren’t quite the same as the full PECOTAs–they are park-neutral, rather than being adjusted for the home park a player plays in. They are not age-adjusted. And for the most part, they do not reflect minor-league performance–we have major-league data for all of MLB history, but very little minor-league data. Still, they do represent a substantial portion of the PECOTA process.

It is time-prohibitive for us to generate full age curves for all of these historic forecasts, but we did adapt a simplified set of age adjustments for past purposes. These simplified PECOTA forecasts aren’t as accurate as the full PECOTAs, but they give us a chance to view how well PECOTA fares over a large swath of history.

There’s only one other projection system available and therefore able to be pitted against PECOTA for such a large part of baseball history: the Marcels, originally developed by Tom Tango. (The version we’re using in these tests was published by Jeff Sackmann.)

I “re-baselined” each forecast for each season in the test by subtracting the average forecast and adding in the average performance of the players forecasted (weighted by playing time, in both instances). Then I took a look at two tests–one is root mean square error, which tells us that 68% of forecasts were within that margin of error. The other is simply counting which forecast was closer to a player’s actual performance. Looking at offensive stats first:

P_OBP_RMSE	P_SLG_RMSE	M_OBP_RMSE	M_SLG_RMSE	P_OBP_SR	P_SLG_SR
0.036	0.066	0.036	0.066	56%	54%

In terms of RMSE, our simplified PECOTA is in a dead heat with the Marcels. (And again, PECOTA is giving Marcels an edge, as it is forecasting everyone for a neutral park; the Marcels make no park adjustments, but most players do not switch teams or parks between seasons.) In terms of “success rate” (in other words, the percentage of head-to-head projection matchups "won"), PECOTA has a slight edge.

Now, for pitching:

P_ERA_RMSE	M_ERA_RMSE	P_ERA_SR
1.22	1.22	48%

Again, RMSE shows a dead heat. In terms of success rate, the Marcels have a slight edge over the simplified PECOTAs.

Like I said, this sort of testing emphasizes breadth rather than depth–PECOTA has foregone several of its advantages, like park adjustments and minor-league data. And yet it’s still producing accurate forecasts.

Now, just as a reminder–PECOTAs are available only to our subscribers (Premium and Fantasy) who are signed up for a whole year. If you haven’t already subscribed, you can do so here. Doing so doesn’t just get you access to PECOTA, but also to our fantasy tools like the Player Forecast Manager and Team Tracker, as well as exclusive access to some of the best baseball writing out there. If you’re already a subscriber, thank you for your support, and I truly hope that you all enjoy what we do as much as we enjoy doing it. I am continually amazed at how intelligent and knowledgeable our readers are, and I really do think that the people who read BP are among the best baseball fans I’ve ever known.

That’s all I've got–have fun, folks.

Colin Wyers is an author of Baseball Prospectus. Follow @cwyers
Click here to see Colin's other articles. You can contact Colin by clicking here