BP Unfiltered

When do you anticipate switching the Depth Charts, PFM, and Spreadsheet to Weighted Means?

"Thanks for writing and for your support of Baseball Prospectus.

Our crew has posted an updated about some of the recent changes to PECOTA. You can find it at
http://www.baseballprospectus.com/article.php?articleid=10226.

We hope this is helpful and we're sorry for the delay in communicating with you. Thank you for your continued support of BP."

Seriously?! This is the response I get from your customer service when I complain that I paid for access to "just the numbers", but find most of those numbers are not available for my draft tomorrow.

It mid March and there's no long term projections, no pitcher weighted mean, no pitcher cards, the hitter cards are "beta", the PFM is using different stats this year because the proper ones are not available. This is a giant problem for fantasy subscribers.

Sorry but a link to this post is not good enough, and does not address the problem.

Helpful update. Thanks for being transparent about it.

Thanks for the update, though it still seems that there are issues with the PECOTA hitter cards that haven't been addressed. One is that several players have screwy things going on around the 90th percentile. Take Adam Jones for example.

His Eq rate stats are .305/.370/.509 and .298/.360 /.509 in his 90th and 80th percentiles, respectively. But the adjusted rate stats are .306/.371/.519 and .308/.370/.535 for those same percentiles. Why is this happening (for him and others)?

Staying with Adam Jones, his Weighted Mean Projections have him at .291/.323/.483. This OBP is lower than that for his 10th percentile. It seems that a similar OBP-suppressing phenomenon is happening for most players. However Dexter Fowler's Weighted Mean OBP is .380 which is higher than his 60th percentile projection.

Just a couple of observations.

There's an ongoing discussion in the Depth Charts, for those interested in the thoughts of some who have followed the updates. Some of us are skeptical.

The depth charts now show the AL at 61 games over .500, which would be impressive, all the more if not for the fact that the NL is projected to be right around .500.

--JRM

Dear BP,

I suggested this a couple of weeks ago. I'll suggest it again now. Ask some of your subscribers to be Beta-testers for the Depth Charts, PFM and Weighted Means, PRIOR to release.

Question - why not include projections for all first rounders from last year? Why are guys like Dustin Ackley and Kyle Gibson excluded, while other rookies with zero pro experience like Stephen Strasburg and Jose Iglesias are projected?

Should we be sending the books back to Amazon? Are THOSE Pecota numbers any good? I can't even tell anymore what's what. The season ended 5.5 months ago---that's a long time to have worked this all out. Hopefully somebody will spend the Summer fixing this. Many drafts are already over this year.

thanks for the update and for explaining the problems and the processes by which you are seeking to solve those problems.

Nevertheless (and I say this with all due respect), I think you should be offering some sort of refund to all of us. I can't speak for anyone else, but 90% of the reason I subscribe to BP is for PECOTA stats to prepare for my fantasy draft. In particular, I'm most interested in UPSIDE, as I am drafting almost exclusively prospects in a 100% keeper league. I pay $40 for a year's subscription so that I can have access to these statistical tools for basically one to two months of the year. I tried to delay the start of my draft further and further in hopes that PECOTA would be fixed before we started making picks, but the other guys in my league understandably wouldn't wait any longer. So while I will still enjoy reading the columns for the rest of the year, I feel like I just wasted $40 (or, to be fair, about $36) on this subscription.

I suggest a partial refund and/or an extension of our subscriptions.

My two cents is this has been (and still is) a major disaster that will affect the BP brand very negatively. I would make a couple of suggestions for your consideration. They have been suggested previously but not acknowledged....

1. Get the Pecota Beta on line January 15 next year.
2. Consider getting customer input before posting.
3. Review the Pecota at basic levels so at least the math is correct. Not +60 wins in the AL and -15 in the NL.
4. Make a big public apology.
5. Offer a discounted subscription rate for renewals.

I'd note the comparisons appear designed to show the newer PECOTA is better.

First, back-testing on one year of data is insufficient.

Secondly, was the back-testing done after filtering out the known information (the 2009 season?) That is, doesn't the 2010 PECOTA have some 2009 season information in it? If you did filter it out, you should have clarified this. If not, the data's value is damaged.

Thirdly, what bias exactly was filtered out?

Fourthly, the BA/OBP/Slg are all worse now than they were with the book's method. You used a measuring system that a cynical unpleasant person might think is designed to hide this.

Fifthly, the changes keep coming. I have to say, if I kept changing a system, I'd think I could fix it up to predict the 2009 players right well by this time.

To quote myself from the Depth Charts thread, I don't believe you when you say PECOTA 2010 is fine now. More very simple errors are being generated with each iteration, so I don't think you believe you either.

--JRM

It's unclear from this post whether 10-year projections for hitters have been fully "fixed" yet, but it doesn't look like it. Basically, most players are projected to remain stable or improve from mid/late-20s through age 35 or so. This is not a normal aging curve, and differs significantly from past long-term PECOTA forecasts. E.g., Dustin Pedroia's TAv taken from his latest 10-year forecast:

Age 26: .305
27: .294
28: .292
29: .290
30: .291
31: .293
32: .292
33: .291
34: .301
35: .292

Other examples (pulled quickly by spot checking random players) include Sizemore, Miguel Cabrera, Adam Lind. Basically almost every good player is projected to experience no decline between age 30 and age 35. This does not correspond to observed reality (http://www.baseballprospectus.com/article.php?articleid=4464 and http://www.baseballprospectus.com/article.php?articleid=9933), or to the way past PECOTA projections looked.

First, I had a hard time finding the 2010 projections. The 'Find a Player' box still takes you to 2009.

Jesus Montero's weighted mean OB of 315 is only 20 pts above his BA of 295, and lower than any of his percentile forecasts. Clearly a miscalculation. But the final nine years of the ten year forecast are built on that number.

It seems to me people are missing the implication here that Nate is not doing the PECOTA adjustments and tweaks as he has always done, probably because he's no longer really an active part of BP. Every year Nate has, in the mid-winter, posted his various PECOTA adjustments and new projects for the year. However, it's a lot easier for the designer to work on a project than it is for people, however brilliant, to pick up that project, learn it and improve it without breaking some things.

Clearly, they have broken some things during the learning process.

"All we are saying ... is give Pease a chance."

(we now return you to the PECOTA/BP-flogging, already in progress)

To the torch-wielding mob: Does rotoworld's projections work better than PECOTA right now? I enjoy reading BP, but I want the best projections possible for my draft in two weeks. Which would you choose?

as someone who's not involved in fantasy baseball, or uses the PECOTA projections for anything more than personal reference...i have to echo others' sentiments about how disappointing this roll-out has been. i'm mystified by the fact that this transition didn't fully undergo thorough QA testing before replacing the existing methodology to produce the data that is the foundation to the core product.

beyond that, y'all could've communicated to your audience a little more regularly, and broadcast in a different way than has been done. if i were reliant on this information for my draft, i'd be furious.

that being said, i'm extremely sympathetic to the BP staff for the heat they're taking, especially considering the years of quality that have gone before 2010. i'm hoping the staff is learning from their experience to ensure history does not repeat itself.

I'm not a programmer and have generated "software" in Excel for Z-scores, regressions, projections, et cetera that could not have possibly amounted to 1/1000th the complexity of what Nate created and tweaked over the years. When I've handed my "software" to a programmer to translate it, they've often laughed at the silly complexity involved in using Excel to kludge the calculations. I imagine successfully converting Nate's work lies somewhere between flapping your arms to fly and blowing someone's head off with a thought.

We're stuck with the name PECOTA for branding reasons, but the product is no longer the same. It isn't clear whether it should be exactly the same or not--certainly asking that question is the correct thing to do. PECOTA was *not* a fantasy baseball tool at first. My impression from the beginning is that the focus is on identifying prospects before they identify themselves or prognosticating breakouts and collapses before they happen. It's too bad that building a successful business in baseball analysis requires turning to the fantasy world. PECOTA wasn't and isn't well built for fantasy--it's an aggregate tool being shoehorned into an individual game.

I buy BP for the articles, but if I bought a fantasy subscription this year, I'd be seriously irked. Drafts are already taking place, and this stuff is still buggy?

Let me just say I would buy a BP subscription just to read Kevin Goldstein's prospect analysis -- for me, PECOTA is just a bonus.

That said, I continue to be dumbfounded by this year's comparables. Is Everth Cabrera really most comparable to the young Hanley Ramirez and Jimmy Rollins, and also very similar to the young Derek Jeter? Can you comment on whether the comps perhaps need to be tweaked along with the 10-year forecasts?

Another comp example: Matt Wieters' top comps are John Christensen, Jarrod Saltalamacchia, Sid Bream, and Steve Decker. Looking at these, and Everth Cabrera's, I'm no longer sure the comp listings have any value whatsoever. (Of course, if Cabrera turns out to be a Hall of Fame player . . .)

*Gnashing of teeth*

I was kinda disappoint that PECOTA wasn't up to snuff when I had my draft last weekend. AND I think BP should have been more upfront about things A LOT earlier. It just seemed like I expected BP people to more forthright about all this PECOTA stuff. Why keep pretending?

Articles keep getting posted, analysis keeps getting churned out, readers keep reading.... But upon what are the articles, analysis and readers depending upon? Are we really to expect some point of clarity which allows us all to move forward with reliable data, predictions and analysis?

I can pick my own fantasy baseball team on my own just fine. Thats on me. But PECOTA.... That's on you guys, BP. I wish you the best and truly hope that you get it all sorted out.

Allow me to join the angry mob here. I'll reiterate what's been said above - the BP brand has been severely, if not permanently, impaired.

Presumably it was clear during last season that a switch over needed to be made from the Nate Silver cobbled-together sheets into something more dynamic (and by the way - I've never understood the Silver hagiography that goes on here, nor been impressed by the next site he's gone on to run, but this makes me wonder if he isn't in fact the only person with a clue around these parts).

With that in mind, why not work on the program throughout the 2009 season and have a 2010 projection ready to roll out around October 1st? You guys shot yourself in the foot by waiting until February to get this going.

And while I've been a defender of the site in the past, you guys reap what you sow when you tell people that "Everything they know about baseball is wrong" and that your forecasts are "deadly accurate" ... and then you deliver a stinker like this.

The reality is, the product you are offering for a price is inferior to those being offered elsewhere for free.

Not a great path to prosperity in my view.

i think toyota has taken less flak than BP has this spring.

I've asked this question previously in ways that haven't generated positive responses -- so now I'll ask it in a positive way. Could anyone please provide some insight into the characteristics of Nate McLouth that would lead him to be ranked by BP as one of the top players for the coming year, though other prognosticators differ on this point? Is there something about his BABIP data, his expected walk rates, an age-fueled increase in AVG, or anything else that is making PECOTA particularly optimistic about his expected production?

Aren't good fantasy owners supposed to look at projections, add in their own conclusions and make their calls accordingly? Before all this noise, I had my PFM set to my league rules, and I draft according to the projections and my assessment on which ones I am on board with and which ones I am not.

The fact that PECOTA has some kinks is not ideal, but who here is drafting blindly with these projections? Marc's guidance, Will's THR's and damn near every article on this site should help all of us form opinions and make the calls.

The book is great, the site is great, the BP team is being upfront about the issues. Let's roll with it and move on.

I will make one suggestion. Either:

A), get those projected records down somehow; or,

B), put up a message box stating that you know they don't add up and you're working on that.

It just looks awful, putting such an obvious error out there for all to see.

So, how exactly are the "2010 Projections" done that are in the box at the top of the box on the player cards page ? I see that it's the Median rate stats, but they are applied to more PA's ? Where does the PA number come from ?

As for the 10 year projections they still don't pass the sniff test. Nate McLouth is not going to show any meaningful decline as a player until age 35 ? Give me a piece of that action. Meanwhile, Adam Jones, who arrived in the majors at a much younger age than McLouth shows a drop off at age 33. Doesn't this fly in the face of what we know about career arc's ? Typically, it's the later arrivers who fade first, right ?

Regarding the issues with PECOTA this year, I clearly overreacted in this threat and for that, I offer my humble apologies.

The bottom line is that the frustration of myself and others relates directly to what we're used to seeing from BP - innovative thinking and great writing. We don't always get everything we want, but the fact that BP provides a forum for their customers to comment/complain/offer feedback is appreciated. Then taking the time to listen and address said feedback is even more appreciated.

I have all the confidence in the world that when you put together this many talented minds, that issues (like PECOTA) will be both addressed and resolved.

I do need to say this, I will not be renewing for a number of reasons: No reliable PECOTA (the silence surrounding the PECOTA issue puzzles and disappoints), no Sheehan, poor customer service, and articles and analyisis which seem to me to be strewn with grammatical, stylistic and most probably mathematical/statistical errors.

On Friday, I openned up the latest update to the Depth Charts and was immediately very upset. The more I look at it, the more upset I get, too. I'm not much of a fantasy player, so its not just the consistency aspect to it. No, what bothers me is that the Texas Rangers managed to magically lose 50 runs of offense during spring training. They lost this by losing a big chunk of their power, and this came from decreased power output from their main power hitters.

Indeed the Rangers, who have been consistently projected as having the highest slugging average in the majors, are now predicted to be 6th, tied with Toronto. This is also the first projection which shows them under 800 runs scored, and its significantly below that level. That doesn't pass the sniff test.

Regarding Jose Reyes, I just noticed that the 2010 projections appearing at the top of the beta card do not match the weighted mean under the 2010 Forecast further down the sheet, nor is it close.

My guess is that the projections at the top are pre-thyroid diagnosis (e.g., 97 runs) and the forecasted numbers in the 2010 Forecast section post-thyroid diagnosis (e.g., 79 runs at the 90% level).

This also raises the question as to which of these numbers, if any, PFM uses, which upon examination appears to be the projections at the top of the page. Unfortunately, these appear to be sorely out of date.

Re: PFM and 50th percentile projections - I still don't see a perfect fit.

in PFM, McLouth is hitting .267. I don't see this AVG in any of his PECOTA percentiles. I do see it in the Weighted Means Spreadsheet though

No mention of the massive changes in projection to players like Alex Rodriguez, Mark Teixeira, Chris Davis and others this go-round?

Guys, I love BP to death and have been a long and ardent follower, but if this is transparency, then I'd hate to see the "wool pulled over our eyes" by you guys.

How does this article explain why just a couple of weeks ago Alex Rodriguez had a .700 SLG% in his 90th percentile and now has a .630? That's not a playing time correction, and it sure as heck isn't defined in any of the changes you outlined above.

PECOTA was worthless weeks ago, and for us -- the subscribers -- to have any sense
of it having been reconciled, it would be nice if you guys would actually fess up to what in the world you did to screw these numbers up so badly and what you did to fix them. Sweeping it under the proverbial rug would not be a good first step, I would think.

I think it's telling that in 2003, when I was first introduced to the site, Baseball Prospectus told me everything I needed to know about statistics and value and the great world of sabermetrics in baseball. Now? I only come here for write-ups on prospects who are evaluated almost solely by scouts. I think the message has been lost somewhere, gentlemen.

I made some comments in the depth charts. It's a little difficult to suss out why there are differences in the projections from various PECOTA sources (two spreadsheets differ, which differ from listed projection, which differ from 50th percentile.) Integer rounding explains a lot of it, but I don't think all of it. Weighted means appear to still be miscalced. Big differences (like Andy Marte's slugging) between projection and 50th percentile are undesirable.

I'd note that any comparisons to past iterations needs to include slash stats or (or preferably and) total value stats, like virtually every other forecasting comparison ever done, including by BP. Failing to include that basic comparison renders such an exercise of little value.

--JRM

There is a pretty interesting discussion regarding this year's PECOTA's over at Tom Tango's site, Inside The Book. The discussion can be found in the comments section of the March 3rd, "Weiters II" blog post. (I won't provide the link, but it should be easy enough to find).

Colin has reported some of his unpublished findings regarding this year's PECOTA.

Oh, and I don't know if this is by design, but this post no longer shows up in the Unfiltered Archives. For some reason the most recent posts that appear are from March 3rd and earlier.