|
BP's Research Director is projected to answer PECOTA questions and anything else you might want to ask.
Colin Wyers: Let's talk PECOTA. And things that aren't PECOTA.
Russ (Wernersville, Pa): I new to BP and was wondering how often the PECOTA projections are updated between now and the start of the season?
Colin Wyers: There's two kinds of updates - updates where we have to rerun the whole PECOTA projection, and updates where we just have to move players around to different teams and parks. (Those sort of adjustments come very late in the PECOTA process, so we don't have to rerun the whole thing to get changes.) I'd expect the later to happen very frequently, depending on what sort of transactions are happening (once teams start doing roster cuts and the like you'll probably see a flurry of those sorts of changes, and then it'll slow down again. The PFM is going to be a quicker way to get those updates than the spreadsheet. (For players in the PFM, the spreadsheet will have their latest PFM-projected stats as of whenever the projections were made).
As for the larger updates, those won't happen very frequently. But we'll still be updating the spreadsheet every few days to keep it in line with the PFM.
BDesrochers (New York, NY): Based on last year's results, it appears that PECOTA did quite a good job with hitters but not such a good job with pitchers relative to other projectors. 1) Do you agree with that statement? 2) If so, do you have any thoughts as to why?
Colin Wyers: You're referring to what Matt Swartz published this morning at Fangraphs, right? I had a chance to read it this morning but not much more than that. As for pitching versus hitting, what I noticed is that the SD of batting was much more similar between forecasting systems than the SD of pitching, and I think that plays a pretty significant part of those sorts of tests. You can get a lower RMSE by taking fewer risks with pitcher projection and being really conservative with the amount of regression you're doing.
Also, Matt's pitching tests contain some things that I'm frankly rather shocked by - his claim that Oliver correlates less well with future pitching performance than prior season ERA is hard to believe, at least for me. There's plenty of things that are hard to believe that are true, of course. But I'd like to hear more about that result, for instance.
rubinr (Seattle): Will PECOTA include Scoresheet/Sim ratings this season?
Colin Wyers: Yes. SS_SIM is generated in the PFM currently, and I'll work on migrating that into the spreadsheet at a later date, for players who aren't part of the PFM.
DanDaMan (SeaCliff): Colin- Reading all the recent talk of PECOTA I wondered how do you guys write the annual without all of the info? or do you just wait for PECOTA before you have all the stats printed out in the book?
Colin Wyers: This year our first PECOTA run happened a few days after the end of the season, and authors had access to it while writing the annual. Of course PECOTA changed many times after that, but authors had some guide to what PECOTA was going to say.
Speaking just for myself, as a consumer of the book (I have very little involvement with the writing aspect of it, as that's a very busy time in the offseason for me with PECOTA) I prefer player comments that tell me something the numbers don't, especially the numbers that are incorporated in the book.
lemppi (Ankeny, IA): Do you see big (or at least notable) regression from Alex Avila and Jhonny Peralta? The seasons posted by these two were key factors in the Tigers winning 95 last year.
Colin Wyers: I think any time you see a guy post a 6 WARP season who has never done anything like that before, you expect a good deal of regression. Peralta's numbers were more in line with his career, although again any time you see an outsized performance you expect some falloff.
Pat (KC): What royals were ranked in Keith Law's Top 100 prospects...Position?
Colin Wyers: I have no idea. (Seriously, I haven't checked yet.) And if I did know, I'd probably suggest that you subscribe to Insider to find his results rather than telling them to you here.
On that note - KG's top 101 will be out on Monday.
dianagram (VORGville): Hi Colin ... thanks for all your great work picking up the pieces of PECOTA.
Is there any hope of making the PFM draft tool a stand-alone instrument, to improve its processing time and eliminate the need to be on-line to use it?
Colin Wyers: The nice thing about PFM as a web page is that it runs on a whole lot of things - you can use it from a Mac, a PC, you can use it on a tablet or a smartphone, you can use it on computers where you don't have administrative rights to run stand-alone software. Running the PFM as a standalone app has the potential to eliminate that benefit. It's something that's asked about every so often, and we'll continue to evaluate doing it. Actually, let me ask all of you - if we were to do an offline PFM, what would you use it for? Where and what would you use to access it?
kcarlson2 (St.Louis, MO): Some months ago you indicated on "The Book" that you were going to review Humphreys' book Wizardry. Did I miss it or haven't you done it yet?
Carlson
Colin Wyers: I haven't yet, sadly. Humphreys is a really smart guy and he has a lot of interesting things to say about defense; the book is rather dry in parts, though, and I wish the publisher had put more effort into properly typesetting his formulas.
His defensive system itself is a fine one (I think it's my favorite of any that I haven't designed myself), and if you have any kind of an interest in defensive analysis it's worth picking up.
score (sheet): Do you know why SS/SIM isn't currently present in PFM? Will it be added later?
Colin Wyers: We (and by we I mean the talented Ben Murphy) is loading the Scoresheet player lists into the back end as we chat. Once those are loaded, I think that'll start generating SS_SIM in the PFM.
tommybones (brooklyn): I've wanted the following PFM enhancement for years:
A way to compare players per game. In other words, players are ranked by including predicted playing time, which puts part time players much lower in rankings. But on a day-by-day basis, you may know that the part timer is going to be in the major league lineup and on a game-by-game basis, he may actually be better than the guy you normally start. In other words, if the PFM generated a daily value, based on the idea that every player listed would be in the lineup, then you can better compare part-time players vs. everyday players when filling out a daily fantasy lineup. For example, perhaps John Mayberry is a better fantasy player when in the lineup than Nick Markakis, even though the PFM ranks Markakis much higher overall. I hope I made this clear.
Colin Wyers: I think I follow, yes. I think this can be accomplished with the current PFM setup if you only use rate stats. If we added more rate stats to the PFM (like RBI/PA, for instance) would that satisfy what you're looking for here?
modofacid (Philly): Colin, If I recall correctly from econometrics (5 years ago). The OLS regression is designed to find the most accurate line (including the intercept). So what is the goal/thought process in changing the intercept?
Colin Wyers: In terms of generating the relative weights (where 1 is the most recent season, etc.) it actually doesn't make much of a difference whether I set the intercept to 0 or let the regression figure it out, as far as the weights I get.
iorg34 (Last place): One thought for an enhancement to PFM: It'd be great if you allowed users to adjust the PECOTA projections based on new information or hunches, and have it change the dollar value output. Any hope for this?
Colin Wyers: It'd be rather difficult to do so, especially without making the PFM a lot slower for everyone.
tommybones (brooklyn): Yes, as long as the option was available to compare everything by PA. So, to compare hitters, I'd love the option to choose AVG, HR/PA, R/PA, RBI/PA, SB/PA as my 5 categories.
Colin Wyers: That's probably not terribly hard to do.
If anyone else is interested in more per-PA stats, let me know so I know how many people would benefit from this.
Pete (NYC): Do you have a PECOTA projection for a 50 Year Old Kate Upton?
Colin Wyers: I'm reasonably certain she won't be playing baseball.
Bill (New Mexico): How does PECOTA handle guys like Colby Rasmus or Milton Bradley (or, on the positive side of the ledger, Josh Hamilton) whose obvious off-the-field issues clearly affect their performances? Does the search for comparables include a search for comparable behavior problems?
Colin Wyers: That isn't something we can quantify a lot of the time. In the case of someone like Hamilton, it may play into some things we CAN quantify, like his playing time.
tfierst (MN): RE: Offline PFM. It'd be useful for drafts with out WIFI. Also I don't use it during the draft because it takes too long to load the page. The draft happens way too quick for that.
Colin Wyers: So I can understand the amount of need out there for this - is there a benefit to running PFM during the draft, as opposed to exporting a CSV out of the PFM and using that during the draft?
Rick (Chicago): PECOTA seems extremely down on Reds pitchers. 24 year old Mike Leake is projected to have a worse year than either of his previous (and only) 2. The skill and performance changes we've seen from 26 year old Johnny Cueto are projected to disappear. Same with Sam LeCure. There just seems to be a massive regression across the board. I'm wary of simply being a homer and wearing rose color glasses, but this seems "off", even accounting for reasonable levels of regression based on performance of year's past and to the mean. What am I missing?
Colin Wyers: Leake is a guy whose outpitched his FIP for two straight seasons. One season he did this by having a really low BABIP, another season he did this by giving up unearned runs at twice the league average. PECOTA doesn't just look at what a guy's done, but how he's done it - some things carry with them a higher predictive value than others, and so Leake's ERAs are maybe not the best reflection of the underlying skills he has.
luftmich (Niles): Colin, I liked what you did with the PFM at the end of the season by making the 2011 accumulated values available. This year, will we be able to not only run projections, but see a player's accumulated values during the season?
Colin Wyers: I believe that's still going to be the case.
And I should note that the PFM is largely done by Rob McQuown and Ben Murphy, who put a lot of hard work into it. I can pass things on to them, just don't want to take credit for their efforts.
Robyn (Sweden): If you were the GM of the Rangers, would you even entertain a long term offer for Josh Hamilton?
Colin Wyers: Entertain? Yes. It depends on the years and the money.
In general, my sense is that long term deals don't account for enough downside risk, and Hamilton's downside risk is greater than otherwise similar ballplayers. Absent the risk of a serious relapse, he seems far more brittle than the typical player. But if he's willing to sign a deal that takes into account those things, then yeah, I'd sign him. I don't know if he's willing to, though, nor do I know if another team is willing to offer him a deal better than that.
batts40 (IL): I run PFM during my draft, put prices on guys as they go in the draft and adjust strategy by inflation, etc.
Colin Wyers:
modofacid (philly): Re: offline PFM.
You would be able Update inflation $$ as the draft occurs through PFM. Also, may solve some of the speed issues you mentioned for Iorg's question.
Colin Wyers:
Brew Jays (Toronto): I won my league last year because of the PFM. An offline version would only make it even better.
The user-centric inflation is crucial, but the biggest negative is waiting for the page to load in the online version.
Colin Wyers:
whonichol (DC): Re: Offline PFM - I agree that it would be usefull as a stand alone app that would increase responsiveness during the draft. I think that in-draft usage is the most useful aspect of PFM...
Colin Wyers:
dianagram (VORGville): "Is there a benefit to running PFM during the draft, as opposed to exporting a CSV out of the PFM and using that during the draft?"
===============
Most certainly, as the PFM algorithms adjust the rankings based on availability at a position and which teams already have those positions filled. (User-centric inflation I believe)
Colin Wyers: Okay, hearing from a lot of people that offline PFM would be useful. We'll look into it. I assume most of you are talking about using a Windows-based laptop, yes?
belewfripp (Knoxville, TN): Hi Colin - First, just let me say that I have really enjoyed the statistical analysis you've brought to BP since you started writing for the website.
Second, I've noticed that other sabermetrically-inclined writers on the web - people like Rob Neyer, Joe Sheehan and Joe Posnanski - tend, when using a thumbnail stat to evaluate a player, to almost always use the version of WAR developed by either Fangraphs or Baseball-Reference.
Does it bother you that, although BP's WARP can be found on a player's Pecota card w/o being a subscriber, it does not seem to be considered the standard or benchmark measurement? And do you have any theories as to why that is? Personally, I usually use BP's version of WARP and started thinking about this when Sheehan wrote in a recent newsletter that Edwin Jackson was a 3-4 WAR/season pitcher and I thought to myself, "No he's not..."
Colin Wyers: My best guess is that, for several years, we basically kept WARP in the basement, on a separate set of player cards and not well integrated into the sortable reports. If we make it hard to use our information, people won't use it. I think we've done quite a bit to correct this in the past year or so, but we know there's a lot more we can be doing to make our cards and sortables more useful. So expect to see more improvements there.
Cardinals645 (Houston): With the new weighting for PECOTA, isn't there a risk that breakouts for younger players, and collapses for older players?
Colin Wyers: It's very, very hard to tell the difference between a breakout and a career year until a player has had a chance to play again. I wrote a lot about this for the upcoming BBTN sequel and Bautista (far more than I can reasonably type into this little chat box). But "fluke" one-year increases are much more common than real breakouts.
It may help to think about it this way - instead of doing as much regression to the league mean, we're doing a bit more of "regressing" a player to his past performance.
myshkin (Santa Clara, CA): I always figured that, these days, the simplest way to do an offline PFM would be to use Javascript and HTML5 client-side storage, for easier cross-platform support (including tablets and smartphones), and somewhat more code overlap with the existing product (?). I have no idea how you've implemented the PFM, though. Would that be feasible?
Colin Wyers: Last I looked into anything like this (and I'm not the expert, by a long shot) different browsers had support for different client-side storage engines. It's a good idea, though.
Steve (Toronto): PECOTA seems a bit low on Brett Lawrie for 2012 based on what I see in the PFM. Your thoughts?
Colin Wyers: In terms of his major league performance, Lawrie (like everyone) is subject to Voros' Law: any player good enough to make it to MLB can do anything in a third of a season or so. So you have to account for that.
Now, Lawrie had very good minor league numbers as well. But you have to look at how those stats translate to the majors. One thing we're doing now (and I plan on writing more about this in the future) is regressing translated minor league stats to a lower level of performance than the major league average, but instead the average performance of players coming up from the minors. That's going to temper his forecast a bit more than if we were regressing to the MLB average.
Bolden (Atlanta): Do you think we will ever see a female player in MLB? If so, how would PECOTA deal with this position?
Colin Wyers: Ever is a long time.
We're certainly not going to see it soon. I think the fundamental thing is that there currently are not any high-level programs where women play baseball currently - typically at the college level and the pro leagues that do exist play softball, not baseball. So to see a female player in MLB you'd probably have to see those convert over to baseball. And even then, there's no guarantee - college and pro basketball for women is very well developed by this point, and I still think we're a ways away from seeing a woman ever play in the NBA (and I understand that's not an apples to apples comparison, because basketball selects heavily based on height).
But if you do get to that point, it'll probably be because you've developed professional or high-level amateur programs that have started to fill into the affiliated minors, so you should have data to go on to make a projection.
ObnoxiousGuy (Chicago): Just poking a sleeping bear with a stick here, but the fangraphs article also claimed that just using SIERA regression was more accurate than PECOTA.
Colin Wyers: I think it's an apples and oranges comparison. The difference between the two in terms of something like RMSE is dwarfed by the difference in terms of SD of the inputs. So by being conservative you get a lower RMSE, but is it worth it? If your projection says that "68% will have an ERA within .41 points of the league average" (which is what Matt reported for his SIERA projections), compared to .72 for PECOTA, you're creating forecasts that are far less informative.
What a forecasting system is supposed to do is help you identify talent - for fantasy or for real-world teams. If you can reduce your RMSE by saying X,Y and Z all have the same expected level of performance, you're not actually contributing anything useful to the decision between X,Y and Z. If you're willing to be less conservative and say that "X is likely to perform better than Y and Z," I think that's more useful.
Ace (Texas): How would PECOTA deal with a player that came out of the closet? I would have to believe the player would face a lot of scrutiny that can't ever be measured by a player evaluation system.
Colin Wyers: To the larger question, and tying in with the earlier question about Hamilton and Bradley - there are always things that factor into a player's performance that we can't predict, because they're "intangible." Players get divorced. Players have sick kids. A player's parent may die.
And we really are unequipped to model how those things affect performance numbers. And I think we always will be. Because different players are suited to responding to them differently, and collecting data is nearly impossible - I think PECOTA should try and handle what it handles well, and you can adjust from there based upon what you know of the intangible aspects.
Colin Wyers: Thanks for coming, everyone. I'm out of time for today, unfortunately, but I'll try and come back and do this again soon. Have a great day.
|
