Moonshot: Prospect Tra... (03/18)
the archives are now free.
13 comments have been left for this article. BP Comment Quick Links
| |
|
|
March 12, 2015 MoonshotAnother Look at Prospect Values
A recurring question this offseason has been about the value of top prospects. We have seen San Diego accomplish a significant rebuild without trading away much of the top of their farm system, soon after Billy Beane was compelled to send BP’s no. 2 rated prospect, Addison Russell, to Chicago for a one-year rental of Jeff Samardzija. Meanwhile, the Cole Hamels saga is ongoing in Philadelphia, with Ruben Amaro, Jr. coyly refusing to ship Hamels for less than a handful of top young talents (or so the reports claim). These scenarios suggest that the value of prospects is a complex and contentious topic, even within baseball’s front offices. Admittedly, I am not the first and will not be the last to approach this question. But the quandary of prospect valuation is worth reexamination both because of its complexity and importance. Others have approached the topic from the standpoint of minor-league statistics, and found some ability to project future success with that information. That’s PECOTA’s beat, though, and any way, previous research has shown that skilled talent evaluators are able to outperform minor-league-based projections. As a way of harnessing the scout’s eyes, I borrowed Chris St. John’s prospect rankings database, which covers various top 100 lists going all the way back to 1990. St. John’s collection aggregates from a wide variety of sources, including BP and Baseball America. The most straightforward way to approach the question of prospect valuation is to build the following simple graph, which charts the average rank of a position player prospect against their lifetime WARP. This method takes the scouts at their word, and simply asks how well their rankings reflect the eventual value of a player.
The shape of the curve is similar to what’s been previously reported, but the magnitude or depth of it is very different. I find that a top-5 hitting prospect is worth on average, about 15 WARP over the course of his playing career. That number comes with a massive variance, of course, ranging all the way from negative WARP to more than 100. But, importantly, the average outcome is substantially higher than lower-ranked players. I will note here that other studies have found lower numbers for the value of top-ranked prospects. There are a couple of ways my study is different. One, I am using a slightly different timeframe (1990-2010[1], although this would tend to decrease the relative value). Two, for now I am excluding pitchers, who may pollute the overall rankings with their injury issues and greater uncertainty (TINSTAAPP). Three, I take the average of all of a player’s appearances, across years and different lists. I find that the average is a better way of examining prospect worth, because it exploits variation between lists as well as variation across years to get a clearer picture of the consensus on each prospect. A guy who is rated very highly for one year is less likely to succeed than one who consistently occupies the top ten for two or three years in a row. Adding additional lists proved surprisingly unhelpful. In the above graph, I show the same curves but generated from the lists of only one publication each: Baseball Prospectus, in blue, and Baseball America, in red. As you can see, they mirror the consensus, in black, with minimal differences. If anything, the addition of more lists tends to dampen the success rate of lower-ranked prospects. Other Factors
It turns out that two years is the sweet spot. Prospects who go more than two years on the lists tend to gradually fall off toward lower contributions, and prospects who appear on only a single year’s list are usually not terribly good prospects to begin with. So, while Buxton is safe, Sano, who has appeared on five consecutive years of lists, is probably beginning to tip toward a worrisome fate. Another factor of interest is age. The younger a prospect is, the better off he is in terms of lifetime WARP production, all else being equal. In theory, age is already baked into some of these prospect lists (depending on the individual philosophies of the evaluators who made each list). However, my results show that a highly rated prospect at 19 is a better bet for lifetime WARP production than one at 26. I don’t think that should surprise anyone. Finally, we can look at the position a player fields. Generally, we should expect that players who field more challenging positions will do better in the long term. Since athletic ability is tied to position, and athletic ability tends to diminish, not increase, over time, players who start from loftier positions on the defensive spectrum have more spots on the diamond to fall down to as they age and become worse fielders. As Sam Miller has noted, a surprisingly large portion of the players currently in MLB played at least some shortstop in the minors—even a few nearly immobile guys (e.g. Mike Morse) who we do not think of as defensive wizards logged some time at SS. Their initial competence may have given them the room to decline into becoming merely poor defenders in MLB, as opposed to unacceptably terrible. The data supports this notion, with some caveats. Since many players try out multiple positions in the minors, there is no perfectly clean way to divide them. One simple partition to make is based on whether a player ever played shortstop. Those who did averaged 10.6 WARP (this is heavily inflated by the tail of players who had superlative careers). Those who didn’t averaged 9.3 WARP, which is a bit less. Similarly, the median WARP of non-shortstops is 2.6, vs. shortstops at 3.7. Admittedly, this is kind of a clumsy way to parse the data, because it tells us nothing about their performance in the time that they played shortstop. They could have been terrible, and switched immediately to first base, for instance, which doesn’t say much positive about their fielding ability. Nevertheless, that someone even attempted to put them there suggests that they could run at least a little bit, and bodes well for their future athleticism as they slide downwards towards DH with age. There was a similar effect, albeit slightly muted, for catchers, as well. If we put all of these factors together into a giant regression[2], and then attempt to predict the WARP of each player, we get a graph that looks like this:
There is a bit of correlation, but it isn’t terribly reliable (R2=.18). This isn’t a sterling R2 number. But we shouldn’t have expected it to be. Forecasting the future production of 18- to 22-year-olds decades in advance is no easy task. Even leveraging teams of trained professionals or heartless but brilliant machines, we can scarcely make headway on this difficult problem. Indeed, the lack of predictability suggests that teams will be disagreeing about the value of prospects as long as there are prospects. In the presence of so much uncertainty, a huckster can argue that his asset is a Hall of Famer while his adversary can wave Delmon Young’s impressive prospect credentials around instead. The truth is that while highly rated prospects are likely, on average, to make substantial contributions to their teams, any individual prospect could be worth as little to the team’s success as a quad-A replacement.
[1] The depth of the curve increases as you move the timeframe back. So, limiting it to 2005 produces a still-steeper curve, with top prospects worth ~20 WARP. I chose the more recent timeframe to get as much of the data as I could. [2] Using a SVM with 5-fold cross-validation, to curtail overfitting.
Robert Arthur is an author of Baseball Prospectus. Follow @No_Little_Plans
|
Five to Watch: National League Starting Pitc...
Robert, this is really good. Is it possible to look at years on list alongside age they first appeared on a list? I wonder if Sano's presence from a very young age -- because he was a foreigner who didn't need to play HS ball in the US -- should be looked at as a positive or a negative, given your findings.
Good idea. I went back and looked at the data, and 2 years on the list is the optimal number, pretty much no matter what the age a prospect started getting ranked is.
There might be something to the foreign player vs. domestic player distinction, though. Maybe Sano was less prepared/needed more development than an equivalent, US-born player. Perhaps the prospect people can weigh in on that, I'm not enough of an expert to know.
My thinking is that it's not so much a knock on those players who appear on this list 3+ years, but a sign of how rare that actually is: you have to be star prospect from the get-go, and have not petered out like so many others.
Whereas it is much more common to make your mark on the prospect charts at A/AA level, follow it up with success at the AA/AAA level, and then be out of the running due to promotion.
I'd be curious to know, within each pool, what was the chance of reasonable success.
It's likely that being on the lists three or more years means you are young, highly regarded, and haven't done anything real enough yet. Probably lots of international 16 year olds in that area. You are right about the guys who come off the list due to promotion. That's why it's the sweet spot. Interestingly, this won't cover the out-of-nowhere guys like Pujols who are drafted in the later rounds and rocket through the system without making much of an appearance on lists. That's fairly rare, though...right?
@therealn0d I'm not aware of any international 16-year-olds on top 100 lists in the past 10 years.
My theory (that perhaps the author could test), is that there is far too much inertia when changing prospect rankings. a.k.a. the VanBenSchoten effect. To test that, you might look at players moved either up or down 25+ spots in a given year, and see how much the out/under-performed their position (in the second year).
The other really interesting thing would be to look at which ranking sources have been leaders vs. followers. That is, if source X ranks a player a certain distance from the average rank, do the rest of the sources tend to follow the next year or not. It's a simpler/quicker test that career WAR, and perhaps more valuable as it would be good to know who is best at predicting future predictions.
This question about inertia is the subject of the next article.
Could also look into the leaders vs. followers phenomenon, but you might run into problems because not all lists are released in the same order every year. Within a year, I expect there is some cross-influence between lists, too.