Prospectus Idol Entry

Nicely written, Brian.

“As a player ages through his twenties, he will on average lose speed, but gain power, strike zone judgement… It would then make sense not to let too much time elapse between the two sets of stats being compared.” Would it not make more sense adjust his statistics based on appropriate aging patterns? Of course, with limited amateur resources that may have been too difficult, but aren’t there some reasonable aging algorithms publicly available?

“The factors will be depressed by a disproportionate number of players at the higher level (particularly in the majors) playing sparingly.” Nice catch. That’s a nuance I never considered before.

“. . . is it correct to base the factors partly on the records of players who failed to advance?” Sure it is. That makes it more useful for assessing if a player could cut it in the majors. If your MLE is accurate, it should reflect that ability.

“The purpose of the MLEs is to show how well a player in the minors will perform, if and when he reaches the majors.” That may be so, but in plain English “Major League equivalents” would be a direct translation of what that player would have done in the Majors, not what he will do when he gets there. We need a better word for this.

“. . . the direct comparison method consistently gives the closest estimate of future MLB performance.” Given that the “MLB direct” method shows the MLE of an A+ level player to have deflated indications of power (doubles and home runs on balls in play), while AA and AAA show inflated indications of power by the same measure, does this pass the smell test?

Whether it does or not, this is an interesting exploration. Perhaps, this study shows that players of future major league ability already demonstrate in A+ whether they are capable or not, while their time in AA and AAA is a waste! Their hit rates on balls in play (SDT or BA/BiP) go down at each level on their way up, because fielding improves at each level. Their walk rates drop and strike out rates rise, because the pitchers they face are better at each higher level. How much of that is due to his improvements in hitting may be very little – as little as his power rates improve going from A+ to MLB over the number of years it takes.

"Would it not make more sense adjust his statistics based on appropriate aging patterns?"

Yes, a finished projections system should include aging patterns, but these tests were to determine which selection criteria were best for a matched pairs comparison, which is the first of several steps before applying age corrections. In that test I was attempting to minimize other influences such as aging. Several projections that I am familiar with deliberately choose to limit their comparisons to adjacent seasons or only same seasons in order to avoid an aging bias. I wanted to find out if this is a wise approach, or if it creates more problems than it solves.

A lot of numbers here. I agree this would have been more interesting with at least a few examples of real life players.

Thanks, Brian. Thinking about this a little further (a good sign):

Wouldn't the final accuracy test with players who actually made the Major Leagues produce a bias against the first two MLE methods which include minor leaguers who did not reach the Majors?

Who were the players you sampled for these data sets? You stated whom their MLEs were compared against in the final test, but I don't see where you stated where the samples come from in the first place other than "matched pairs of batting data". I know you were limited in word count, but the data selection process is rather critical in such an analysis.

I agree as well about needing examples. I got to the point where I could figure out what you were analyzing if I really wanted to, but then realized I could just find the Marcel system (which has existed for longer) and use that instead...

I also wonder about the level of precision with the calculations. If you multiply a .00002 by a .00003, are you getting .00006, .0001 or .0000 (depending on whether you are calculating or rounding up/down). In this kind of article, where a difference of .0001 is significant, it'd be important to include a statement about what rounding, if any, you are using.

Brian,

I really had to take my time reading your piece, mainly because I wasn't clear on what your actual goal was until the very end--not really a big deal, because you did make it clear eventually.

I was happy to see that you included both types of comparisons, and not just the ones that worked. I beleive this kind of reporting/analysis is the most informative.

Lastly, I think it is important to include all of the players at the lower levels, and not just the handful that make it to the MLB. After all, we are trying to forecast prospects, and a team won't know which of their prospects will make the majors. If I understand your piece correctly, the numbers have been adjusted using only those that made the majors. But if we then apply these numbers to all prospects, won't it make some look better than they actually are? If I were a GM, I would rather err on the side of a prospect doing better than I projected (using deflated numbers) than spending money, time, etc. on a prospect who was not as good as I thought (because of inflated numbers).

Overall I enjoyed this article, and can't wait to see more!

The factors were calculated in four different ways. The results of each were tested with the 368 rookies in MLB from 2003 to 2008.

I didn't do any rounding other than telling Access to display 3 deciamls.

The purpose the the MLE fctors is to translate minor league stats into 'equivalent' major league stats. How can I test the accuracy of the translation on players who did not perform in both situations?

Say I have a set of ten x,y pairs, each in two different coordinate systems. I can compare the two sets of data to derive a matrix to convert from one to another. Then each of the points are run through this matrix to compare their predicted location to their observed, the residual error. Once the residuals are sufficiently small, the conversion is deemed reliable, and any point from coordinate system one can be converted to system two. The complication in MLEs is that there are more than two systems, with six levels of minors below MLB.

What is the method of modeling the talent levels of the various minor leagues that will create the smallest residual errors? I listed four different methods, and calculated the error rates for each. We need to avoid picking a method because inutitively it sounds good without firt testing it against the alternatives.

Brian, this is an excellent article but a rigorous article. It's tough communicating the results of so much research in so short a piece. Furthermore, while I enjoyed your research and your results the most, I'm not sure that BP is aiming for articles that demand quite so much concentration and study to comprehend. (Well, articles that require so much concentration and study for ME to comprehend...)

In any case, thanks for sharing the research, and best of luck: I know your work, and I want the winner of this contest to be a writer capable of bringing to BP the sort of insight and analysis that one always finds in your articles.

I'm adding my judging comment to each article:

Cartwright, Brian -- 7. I think this is a BP quality piece. Most of it is way above my head and therein lies my problem with it. I just don't grasp most of it and he makes little effort to bring it down to a level where I get any sort of takeaway or even an explanation. As an example he starts off with a question about "elapsed time" - I'm not sure what it means and I don't see where he answered it in the piece. There's going to be an element of BP readers that loves this and I don't blame them, but there's going to be a lot of people that just don't get it.

i still don't get it.

and the time constraint? heck, you've got until friday to get the next piece done. and then the next week ... and the next week ...

'we do this every day' - earl weaver

Mitchel Lichtman said recently on his blog that in calculating his unpublished projections, he only compares players in different levels in the same season. The reasoning is that if you compare 2007 to 2008 stats, the player's true talent may have changed during that time. That approach limits you to a chaining process to compare the lower minors to MLB. If I compare two stat samples that are four years apart, the odds that the player's talent has changed (the thing I'm NOT trying to measure at this step) is higher, but it gives me an opportunity to directly compare a player's performance in Class A to his performance in MLB. Which method gives the best results?

Brian, little advice: Stay out of the comments. Seriously. If you need to comment on your own work then your work wasn't written well enough in the first place.

And Will, I believe the "With such a time constraint..." hanger was the lead of the next paragraph, not a complaint about working conditions. Although I suppose it could be both.

You know, the more hours that pass after my reading this, the more I understand how important this article is. I understood the first time what you'd proven, Brian, but I'm getting a better grasp with reflection of why your findings are important.

Thanks, Brian. I'm realizing that this is not just a great contest entry, but that it's also going to be one of those articles that might influence thought in the community for some months to come. It took me a while to appreciate what you'd written, but it was well worth the time.

Well thought out techniques and I liked the process of examining the bias inherent in each of the four methods. I find the debate over the end goal of MLEs to be interesting. Is the idea to find out 1)how any player from minor leagues would have done in the big leagues in that year or 2)to find out how the guys that will eventually get there can be expected to perform. I'm not sure I believe there is any mathematical way to determine #1. Very rough approximations are a possibility but ultimately there are so many factors involved that I just don't think it's possible to acheive much accuracy at all. #2 seems slightly more feasible, and this seems to be what you were getting at with the MLB Direct method. I like the fact that the translation factors were raised by this method b/c many MLEs seem to grossly undervalue the most talented players (at least when looking back in hindsight). Nice work.

I really liked a lot of what's going on here, but I've got a Master's degree in Statistics, and I have no idea what all of the statistics you report in the final table mean.

Also, a quick question:

To test the accuracy of the various projection methods, you use the actual accumulated MLB statistics from a group of player that, by necessity, have reached the bigs. Wouldn't this bias your results in favor of the projection methods that, similarly, only look at players that reached the big leagues?

Similarly, any system for translating minor league statistics to the big leagues will necessarily be optimized for translating minor league statistics for players that reach the big leagues. Since in practice, we will inevitably end up using such projections on players who won't reach the majors, won't this introduce unaccounted for variation into the model? Is there any way to deal with this? Is it even important to?

Comments and some things I've learned this week -
I've been a BP subscriber for about 10 years, and this is the fourth site where I have published articles. One of the problems when writing for a new audience is knowing the type of article they expect. When I first started reading BP, it was a stats site, but apparently is not so much anymore, although there are some (many?) of us who wish it still was. I am still primarily a stats guy, and would like to write mainly stats articles for BP, but wherever I write will be learning how to tailor my message to the audience.

As someone who works with the stats virtually everyday, the numbers quoted in the article were in my language, and had lots of meaning for me. When someone says "superior projection of a few hundreths of OBA+Slg" I realize I have not made it clear in terms the readers are familiar with.

Restating the test results:
BA OB SA
270 327 427 Marcel of MLB of 368 players in test sample
267 323 419 A+ using MLB Direct
242 295 368 A+ using MLB chained
234 284 345 A+ using MinPA Chained
229 279 338 A+ using All Chained

Collectively the 368 players had a MLB line of 270/327/427 in their first 2-3 seasons. Using only batting data from High A and below, the MLB Direct approach would tells us that they project to 267/323/419, very close. A projection system that uses an All Chained method would project these same players, based on the same set of stats, to have a 229/279/338 line, which is clearly not good enough to even play in the majors, instead of MLB average, and therefor not a useful method.

I'm guessing this is not an original point, but the big problem with MLEs seems to be the interpretation. The desired (in my opinion) interpretation is, "What can we expect a player in Double A to produce if promoted to the majors given his stats?" But, instead, we get the expectation of Double A players _that were promoted_. This is a selected sample comprised of players who, on average, are more ready for the next level given their stats. If you randomly promoted a player, it's unlikely you'd get the same stats. The difference is basically whether you want MLEs to be descriptive of what happens or predictive of what would happen in other situations.

I bring this up because the different methods described above are different ways to select the sample. You're not just changing the "elapsed time" or accounting for the "pinch hit penalty." You're also changing the bias due to selection. By selecting based on plate appearance per game, you're dropping players that started in Triple A but were only good enough to pinch hit in the majors. But you're keeping players that started in Triple A but were good enough to play regularly in the majors. Your sample ends up only including the successful players. Similar points can be made for the other permutations.

This isn't a fatal flaw, but I think the differences in the results need to also be discussed in this light. Thoughts?

MLB Chained and MLB Direct used the same sets of players, the same stats. If there are biases in the selection of players, it is cancelled out. The difference in these two methods is whether A+ is being compared to MLB, or A+ to AA to AAA to MLB. For the same players, chaining gives a result that has more error, an error that says the test players (as a group) are not good enough to even play in MLB, when in fact they performed at MLB average. If there's a bias that creates a result that's 95% of true value, by chaining the result is 90% in AA, 85% in A+, 81% in A, 74% in A-...it just keeps getting worse the more steps away from MLB.