Another fantastic article by IMO the most important guy at BP right now.

Keep it up, Mr. Swartz.

This is a good question to ask, Matt.

Industrial engineers use advanced statistical tools in the endless pursuit of 100% yield. And even when all the variables are accounted for, we have the occasional hiccup.

Then there's the notion that making the "ideal" statistical move is an absolute. Certainly there are competing statistical methodologies that produce varying outcomes. Even the most thorough organizations like the Red Sox clearly make moves at odds with the data (see Lugo, Julio).

The data we get from PECOTA is a forecast of player performance on a macro scale (i.e., over the course of a season). Then we further scale that to predict a given performance, matched with other given performances, will produce a larger outcome (i.e., team wins and losses). If memory serves, the Red Sox under Francona have as often as not underperformed their Pythagoream record. That sometimes gets lost when we consider their overall record in that span. (When your guys win a World Series, you don't worry so much that the data say they should have won 102 games, not just 96.) Maybe there's a wrinkle in there on the scaled level that still needs to be worked out, and/or perhaps there's work to be done for in-game statistical use.

I think it would be interesting to see how predictions made through a non-sabermetric approach would compare, which would give us a better idea of whether PECOTA and other sabermetric projections are unique in overrating certain teams, or if "everyone" is overrating those teams. In the case of Cleveland, for instance, I'm sure that the consensus mainstream preseason predictions for 2006 and 2008 were for them to finish 1st or 2nd in the Central.

Of course it's harder to pin down a win estimate for each team, but perhaps one could look at predicted place in the standings instead of wins.

First of all, excellent article Matt.

One thing I'd be interested in though: How do the results change if Actual Wins are replaced by Third Order Wins (or Pythagorean Wins or something similar) in the above table? Just curious to see if Pecota is overrating a team's overall ability, or if the problem lies in Pecota not being able to distinguish teams that, for one reason or another, are more likely to under-perform their expected win %

A very interesting article, Matt. Good work.

If you regress Actual Wins on Saber Usage and Average Payroll (plus a constant), the coefficient on Saber Usage is insignificantly different from zero. I.e., there's no evidence in this data (about which more later) that sabermetric capability has an independent impact on winning, apart from payroll. Though it is worth emphasizing that "absence of evidence is NOT evidence of absence" - this is a small data set and significance is difficult to achieve in that context.

If you regress PECOTA Projected Wins on Saber Usage and Average Payroll, Saber Usage has a positive and statistically significant coefficient. This is, basically, your conclusion that saber-savvy teams pick the same players that PECOTA likes.

One limitation here is the survey of Saber Usage: apart from the limited number of respondents, team's front offices are black boxes. There are many teams that have hired sabermetricians, but we have no idea which ones actually listen to them - or, if they do, what weight they attach to the sabermetricians' views.

On the whole, though, this is a very thought-provoking piece.

Good Read. More of this please.

I look at the "Moneyball approach" and sabermetrics as two different, but related, ideas. From my understanding, the Moneyball approach is mostly about identifying skills (or classes of skills) that are undervalued by the market and making them a target of your organization. The A's used sabermetrics to identify that OBP was such a skill. The Rays' sudden rise in 2008 was largely a result of their improvement on defense, something thought to be undervalued by the market. However, I'm not sure that sabermetrics was how they accomplished this; it might have involved a large amount of scouting instead (I don't know for sure).

I just want to know why PECOTA couldn't see the Nate McClouth meltdown coming

Without doing any real math, what looks overrated is teams that ended up having bad bullpens and what looks underrated are often teams with good bullpens.

Perhaps it's because that's the highest variance or perhaps it's in the playing time dynamic or possibly just how PECOTA handles relievers.

GREAT ARTICLE- one of the things I MOSt love about BP is your willingness to question your results.

Of course PECOTA is just a fomula and does not know or care how decisions are made- it can only process the data it is given.

However, nothing is really ever 100% objective. By quantifying a process, you eliminate a lot of subjectivity, but subjectivity is still introduced in the system by the initial decisons of the people who created the PECTOA program. Great job by you in at least identitfying and trying to measure this effect.

Excellent article. The one issue I have is that if teams aren't saber-oriented then they must be scout-oriented. In other words, their decisions must come from some philosophy. Asking BP staffers will get you mostly saber-oriented opinions and they are likely to think good teams follow their philosophy (i.e. "The Yankees are good so they must be sabermetrically oriented.")

I think if you ask a bunch of scouts which team most utilizes scouting information you won't get a unanimous response of Kansas City, the way BPers proudly claimed Boston.

I think they tried to give it a fair shake when I see where Pittsburg is, for example, but I think comments like "Sabermetric teams are doing better overall compared to teams that do not rely as much on sabermetrics" have to be taken with a grain of skepticism given the bias of the response group.

Several problems, some of which you and others have already mentioned:

1) Potential bias of authors thinking teams PECOTA likes are more sabery.

2) Authors ignoring the timeframe of the question. Do people really think the Pirates and Mariners' major league teams over the 2005-2010 are the product of sabery front offices? I doubt it.

3) The numbers aren't independent. If the Angels are 9 wins/year better than PECOTA thinks, and they play the Mariners and A's in 1/9 of their games, those teams are going to lose a win a year even if their own talent was correctly assessed.

4) PECOTA projected standings are largely dependent on subjective guesses at playing time and injuries. I'd be interested in seeing whether individual players on supposedly sabery teams underperformed their PECOTAs.

5) There's a negative correlation between payroll and saberness, and high payroll teams are more likely to add talent during the season (while low payroll teams are more likely to shed talent).

One thing that you learn in econometrics or other statistical courses is to study outliers if you want to understand better where a particular predictive equation (PECOTA)over or underestimates the phenomenon it is measuring (in this case, wins for MLB teams from 2006-2010) .

So, the question from the excellent Matt Schwartz article is are there characteristics that that the outliers on both ends of the distribution share that PECOTA either over or undervalues? On the underestimating side (ie, the teams with a negative difference between PECOTA projected wins and actual wins) there are eight teams PECOTA underestimated wins by 5 or more (using rounding). What characteristics might the Angels, Twins, Blue Jays, White Sox, Rangers, Padres and Rockies share? It might they are less sabermetrically oriented, but the fact the Rangers, Padres and Blue Jays are all in the cohort make me think it's not that. What might it be? A commenter above suggeseted good bullpens and that might be worth looking at. The Twins and Angels anyway, perhaps share a philosophy that is more oriented away from OBP relative to BA. Could OBP could be overvalued by PECOTA in predicting wins? Another characteristics these teams perhaps share is a pitching staff that issues few walks? Maybe PECOTA doesn't give enough credit to teams that limit walks? Obversely, perhaps PECOTA overvalues strikeouts?

Something that has always struck me about where PECOTA and other predictive equations is the difference between projected wins and actuals win, when it is a consistent phenomenon (ie, not just a single year) is the difference between predicted and actual wins might be a proxy for manager skill. Is it possible the eight teams that best overperform on PECOTA are being managed by good managers relative to other teams? Obversely, maybe the Indians and Pirates in particular have suffered from poor managers? It's a crude tool, but no one has come up with good metrics for measuring the impact of a manager (probably for good reason) but everyone who follows the game recognizes the manager matters, at least to some degree and we know manager skill is variable. Perhaps PECOTA error is a measurement of manager effect??

Could this be a matter of PECOTA systematically underrating the ever nebulous but scouting friendly "tools." Perhaps players without quite the statistical track record (especially in the minor leagues) are skewing the results the opposite direction because the system fails to foresee breakouts, rather than under-performance by those with solid track records.

A few thoughts:
- Where does the ability of an organization to develop come in to the calculus? Presumably, PECOTAs projections for minor leaguers is less accurate than with known commodities, thereby biasing its projections against teams with lots of young talent.
- What about defense. If strong team defense tends to be a characteristic of traditionalist teams and is absent or undervalued by PECOTA, this is an obvious place for error.
- What level of error do we see here and to what degree can that be explained by things which are essentially random (e.g. significant injuries) which just randomly happen to be correlated with sabermetric leaning over this time period. Are we perhaps making a 'post hoc, ergo propter hoc' mistake?
- Or if we believe that keeping players healthy is a skill which an organization can possess, its absence from the model could be reflected here.
- Lastly, what about the value gained (or lost) through synergies (on incongruities) in roster construction. Take a great defensive SS and a ground ball pitcher. PECOTA will not account from the ways in which they will benefit each other in its projections. Are non-saber teams building more efficient synergistic rosters?

Interesting article.
The woeful Indians are the poster boy for ridiculously overrating "good" organizations as Sheehan put it.
That said, I don't think Shapiro's problem is sabremetrics. It's just his inability to build a complete baseball team. For a team with a moderate-low payroll to spend $10Mill on Kerry Wood is just nutty.

Matt, in your opinion, is it at all possible that it's sabermetric's possible flaws themselves that are causing PECOTA to overproject team wins? By flaws I am talking mainly about overvaluing OBP and strikeouts?

This is somewhat tangential, but I would love to see a thorough, objective analysis of Billy Beane's performance over the past five years. Obviously, the end results have not been that good; but would like to see evidence for and against the thesis that he is no longer a top GM.

I read the article and skimmed the 41 (When I posted) replies. My question/observation:

Do bad teams generally under-perform while good teams over-perform?

Any predictions are typically conservative whether for players or teams. That said, team predictions are compiled projections of the players that make up those teams at the beginning of the season.

Trades and other roster moves are considered to not make a large difference, but they do make a difference. Teams under-performing dump talent and teams over-performing acquire talent?

Just curious.