November 7, 2012

Pebble Hunting

Are GMs Smart or Not Smart?

by Sam Miller

The answer is they are smart. Come on guys, be serious. Of course they're smart. 

But...

***

Once a year, ESPN’s Jerry Crasnick polls baseball executives on Hot Stove topics. This year Crasnick surveyed “22 general managers, assistant GMs, advisers, scouting directors and talent evaluators in the field for their opinions on seven questions that are likely to drive media coverage,” and the results are characteristically fun: Josh Hamilton to the Brewers! One year and $2 million for Melky! Many other things!

It’s one of my favorite pieces each offseason, and I hope it goes on forever, or at least until I die, because it makes me sad to think that the world will keep going when I’m gone. The hardest part of writing about baseball is the realization that, by not being in the room, I’m getting maybe two percent of the relevant information, which is probably canceled out by the flood of misinformation that does make it to me by virtue of being false and thus having nobody trying to protect it from the public. The guys Crasnick talks to are in the room, and they’re friends with other people in other rooms, and because they’re anonymous there are no obvious incentives for them to lie, so it’s fascinating to read what they believe to be true.

But “believe to be true” is still not necessarily true, and I wondered how much extra information these guys really have. Crasnick has been doing this long enough that a pretty good archive of these pieces is available. Thanks to all those phone calls Crasnick made, we can now review the executives’ predictions from 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, and 2011.

I put every prediction into a spreadsheet, a simplified version of which appears below, a more detailed version of which you can see here. I limited it to predictions that are specific and not vague. For instance, I deemed “Which contract was the worst signed so far” not specific enough, but “Which of (five players listed) will be best next year” specific enough. With one exception, I ignored executives' salary predictions, because of the wide range of responses and because the multivariable aspect of contracts (length + AAV) makes it hard to evaluate each prediction's correctness.

Year	Question	Consensus	Correct?	Type
2003	Who is most likely to be wearing a different uniform in 2004 -- Alex Rodriguez, Garciaparra or Griffey Jr.?	Garciaparra	No	1
2003	If the price was right, which shortstop would you sign, Miguel Tejada or Kaz Matsui?	Tejada	Yes	2
2003	Where will Vladimir Guerrero play in 2004?	Baltimore	No	1
2003	Kevin Millwood or Bartolo Colon?	Colon	No	2
2004	Where does Adrian Beltre sign, and for how much?	Seattle	Yes	1
2004	Will Billy Beane trade Tim Hudson?	Yes	Yes	1
2004	Where will he (Hudson) go?	Dodgers	No	1
2004	Will Jason Giambi be the Yankees' Opening Day first baseman?	No	No	1
2005	Where will Manny Ramirez be playing on Opening Day?	Boston	Yes	1
2005	Which team will sign Paul Konerko, and how much will it take?	Chicago	Yes	1
2005	Will Roger Clemens return to pitch in 2006?	Yes	Yes	1
2005	If money were no object, which premium closer would you rather sign -- Billy Wagner or B.J. Ryan?	Wagner	Yes	2
2005	Will Rafael Palmeiro return to play in 2006?	No	Yes	1
2005	Of Jeff Bagwell, Mike Piazza, Sammy Sosa or Frank Thomas, which player is most likely to have a productive, season?	Thomas most, Sosa least	Yes	2
2006	Given a choice between Barry Zito and Daisuke Matsuzaka, which free-agent pitcher do you prefer?	Matsuzaka	yes	2
2006	Where do you think Barry Bonds will sign?	Giants	Yes	1
2006	Alfonso Soriano or Carlos Lee?	Soriano	Yes	2
2006	Where will Roger Clemens be pitching in 2007, or will he retire?	Astros	No	1
2006	Which player with a no-trade clause -- Manny Ramirez, Alex Rodriguez or Pat Burrell -- is most likely to be traded?	Burrell	No	1
2006	Which "second tier" starter do you like best: Adam Eaton, Ted Lilly, Gil Meche, Vicente Padilla or Randy Wolf?	Lilly	Yes	2
2007	Where do you expect Alex Rodriguez to be playing next year?	Angels	No	1
2007	Where do you expect Barry Bonds to be in 2008?	A's or retire	By a small plurality	1
2007	Who would you want in center field, Torii Hunter or Andruw Jones?	Hunter	Yes	2
2007	What will the Minnesota Twins do with Johan Santana?	Keep and/or extend	No	1
2008	Which team will sign left-hander CC Sabathia?	Yankees	Yes	1
2008	Which team will sign Manny Ramirez?	Dodgers	Yes	1
2008	Will the Padres trade pitcher Jake Peavy?	Yes	No	1
2008	Will the Rockies trade outfielder Matt Holliday?	Yes	Yes	1
2008	Who would you rather sign to a four-year deal, Derek Lowe or A.J. Burnett?	Lowe	No	2
2008	Left fielder Pat Burrell or right fielder Bobby Abreu?	Burrell	No	2
2008	Will Ken Griffey Jr. play in 2009? If so, where?	Yes	Yes	1
2009	If money and length of contract were no object, which outfielder would you sign: Holliday or Bay?	Holliday	Yes	2
2009	Where will John Lackey, the top starting pitcher on the market, sign in the winter?	Yankees	No	1
2009	Which free agent do you most expect to be back with the Yankees in 2010 -- Pettitte, Damon or Matsui?	Pettitte	Yes	1
2009	Where will Aroldis Chapman sign? For how many years and how much money?	Yankees	No	1
2009	Which player is most likely to be traded during the winter -- Roy Halladay, Adrian Gonzalez or Felix Hernandez?	Gonzalez	No	1
2009	Which could pay the most dividends on a one-year contract: J. Smoltz, R. Johnson, K. Griffey Jr., P. Martinez or J. Thome?	Smoltz	No	2
2010	Which team will sign Cliff Lee?	Yankees	No	1
2010	If you had an outfield opening, which guy would you take: Carl Crawford or Jayson Werth?	Crawford	No	2
2010	Do you think the Royals will trade Zack Greinke this winter?	No	No	1
2010	Which soon-to-be free agent do you think is more likely to get traded this offseason: Prince Fielder or Adrian Gonzalez?	Fielder	No	1
2010	Which has the most left: Lance Berkman, Manny Ramirez, Vlad Guerrero, Hideki Matsui, Jim Thome or Johnny Damon?	Thome	No	2
2010	Which pitcher coming off arm problems would you be most interested in taking a shot on? Webb, Young, Bedard or Francis?	Webb	No	2
2010	Assuming Jeter signs with the Yankees, what will he get in years and dollars?	3/$55 million	Yes	1
2011	Who has the best chance of rebounding next season? Dunn, Crawford or Werth.	Crawford	No	2
2011	Which 2011 September-collapse team has a better chance of making the playoffs next year: Boston or Atlanta?	Boston	No	2

The last column splits the predictions into two categories: 1 for predicting player movement; 2 for predicting player performance. We'll get to that later.

So that gives us one prediction (the Bonds prediction from 2007) that you could argue was right or you could argue was wrong. Of the other 45, there are 21 correct predictions and 24 incorrect predictions. Worse than coin flips!

(Quick aside: this exercise may remind you of Ben Lindbergh's piece last month on predictions that managers, executives, and players made for themselves before the season. It is similar, with two significant differences that I believe advance the discussion. One is that the predictions Ben collected were made publicly and on the record, so the predictors had incentive to lie, spin, or boast. The other is that Ben's piece had zero tables and my piece has two tables.)

Except that most of these aren’t either/or questions; there’s not really a one-in-two chance of picking which team will sign Aroldis Chapman, when there are 30 potential answers. There’s not really a one-in-30 chance, either, as not every team was linked to Chapman, had scouted Chapman, etc. So it’s a bit trickier to compare the executives to random chance.

But if we split these up into categories by how many options there are, then we get a bit more insight:

Possible answers	#	Correct rate
2	21	52%
3, 4, 5, or 6	10	30%

The remaining 15 have an unknown number of true options. But of the questions that have a finite number of possible answers, the executives’ success rate is just a shade better than random chance: 52 percent rather than 50 percent when there were two possible answers; 30 percent instead of the 26 percent random expectation on the three-to-six-answer questions.

Surprisingly, to me, the executives don’t seem to do any better or worse when they’re predicting player performance than when they’re predicting transactions. They were 47 percent successful when asked which player or team would be best; they were 46 percent successful when asked where a player would sign, whether he would be traded, of it he would return the next season.

Obviously I’m going to include a few of the most excellent excerpts. Three:

• "I saw (Kaz) Matsui four years ago, and I thought he was a better player than Ichiro," said an AL GM. (2003)
• "If the Yankees were able to lock Lackey into a rotation with Sabathia and Burnett, it could be the start of another dynasty," a National League assistant GM said. (2009)
• Which player is more difficult to trade -- Cubs outfielder Milton Bradley, because of his reputation and other baggage, or Blue Jays outfielder Vernon Wells, because of his contract? Responses: Wells 20, Bradley 0. (2009. Bradley was traded that offseason in a bad-contract swap. Wells was traded for value a year later.)

***

I think there are two possible ways of explaining this low success rate. One is based on the work of Philip Tetlock, who spent 20 years studying thousands and thousands of pundit predictions. He concluded that expert pundits are barely more predictive than random chance. 

First, as the skeptics warned, when hordes of pundits are jostling for the limelight, many are tempted to claim that they know more than they do. Boom and doom pundits are the most reliable over-claimers.

There's a difference here, in that the "experts" we're evaluating aren't in the limelight and have no obvious incentive to over-claim. But it is possible that, because they self-identify as experts, they naturally view and present themselves as more certain than they should. It might also be the case that it is more satisfying to have an interesting opinion (Kaz Matsui > Ichiro; Alex Rodriguez most likely to sign with the Marlins; Melky Cabrera to sign for one year and $2 million) than a conservative but accurate opinion. 

The other is that baseball falls under the efficient-market hypothesis: 

In finance, the efficient-market hypothesis (EMH) asserts that financial markets are "informationally efficient". In consequence of this, one cannot consistently achieve returns in excess of average market returns on a risk-adjusted basis, given the information available at the time the investment is made.

Crasnick is asking these questions specifically because they are perceived, based on publicly available information, to be difficult to answer with much confidence. He is asking GMs because they are perceived to have more than the publicly available information. But it might just be the case that, when it comes to evaluating major-league players and predicting offseason moves, most of the relevant data is already publicly available and priced into our expectations; that the relevant data that isn't publicly available is not shared between teams; and that, when it is shared between teams, it tends to trickle out to the public, via excellent reporters like Crasnick. In other words: for the purposes of these two types of questions, there's not a significant advantage to being in the room. 

***

Like I said at the beginning, of course GMs are smart. A winning organization depends on so many skills in the front office: contract negotiating, strategizing, scouting for weaknesses, managing risk, motivating players, motivating non-players, generating revenue, working with other teams' front offices, keeping players healthy, developing young players, collecting information, measuring and assessing performance accurately, approving timecards. GMs are, presumably, smart at those things, and many others.

But predicting baseball might just be impossible, and a team that puts too much faith in its own predictions might be falling into a trap. So don't feel bad if you're bad at it. It's not your fault. It's baseball's fault.

Sam Miller is an author of Baseball Prospectus. Follow @SamMillerBB
Click here to see Sam's other articles. You can contact Sam by clicking here