December 4, 2015

Prospectus Feature

Good Scouts Look Away

by Ezra Wise

As a reader of this site, you’re inevitably well aware that statistics, biographies, analysis, scouting reports, rankings, and video—whether for pro players or amateurs—are but a mouseclick away. There’s a whole lot of baseball information out there, and while it’s plentiful and easily accessible, increased volume necessarily comes with more noise to filter through. The Baseball Operations sub-department most affected by the information boom of the past decade is of course Analytics, but the degree to which increased information availability has affected Scouting goes overlooked. As we’ll explore in this piece, scouts are tasked with filtering and, in many cases, flatly excluding large batches of available information in the interest of maintaining the originality and validity of their evaluations.

Scouting is a necessarily subjective exercise, but one is nonetheless obligated to transpose those subjective inputs onto the decidedly more objective palate that is the 2-8 scale. Consequently, a formalized methodology for regulating the information sources that are deemed admissible, versus those deemed inadmissible, is necessary to ensure that all scouts within a department are arriving at their grades via the same collection of inputs, and that extraneous information sources don’t corrupt the evaluation process.

Player tools, makeup, and physical traits are the primary sanctioned inputs, while unsanctioned inputs are typically those that reflect the opinions of others; those that convey only circumstantial evidence about the player; and those that might lead to anchoring on quantitative or pseudo-quantitative information. It’s no surprise that these tenets of scouting are so revered. After all, they appeared in the Old Testament and were recited famously by Samuel L. Jackson in Quentin Tarantino’s 1994 masterpiece, Pulp Fiction.

“The path of the righteous talent evaluator is beset on all sides by the inequities of the industry consensus and the tyranny of misleading statistics. Blessed is he, who in the name of sound talent evaluation methods and the maintenance of originality, bases his grades solely on the tools, physical projection, and makeup he perceives with his eyes and ears, for he is truly a man of conviction and the finder of unrecognized value. And he will eschew, with great discipline and discerning character, those inputs that would threaten to corrupt and diminish the validity of his reports. And they will know his process is sound when he enters his reports and pref lists into the organization’s scouting database!”

My initial response was, “what?” One is not supposed to base tool grades and OFPs on all available information sources? After a bit of reflection, however, the benefits of information exclusion came into further focus once I fully recognized that scouting input is but one important piece of a much larger puzzle. Most of the information that a scout should exclude is already available for a GM to consider; in many cases, there are specialists elsewhere in the front office who have expertise in a certain type of data surpassing the expertise of the scout. The value of a scout's information is maximized when it’s conceived independently, without attempting to account for the other pieces of the puzzle.

The Siren Song of the Industry Consensus
While it’s impossible to exist in a vacuum, a well-conceived report eschews the perceived consensus opinion of the industry. It’s undoubtedly safer and easier to fall in line with the crowd, but such an approach defies the overriding objective of scouting—to deliver original, unbiased, well-supported reports based solely on the opinions of the evaluator. Consider an amateur ranked as the best player in a given area by Perfect Game, Baseball America, Prep Baseball Report, etc. Associate and part-time scouts rave about him; competing scouts drool over him; and college coaches all scramble to renew their NAMBLA memberships. Essentially, the consensus opinion of the player is that he should be atop every area scout’s pref list. However, if after seeing the hypothetical player a few times, a given area scout sees a Role 4 player where others see a Role 6 or 7 he’s faced with a dilemma. Go with the consensus? Input a hedged/regressed grade of 5? Or put down the 4 his eyes are pleading with him to input? The answer is unequivocally the latter, as the consensus opinion is merely an aggregation of other people’s opinions. While gathering information on the industry’s consensus opinion of a player is important for the purposes of estimating how other clubs perceive the player’s value, this information is not admissible when building grades and explaining those grades in the written portion of a report.

Circumstantial Evidence
A given player’s draft position, signing bonus, salary, trade history, depth chart placement, actual role, current level, and award résumé suggest information about his tools, skills, and value, but they are irrelevant in a scouting context. A scout’s job is to see through the noise and offer competing perspectives when they’re warranted. Fulfilling this directive is impossible if an individual is a slave to circumstantial evidence. For example, a recent draftee’s signing bonus is a reflection of one club’s opinion of the player’s value at a particular moment in time—not some immutable truth , which is ultimately what a scout is trying to get at. A bonus baby should theoretically be given the same consideration as a player who signed as a minor-league free agent. If a player shows the tools, body and makeup to warrant a particular grade, then he needs to receive that grade whether he was taken in the first round or signed off the scrap heap.

A similar logic defines the inadmissibility of current role and level information. Consider the no. 3 starter on a low second-division club. Just because he’s occupying the no. 3 spot in a major-league rotation does not necessarily mean that he’s a no. 3 starter in a Platonic sense or that he’s even worthy of a big-league job. In fact, it’s very likely he would be occupying a much lower role on a marginally better team—but accounting for such situation-dependent information does not fall within the purview of the scout’s job responsibilities and improperly doing so necessarily diminishes his value to the organization.

Further, a player’s present level should not be treated as an indicator of his present value. Scouting inputs are intended to be context-independent and should thus theoretically transcend the various levels of organized play. In general, talent is allocated very efficiently in the professional ranks, meaning that if there is a player capable of providing present value on an major-league roster, it’s likely that he’s either already on a major-league roster or on the periphery of one (with the exception of players being held back because of development or service-time reasons). That being said, talent allocation is not perfect.

Consider 2015 Rule 5 draftee, Odubel Herrera, who not only managed to stick on the Phillies’ roster but also posted the best performance on the club during the 2015 campaign. Essentially, the Phillies saw a present Role 4-caliber or better player who was being treated by the Rangers as if his optimal present Role was that of a high minor leaguer, or what would be referred to as a Role 1,2, or 3, depending on the particulars of the Role system in use. If the evaluators who ended up pushing for Herrera were anchored on his present level and/or got caught up speculating about the reasons that might account for Herrera’s placement in Double-A instead of the Big Club, it’s unlikely that Herrera would have come to be a Phillie.

Statistics
There isn’t universal agreement in the scouting community about how stats should be treated when building evaluations. Some clubs encourage their scouts to consult statistics, while others treat statistics as an inadmissible information source. The primary issue with invoking statistical information in a scouting context is that, like navigating circumstantial evidence, it necessarily diminishes one’s ability to offer an original opinion of a player—especially if said opinion strongly disagrees with what’s suggested by his historical statistical performance. In other words, if scouts were beholden to stats, they’d be less inclined to go out on a limb and assert that a player’s true talent has changed in some way.

While a wholly stats-driven projection system is necessarily beholden to past statistical performance, a scout is not. Thus, he’s capable of picking up on sudden and discernible changes in true talent or expected performance in a new role (e.g. starter to reliever) much quicker than his automated counterparts are. This, of course, is one of the many ways scouts provide value to their respective organizations. It’s undoubtedly difficult to go against the grain and assert opinions that conflict with past statistical performance, but doing so is sanctioned and often encouraged so that the most realistic assessment of a player can be constructed when in-office personnel are building a complete picture of the player by aggregating all of the diverse information sources they have access to.

Ideal Scouting Conditions
Imagine scouting a game at an unknown level of play that features nameless, history-less players. While this scenario is realistically impossible to create, since the game of baseball exists to please fans who construct their fandom primarily on contextual knowledge of players and their unique histories (both biographical and statistical), the conditions outlined above would allow for the most valuable, actionable scouting insights to be achieved. Without consensus opinion, circumstantial information, and past statistical performance to anchor on, the entirety of one’s scouting faculties would be devoted to evaluating only the inputs observable on the field, thus producing reports almost completely devoid of bias. Even though it’s impossible to exist in the information vacuum outlined above, approaching scouting with context-independence in mind is nonetheless incredibly useful as a guiding framework. Take a stand, show conviction, and report what you see—not what you’re supposed to see. Happy Scouting!

Ezra Wise is an author of Baseball Prospectus. Follow @EzDW24
Click here to see Ezra's other articles. You can contact Ezra by clicking here