BP Unfiltered

How good a job have we done at using the data we have to evaluate umpires? I've seen some metrics like overall percentage of correct calls, but that doesn't take into account that it's the close calls that are difficult. It's sufficient to compare umpires to each other, but not to establish an absolute level of performance.

The reason I ask is that I'd wonder whether something more like what they have in tennis would work, where the machines indicate the result for close calls and people continue to handle the routine calls and can overrule the machine if obviously wrong. But to know if that makes sense, we'd have to have an idea of the error rates for the machines and the umpires for close calls.

@Mike

Yeah something like that. Kinda science fiction like though.

Has there been any discussion of the height of the strike zone? Specifically, how well does pitch/fx (or any technological solution) deal with the variance in batter heights and stances? Seems whenever I watch the FoxTrax box it's always the same height, regardless of who is batting.

While I am pretty confident that we can use a device to measure weather a pitch crossed the plate or not, I'm not sure we are as good at determining whether it was high or low. How is that calibrated from batter to batter, and how much judgment/error is involved in the process?

I'm guessing that most umps already know what areas they struggle with -- or at least vary from their peers. The hard part, I have to believe, is translating that knowledge in to what they're seeing on the field.

It would seem we could pretty easily provide umpires real-time feedback via a simple LED indicator built in to the device they currently use to track outs and pitch counts.

It would be used solely for their own edification, helping them to correct any systematic biases. As with all professionals, they will resist feedback systems which are (seemingly) punitive rather than instructive.

I think a lot of good could be done by simply taking steps to make pitchf/x a tool which helps them do their jobs better rather than a way to show fans when they screwed up.

(Direct reply not working, so bear with me here.)

"Perfection is not an attainable goal with any ball-strike detection system. We're going to fall short whether we use human umpires or technology."

To see how counter-productive this statement is, consider the following: "Perfection is not an attainable goal with any forecasting system. We're going to fall short whether we use statistical methods or astrology."

Perfectly true statements, but they add (at best) nothing to a discussion of the relative methods of statistical forecasting and astrology.

If there is an argument in favor of human umpires, it does not begin this way.

"The question is how close we can get to a perfect ball-strike detection system, at reasonable cost, without too much disruption to the game."

No. The question is how much better than human umpires can we get, at reasonable cost, without too much disruption to the game. We measure success by how much better off we are, not by how close to perfection we can get.

@Fantasyking (and reply): The question of strike zone adjustment to batter height and stance is usually raised as a barrier to automated ball and strike calls, but I never hear anyone mentioning the fact that there's very little evidence about how well (or poorly) human umpires do this. The fuzziness in the top boundary of the strike zone is well known, and I see no evidence (with the possible exception of Rickey Henderson) that umpires ever take into account a hitter's normal batting stance in their calls.

Finally, a question: I'm clearly not understanding why calibration is considered such a problem. Any three cameras can triangulate a position. Home plate doesn't move. How hard is it to sight on reference objects placed at the corners of the plate, at known heights, just prior to game time? That would correct for any changes in camera orientation since the last calibration. If the positional assessment is accurate to an inch or two, but the orientation is a concern, wouldn't that fix the problem? What am I missing here?

I certainly agree that everything needs to be compared against how umpires really perform in practice. We're on the same page there. Can't wait to see your research.

I do still see a difference between "how much better than X is Y for our purposes?" and "how much closer to perfection is X than Y?". The ranking of X and Y is the same for both questions, but the importance of the absolute difference (or the perception of it) is not. People tend to rate the importance of a difference as a percent of the scale. If X falls 82% shy of perfection and Y falls 79% shy, that looks like a pretty small difference -- until you realize that you're talking about slugging average, and perfection is a 4.000 ...