January 28, 2013
Fact or Fiction: The Verducci Effect
Last week, Sports Illustrated writer and Jason Parks man-crush Tom Verducci put out his annual column warning about a specific type of player: A young pitcher (25 or younger) who saw a significant increase in his workload in the previous season over the season before that (defined as an increase of at least 30 innings, including postseason and minor-league work). Verducci claims that this sort of pitcher is in danger of either a significant injury and/or a performance decline in 2013 because his 2012 was much busier than his 2011. It's a proposition that's become known as the Verducci Effect.
Let's start with the positive: This is a perfectly reasonable hypothesis. It makes sense intuitively. If you are training to run a marathon, you don't go from maxing out at one mile to maxing out at 10 in a week. You have to build up your endurance along the way. Rick Peterson, who has been a pitching coach for several teams, has given the Verducci Effect his imprimatur. There have been several cases fitting the criteria in which pitchers have gone on to suffer major injuries or major declines in performance. We have face validity through imperfect analogy, expert testimony, and case examples. It's not silly to believe that Verducci's hypothesis could be true.
But, as someone who has taught both research methods and statistics, I see a few problems. None of those forms of evidence (analogy, testimony, or example) are legal tender as proof of anything. That's not how science works. If it were, Jenny McCarthy would be Surgeon General. Verducci himself states that the “25 or under, 30 or more” rule of thumb is "is not a scientific, predictive system." He's right. There are plenty of problems with how Verducci justifies his hypothesis. It might be minimally decent science to add the disclaimer, but it's irresponsible journalism. If you don't have good evidence for a theory, why write an article at a highly visible outlet and name names?
What's even more frustrating about the Verducci Effect is that it's presented in the form of a testable hypothesis. And there have been people who have tested it. David Gassko at The Hardball Times looked at the group-level stats of Verducci identified pitchers with another group of under-25s who did not increase their workload. There was little evidence for the Verducci Effect. BP's own Derek Carty used a similar matched-pairs methodology and found comparable pitchers for each pitcher on Verducci's list, but who did not increase their workload by 30 IP or more. Again, no difference. Jeremy Greenhouse combined Gassko's approach with some regression-based analysis and found no evidence to support Verducci. J.C. Bradbury used a moderator model regression (and random effects!) to study whether the "Rule of 30" had support. Short answer: no.
So far, it's not looking good for Verducci, but it could be argued that each of the above studies had some small flaw in it. I propose to go a little bit deeper than some of these previous studies did and with somewhat more complete data. Let's give the Verducci Effect a full, impartial examination and then decide what to do about it.
Warning! Gory Mathematical Details Ahead!
Age was calculated as of April 1st (Opening Day) for all players. Innings pitched included minor-league, regular season MLB, and postseason innings. Per Verducci's criteria, I looked for all players who were 25 or younger and who had an increase in workload of more than 30 innings in the previous season.
I compared only major-league performance in the regular season and required that the pitcher have 50 major-league IP both in the year previous (when he amassed the 30 extra innings) and the "year after" in which the Verducci rule predicts doom. This allows a sample size that made it reasonably likely that I would be getting a good read on performance stats like strikeout rate, rather than making judgments on 20 MLB hitters faced in one year.
There were 75 "Verduccis" and 137 controls. To detect a medium-sized effect in a t-test (Cohen's d' = .5 for the initiated), you need a minimum of 64 per group. For a chi-square, you need about 58. With a bigger sample size, we could detect smaller and smaller differences between the groups, but there comes a point where, if the impact of being a Verducci Effect pitcher is that small, no one would bother writing a column about it. This sample size should be OK for our purposes.
I tested all of these by chi-square. Only one specific body part came in significant (fingers), but the "any arm injury" and "any injury" categories had p-values of .072 and .048, respectively. A Verducci victory? From the looks of it, those in the Verducci group really are more likely to have an injury.
Not so fast. One good reason to have an increase of more than 30 innings is to have ended your season two years ago with an injury and then come back to pitch a full season. It stands to reason that the Verducci group may include certain pitchers who have an injury history, and as such they would be more prone to injury.
To test this, I ran a logit regression with two binary predictors, Verducci group vs. control, and whether two years ago the pitcher had an injury to that body part. Results were pretty much the same. While Verducci status did not affect individual body parts, the findings around "any arm injury" were much the same, with a significance value of .088. So, there's still marginal evidence of increased arm injury risk, and it's not just previous history driving it. Verducci is vidicated!
Before you pop that champagne, hang on. I ran a t-test between the Verducci group and the control group to look at who missed more games as the result of injury. The Verducci group missed an average of 15.08 games over the course of a season, while the control group clocked in at 21.09 games. When I looked specifically at those who sustained an injury, the Verducci group missed an average of 22.17 games from those injuries while the control group missed an average of 39.04. Both differences are significant.
To say that pitchers on the Verducci list are more likely to experience an injury is correct. However, it's actually the control group that is more likely to land on the DL. In fact, 29.2 percent of controls spent time on the DL compared to 24.0 percent of the Verducci group (not significant, for the record). Perhaps we might interpret this in context. Teams probably treat their young pitchers gingerly, especially in the context of having pushed them a bit in the previous year. Maybe they are a little more careful with minor injuries and push back a start here and there. But there's no evidence that Verducci Effect pitchers are more likely to sustain serious injuries. In fact, what evidence is there runs in the opposite direction.
Even if these numbers were significant, we're talking about small absolute differences between groups. The real story is that pitchers in general tend to get hurt. If about a quarter of young pitchers spend time on the DL, then any list of young pitchers made before a season, even one drawn at random, will have several victims of injury. It's not being under 25 and overworked. It's being a pitcher that's the problem.
I even looked at whether there would be an association between Verducci status and whether there would be a one-run spike in ERA. Verducci Effect pitchers (15.9 percent) were actually a little less likely than controls (19.4 percent, difference not significant) to have that happen. Again, there will be guys who fall apart in any list of pitchers. It doesn't seem to be associated with being young and extending yourself in the previous year.
This one wasn't even, "Well, it's a complicated story and I could see how at a glance, you might make that mistake..." Whether the analyses were simple or a little more sophisticated, there is very little evidence that Verducci status has any effect on performance.
Does it make a difference if you are big or little?
This is also testable. The number of innings criteria is quick to calculate. For size, I used the player's listed height and weight to generate a
I isolated only the Verducci pitchers, since they are the only ones affected, and tested whether
Simmah Down Now
The Verducci Effect is a case of speculation mixed with a really poor understanding of the scientific method, and that is a dangerous combination. It gives the illusion of knowledge, and that's more dangerous than simply not knowing something. It's tempting to want to grab onto the Verducci explanation, especially when a young pitcher with so much promise suffers such a large setback, because a wrong explanation feels better than no explanation.
With that said, my findings are not a license for teams to go out and Mark Prior-ize their pitchers. You can blow out a young (or old) arm from overuse. It's just that the Verducci formulation isn't a good guide to figure out who is at risk. I'll suggest that the reason has to do with how Verducci defines his sample.
Consider the ways in which a pitcher can pile on extra innings, and what effects we might expect that pathway to have for him with respect to injury risk and performance.
I could keep going. What should be apparent is that Verducci Effect pitchers are a very mixed bunch. There are plenty of ways to get on the list, and different ones might bode well, poorly, or neutrally for a pitcher's future. The problem of the Verducci Effect formulation is that the sample is far too heterogeneous to expect coherent effects out of it. Maybe the real frontier here is in breaking players down into sub-groups based on how they got onto that list to begin with. It's much more complex, doesn't fit nicely onto the page of a magazine, and it's the way that real research is done.
So here's to hoping that I don't have to resurrect this column a year from now. It's time to just admit that the Verducci Effect doesn't hold water and move on.