January 28, 2013

Baseball Therapy

Fact or Fiction: The Verducci Effect

by Russell A. Carleton

Last week, Sports Illustrated writer and Jason Parks man-crush Tom Verducci put out his annual column warning about a specific type of player: A young pitcher (25 or younger) who saw a significant increase in his workload in the previous season over the season before that (defined as an increase of at least 30 innings, including postseason and minor-league work). Verducci claims that this sort of pitcher is in danger of either a significant injury and/or a performance decline in 2013 because his 2012 was much busier than his 2011. It's a proposition that's become known as the Verducci Effect.

Let's start with the positive: This is a perfectly reasonable hypothesis. It makes sense intuitively. If you are training to run a marathon, you don't go from maxing out at one mile to maxing out at 10 in a week. You have to build up your endurance along the way. Rick Peterson, who has been a pitching coach for several teams, has given the Verducci Effect his imprimatur. There have been several cases fitting the criteria in which pitchers have gone on to suffer major injuries or major declines in performance. We have face validity through imperfect analogy, expert testimony, and case examples. It's not silly to believe that Verducci's hypothesis could be true.

But, as someone who has taught both research methods and statistics, I see a few problems. None of those forms of evidence (analogy, testimony, or example) are legal tender as proof of anything. That's not how science works. If it were, Jenny McCarthy would be Surgeon General. Verducci himself states that the “25 or under, 30 or more” rule of thumb is "is not a scientific, predictive system." He's right. There are plenty of problems with how Verducci justifies his hypothesis. It might be minimally decent science to add the disclaimer, but it's irresponsible journalism. If you don't have good evidence for a theory, why write an article at a highly visible outlet and name names?

What's even more frustrating about the Verducci Effect is that it's presented in the form of a testable hypothesis. And there have been people who have tested it. David Gassko at The Hardball Times looked at the group-level stats of Verducci identified pitchers with another group of under-25s who did not increase their workload. There was little evidence for the Verducci Effect. BP's own Derek Carty used a similar matched-pairs methodology and found comparable pitchers for each pitcher on Verducci's list, but who did not increase their workload by 30 IP or more. Again, no difference. Jeremy Greenhouse combined Gassko's approach with some regression-based analysis and found no evidence to support Verducci. J.C. Bradbury used a moderator model regression (and random effects!) to study whether the "Rule of 30" had support. Short answer: no.

So far, it's not looking good for Verducci, but it could be argued that each of the above studies had some small flaw in it. I propose to go a little bit deeper than some of these previous studies did and with somewhat more complete data. Let's give the Verducci Effect a full, impartial examination and then decide what to do about it.

Warning! Gory Mathematical Details Ahead!
I pulled together the following databases:

• Minor-league pitching data from 2006-2012.
• Historical regular and postseason pitching data.
• The BP injury database, as compiled by Corey Dawkins and Dan Turkenkopf. I am indebted to Ryan Lind for his work in converting the file for me.

Age was calculated as of April 1st (Opening Day) for all players. Innings pitched included minor-league, regular season MLB, and postseason innings. Per Verducci's criteria, I looked for all players who were 25 or younger and who had an increase in workload of more than 30 innings in the previous season.

Methodological Issues
There are some methodological issues to resolve. As a comparison group to the Verducci group, I selected all pitchers who were 25 or younger but did not have a 30 IP jump, but even this proved to have some hidden biases. Pitchers in the Verducci group were much more likely to work primarily as starters. It's not surprising, because it's a lot easier to get 30 extra innings when the baseline usage is 170 than it is when the baseline usage is 50. To that end, I considered only pitchers who started 80 percent of their games. (Later, I took off that filter. It didn't make a difference.)

I compared only major-league performance in the regular season and required that the pitcher have 50 major-league IP both in the year previous (when he amassed the 30 extra innings) and the "year after" in which the Verducci rule predicts doom. This allows a sample size that made it reasonably likely that I would be getting a good read on performance stats like strikeout rate, rather than making judgments on 20 MLB hitters faced in one year.

There were 75 "Verduccis" and 137 controls. To detect a medium-sized effect in a t-test (Cohen's d' = .5 for the initiated), you need a minimum of 64 per group. For a chi-square, you need about 58. With a bigger sample size, we could detect smaller and smaller differences between the groups, but there comes a point where, if the impact of being a Verducci Effect pitcher is that small, no one would bother writing a column about it. This sample size should be OK for our purposes.

Injuries
The BP Injury database is nice in that it goes back to 2002, lists injuries sustained in the minors, gives details on what happened, and tells how many games the player missed. I ran a series of codes for whether or not a pitcher suffered any injury at all (even a day-to-day one), whether he spent time on the DL, and whether he sustained an injury to several specific parts of the arm (e.g., shoulder, elbow, wrist), or to any part of his arm in general. I also calculated the total number of games a pitcher missed due to his injuries.

The results:

Body Part	Control	Verducci
Shoulder	12.4%	17.3%
Elbow	10.2%	13.3%
Forearm	6.6%	4.0%
Hand	1.5%	5.3%
Fingers	5.1%	12.0%
Thumb	0.7%	2.7%
Upper Arm	1.5%	0.0%
Wrist	1.5%	0.0%
Any Arm Injury	32.8%	45.3%
Any Injury	54.0%	68.0%

I tested all of these by chi-square. Only one specific body part came in significant (fingers), but the "any arm injury" and "any injury" categories had p-values of .072 and .048, respectively. A Verducci victory? From the looks of it, those in the Verducci group really are more likely to have an injury.

Not so fast. One good reason to have an increase of more than 30 innings is to have ended your season two years ago with an injury and then come back to pitch a full season. It stands to reason that the Verducci group may include certain pitchers who have an injury history, and as such they would be more prone to injury.

To test this, I ran a logit regression with two binary predictors, Verducci group vs. control, and whether two years ago the pitcher had an injury to that body part. Results were pretty much the same. While Verducci status did not affect individual body parts, the findings around "any arm injury" were much the same, with a significance value of .088. So, there's still marginal evidence of increased arm injury risk, and it's not just previous history driving it. Verducci is vidicated!

Before you pop that champagne, hang on. I ran a t-test between the Verducci group and the control group to look at who missed more games as the result of injury. The Verducci group missed an average of 15.08 games over the course of a season, while the control group clocked in at 21.09 games. When I looked specifically at those who sustained an injury, the Verducci group missed an average of 22.17 games from those injuries while the control group missed an average of 39.04. Both differences are significant.

To say that pitchers on the Verducci list are more likely to experience an injury is correct. However, it's actually the control group that is more likely to land on the DL. In fact, 29.2 percent of controls spent time on the DL compared to 24.0 percent of the Verducci group (not significant, for the record). Perhaps we might interpret this in context. Teams probably treat their young pitchers gingerly, especially in the context of having pushed them a bit in the previous year. Maybe they are a little more careful with minor injuries and push back a start here and there. But there's no evidence that Verducci Effect pitchers are more likely to sustain serious injuries. In fact, what evidence is there runs in the opposite direction.

Even if these numbers were significant, we're talking about small absolute differences between groups. The real story is that pitchers in general tend to get hurt. If about a quarter of young pitchers spend time on the DL, then any list of young pitchers made before a season, even one drawn at random, will have several victims of injury. It's not being under 25 and overworked. It's being a pitcher that's the problem.

Performance
This one was easier than I thought it would be. Again, my control group were all pitchers 25 or younger who had not increased their workload by 30 IP in the previous year. Verducci status was not associated with differences year over year in strikeout rate (per batter faced), walk rate, home run rate, or ERA (i.e., his strikeout rate went from 12.1 percent to 13.2 percent for a difference of 1.1). This held after controlling for the previous season's number to account for possible ceiling or floor effects. I tried running predicting strikeout rate using last year's rate and Verducci status. Nothing.

I even looked at whether there would be an association between Verducci status and whether there would be a one-run spike in ERA. Verducci Effect pitchers (15.9 percent) were actually a little less likely than controls (19.4 percent, difference not significant) to have that happen. Again, there will be guys who fall apart in any list of pitchers. It doesn't seem to be associated with being young and extending yourself in the previous year.

This one wasn't even, "Well, it's a complicated story and I could see how at a glance, you might make that mistake..." Whether the analyses were simple or a little more sophisticated, there is very little evidence that Verducci status has any effect on performance.

Does it make a difference if you are big or little?
In his article, Tom argues that there two mitigating factors. One is that players who are bigger will be less likely to regress or face injuries. Bigger bodies are meant to handle bigger workloads. (This one gets thrown a lot... it makes sense, but has anyone ever done this study?) Additionally, Verducci suggests that the number of innings by which the pitcher exceeded the 30-inning threshold matters. (Guys like Chris Sale, who increased his 2012 workload by more than 120 innings, are at particularly great risk, according to Verducci.)

This is also testable. The number of innings criteria is quick to calculate. For size, I used the player's listed height and weight to generate a BMI (Body Mass Index) score. This isn't a perfect correlate of what we're getting at, but it'll do for now.

I isolated only the Verducci pitchers, since they are the only ones affected, and tested whether BMI, workload increase above 30 innings, or an interaction between the two produced some sort of significant effect either on likelihood of injuries or performance.

No.

Simmah Down Now
To all the fans of teams who had a guy or two on the Verducci list, you can calm down (for a moment... in a minute we'll talk about why all hope may still be lost). For all you fantasy owners preparing for your draft, please ignore the list.

The Verducci Effect is a case of speculation mixed with a really poor understanding of the scientific method, and that is a dangerous combination. It gives the illusion of knowledge, and that's more dangerous than simply not knowing something. It's tempting to want to grab onto the Verducci explanation, especially when a young pitcher with so much promise suffers such a large setback, because a wrong explanation feels better than no explanation.

With that said, my findings are not a license for teams to go out and Mark Prior-ize their pitchers. You can blow out a young (or old) arm from overuse. It's just that the Verducci formulation isn't a good guide to figure out who is at risk. I'll suggest that the reason has to do with how Verducci defines his sample.

Consider the ways in which a pitcher can pile on extra innings, and what effects we might expect that pathway to have for him with respect to injury risk and performance.

• He could show genuine improvement in his ability to pitch, and his manager will want him to be on the mound more. This may or may not increase his susceptibility to injury, but since the improvement is genuine, we'd expect performance to stay steady or even improve.
• He could hit a vein of good luck which is mistaken for genuine improvement and be left on the mound longer. Baseball is a game in which luck plays a bigger part than some people want to admit, and consequently, it’s often interpreted as skill. His performance will likely regress. Injury risk may or may not increase.
• He could have one of those years with his BABIP. Figure that a starter faces 800 batters in a year and roughly 600 balls in play. If his BABIP goes from .300 to .250, that's 30 extra outs. Even if he doesn't really face too many more hitters, that's a nice down payment on his extra 30 innings. You can't make the case that he's really over-extended himself, but his performance will probably regress.
• His team could make a deep playoff run, and he makes four extra starts in the postseason. Maybe he was pitching over his head to get them there. Maybe he really improved. Maybe the team got there in spite of his awful performance.
• He could suffer a season-ending injury two years ago, pitch a full season last year, and come into this season with a Verducci tag and an elbow that has already had surgery. Performance is anyone's guess, but injury risk is probably higher.
• He could have made mechanical adjustments that not only cleaned up his delivery and enable him to last longer in games, but also make him less likely to be injured. The improved mechanics might make him a better pitcher or just able to sustain the same performance deeper into a game. The effect on performance is an unknown, but he's probably less likely to suffer an injury.

I could keep going. What should be apparent is that Verducci Effect pitchers are a very mixed bunch. There are plenty of ways to get on the list, and different ones might bode well, poorly, or neutrally for a pitcher's future. The problem of the Verducci Effect formulation is that the sample is far too heterogeneous to expect coherent effects out of it. Maybe the real frontier here is in breaking players down into sub-groups based on how they got onto that list to begin with. It's much more complex, doesn't fit nicely onto the page of a magazine, and it's the way that real research is done.

So here's to hoping that I don't have to resurrect this column a year from now. It's time to just admit that the Verducci Effect doesn't hold water and move on.

Russell A. Carleton is an author of Baseball Prospectus. Follow @pizzacutter4
Click here to see Russell's other articles. You can contact Russell by clicking here