May 11, 2005

Prospectus Q&A

Mark Johnson

by Thomas Gorman

Printer-friendly

After a number of years as a software engineer and database designer for Silicon Valley technology firms, Mark Johnson jumped into sports with a consulting job for the MLB commissioner's office. He explored the statistical implications of various potential rule changes on a committee that included John Henry, owner of the Red Sox, and Rob Manfred, head of the Labor Relations Department at MLB. From that experience Johnson got hooked on applying his mathematical background to sports management and eventually ended up serving as Senior Analyst of Baseball Development for the 2004 NL Champion St. Louis Cardinals. Now back in Silicon Valley, Johnson sat down with BP during a recent San Jose Giants game to discuss his background, his experience with the Cardinals, and where he sees the the most valuable applications of sabermetrics, both now and in the future.

Baseball Prospectus: Could you begin by telling our readers a little bit about your academic background? In the baseball industry it is a bit atypical.

Mark Johnson: I was an undergraduate at Indiana University where I studied computer science and mathematics. During five years there I earned a bachelor's degree in computer science and a master's in mathematics. Immediately following that I went to Princeton University, where I earned a master's and a PhD in applied and computational mathematics. My areas of research were mostly in theoretical mathematics, but I tried to mix in as much computational work as I could into solving theoretical mathematical problems. Following that I went to the University of Minnesota for one year as a post-doctoral researcher before deciding to leave the academic world and jump into the Silicon Valley workforce.

BP: You first became involved in sports by doing some consulting for the commissioner's office, is that correct?

MJ: In a professional sense, yes. I actually have experience from my undergraduate years; I was a student manager for Bob Knight during my four years at Indiana. In fact I worked closely with, and was roommates with Lawrence Frank and Calbert Cheaney. I also, of course, worked closely with Coach Knight and some of the assistants. That was my first real exposure to organized sports, but in a professional sense my first experience was working as a consultant to the commissioner's office in the summer of 2003.

BP: What exactly were you working on for the commissioner's office?

MJ: My brother (Matthew S. Johnson, a professor of Statistics at Baruch College of the City University of New York) and I were hired as consultants to the commissioner's initiative, which is a group organized to evaluate possible changes to baseball, and changes to rules. We were brought in to add a little statistical analysis to some of the decisions that the people in the commissioner's office as well as other people within the initiative were looking to address. They had plenty of evidence to support some of these decisions but they wanted to add a statistical component to the final decision-making process. So we helped in that regard.

BP: Did you find the commissioner's office was receptive to that statistical point of view, and was any of the work you did put into action, anything fans might have seen?

MJ: They absolutely were receptive. I was working on a subcommittee consisting of Rob Manfred and John Henry, the owner of the Boston Red Sox, who as many people know is a very analytically-minded person. He was definitely very interested in some of the analytical results that we came up with, and these results were definitely weighed heavily in the final decision.

I can't really speak about the details of what we worked on, however it did have to do with some scheduling questions that they were asking, and the outcome of all those results was that nothing changed. The rules never changed, and whether or not it was the result of our work that led to those decisions is beyond my understanding.

BP: How did you end up working with the St. Louis Cardinals? Who brought you in, and what was your position supposed to entail?

MJ: After I worked for the commissioner's office I was putting in my best effort to try to create a niche of moving analytics into sports, so I spent a lot of time on the phone talking with athletic directors from universities, with commissioners from several different major collegiate conferences, speaking with the NFL commissioner's office, obviously continued speaking with Major League Baseball's commissioner's office, and started to think about how I could start talking to actual clubs. I was put in touch through some colleagues with Jeff Luhnow, who had recently been hired by the St. Louis Cardinals. Upon meeting him and talking with him about some of the ideas that I had and some of the approaches I could bring to the table, Jeff eventually hired me on in January of 2004.

The original charter for my position was to, at a really broad-stroke level, try to bring some innovative technology into baseball decision-making. So there was a combination of both my software engineering skills and database management skills as well as my mathematical skills that were used in the position, both to help collect and disseminate information from large amounts of data. So, in a sense, part of the role was mathematical and another part was software engineering and database related.

BP: How did first experiences with the Cardinals go?

MJ: Jeff was relatively new at the time that I started, so it was a learning process to understand where we would have the most effect within the organization. As you know, the baseball schedule has a lot of very distinct decision-making points throughout the season. For me, when I started in St. Louis in the beginning of February, the first real decision-making point on the horizon, in my eyes, was the June draft. Between the time I started and the beginning of June I first had to learn more about the actual process--how the scouting organization was put together--and also try to understand and collect as much information as I could about high school and collegiate players, and then to try to disseminate something from that information that could help us make better decisions on draft day, or in the months leading up to draft day.

So that was really the first objective, to understand the scouting process, and the draft itself, to try to bring in as much of my mathematical and software skills as I could in time for that draft. Beyond that, the other important deadlines that I played a little bit of a role in were the trade deadline in July as well as the minor league free agent season in October and subsequently the major league free agent season.

BP: It sounds like you were using your mathematical background in player evaluation...

MJ: In the baseball analytical world, at least the way that I look at it, I think there are three categories where statistical analyses can help decision makers. One is at the commissioner's office level, when you evaluate rules to try to understand the impact that rule changes might have on the game. The second level is looking at player evaluation, it's more of a front office decision-making process. That's putting together rosters, trying to understand the value of players to your team or the impact of certain contracts. The third level is the actual in-game strategy, what sort of strategies can you bring to the field.

My experience with the commissioner's office definitely fell into that first category, and my experience with St. Louis definitely fell into the second, which was helping the front office put together rosters, which like you said, falls into the area of player evaluation techniques.

BP: How have the Cardinals balanced various approaches to baseball operations decision-making? There have been guys like you and Jeff Luhnow internally, others on the Board of Advisors, and then there are the more traditional scouts and player development people. How does the front office balance these voices?

MJ: Each one of the decisions that I was somehow involved with were being made by different people in the organization who combined information in different ways. My objective was to basically be the "mathematical scout," and to give these decision-makers the mathematical scouting report, so to speak. In a lot of the decisions that were made last season there was a mix of scouting reports and statistical scouting reports and each person handles the combination of that information differently.

In the case of the amateur draft we have a lot of players that we don't know a lot about. We have scouts that go watch these players only a few times in the course of their careers, so in that sense the statistical information has more of an impact. It's, relatively speaking, a lot more information than we had beforehand. When you look at major league player evaluation, people have a lot of information about those players already. They've been watching them for years. They have many scouting reports on these players, they have a lot of video tape on these players. In my opinion, the statistical information is still valuable in those cases, absolutely it's still valuable, but maybe not, relatively speaking, as valuable as having additional statistical information on the larger pool of players from whom you need to make your selections. So at the amateur draft I think statistical evaluation has more impact, is the conclusion I've come to, rather than at the major league level.

BP: Without getting into any areas that are covered by your non-disclosure agreement, is the way that a team like the Cardinals goes about evaluating players radically different from the statistical evaluation that you would see at Baseball Prospectus or at other venues that do that type of analysis? Is there stuff that's just a generation ahead, or is it relatively similar?

MJ: I would say it's more of the latter. There is a real limitation in this business that lies in the granularity of data that's available. I think that most organizations that are doing any sorts of statistical analysis, whether that be Baseball Prospectus or the St. Louis Cardinals, is that we are all working with fundamentally the same core data, which have certain limitations to it. I think if you put a few smart people on top of that data that they are going to reach some pretty similar conclusions. Definitely with different approaches, but some similar conclusions come out the other end.

I think the real difference-maker is in establishing ways to get better granularity of data at every level. For example, at the major league level defense is the next chapter that people want to write and frankly, with all the data that I've seen available to clubs like the St. Louis Cardinals, the data aren't really at a granularity that's fine enough to make any real solid conclusions. At the other end of the spectrum, at the amateur draft, one would have a really hard time finding significant amounts of, let's say, high school statistical information. There's still a question of whether that information would be useful, but the story there is still that the data is relatively difficult to find.

I've digressed. To get back to your first question: I think that most organizations are probably doing slight variations of the same thing. I think the real difference maker will be in collecting more data. Certain organizations may be investing money in finding a greater granularity of data, I don't know, but if they are they're probably at the next level.

BP: Especially as it regards major league free agents, do you find that teams are spending much energy or time trying to figure out "the market"--the finances, the competition, what a guy is worth, quantifying value in dollar values? Every off-season we hear about the deals that "blew the market" with this offer for that guy that set up new comps for those players down the road. Is there time spent on this sort of analysis?

MJ: Clubs definitely do a significant amount of research on the market conditions before free agent offers are made and during the preparation for arbitration. I haven't really seen much compelling work in the publicly available literature (Ed note: Check out the work done by Baseball Prospectus' Ben Murphy, following the work of the late Doug Pappas), and I am sure that some clubs would stand to gain from understanding this market a bit better, but yes, clubs definitely do some work at evaluating the market.

BP: The original reason you came into the organization was to build a database for the team. Is that just a collection of statistics or is it more than that? What type of data warehousing does a major league team do to help them make decisions?

MJ: The most crucial pieces of information, as far as the front office is concerned, are just being able to collect scouting reports and being able to find information from scouting reports. There is a process that begins out at the ballyard, with a scout writing down notes about a specific player. That information needs to find its way back to the front office. Now in St. Louis, we used some third-party software to help in that process, but at the end, before decisions are made, we sometimes need to collect all of our information and put it into a sensible and accessible presentation before the decisions are finally made. So in addition to helping with constructing player evaluation models, I was also involved in collecting some information from these scouting reports and putting that into final reports that the decision makers would use.

BP: As someone with a thorough professional background in mathematics and analytical-approaches to decision-making, how do you view the current state of publicly-available baseball analysis? Are there strengths and weaknesses to the body of work that's out there that jump out to someone like yourself who has used the same sorts of approaches with other problems in different fields?

MJ: This is a great question. To be completely honest, when I first started doing a background study on publicly available baseball research papers, I grew pretty frustrated pretty fast. I quickly convinced myself that I would need to reinvent the wheel, so to speak, in any area where real decisions were going to be made as the result of my work. I encountered cases where people who had developed "predictive" models never tested how well their models actually predicted; I encountered cases where the contents of the paper contradicted the conclusions that were stated, and on and on. I really do believe that in some areas, it is more efficient to start from scratch than to try to separate the signal from the noise in the literature. Don't get me wrong, there is some quality work out there, so you have to know where to look.

BP: As someone who has worked on analytical approaches to baseball decision-making from both the outside and inside of a major league club, where do you see the next frontiers in sabermetrics? What questions do you think are most desperately in need of answers? Are there approaches that have been heretofore ignored? Is there anything that you yourself are working on independently?

MJ: There are two parts to this answer, in my mind. On the one hand, as I mentioned earlier, I think a next frontier lies more in the collection of finer-granularity data of major league baseball events as well as more thorough injury databases. Beyond better major league data, there definitely is a need for better collegiate and high school, as well as international, information as well.

On the other hand, I think that there really is great possibility for bringing some cutting-edge machine learning technologies from the academic world into building more robust predictive tools in baseball. The academicians continue to push the envelope with machine learning in many scientific applications, so staying on top of these technologies...or working with people who are on top of them, should shine some new light on separating the random components of baseball observations from the informative.

As far as what I am doing now, I have started assembling a consortium of a few statistics and applied math professors who I know from my graduate school days. Several of us will be meeting in a few weeks to talk about where we want to take this work, but I do believe sports and academics can be brought together in a practical way.

BP: What do you mean by "machine-learning technologies?"

MJ: Machine-learning refers to computational approaches toward finding patterns and structure in data. Voice recognition, as used in phone menu systems, or character recognition, as used when scanning entire libraries into a computer database are built using machine learning methods.

In baseball analysis, once we establish how to measure a player's contribution towards a team's success, we can start to think about the next step of how to predict it. In a simplified view, we take every piece of information about every person who's played the game and throw it into a big pot. We then let a machine learning system loose in the pot, sifting through all the data toward finding the information that is most useful in predicting future success, which is what statistical player evaluation techniques are aiming for. Neural nets, decision trees, support vector machines...those are some of the machine-learning buzzwords people might have heard of.

BP: You left the Cardinals at the end of the season and returned to Silicon Valley, where you've gone back to doing the sorts of tech work that you did before you got into baseball. Do you see yourself getting back into sports, perhaps going back to work for another baseball club in the future, or is that it for you?

MJ: I definitely see myself being involved with sports in some capacity in the future. I continue to talk with executives in baseball as well as other sports. I currently have returned to Silicon Valley; until November, I'm sitting out a one year non-compete clause that was a part of my employment in St. Louis. I do hope at some point in the future to be involved with a baseball organization again.