A neural-net Hall of Fame prediction method
"Predicting Baseball Hall of Fame Membership using a Radial Basis Function Network," by Lloyd Smith and James Downey
This JQAS article, from the most recent issue, is a new system to empirically predict who is and who is not in the Hall of Fame. But it's not a series of formulas -- it's an algorithm called a "radial basis function network," which is a type of neural net. I don't know much about this kind of thing, but it's called a "machine learning approach" because the algorithm figures out the algorithm, as it were.
The advantage, it seems to me, is that you don't have to figure out an algorithm yourself. But the disadvantage is that you have no idea what the "real" qualifications for the HOF are -- the neural net spits out a probability for each player, but you have no idea were that came from. As the authors note,
"A disadvantage of the approach described here is that the neural network model is opaque -- it is impossible to understand, with any degree of confidence, why the model fails to classify a player such as Lou Brock as a Hall of Fame member."
That, perhaps, is an unfortunate choice of example. Obviously, Brock is in the Hall of Fame because of his stolen bases. But the authors didn't feed steals into the model. Indeed, the words "stolen base" don't appear anywhere in the paper!
What *does* the model include? For pitchers, it considers: wins, saves, ERA, winning percentage, win shares, and number of times selected to the All Star Game. And it seems to do a reasonable job distinguishing HOFers from non. For pitchers retiring between 1950 and 2002, it makes only six errors -- it mistakenly calls Billy Pierce, Lee Smith, and John Wetteland Hall of Famers; but omits Fergie Jenkins, Hoyt Wilhelm, and Dennis Eckersley.
For hitters, the model includes: hits, HR, OPS, WS, and again All-Star selections. This time the algorithm misidentifies 13 players (Rice was listed as an error, but now is not). The incorrect selections are: McGwire, Dawson, Garvey, Baines, Santo, and Parker. The incorrect omissions are Brock, Appling, Yount, Kiner, Boudreau, Campanella, and Jackie Robinson.
Many of the errors are understandable; McGwire, Campanella, and Robinson, for instance, whose status is heavily influenced by factors other than their statistics. But a couple of the mistakes arise from the choice of data; Brock, of course, but also Robin Yount, who winds up misclassified because he had only three all-star selections -- by far the lowest of any HOFer in the 1950-2002 era. (The next lowest was 6, by Willie McCovey and Billy Williams. And all of the missing HOFers that the model failed to predict had on the list had 8 or fewer.)
The authors defend the use of All-Star selections on the basis that it's a proxy for position played; that's somewhat reasonable, and I guess that's why it somewhat works.
Anyway: is this method better than others, most notably Bill James' algorithms? Strangely, although the authors cite both of James' methods, they don't compare them to their own. My guess is that Bill's methods are probably at least as accurate as the ones in this paper. And Bill's have the advantage that we actually learn something from them -- they help us figure out what it takes to get into the Hall of Fame. The method in this paper, while perhaps being objective, accurate, and complex, doesn't tell us anything except its predictions, and so we don't learn very much about baseball from it.
P.S. The paper assumes that all sabermetrics comes from SABR. This is, of course, not true.