Two nice statistical analogies

I'm always trying to find good analogies to help explain statistical topics. Here's a couple of good ones I've come across lately, that I'll add to the working list I keep in my brain.


1.

Here's Paul Bruno, explaining why r-squared is not necessarily a good indicator of whether or not something is actually important in real life:

"Consider 'access to breathable oxygen'. If you crunched the numbers, you would likely find that access to breathable oxygen accounts for very little – if any – of the variation in students' tests scores. This is because all students have roughly similar access to breathable oxygen. If all students have the same access to breathable oxygen, then access to breathable oxygen cannot 'explain' or 'account for' the differences in their test scores.

"Does this mean that access to breathable oxygen is unimportant for test scores? Obviously not. On the contrary: access to breathable oxygen is very important for kids’ test scores, and this is true even though access to breathable oxygen explains ≈ 0% of their variation."

Great way to explain it, and an easy way to understand why, if you want to see if a factor is important in a "breathable oxygen" kind of way, you need to look at the regression coefficient, not the r-squared.


2.

This sentence comes from Jordan Ellenberg's mathematics book, "How Not To Be Wrong," which I talked a bit about last post:

"The significance test is the detective, not the judge."

I like that analogy so much I wanted to start by putting it by itself ... but I should add the previous sentence for context:

"A statistically significant finding [only] gives you a clue, suggesting a promising place to focus your research energy. The significance test is the detective, not the judge."         [page 161, emphasis in original]

(By the way, Ellenberg doesn't put a hyphen in the phrase "statistically-significant finding," but I normally do. Is the non-hyphen a standard one-off convention, like "Major League Baseball?")

(UPDATE: this question now answered in the comments.)

The point is: one in twenty experiments would produce a statistically-significant result just by random chance. So, statistical significance doesn't mean you can just leap to the conclusion that your hypothesis is true. You might be one of that "lucky" five percent. To be really confident, you need to wait for replications, or find other ways to explore further.

I had previously written an analogy that's kind of similar:

"Peer review is like the police deciding there's enough evidence to lay charges. Post-publication debate is like two lawyers arguing the case before a jury."

Well, it's really the district attorney who has the final say on whether to lay charges, right?  In that light, I like Ellenberg's description of the police better than mine. Adding that in, here's the new version:  

"Statistical significance is the detective confirming a connection between the suspect and the crime. Peer review is the district attorney deciding there's enough evidence to lay charges.  Post-publication debate is the two lawyers arguing the case before a jury."

Much better, I think, with Ellenberg's formulation in there too.

JSE: Rodney Fort on sabermetrics and salary data

The February, 2010 issue of the "Journal of Sports Economics" contains eight articles, all of which are on topics of interest to sabermetricians. I'll try to review them all over the next couple of weeks. [2015 update: well, that didn't happen!]

I've already talked about one of them: the one by David Berri and JC Bradbury, "Working in the Land of the Metricians." This post is on one of the other seven articles: "Observation, Replication, and Measurement in Sports Economics," by Rodney Fort.

Dr. Fort's research here shows that the reliability of MLB salary data, as found on the internet and elsewhere, is somewhat variable. For instance, of the 13 sources of payroll data that Fort found, seven of them have data for 1999. Those estimates of average player salary are:

$1,567,873
$1,604,972
$1,569,000
$1,733,557
$1,926,565
$1,720,050
$1,609,418

The difference between the high and low is 23%, which is a lot. Fort guesses that some of the difference is what time of year the calculations were made, since rosters change continuously, and that seems like a reasonable explanation. In 2009, the USA Today data shows 812 players, as opposed to the 750 players that would appear if opening day rosters only were used.

Still, for the years there are multiple estimates, why does every one seem to be different? These are very good questions to be asking.

The article contains multiple (different) estimates for every year between 1976 and 2006. I suspect the discrepancies won't make a huge difference in most studies, but, still, accurate data is better than inaccurate data, and I didn't realize before Dr. Fort's article that the differences were so large.

-----

Fort's article serves as an introduction to his guest-edited "Special Issue on the Challenges of Working With Sports Data," and so he also talks about the sabermetric community, and his personal interactions with us. One of those examples mentions me:

"I have had a couple of informative interactions at ... Phil Birnbaum's [blog] ... Recently, Birnbaum attempted to decompose team valuations from Forbes into cash flow value and "Picasso Value". The posts that followed were all of a mind on improving Birnbaum's estimation approach. I posted a suggestion that his approach did not include other monetary values of ownership (documented in some of my work). Thus, he ran the danger of "inflating" Picasso value. Birnbaum acknowledged the input but the flow of discussion did not change."

Dr. Fort has a point -- he did disagree with my post, and those of commenters, and, indeed, the flow of discussion didn't change. In the first post, Fort wrote that Forbes' team values didn't include the value of tax breaks and corporate cross-overs (such as when a team sells broadcasting rights to its own parent company at a discounted rate, thus keeping the team's stated profits artificially low). I should have acknowledged that in a subsequent comment.

In the second post, Dr. Fort mentioned that again, and I asked him what his estimate of Picasso Value is in the light of his estimates of the cash flow value of owning a team. He argued that in the absence of hard evidence for consumption value (i.e., evidence that team owners are willing to accept less profit for the thrill of owning the team), we should assume that value is zero. There, I disagree -- even accepting Dr. Fort's estimates of other-than-operating-proft, I think there is still evidence of consumption value. But, I should have said that online at the time. Certainly, when an expert like Dr. Fort drops by with some relevant knowledge on an issue, we absolutely should acknowledge the contribution. My apologies to Dr. Fort that such was not the case.

-----

In another example, Dr. Fort talks about his experience at an American Statistical Association conference, where a bunch of academic statisticians had a session to debate how to figure out the best slugger in baseball history:

"The primary issue for nearly all in the room had to do with holding all else constant (dead ball era, end of the spitball, changes in rules ... and so on), not at all unfamiliar to economists. I interrupted the back-and-forth question and answer session and offered a normalized pay measure: the highest-paid slugger must surely be the best. The rest of the discussion held fast to its previous course ..."

Here, I'd argue that Fort's suggestion is not really answering the question that was asked. The statisticians are asking, what is the proper algorithm for calculating how good the player is? And Fort is saying, "don't bother! The team GMs already know the algorithm. If you want to know who the best player is, just look at their payrolls."

That's fair enough. But it's only natural that the statisticians won't be satisfied by that: they don't just want the answer, they want to know how to calculate it.

This narrative reminds me of an old story: A physics student sits down to an exam, and one of the questions is: "show how you can use a barometer to determine the height of a tall building." The student is stumped at first, but then writes: "Find the superintendent of the building, and say to him: 'If you tell me how tall your building is, I will give you this fine barometer.'"

-----

In both the introduction and conclusions, Fort comments on the issues raised by the Berri/Bradbury paper:

Berri and Bradbury relate to us their extensive experience with the SABR-metric and APBR-metric communities ... There are tensions and jealousies aroused over the issue of proper credential and peer reivew. There is the issue of proper assignation of credit; when incorporating a technique or measurement developed outside the [academic] peer-review process, what to do? Finally, there is the issue of dealing online with these fellow travelers who often adopt the anonymity of pseudonyms. ...

"I cannot help but think that part of the tension with the metricians revolves around desires for immortality in the naming of a thing. This seems patently absurd to me ... In sports, the authors point this out for the case of Dobson's creation of batting average in 1872. I do not ever remember seeing anybody cite Gauss (1821) and Karl Pearson (1984) on the origins of the standard deviation or the subsequent coinage of the variance by Ronald Fisher (1918). And I doubt these authors took much umbrage over the fact that their inventions simply became household terms in record time. However perhaps, we are wrong in this; if someone grouses, perhaps we should respond."

Hmmm ... from my standpoint, I don't think naming is the issue at all. I agree with Dr. Fort here that it's perfectly reasonable to omit citations for results that have become household terms. We all talk about "DIPS" or "Runs Created" or "Linear Weights" without mentioning Voros McCracken or Bill James or Pete Palmer every time; that's just normal. And I don't think I've ever seen any sabermetrician anywhere demanding that his particular term be used, or that anything be named after him.

One of the things we *do* expect from the academic community is that they be aware of our own research and conventions, and, unless they disagree with the findings, that they show as much respect for them as they would if the research had been published academically. That's just common sense, isn't it? Take DIPS, for example. Do a couple of quick Google searches, and you should find at least 20 studies testing and confirming the hypothesis. But when Berri and Bradbury write about it, in this very same journal as Fort's article, what do they do? They mention Voros McCracken, and then do a quick regression and announce that this "supports" DIPS. Which is fine, but, if those twenty studies were published academically, they wouldn't dare omit those citations. The omission gives the reader the (perhaps unintended) impression that the subject hasn't been studied yet.

It would have been very easy for Berri and Bradbury to have quoted existing research in addition to their own. They could have said (taking two of the many existing studies at random), " the DIPS theory was repeatedly tested and supported in numerous studies after McCracken's article; for example, Tippett (2003) and Gassko (2005). A simpler confirmation is that we now calculate the persistence over time of batting average on balls in play, and find it much lower than for other statistics."

They deliberately chose not to do that.

Most interestingly, Dr. Fort writes,

"Do not get me wrong. The internet seems a place of great potential. Suppose the living Nobel Prize winners start a blog. An idea occurs, they debate it, the courageous among us try to contribute, and the discipline moves forward on that issue. Few would doubt the value ... "

And you know what? That's exactly what's happening! Of course, there's no Nobel in Sabermetrics, but the best and brightest minds in the field are already online doing exactly what Dr. Fort wishes the best economists would do. And that's why the field moves so fast.

With perhaps the exception of computer software development, I bet that there is no other field of human knowledge today in which you can see progress move so quickly in real time, where the best minds in the field publish so much excellent, rigorous work so reliably, where collaboration happens so easily and instantly, and where even an unknown can have something to contribute to the work of even the most experienced sabermetrician.

Fort continues,

"In the publish-or-perish world [of academic economists, we] all know why this has not happened yet. The best we get is editorials by Nobel Prize winners, occasionally scolding each other. And the worst we get is anonymous yelling at each other behind pseudonyms in blog discussion areas. .... This may be entertaining ... but I wonder how it will all play out in the rigorous pursuit of answers to sports economics questions."

Well, in economics, maybe; I'll take Dr. Fort's word for it. But in sabermetrics, the internet has indeed provided the ideal breeding ground for the rigorous pursuit of answers.

As Dr. Fort implies, the incentives facing academic economists are very different from those facing amateur sabermetricians, or even professional sabermetricians (who write books or work for teams). To avoid "perishing," and suffering in their career, academics are forced to write formally and rigorously, submit to peer review by two or three colleagues, wait several months (or more) before getting a response, and publish in journals where relatively few people will read their work.

In contrast, non-academic sabermetricians have different goals: a love for baseball, a dream of working for a ballclub, the recognition and status in the community, or, yes, even just the thrill of scientific discovery. To those ends, they have incentives to explain their ideas informally, publish online instantly, have their work peer-reviewed within minutes or hours, have all the best researchers in the field be aware of their research almost immediately, and maintain credibility in the community by engaging critics and acknowledging shortcomings in their work.

My prediction: over the next few years, non-academic sabermetrics will gain more and more credibility, as more and more of it comes out of major league front offices, and journalists continue to acknowledge it. In that light, academia will find it harder and harder to ignore the accepted, established and growing body of knowledge on what will appear to be flimsy procedural grounds. (Does anyone outside of academia care that Bill James' ideas weren't peer reviewed?)

My view is that the academics, with incentives to maintain their established and expensive barriers to publication, will never be able to keep up with the freewheeling, open-source, dynamism of the non-academic crowd. Eventually, economists will come to the realization that progress in sabermetrics is best done by the non-academic community of sabermetricians, just as the invention of better computers is best done by the non-academic community of computer manufacturers. Academics will still publish papers of interest to sabermetricians, and influence their work, just as materials scientists and computer theoreticians publish work that's of interest to Apple and Google. But the bulk of sabermetrics itself will be seen as coming from non-academic sabermetricians, not academic economists.

It's not a knock against economists, who are certainly as capable of doing sabermetrics as anyone. It's just that academic processes are too rigorous and expensive for a world that moves so fast. Academia, I think, will stick to areas where it has a monopoly, like the more traditional areas of economics, or expensive fields like physics.

Chopped liver II

David Berri and J.C. Bradbury have a new paper out. Called "Working in the Land of the Metricians," it purports to be a guide for Ph.D. sports economists in how to interact with sabermetricians.

The subject is an important one, but, sadly, the paper isn't very constructive. There's a good quote-by-quote critique of the paper at Tango's blog, which I agree with substantially enough that I can just refer you there. (Also, since the paper is gated, you can get a good feel for what's happening in it by reading Tango's take.)  

UPDATE: the paper is now available in full (.pdf) at Dave Berri's site.

For my part, I'll just concentrate on one major point -- that Berri and Bradbury still treat the non-academic community's work as if it barely has any value at all. And they're pretty forthright about it. Despite paying lip service to the idea that maybe non-academic researchers have *something* to contribute, the authors remain wilfully blind to 98 percent of the sabermetric community's contribution to the field.

Indeed, Berri and Bradbury explicitly refuse to recognize that the active research community has any expertise at all:

"Birnbaum considers sabermetricians to be "no less intelligent than academic economists" and superior to economists in their understanding of baseball. This statement reveals a curious worldview. On one hand, the aspect that is universal across both groups -- members of both communities have been devoted sports fans since an early age -- is considered unique to the nonacademic sports anaysts. On the other hand, when it comes to the aspect that is unique to academics -- academia normally involves many years of advanced training and requires its participants to be judged competent by their peers in a "publish or perish" environment -- metricians demand equal recognition. In our view, this mentality begets misplaced confidence."

Get what they're saying here? Bill James, Tom Tango, Mitchel Lichtman, Andy Dolphin, Pete Palmer, those guys hired by baseball teams, all those other sabermetric researchers -- they're not baseball experts at all. They're *just sports fans*. How could they possibly know more about analyzing baseball than any other fan, unless they've formally studied econometrics?

It's astonishing that Berri and Bradbury could possibly believe that economists, even sports economists, know more about baseball than these guys, who have built their careers around analyzing the game. Equally astonishing is their implication that we *are* less intelligent than they are. At first, I thought that they didn't realize what they were saying. But, no, it seems pretty clear that they *do* believe it.

What has become apparent in sabermetricians' debates with Bradbury (and, to a lesser extent, Berri, who doesn't engage us as much) is that they disagree with us on almost every point we make, and that they seem to be uncomfortable arguing informally. I can't recall a single time that either of them has conceded that they're wrong, even on a small point. Economists are supposed to be fond of hypotheticals, simple models, and playful arguments to illustrate a point (see this Paul Krugman column), but Bradbury and Berri, not so much. Attempts to try to describe their models with simplified analogies are usually met with detailed rebukes about how we don't understand their econometric methods.

In that light, it's easier to see where they're coming from. They believe that (a) only formal, peer-reviewed research counts as knowledge, and (b) all us non-peer-reviewed people have been wrong every time we've disagreed with their logic. If both of those were actually true, they'd be right -- we'd just be ignorant sports fans who don't have any expertise and need to be educated.

As for the sabermetric findings that they actually use, like DIPS and OPS ... Berri and Bradbury seem to consider them a form of folk wisdom that the unwashed baseball fans managed to stumble upon, and argue that economists should not accept them until they've been verified by regression methods that would pass peer review and be publishable in academic journals.

Back in 2006, in the post that Bradbury and Berri quoted above, I accused Bradbury of ignoring the findings of sabermetricians in one of his papers. At the time, I thought perhaps I was too harsh. I was wrong. This paper shows that he truly believes that, as non-peer-reviewed sports fans with no special expertise, our research findings are unworthy of being cited.

Indeed, in Bradbury's exposition (I am assuming all the arguments in the baseball portion of the current paper are Bradbury's, although Berri is still listed as co-author), he treats the history of sabermetric knowledge as if it were mostly a series of academic papers. That's absurd; it's indisputable that, conservatively, at least 90 percent of our sabermetric knowledge came from outside academia. If you were writing the history of research about Linear Weights, what would you include? Think about it for a minute.

Ready? Here's how Bradbury sees it:

--In 1992, academic A.A. Blass published a study where he estimated linear weights by regression.

--In 2005, academic Ted Turocy published a paper which highlighted the "omitted variable" bias making the results not as accurate as they could be.

--In 1963, academic George Lindsey had published a paper with a rudimentary form of the same equation, but not using regression.

--In 1984, sabermetricians John Thorn and Pete Palmer "popularized" and "updated" Lindsey's work.
--In 2003, academics Jim Albert and Jay Bennett compared the two approaches.

Got it? Four academic papers (Albert/Bennett is actually their book, Curve Ball, but no matter) and Thorn/Palmer. On top of that, the only mentions of Thorn and Palmer are their "updating" and "popularizing". Each of the academics' work, on the other hand, is described in some detail.

Is that how you would characterize the state of knowledge, that the history of accumulated knowledge about Linear Weights comes from these five academics and a cursory contribution from Pete Palmer? I mean, come on. There's a huge literature out there if you look outside academia, including studies on various improvements to the original.

It gets worse, in the DIPS discussion.

Bradbury starts off by reviewing the seminal Gerald Scully paper. In 1974, Scully (a famed sports economist who passed away in 2009) published a paper that tried to value players' monetary contributions to their teams. Regrettably, he used strikeout-to-walk ratio as a proxy for the pitcher's value to his team's success on the field. That's not a particularly good measure of the value of a pitcher; ERA is much better. Even before sabermetrics, everyone knew that, including casual baseball fans and Joe Morgan. And if they didn't, Bill James would have made it clear to them in his Abstracts, so there was no excuse for a baseball researcher to not be aware of that after, say, 1983.

And Bradbury acknowledges that, that ERA was perceived to be better than K/BB ratio. Does he cite common sense, conventional wisdom, or Bill James? Nope. He cites two academic papers from the 1990s. No, really:

"... [Andrew] Zimbalist (1992b) and [Anthony] Krautmann (1999) argue that ERA is a better measure of pitcher quality ..."

So, before 1992, nobody else argued it? Well, OK, if you say so.

Anyway, with that established, Bradbury continues. In 2001, Voros McCracken came along, and, in an "essay" on the "popular sabermetric Web site Baseball Prospectus," he "suggested" that pitchers have little control over what happens to balls in play. At this point, Bradbury checks the correlation of a pitchers' BABIP in consecutive seasons, and finds it's fairly low (.24).

"This supports McCracken's assertion," he writes.

Okay, thanks for the verification! But, er, actually, it's not like the sabermetric community was just sitting on its hands the past eight or nine years, staring at McCracken's hypothesis and wondering if someone would ever come along and tell them if it was true. Sabermetricians of all sorts have done huge volumes of work to refine and prove versions of the DIPS hypothesis. For a long time, you couldn't hit any sabermetric website without DIPS this and DIPS that hitting you from every angle, with all kinds of theories about it tested and studied.

It's perfectly fine that Bradbury verifies McCracken with his quick little regression here, but why wouldn't he acknowledge everyone else, by, for instance, saying that his study is an example of the kinds of confirmation studies that the sabermetric community has been doing since 2001? In light of the rest of the article, which implies that sabermetricians are just unsophisticated baseball fans, that omission would reasonably lead readers to incorrectly assume that Bradbury's little study is the first of its kind.

But, instead, Bradbury ignores all those years of sabermetric study of the DIPS issue, just because it wasn't academically published or peer-reviewed. That's as wrong now as it was when I wrote about it in 2006 -- especially in an essay that's ostensibly suggesting that academics can benefit from outside research.

The DIPS approach is as well accepted in sabermetrics as the Coase Theorem is accepted in economics. The difference is, if I published something mentioning Coase's hypothesis, and then published a study "supporting" it without citing any other study or mentioning that it's a canon of the economics literature, Bradbury would go ballistic, laying into my ignorance like ... like Tiger Woods on a supermodel. (Sorry.) The other way around, though, and it's all OK.

Anyway, that's just the prelude -- this is the point where Bradbury's argument gets really bizarre.

Why did Bradbury bring up DIPS? Because it shows that, instead of using ERA to evaluate the skill of a pitcher, it's better to just use walks, strikeouts, and home runs allowed, thus eliminating a lot of random chance from the pitcher's record. That part is absolutely fine. But then he comes back to Scully.

Remember when Scully did his 1974 study that used strikeout-to-walk ratio as a measure of a pitcher's value? And everyone agreed that he should have used ERA instead? Well, now, hang on! McCracken has shown us that if we ignore a pitcher's balls in play, we can get a better measure of his talent than if we use ERA. Eliminating balls in play leaves only BB, K, and HR. That means strikeouts and walks are really important. And, in turn, that means that Scully was actually correct back in 1974 when he emphasized strikeouts and walks by using K/BB ratio as his statistic of choice!

To make this bizarre argument Bradbury ignores the fact you have to combine K and BB in a very specific way, or it doesn't work well at all, and that K/BB ratio is still worse than ERA. And HR are important too. But, argues Bradbury, at least Scully was right in that it had to be K and BB. Maybe he wasn't completely correct, but he had the right idea.

Why did Gerald Scully choose to value pitchers by K/BB? According to Bradbury, it wasn't because he just gave it a guess. He did it because he intuitively anticipated that DIPS was true. Voros McCracken, the non-academic sabermetrician, just served later to confirm Scully's original insight.

See? The academics were right all along!

If that sounds ludicrous, it is. I really hope you don't believe what I'm saying here ... I hope you're thinking that nobody who could actually make that argument with a straight face, that Gerald Scully's choice of K/BB ratio as his metric is an anticipation of a completely different, complex, 12-part formula for DIPS ERA that happens to partly depend on K and BB. I hope you believe that I'm making it up.

But I'm not. That's actually what Bradbury argues! Here are the quotes:

"If a measure varies considerably for an individual player over time, it is likely that the measure is heavily polluted by luck ... Scully appeared to understand this point with his choice of the strikeout-to-walk ratio to measure pitcher quality.

"Later research from outside academia [McCracken] confirmed Scully's original approach and suggested [also] including home-run prevention when evaluating pitchers. ...

"Consequently, we see Scully's general approach is confirmed by a metrician, demonstrating what the nonacademic sports research community can contribute to sports economics research."

Could there be a sillier, more self-serving rationalization?

Oh, and by the way ... it seems that Scully wasn't even a baseball fan at the time he started writing his study. How does Bradbury think Scully was able to anticipate a result that wouldn't become apparent until 27 years later? A result, moreover, that pertained to a sport Scully probably didn't know that much about, in a field of science that didn't even exist at the time, that wouldn't emerge until years of study by non-academic researchers, and that was so surprising it shocked even the most veteran researchers in the field?

Maybe it was that "years of advanced training" in economics -- obviously so much more valuable than the non-expertise in sabermetrics that Voros McCracken, a mere baseball fan, couldn't possibly have had.

Evaluating scientific debates: some ramblings

Last week's renewed debate on JC Bradbury's aging study (JC posted a new article to Baseball Prospectus, and comments followed there and on "The Book" blog) got me thinking about some things that are tangential to the study itself ... and since I have nothing else to write about at the moment, I thought I'd dump some of those random thoughts here.


1. Peer review works much better after publication than before.

When there's a debate between academics and non-academics, some observers argue that the academics are more likely to be correct, because their work was peer reviewed, while the critics' work was not.

I think it's the other way around. I think post-publication reaction, even informally on the internet, is a much better way to evaluate the paper than academic peer review.

Why? Because academic peer reviewers hear only one side of the question -- the author's. At best, they might have access to the comments of a couple of other referees. That's not enough.

After publication, on the internet, there's a back and forth between people on one side of the question and people on the other. That's the best way to get at the truth -- to have a debate about it.

Peer review is like the police deciding there's enough evidence to lay charges. Post-publication debate is like two lawyers arguing the case before a jury. It's when all the evidence is heard, not just the evidence on one side.

More importantly, no peer reviewer has as good a mastery of previous work on a subject than the collective mastery of the public. I may be an OK peer reviewer, but you know who's a better peer reviewer? The combination of me, and Tango, and MGL, and Pizza Cutter, and tens of other informed sabermetricians, some of whom I might only meet through the informal peer review process of blog commenting.

If you took twelve random sabermetricians whom I respect, and they unanimously came to the verdict that paper X is flawed, I would be at least 99% sure they were right and the peer reviewer was wrong.


2. The scientific consensus matters if you're not a scientist.

It's a principle of the scientific method that only evidence and argument count -- the identity of the arguer is irrelevant.

Indeed, there's a fallacy called "argument from authority," where someone argues that a particular view must be correct because the person espousing it is an expert on the subject. That's wrong because even experts can be wrong, and even the expertest expert has to bow to logic and evidence.

But that's a formal principle that applies to situations where you're trying to judge an argument on its merits. Not all of us are in a position to be able to do that all the time, and it's a reasonable shortcut in everyday life to base your decision on the expertise of the arguer.

If my doctor tells me I have disease X, and the guy who cleans my office tells me he saw my file and he thinks I really have disease Y ... well, it's perfectly legitimate for me to dismiss what the office cleaner says, and trust my doctor.

It only becomes "argument from authority" where I assert that I am going to judge the arguments on their merits. Then, and only then, am I required to look seriously at the office cleaner's argument, without being prejudiced by the fact that he has zero medical training.

Indeed, we make decisions based on authority all the time. We have to. There are many claims that are widely accepted, but still have a following of people who believe the opposite. There are people who believe the government is covering up UFO visits. There are people who believe the world is flat. There are people who believe 9/11 was an inside job.

If you're like me, you don't believe 9/11 was an inside job. And, again, if you're like me, you can't actually refute the arguments of those who do believe it. Still, your disbelief is rational, and based solely on what other people have said and written, and your evaluations of their credibility.

Disbelieving solely because of experts is NOT the result of a fallacy. The fallacy only happens when you try to use the experts as evidence. Experts are a substitute for evidence.

You get your choice: experts or evidence. If you choose evidence, you can't cite the experts. If you choose experts, you can't claim to be impartially evaluating the evidence, at least that part of the evidence on which you're deferring to the experts.

The experts are your agents -- if you look to them, it's because you are trusting them to evaluate the evidence in your stead. You're saying, "you know, your UFO arguments are extraordinary and weird. They might be absolutely correct, because you might have extraordinary evidence that refutes everyone else. But I don't have the time or inclination to bother weighing the evidence. So I'm going to just defer to the scientists who *have* looked at the evidence and decided you're wrong. Work on convincing them, and maybe I'll follow."

The reason I bring this up is that, over at BPro, MGL made this comment:

"I think that this is JC against the world on this one. There is no one in his corner that I am aware of, at least that actually does any serious baseball work. And there are plenty of brilliant minds who thoroughly understand this issue who have spoken their piece. Either JC is a cockeyed genius and we (Colin, Brian, Tango, me, et. al.) are all idiots, or..."

Is that comment relevant, or is it a fallacious argument from authority? It depends. If you're planning on reading all the studies and comments, and reaching a conclusion based on that, then you should totally ignore it -- whether an argument is correct doesn't depend on how many people think it is.

But if you're just reading casually and trying to get an intuitive grip on who's right, then it's perfectly legitimate.

And that's how MGL meant it. What he's saying is something like: "I've explained why I think JC is wrong and I'm right. But if you don't want to wade through all that, and if you're basing your unscientific decision on which side seems more credible -- which happens 99% of the time that we read opposing opinions on a question of scientific fact -- be aware that the weight of expert opinion is on my side."

Put that way, it's not an appeal to authority. It's a true statement about the scientific consensus.


3. Simple methods are often more trustworthy than complex ones.

There are lots of studies out there that have found that the peak age for hitters in MLB is about 27. There is one study, JC Bradbury's, that shows a peak of 29.

But it seems to me that there is a perception, in some quarters, that because JC's study is more mathematically sophisticated than the others, it's therefore more trustworthy. I think the opposite: that the complicated methods JC used make his results *less* believable, not more.

I've written before about simpler methods, in the context of regression and linear weights. Basically, there are two different methods that have been used to calculate the coefficients for the linear weights formula. One involves doing a regression. Another involves looking at play-by-play data and doing simple arithmetic. The simple method actually works better.

More importantly, for the argument I'm making here, the simple method is easily comprehensible, even without stats classes. It can be explained in a few sentences to any baseball fan of reasonable intelligence. And if you're going to say you know a specific fact, like that a single is worth about .46 runs, it's always nicer to know *why* than to have to trust someone else, who used a mathematical technique you don't completely understand.

Another advantage of the simple technique is that, because so many more people understand it, its pros and cons are discovered early. A complex method can have problems that don't get found out until much later, if ever.

For instance, how much do hitters lose in batting skill between age 28 and age 35? Well, one way to find out is to average the performance of 28-year-olds, and compare it to the averaged performance of 29-year-olds, 30-year-olds, and so on, up to 35-year-olds. Pretty simple method, right, and easy to understand? If you do it, you'll find there's not much difference among the ages. You might conclude that players don't lose much between 28 and 35.

But there's an obvious flaw: the two groups don't comprise the same players. Only above-average hitters stay in the league at 35, so you're comparing good players at 35 to all players at 28. That's why they look similar: the average of a young Joe Morgan and a young Roy Howell looks similar to the average of an old Joe Morgan and a retired, zero-at-bat Roy Howell, even though they both Morgan and Howell each declined substantially in the intervening seven years.

Now that flaw ... it's easy to spot, and the reason it's easy to spot is that the method is simple enough to understand. It's also easy to explain, and the reason it's easy to explain is again that the method is simple enough to understand.

If I use the more complicated method of linear regression (and a not very complicated regression), and describe it mathematically, it looks something like this:

"I ran an ordinary least squares regression, using the model P(it) = ax(it) + b[x(it)^2] + e, where P(it) is the performance of player i at age t, x(it) is the age of player i at age t, and all player-seasons of less than 300 PA were omitted. The e is an error term, assumed iid normal with mean 0."

The flaw is actually the same as in the original, simpler case, the fact that the sample of players is different at each age. But it's harder to see the flaw way, isn't it? It's also harder to describe where the flaw resides -- there's no easy one-sentence explanation about Morgan and Howell like there was before.

So why would you trust the complicated method more than the simple one?

Now, I'm not saying that complexity is necessarily bad. A complex method might be more precise, and give you better results, assuming that there aren't any flaws. But, you still have to check for flaws. If the complex method gives you substantially different results (peak age 29) from the simple methods (peak age 27), that's a warning sign. And so you have to explain the difference. Something must be wrong, either with the complex method, or with all the simple methods. It's not enough to just explain why the complex method is right. You also have to explain why the simple methods, which came up with 27, came out so wrong.

In the absence of a convincing explanation, all you have are different methods, and no indication which is more reliable. In that case, why would you choose to trust the complicated method that you don't understand, but reject a simple methods that you *do* understand? The only reason for doing so is that you have more faith that whoever introduced the complicated method actually got everything right, the method and the calculations and the logic.

I don't think that's justified. My experience leads me to think that it's very, very risky to give that kind of blind trust without understanding the method pretty darn well.

Who is a sabermetrician?

J.C. Bradbury says he's not a sabermetrician:

"... I shouldn’t be considered a sabermetrician. I have never claimed to be a member of this community. What I do is apply my knowledge from my economics training and experience with analyzing data to issues in baseball."

I find this kind of confusing – If JC analyzes issues in baseball, statistically, then *of course* he's a sabermetrician. Perhaps there's a confusion in regards to the definition – does sabermetrician mean "one who does sabermetrics," or "one who is expert in the field of sabermetrics"? I suppose JC could be the one but not the other – if I do some analysis using economic reasoning, I'm doing economics, but that doesn't mean I'm an economist.

In any case, where all this came from is a post by Rob Neyer, pointing out that JC and Tom Tango are in strong disagreement on the Raul Ibanez signing. JC has Ibanez being worth $46 million over three years, while Tango has him worth only about $10 million. That is, to say the least, a substantial difference.

Rob uses this example to show how there are, and always will be, major sources of disagreement in the sabermetric community. I'm not completely with Rob on this one – I think conflicts of this magnitude are fairly rare. My take is that JC has a methodology for valuing players that most other sabermetricians think is incorrect (I, myself, disagreed with him here). As Mitchel Lichtman says in one of the comments,

"There are plenty of experts with respect to projections, and I don't think - in fact I know - that any of them will project his value at anything close to 14mm per year."

If that's the case, that JC's opinion is an outlier, then it might not be that there's long-term disagreement, but simply that somebody – perhaps JC, perhaps everyone else -- is just plain wrong. And, in legitimate science, when someone is wrong, it usually gets corrected pretty quick.

-----

JC's post also goes on to argue that academic research in sabermetrics is more reliable than informal online discussions:

"... much what passes for research within [the sabermetric] community is not sufficiently rigorous to reach the conclusions often claimed. There are many academic researchers from a variety of fields who have significantly advanced the understanding of baseball that receive scant mention in the sabermetric community. For example, Michael Schell’s Baseball’s All-Time Best Sluggers is the most thorough treatise on hitting ever written; yet, few individuals mention his work or attempt to replicate his methods. You rarely see economists Gerald Scully or Tony Krautmann mentioned when attempting to value players, despite the fact that their methods were published in reputable peer-reviewed economics journals, where established experts vetted their work. Academics are not always right, but I believe the checks ensure they are more likely to reach correct conclusions than informal online discussions."

Not to beat a dead horse, but, as I have written before, I'm not very impressed with the output of the academic community, when it comes to sabermetrics. I think the peer review to which JC refers is often not capable of spotting flawed analysis. For my own part, I would certainly prefer to have a paper of my own vetted by Tango or MGL than by an anonymous academic economist who is unlikely to have a strong grounding in sabermetrics. I think experienced amateur sabermetricians are much more likely to find legitimate flaws in my work than professional economists.

However, I haven't seen the academic examples JC mentions. It's certainly possible that Schell, Scully and Krautmann have done work that we non-academics are ignoring. Are any readers familiar with Schell's book, or the Scully and Krautmann valuation methods? If someone can point me to the relevant journal articles, I'll post a review.

UPDATE: Dave Berri comments on his blog.

If it's peer reviewed, journalists won't question it

Does the media lose its sense of balance when dealing with peer-reviewed studies? Apparently, some won't even ask mildly skeptical questions, even on politically-sensitive issues. One newspaper editor says,

"We are dealing with a peer-reviewed journal study, and I don't feel at all comfortable going beyond what they are publishing. That is not our role."

This is on a study about marijuana smoking possible causing gum disease. If even the science beat won't question academic studies, what hope is there that sports editors will?