Experts

Bill James doesn't like to be called an "expert." In the "Hey Bill" column of his website, he occasionally corrects readers who refer to him that way. And, often, Bill will argue against the special status given to "experts" and "expertise."

This, perhaps understandably, puzzles some of us readers. After all, isn't Bill's expertise the reason we buy his books and pay for his website?  In other fields, too, most of what we know has been told to us by "experts" -- teachers, professors, noted authors. Do we want to give quacks and ignoramuses the same respect as Ph.Ds?

What Bill is actually arguing, I think, is not that expertise is useless -- it's that in practice, it's used to fend off argument about what the "expert" is saying.  In other words, titles like "expert" are a gateway to the fallacy of "argument from authority."

On November 8, 2011 (subscription required), Bill replied to a reader's question this way:

"I've devoted my whole career to battling AGAINST the concept of expertise. The first point of my work is that it DOESN'T depend on expertise. I am constantly reminding the readers not to regard me an expert, because that doesn't have anything to do with whether what I have to say is true or is not true."

In other words: don't believe something because an "expert" is saying it. Believe it because of the evidence. 

(It's worth reading Bill's other comments on the subject; I wasn't able to find links to everything I remember, but check out the "Hey Bill" pages for November, 2011; April 18, 2012; and August/September, 2014.)

Anyway, I'd been thinking about this stuff lately, for my "second-hand knowledge" post, and Bill's responses got me thinking again. Some of my thoughts on the subject echo Bill's, but all opinions here are mine.

-------

I think believing "experts" is useful when you're looking for the standard, established scientific answer.  If you want to know how far it is from the earth to the sun, an astronomer has the kind of "expertise" you can probably accept.

We grow up constantly learning things from "experts," people who know more than we do -- namely, parents and teachers. Then, as adults, if we go to college, we learn from Ph.D. professors. 

Almost all of our formal education comes from learning from experts. Maybe that's why it seems weird to hear that you shouldn't believe them. How else are you going to figure out the earth/sun distance if you're not willing to rely on the people who have studied astronomy?

As I wrote in that previous post, it's nice to be able to know things on your own, directly from the evidence. But there's a limit to how much we can know that way. For most factual questions, we have to rely on other people who have done the science that we can't do.

-------

The problem is: in our adult, non-academic lives, the people we call "experts" are rarely used that way, to resolve issues of fact. Few of the questions in "Ask Bill" are about basic information like that. Most of them are asking for opinion, or understanding, or analysis. They want to pick Bill's brain.

From 1/31/2011: "Would you have any problem going with a 4-man rotation today?"

From 10/7/2013: "Bill, you wrote in an early Abstract that no one can learn to hit at the major league level. Do you still believe that?"

From 10/29/2012: "Do you think baseball teams sacrifice bunt too much?"

In those cases, sure, you're better off asking Bill than asking almost anyone else, in my opinion. Even so, you shouldn't be arguing that Bill is right because he's an "expert."  

Why?  Because those are questions that don't have an established, scientific answer based on evidence. In all three cases, you're just getting Bill's opinion. 

Moreover: all three of those issues have been debated forever, and there's *still* no established answer. That means there are opinions on both sides. What makes you think the expert you're currently asking is on the correct side? Bill James doesn't think a four-man rotation is a bad idea, but any number of other "experts" believe the opposite. 

Subject-matter experts should agree on the basic canon, sure. It should be rare that a physics "expert" picks up a textbook and has serious disagreements with anything inside.

But: they can only agree on answers that are known. In real life, most interesting questions don't have an answer yet. That's what makes them so interesting!

When will we cure cancer? What's the best way to fight crime? When should baseball teams bunt? Will the Seahawks beat the spread?

Even the expertest expert doesn't know the answer to those questions. Some of them are unknowable. If anyone was "expert" enough to predict the outcome of football games, he'd be the world's richest gambler. 

-----

All you can really expect from an expert is that he or she knows the state of the science.  An expert is an encyclopedia of established knowledge, with enough understanding and experience to draw inferences from it in established ways.

Expertise is not the same as intelligence. It is not the same as wisdom. It is not the same as insight, or freedom from bias, or prescience, or rationality.

And that's why you can get different "experts" with completely different views on the exact same question, each of them thinking the other is a complete moron. That's especially true on controversial issues. (Maybe it's not that controversial issues are less likely to have real answers, but that issues that have real answers are no longer controversial.)

On those kinds of issues, where you know there are experts on both sides, you might as well flip a coin as rely on any given expert.

And hot-button issues are where you find most of the "experts" in the media or on the internet, aren't they?  I mean, you don't hear experts on the radio talking how many neutrons are in an atom of vanadium. You hear them talking about what should be done to revive the sagging economy. Well, there's no consensus answer for that. If there were, the Fed would have implemented it long ago, and the economy would no longer be sagging. 

Indeed, the fact that nobody is taking the expert's advice is proof that there must be other experts that think he's wrong.

Sometimes, still, I find myself reading something an expert says, and nodding my head and absorbing it without realizing that I'm only hearing one side. We don't always conciously notice the difference, in real time, between consensus knowledge and the "expert's" own assertions. 

Part of the reason is that they're said in the same, authoritative tone, most of the time. Listen to baseball commentators. "Jeter is hitting .302." "Pitching is 75 percent of baseball." You really have to be paying attention to notice the difference. And, if you don't know baseball, you have no way of knowing that "75 percent of baseball" isn't established fact! At least, until you hear someone dispute it.

Also, I think we're just not used to the idea that "experts" are so often wrong. For our entire formal education, we absorb what they teach us about science as unquestionably true. Even though we understand, in theory, that knowledge comes from the scientific method ... well, in practice, we have found that knowledge comes from experts telling us things and punishing us for not absorbing them.  It's a hard habit to break.

------

The fact is: for every expert opinion, you can find an equal and opposite expert opinion. 

In that case, if you can't just assume someone's right just because he's an expert, can you maybe figure out who's right by *counting* experts?  

Maybe, but not necessarily. As Bill James wrote (9/8/14),

"An endless list of experts testifying to falsehood is no more impressive than one."

It used to be that an "endless list" of experts believed that W-L record was the best indication of a pitcher's performance. It used to be that almost all experts believed homosexuality was a disease. It used to be that almost no experts believed that gastritis was caused by bacteria -- until a dissenting researcher proved it by drinking a beaker of the offending strain. 

Each of those examples (they're mine, not Bill's) illustrates a different way experts can be wrong. 

In the first case, pitcher wins, the expert conventional wisdom never had any scientific basis -- it just evolved, somehow, and the "experts" resisted efforts to test it. 

In the second case, homosexuality, I suspect a big part of it was the experts interpreting the evidence to conform to their pre-existing bias, knowing that it would hurt their reputations to challenge it. 

In the third case ... that's just the scientific method working as promised. The existing hypothesis about gastritis was refuted by new evidence, so the experts changed their minds. 

Bill has a fourth case, the case of psychiatric "expert witnesses" who just made stuff up, and it was accepted because of their credentials. From "Hey Bill," 11/10/2011 and 11/11/2011:

"Whenever and wherever someone is convicted of a crime he did not commit, there's an expert witness in the middle of it, testifying to something that he doesn't actually know a damned thing about.  In the 1970s expert witnesses would testify to the insanity of anybody who could afford to pay them to do so."

"Expert witnesses are PRAISED by professional expert witnesses for the cleverness with which they discuss psychological concepts that simply don't exist."

In none of those cases would you have got the right answer by counting experts. (Well, maybe in the third case, if you counted after the evidence came out.)  

Actually, I'm cheating here. I haven't actually shown that the majority isn't USUALLY right. I've just shown that the majority isn't ALWAYS right. 

It's quite possible that those four cases were rare exceptions: that, most of the time, when the majority of experts agree, they're generally right. Actually, I think that's true, that the majority is usually right -- but I'm only willing to grant that for the "established knowledge" cases, the "distance from the earth to the sun" issues. 

For issues that are legitimately in dispute, does a majority matter?  And does the size matter?  Does a 80/20 split among experts really mean significantly more reliability than a 70/30 split?  

Maybe. But if you go by that, it's not *knowing*, right?  It's just handicapping. 

Suppose 70% of doctors believe X, and, if you look at all times that seventy percent of doctors believed something else, 9 out of 10 of those beliefs turned out to be true. In that case, you can't say, "you must trust the majority of experts."  You have to say, at best, "there's a 9 out of 10 chance that X is true."

But maybe I can say more, if I actually examine the arguments and evidence.

I can say, "well, I've examined the data, and I've looked at the studies, and I have to conclude that this is the 1 time out of 10 that the majority is dead wrong, and here is the evidence that shows why."  

And you have no reply to that, because you're just quoting odds.

And that's why evidence trumps experts. 

Here's Bill James on climate scientists, 9/9/2014 and 9/10/2014:

"[You should not believe climate scientists] because they are experts, no. You should believe them if they produce information or arguments that you find persuasive. But to believe them BECAUSE THEY ARE EXPERTS -- absolutely not.

"It isn't "consensus" that settles scientific disputes; it is clear and convincing evidence. An issue is settled in science when evidence is brought forward which is so clear and compelling that everyone who looks at the evidence comes to the same conclusion. ... The issue is NOT whether scientists agree; it is whether the evidence is compelling."

If you want to argue that something is true, you have two choices. You can argue from the evidence. Or, you can argue from the secondhand evidence of what the experts believe. 

But: the firsthand evidence ALWAYS trumps the secondhand evidence. Always. That's the basis of the entire scientific method, that new evidence can drive out an old theory, no matter how many experts and Popes believe they're wrong, and no matter how strongly they believe it.

You're talking to Bob, a "denier" who doesn't believe in climate change. You say to Bob, "how can you believe what you believe, when the scientists who study this stuff totally disagree with you?"

If Bob replies, "I have this one expert who says they're wrong" ... well, in that case, you have the stronger argument: you have, maybe, twenty opposing experts to his one. By Bob's own logic -- "trust experts" -- the probabilities must be on your side. You haven't proven climate change is real, but you've convincingly destroyed Bob's argument. 

However: if Bob replies, "I think your twenty experts are wrong, and here's my logic and evidence" -- well, in that case, you have to stop arguing. He's looking at firsthand evidence, and you're not. Your experts might still be right, because maybe he's got bad data, or he's misinterpreting his evidence, or his worthless logic comes out of the pages of the Miss America Pageant. Still, your argument has been rendered worthless because he's talking evidence, which you're not willing or able to look at directly.

As I wrote in 2010,

"Disbelieving solely because of experts is NOT the result of a fallacy. The fallacy only happens when you try to use the experts as evidence. Experts are a substitute for evidence. 

"You get your choice: experts or evidence. If you choose evidence, you can't cite the experts. If you choose experts, you can't claim to be impartially evaluating the evidence, at least that part of the evidence on which you're deferring to the experts. 

"The experts are your agents -- if you look to them, it's because you are trusting them to evaluate the evidence in your stead. You're saying, "you know, your UFO arguments are extraordinary and weird. They might be absolutely correct, because you might have extraordinary evidence that refutes everyone else. But I don't have the time or inclination to bother weighing the evidence. So I'm going to just defer to the scientists who *have* looked at the evidence and decided you're wrong. Work on convincing them, and maybe I'll follow."  

In other words: it's perfectly legitimate to believe in climate change because the scientific consensus is so strong. It is also legitimate to argue with people who haven't looked at the evidence and have no firsthand arguments. But it is NOT legitimate to argue with people who ARE arguing from the evidence, when you aren't. 

That they're arguing first-hand, and you're not, doesn't necessarily mean you're wrong. It  just means that you have no argument or evidence to bring to the table. And if you have no evidence in a scientific debate, you're not doing science, so you need to just ... well, at that point, you really need to just shut up.

The climate change debate is interesting that way, because, most of the activist non-scientists who believe it's real really haven't looked at the science enough to debate it. A large number have *no* firsthand arguments, except the number of scientists who believe it. 

As a result, it's kind of fun to watch their frustration. Someone comes up with a real argument about why the data doesn't show what the scientists think it does, and ... the activists can't really respond. Like me, most have no real understanding of the evidence whatsoever. They could say, like I do to the UFO people, "prove it to the scientists and then I'll listen," but they don't. (I suspect they think that sounds like they're taking the deniers seriously.)

So, they've taken to ridiculing and name-calling and attacking the deniers' motivations. 

To a certain extent, I can't blame them. I'm in the same situation when I read about Holocaust deniers. I mean the serious ones, the "expert" deniers, the ones who post blueprints of the death camps and prepare engineering and logistics arguments about how it wasn't possible to kill that many people in that short a time. And what can I do?  I let other expert historians argue their evidence (which fortunately, they do quite vigorously), and I gnash my teeth and maybe rant to my friends.

That's just the way it has to be. You want to argue, you have to argue the evidence. You don't bring a knife to a gunfight, and you don't bring an opinion poll to a scientific debate.

Sabermetrics vs. second-hand knowledge

Does the earth revolve around the sun, or does the sun revolve around the earth?

The earth revolves around the sun, of course. I know that, and you know that.

But do we really? 

If you know the earth revolves around the sun, you should be able to prove it, or at least show evidence for it. Confronted by a skeptic, what would you argue?  I'd be at a loss. Honestly, I can't think of a single observable fact that I could use to make a case.

I say that I "know" the earth orbits the sun, but what I really mean by that is, certain people told me that's how it is, and I believe them. 

Not all knowledge is like that. I truly *do* know that the sun rises in the east, because I've seen it every day. If a skeptic claimed otherwise, it would be easy to show evidence: I'd make sure he shared my definition of "east," and then I'd wake him up at 6 am and take him outside.

But that sun/earth thing?  I can only I only say I "know" it because I believe that astronomers *truly* know it, from direct evidence.

------

It occurred to me that almost all of our "knowledge" of scientific theories comes from that kind of hearsay. I couldn't give you evidence that atoms consist, roughly, of electrons orbiting a nucleus. I couldn't prove that every action has an equal and opposite reaction. There's no way I could come close to figuring out why and how e=mc^2, or that something called "insulin" exists and is produced by the pancreas. And I couldn't give you one bit of scientific evidence for why evolution is correct and not creationism. 

That doesn't stop us from believing, really, really strongly, that we DO know these things. We go and take a couple of undergraduate courses in, say, geology, and we write down what the professors tell us, and we repeat them on exams, and we solve mathematical problems based on formulas and principles we are told are true. And we get our credits, and we say we're "knowledgeable" in geology. 

But it's a different kind of knowledge. It's not knowledge that we have by our own experience or understanding. It's knowledge that we have by our own experience of how to evaluate what we're told -- how and when to believe other people. We extrapolate from our social knowledge. We believe that there are indeed people, "geologists," who have firsthand evidence. We believe that evidence gets disseminated among those geologists, who interact to reliably determine which hypotheses are supported and which ones are not. We believe that, in general, the experts are keeping enough of a watchful eye on what gets put in textbooks and taught at universities, that if Geology 101 was teaching us falsehoods, they'd get exposed in a hurry.

In other words, we believe that the system of scientists and professors and Ph.D.s and provosts and deans and journals and textbook publishers is a reliable separator of truth from falsehood. We believe that, if the earth really were only 6,000 years old, that's what scientists would be telling us.

------

Most of the time, it doesn't matter that our knowledge is secondhand. We don't need to be able to prove that swallowing arsenic is fatal; we just need to know not to do it. And, we can marvel at Einstein's discovery that matter and energy are the same thing, even if we can't explain why.

But it's still kind of unsatisfying. 

That's one of the reasons I like math. With math, you don't have to take anyone's word for anything. You start with a few axioms, and then it's all straight logic. You don't need geology labs and test tubes and chemicals. You don't need drills and excavators. You don't actually have to believe anyone on indirect evidence. You can prove everything for yourself.

The supply of primes is infinite. No matter how large a prime you find, there will always be one larger. That's a fact. If you like, you can look it up on the internet, or ask your math teacher, or find it in a textbook. It's a fact, like the earth revolving around the sun.

If you do it that way, you know it, but you don't really KNOW it. You can't defend it. In a sense, you're believing it on faith. 

On the other hand, you can look at a proof. Euclid's proof that there is no largest prime number is considered one of the most elegant in mathematics. The versions I found on the internet use a lot of math notation, so I'll paraphrase.

-----

Suppose you have a really big prime number, X. The question is: is there always a prime bigger than X?  

Try this: take all the numbers from 1 to X, and multiply them together: 1 times 2 times 3 .... times X. Now, add 1. Call that really huge number N. That huge N is either prime, or is the product of some number of primes. 

But N can't be divisible by X, or anything less than X, because that division has to always leave a remainder of 1. Therefore: either N is prime, or, when you factor N into other primes, they're all bigger than X. 

Either way, there is a prime bigger than X.

------

I may not have explained that very well. But, if you get it ... now you know that there is no highest prime. If you read it in a book, you "know" it, but if you understand the proof, you KNOW it, in the sense that you can explain it and prove it to others.

In fact ... if you read it in a textbook, and someone tells you the textbook is wrong, you may have some doubt. But once you see the proof, you will *never* have doubt (except in your own logic). Even if the greatest mathematician in the world tells you there's a largest prime, you still know he's wrong. 

-----

In theory, everything in math is like that, provable from axioms. In practice ... not so much. The proofs get complicated pretty quickly. (When Andrew Wiles solved Fermat's Last Theorem in 1993, his proof was 200 pages long.)  Still, there are significant mathematical results where we can all say we know from our own efforts. For years, I wondered why it was that multiplication goes both ways -- why 8 x 7 has to equal 7 x 8. Then it hit me -- if you draw eight rows of seven dots, and turn it sideways, you get seven rows of eight dots.

There are other fields like math that way ... you and I can know things on our own, fairly easily, in economics, and finance, and computer science. Other sciences, like physics and chemistry, take more time and equipment. I can probably prove to myself, with a stopwatch and ruler, that gravitational acceleration on earth is 9.8 m/s/s, but there's no way I could find evidence of what it is on the moon. 

But: sabermetrics. What started me on all this is realizing that the stuff we know about sabermetrics is more like infinite primes than like the earth revolving around the sun. Active researchers don't just know sabermetrics because Bill James and Pete Palmer told us. We know because we actually see how to replicate their work, and we see, all the way back to first principles, where everything came from. 

I can't defend "e equals mc squared," but I can defend Linear Weights. It's not that hard, and all I need is play-by-play data and a simple argument. Same with Runs Created: I can pull out publicly-available data and show that it's roughly unbiased and reasonably accurate. (I can even go further ... I can take partial derivatives of Runs Created and show that the values of the individual events are roughly in line with Linear Weights.)

DIPS?  No problem, I know what the evidence is, there, and I can generate it myself. On-base percentage more important than batting average?  Geez, you don't even need data for that, but you can still do it formally if you need to without too much difficulty. 

For my own part -- and, again, many of you active analysts reading this would be able to say the same thing --  I don't think I could come up with a single major result in sabermetrics that I couldn't prove, from scratch, if I had to. Even the ones from advanced data, or proprietary data, I'm confident I could reproduce if you gave me the database.

For all the established principles that are based on, say, Retrosheet-level data ... honestly, I can't think of a single thing in sabermetrics that I "know" where I would need to rely on other people to tell me it's true. That might change: if something significant comes out of some new technique -- neural nets, "soft" sabermetrics, biomechanics -- I might have to start "knowing" things secondhand. But for now, I can't think of anything.

If you come to me and say, "I have geological proof that the earth is only 6,000 years old," I'm just going to shrug and say, "whatever."  But if you come to me and say, "I have proof that a single is worth only 1/3 of a triple" ... well, in that case, I can meet you head on and prove that you're wrong. 

I don't really know that creationism isn't right -- I only know what others have told me. But I *do* know firsthand what a triple is worth, just as I *do* know firsthand that there is no highest prime. 

------

And that, I think, is why I love sabermetrics so much -- it's the only chance I've ever had to actually be a scientist, to truly know things directly, from evidence rather than authority.

I have a degree in statistics, but if nuclear war wiped out all the statistics books, how much of that science could I restore from my own mind?  Maybe, a first-year probability course, at best. I could describe the Central Limit Theorem in general terms, but I have no idea how to prove it ... one of the most fundamental results in statistics, one they teach you in your first statistics class, and I still only know it from hearsay.

But if nuclear war wipes out all the sabermetrics books ... as long as someone finds me a copy of the Retrosheet database, I can probably reestablish everything. Nowhere near as eloquently as Bill James and Palmer/Thorn, and I'd probably wouldn't think of certain methods that Tango/MGL/Dolphin did, but ... yeah, I'm pretty sure I could restore almost all of it. 

To me, that's a big deal. It's the difference between knowing something, and only knowing that other people know it. Not to put down the benefits of getting knowledge from others -- after all, that's where most of our useful education comes from. It's just that, for me, knowing stuff on my own ... it's much more fulfilling, a completely different state of mind. As good as it may be to get the Ten Commandments from Moses, it's even better to get them directly from God.