Sabermetrics and Malcolm Gladwell's "Blink"

I just finished reading "Blink," my Malcolm Gladwell. (The book actually came out in 2005, but I'm slow.)

The book's subtitle is "The Power of Thinking Without Thinking." Purportedly, it's about intuition and non-analytical ways of solving problems. But I took away something different. To me, the common thread in the book is that sometimes, the way we try to solve complex problems doesn't always lead to the best result, and simpler methods, that you wouldn't think would be good enough, work much better.

What are these other simpler methods? The one most mentioned in reviews of the book I've seen is plain intuition. When the Getty Museum was approached to purchase what was purportedly an ancient Greek statue in 1983, they did all the requisite scientific analysis to rule out forgery. The marble was indeed of a kind found in Greece, and the surface had tarnished exactly as you'd expect of a statue that old.

But when experts viewed it, they immediately, within two seconds, knew it was a fake. How? They couldn't explain, and still can't. It just seems that with their years of expertise, they immediately felt something wasn't right. Their exact methods were within the "locked door" of their consciousness, and they still can't explain how they knew they were looking at a forgery.

So from the book's title, and the first chapter, and the reviews, you'd think the book was about how intuition trumps analysis – that scouts (to switch to a sports analogy) can spot things that sabermetricians can't. But, surprisingly, the rest of the book is kind of the reverse. It's not about how analysis isn't as good as intuition. It's about how a lot of analysis is wrong, and how simpler analyses are better. Instead of "intuition is better than analysis," it's more like "the kind of analysis that your intution suggests is wrong." To me, aside from the first chapter, the book vindicates sabermetrics over traditional scouting.

For instance, take Gladwell's example of trying to figure out whether a particular married couple will stay together or get divorced. When psychologists were allowed to watch and listen to the couples converse, they were able to guess right only 50% of the time, no better than luck.

But one researcher, John Gottman, figured out a better way. He analyzed a bunch of conversations and scored the partners on twenty different aspects of their argument – disgust, contempt, anger, defensiveness, stonewalling, whining, and so forth. Based on that coding, he was able to find a formula that was accurate over 95% of the time, given only an hour's worth of data. It turns out that "contempt" is the number one predictor of divorce; "stonewalling," "defensiveness," and "criticism" are the next three (in some order). From those four measurements, you can apparently get a phenomenal predictor.

The obvious analogy is forecasting a baseball player's performance. A scout, or a sportswriter, would look at the guy, and see how fast he is, and how smooth his swing looks, and what his batting stance looks like, and so on. But if you know what's relevant and what's irrelevant, you can limit yourself to a few basic statistics, and the guy's age; you wind up with something like the Marcels, with accuracy pretty close to the maximum.

Most of the book similarly supports the idea of analysis – and that the analysis that works better is often a lot simpler than the analysis that doesn't. When a patient walks into the emergency room with chest pain, how do you know if it's a serious heart problem or not? Every doctor uses his own intuition, with mixed results. But, at one Chicago hospital, an admistrator named Brendan Reilly came up with a simple algorithm that beat the doctors handily – 95 percent correct, versus "between 75 and 89 percent" correct. As Gladwell writes,

"For all the rigor of his calculations, it seemed that no one wanted to believe what he was saying, that an equation could perform better than a trained physician."

That seems like a defense of analysis over intuition rather than the other way around, doesn't it?