Rating systems and rationalizations

The Bill Mazeroski Baseball Annuals, back in the 80s, had a rating system that didn't make much sense.  They'd assign each team a bunch of grades, and the final rating would be the total.  So you'd get something out of 10 for outfielders, something out of 10 for the starting pitching staff, and something out of 10 for the manager, and so on.  Which meant, the manager made as much difference as all three starting outfielders combined.

Maclean's magazine's ratings of Canadian universities are just as dubious.  Same idea: rate a bunch of categories, and add them up.   I've been planning to write about that one for a while, but I just discovered that Malcolm Gladwell already took care of it, three years ago, for similar American rankings.

In the same article, Gladwell also critiques the rating system used by "Car and Driver" magazine.

In 2010, C&D ran a "comparo" of three sports cars -- the Chevy Corvette Grand Sport, the Lotus Evora, and the Porsche Cayman S.  The Cayman won by several points:

193 Cayman
186 Corvette
182 Evora

But, Gladwell points out, the final score is heavily dependent on the weights of the categories used.  Car and Driver assigned only four percent of the available points to exterior styling.  That makes no sense: "Has anyone buying a sports car ever placed so little value on how it looks?"

Gladwell then notes that if you re-jig the weightings to make looks more important, the Evora comes out on top:

205 Evora
198 Cayman
192 Corvette

Also, how important is price?  The cost of the car counted for only ten percent of C&D's rating.  For normal buyers, though, price is one of the most important criteria.  What happens when Gladwell increases that weighting relative to the others?

Now, the Corvette wins.

205 Corvette
195 Evora
185 Cayman

------

Why does this happen?  Gladwell argues that it's because Car and Driver insists on using the same weightings for every car in every issue in every test.  It may be reasonable for looks to count for only four percent when you're buying an econobox, but it's much more important for image cars like the Porsche.  

"The magazine’s ambition to create a comprehensive ranking system—one that considered cars along twenty-one variables, each weighted according to a secret sauce cooked up by the editors—would also be fine, as long as the cars being compared were truly similar. ...  A ranking can be heterogeneous, in other words, as long as it doesn’t try to be too comprehensive. And it can be comprehensive as long as it doesn’t try to measure things that are heterogeneous. "

I think Gladwell goes a bit too easy on Car and Driver.  I don't think the entire problem is that the system tries to be overbroad.  I think a big part of the problem is that, unless you're measuring something real, *every* weighting system is arbitrary.  It's true that a system that works well for family sedans might not work nearly as well as for luxury cars, but it's also true that the system doesn't necessarily work for eiher of them separately, anyway!

It's like ... rating baseball players by RBIs.  Sure, it's true that this system is inappropriate for comparing cleanup hitters to leadoff men.  But even if you limit your evaluation to cleanup hitters, it still doesn't do a very good job. 

In fact, Gladwell shows that explicitly in the car example.  His two alternative weighting systems are each perfectly defensible, even within the category of "sports car".  Which is better?  Which is right?  Neither!  There is no right answer.

What I'd conclude, from Gladwell's example, is that rating systems are inappropriate for making fine distinctions.  Any reasonable system can tell the good cars from the bad, but there's no way an arbitrary evaluation process can tell whether the Evora is better than the Porsche.  It would always be too sensitive to the weightings.

In fact, you can always make the result come out either way, and there's no way to tell which one is "right."  In fact, there's no "right" at all, because "better" has no actual definition.  Your  inexpressible intuitive view of "better" might involve a big role for looks, while mine might be more weighted to handling.  Neither of us is wrong.

However: most people's definitions of "better" aren't *that* far from each other.  We may not be able to agree whether the Porsche is better than the Corvette, but we definitely can agree that both are better than the Yugo.  Any reasonable system should wind up with the same result.

Which, in general, is what rating systems are usually good for: identifying *large* differences.  I may not believe Consumer Reports that the Sonata (89/100) is better than the Passat (80) ... but I should be able to trust them when they say the Camry (92) is better than the Avenger (43).  

-------

In the March, 2004, issue, Car and Driver compares six electric cars.  The winner was the Chevrolet Spark EV, with 181 points out of 225.  The second place Ford Focus Electric was only eight points behind, at 173.

That's pretty typical, that the numerical ratings are close.  They're always much closer than they are in Consumer Reports.  I dug out a few back issues of C&D, and jotted down the comparo scores.  Each row below is a different test:

189 - 164
206 - 201 - 200 - 192
220 - 205
196 - 190 - 184 - 179

All are pretty close -- the biggest gap from first to last is 15 percent.  Although, I deliberately left out the March issue: there, the gap is bigger, mostly because of the electric Smart car, which they didn't like at all:

181 - 173 - 161 - 157 - 153 - 126

Leaving out the Smart, the difference between first and last is 18 percent.  (For the record: Consumer Reports didn't rate the electric Smart, but they gave the regular one only 28/100, the lowest score of any car in their ratings.)

Anyway, as I said, the Spark beat the Focus by only 8 ratings points, or five percent.  But, if you read the evaluations of those two cars ... the editors like the Spark *a lot more* than the Focus.  

Of the Spark, they say,

"Here's a car that puts it all together ... It's a total effort, a studied application of brainpower and enthusiasm that embraces the electric mandate with gusto ... Everything about the Spark is all-in. ... It is the one gold that sparkles."

But they're much more muted when it comes to the Focus, even in their compliments:

"The most natural-feeling of our EVs, the Focus delivers a smooth if somewhat muted rush of torque and has excellent brakes. ... At low speeds ... you can catch the motor clunking ... but otherwise the Focus feels solid and well integrated. ... What the Focus Electric really does best is give you a reason to go test drive the top-of-the-line gas-burning Focus."

When Car and Driver actually tells you what they think, it sounds like the cars are worlds apart.  All that for eight points?  Actually, it's worse than that: the Spark had a price advantage of seven points.  So, when it comes to the car itself, the Chevy wins by only *one point* -- but gets much, much more appreciation and plaudits.

What's going on?  Gladwell thinks C&D is putting too much faith in its own rating:

"Yet when you inspect the magazine’s tabulations it is hard to figure out why Car and Driver was so sure that the Cayman is better than the Corvette and the Evora."

I suspect, though, that it's the other way around: after they drive the cars, they decide which they liked best, then tailor the ratings to come out in the right order.  I suspect that, if the ratings added up to make the Focus the best, they'd say to each other, "Well, that's not right!  There must be something wrong with our numbers."  And they'd rejig the category scores to make it work out.

Which probably isn't too hard to do, because, I suspect, the system is deliberately designed to keep the ratings close.  That way, every car looks almost as good as the best, and everybody gets a good grade.  A Ford salesman can tell his customer, "Look, we finished second, but only by 8 points ... and, 7 of them were price!  And look at all the categories we beat them in!"

That doesn't mean the competition is biased.  The magazine is just making sure the best car wins.  Car and Driver is my favorite car magazine, and I think the raters really know their stuff.  I don't want the winner to go the highest-scorer of an arbitrary point system ... I want the winner to be the one the magazine thinks is best.  That's why I'm reading the article, to get the opinions of the experts.  

So, they're not "fixing" the competition, as in making sure the wrong car wins.  They ARE "fixing" the ratings --  but in the sense of "repairing" them.  Because, if you know the Spark is the best, but it doesn't rate the highest, you must have scored it wrong!  Well, actually, you must have chosen a lousy rating system ... but, in this case, the writer is stuck with the longstanding C&D standard.

-------

"Fixing" the algorithm to match your intuition is probably a standard feature of ranking systems.  In baseball, we've seen the pattern before ... someone decides that Jim Rice is underrated, and tries to come up with a rating that slots him where his gut says he should be slotted.  Maybe give more weight to RBIs and batting average, and less to longevity.  Maybe add in something for MVP votes, and lower the weighting for GIDP.  Eventually, you get to a weighting that puts Jim Rice about as high as you'd like him to be, and that's your system.

And it doesn't feel like you're cheating, because, after all, you KNOW that Jim Rice belongs there!  And, look, Babe Ruth is at the top, and so is Ted Williams, and a whole bunch of other HOFers.  This, then, must be the right system!

That's what always has to happen, isn't it?  Whether you're rating cars, or schools, or student achievement, or fame, or beauty, or whatever ... nobody just jots a system down and goes with it.  You try it, and you see, "well, that one puts all the small cars at the top, so we've rated fuel economy too high."  So you adjust.  And now you see, "well, now all the luxury cars rate highest, so we better increase the weighting for price."  And so on, until you look at the results, and they seem right to you, and the Jim Ricemobile is in its proper place.

That's another reason I hate arbitrary rankings: they're almost always set to fit the designer's preconceptions.  To a certain extent, rating systems are just elaborate rationalizations.