Rating battery life on a 100-point scale
I've written before about how Consumer Reports (CR) uses a 100-point system for its product ratings. In their latest issue, they use that same system to rate AA batteries, and I suspect the ratings turned out so misleading that CR wound up fooling its own editorial staff!
CR rated 13 brands of alkaline batteries, and two brands of lithium batteries. In the alkaline category, Kirkland Signature (Costco's house brand) was rated "best buy." It was the third-best alkaline, and, at 27 cents a battery, the least expensive by far. Most of the others were between $0.75 and $1.00 (although they would have been cheaper if CR had priced them in a bulk pack, like the Kirkland).
The two lithium batteries rated the highest of all, but they cost more than $2 each.
Now, suppose I'm not near a Costco, and need batteries. My choice is between the high-rated Duracell alkaline, at $1.20, and the Energizer Ultimate Lithium, at $2.50. Which should I buy?
There's no way to tell from the article. Why? Because all we have is that 100-point scale. That doesn't help much. Why doesn't CR just tell us how long each battery lasted, so we can do our own cost/benefit calculation?
It's not quite that simple, you could argue. Batteries perform differently in "high drain" and "low drain" applications. CR tested both -- it used a flashlight for its low-drain test, and a toy for its high-drain test. Then it combined the two, somehow, to get the rating. But, couldn't they have combined them in such a way that the ratings are roughly proportional to how long the batteries last?
I found a 2012 "battery showdown", from BitBox, that gives you actual data. Here's their graph of how much power you get from different brands of battery at high-drain (before the voltage drops below 0.8V). The lithiums are the two at the top, the alkalines are the large cluster in the middle, and the cheap carbon-zinc batteries (which CR didn't test) are the poor performers at the bottom.
Looking at their chart of numbers ... the Energizer Ultimate Lithium, it appears, lasts around 3.1 times as long as the Costco alkaline in high-drain applications. At low-drain, the lithium lasts 1.7 times as long.
That's consistent with what I had previously understood -- that lithium batteries are by far the best, but shine more in high-drain applications than low-drain applications.
Strangely, the CR chart might lead you to expect exactly the opposite! CR rated the lithium batteries "excellent" (their maximum rating) in both applications. That "tied" eight of the 13 alkalines in the high-drain test, but only one in the low-drain test. Based on those ratings, a reader would be forgiven for concluding that lithium batteries give you more leverage in low-drain uses. (In fairness, the text of the article does give the correct advice, although it doesn't explain why the chart seems to imply otherwise.)
Anyway, combining the two factors, 3.1 and 1.7, we might choose to conclude that the lithiums last maybe two-and-a-half times as long as the alkalines.
But CR's ratings give no clue that the difference is that large. All they tell us is that the lithium grades a 96/100, and the Costco alkaline grades an 84/100. In other words: CR gives the lithium 14% more points for 150% more performance.
Which, I guess, has to be the case, given the rating system. If you give the lithium a perfect score of 100, you'd have to give the alkalines 40 or less. And they can't do that, since, to CR, 40/100 can only mean "poor."
The article goes on to say,
"The top-scoring  alkaline battery -- Duracell Quantum -- was not significantly different from the high-scoring [94 and 96] lithium models ..."
That, I believe, is just plain false. A quick Google search of bloggers who tested the Quantums suggest that, at best, they're a bit better than other alkalines, but nowhere near as good as lithiums. So, CR winds up telling us a battery that lasting twice as long does not make a battery "significantly different."
Well, it might have been a misapplication of the normal criteria for "significantly different." In their longer ratings articles, CR includes a disclaimer in their ratings: "differences of fewer than X points aren't meaningful."
For the batteries ... sure, lower in the rankings, five points isn't significant. I fully agree that the Rayovac at 78/100 isn't significantly different from the CVS at 82/100. But it's absolutely not true that the Quantum at 91/100 is anywhere near as good as the lithium at 94/100. The rating system might work in the middle, but it fails at the top.
That's how, I think, CR wound up fooling itself. The writers looked at the ratings, and thought, "hey, it's only three points!"