Consumer Reports is starting to bug me

Consumer Reports magazine has been frustrating me recently. They seem to know what they’re doing when it comes to testing products, but in anything involving numbers or analysis – consumer sabermetrics, say – they come up short.

Here are three examples from the October, 2007 issue, which arrived in my mailbox last week. These aren’t the only things in the issue that bother me, but they’re the three biggest.

----

Jack is a hypothetical retiree in a sidebar in CR's "Your Money" column. The question: should Jack start collecting Social Security at age 62, or should he wait until age 70?

If he starts at 62, he'll receive $19,320 a year (adjusted annually for inflation). Or, if he waits until age 70, he'll start receiving $35,240 per year. Which is better, assuming Jack will live to age 90?

According to the "CR Money Lab," the first option gives Jack total payments of $542,570 (in 2007 dollars). That's 28 years times $19,320. The second option gives him 20 years times $35,240, which works out to $709,745. (These numbers aren't exactly correct; I assume there are rounding errors somewhere.)

28 years times $19,320 = $542,570
20 years times $35,240 = $709,745

Therefore, CR says, the second option is better.

Has CR never heard of the time value of money? A dollar today is worth more than a dollar eight years in the future, because of interest. CR’s calculation treats them as equal, in violation of basic principles of Finance 101. As Steven E. Landsburg writes (perhaps a little too harshly): "when college sophomores treat a dollar paid 20 years from now as the equivalent of a dollar paid today, we usually advise them that they have no talent for economics."

Assuming a real discount rate of 3%, the first option gives Jack $381,842 (in 2007 dollars, adjusted for inflation and time). The second option is worth $314,033. It’s actually the first option that’s better!

The higher the discount rate, the better the first option looks. If you use 4%, it wins by $341K to $237K. To get the two options equal, you have to lower the discount rate to about 1.8%.

Which option to choose depends on your personal discount rate, which is another way of saying it's your personal situation and preference. As CR does admit, your economic circumstances are a factor in which option is better for you.

Here’s one last way of looking at it. Suppose you don’t need the money until age 70, but decide on the first option anyway. From age 62 to 69, you invest all your benefits at a real rate of 4%. At age 70, those eight years of benefits will have grown to $185,139.

Now, even after 70, you’ll still be receiving only $19,320 a year, as opposed to the $35,240 you would have got had you waited. Your shortfall is $15,920 per year. But the lump sum of $185,139 should cover the shortfall for quite a few years, especially if invested well. You could probably even buy an annuity with that $185K that would top up your annual benefit, bringing it close to what you would have got by waiting.

Indeed, this kind of actuarial logic is probably how the government sets the benefits in the first place: so no matter which of the two options you take, the expected total payout, adjusted for time value, is roughly the same. CRs oversimplified calculation is simply not correct.

----

In the October issue’s letters section, a doctor writes in to explain that when drug companies send him free samples, he uses them "to let patients try the medication for free to see if it is effective and tolerable."Sounds like a win/win, right? Not to CR. An unnamed editor replies,

"Free samples are not free. They are part of the drug company’s advertising budget and contribute to the high cost of drugs. The free sample is a tool to tune the patient in to brand-name recognition, so that when it runs out they will stick with the same brand, despite the expense. There might be less-expensive drugs that are just as effective."

This is wrong.

First, the free samples might actually lower the cost of the drug. Most of the drug companies’ costs are in R&D; it can take many millions of dollars in research to come up with one marketable drug. But once the drug is approved, the cost of actually making the pills is minimal. So the more patients taking the drug, the lower the drug company can price it and still recoup their costs. If the free samples prove to patients that the drug will help them, the drug sells more, and prices can come down.

Suppose that without free samples, the demand curve is such that 1,000 people will use the drug at $100, but 1,500 will use the drug at $60. The company would make more money pricing the drug at $100. Suppose that after giving out samples, 2,000 will use the drug at $100, but 5,000 will use the drug at $60. Now, the company will drop the price to $60, so it can earn $300,000 instead of $200,000.

CR should look at it this way: if I were the only reader of their magazine, could they afford to sell it for $3 a copy?

Second, we patients aren’t as dumb as CR thinks we are. If we like our free sample of drug X, we might stick to it even though we know that there might be cheaper drugs that work too. But, in most cases, it’s just not worth the risk. If drug X is $100 a month, and drug Y is $80 a month, it’s perfectly reasonable to pay the extra $20 for a drug you know will work, instead of taking a chance on an untried alternative. (This arguably doesn't apply to generics, which are actually the same compound as the brand name. But CR isn’t talking about generics.)

Medicine is not a game, where the objective is to experiment with treatments to make the drug companies’ profits as low as possible. Switching to a new drug, if the old one works just fine, can be rather unpleasant. If a $1 pill keeps my diarrhea under control, I ain’t switching to a different compound just because it only costs 90 cents. At least not before a big date, or a job interview.

And keep in mind that these are prescription drugs we’re talking about, drugs the patient can’t get without a doctor’s approval. If there are cheaper alternatives, I expect my doctor to recommend them. If my insistence on the same drug doesn’t make medical sense, I expect my doctor to try to talk me out of it. But if I’m happy to get a free sample, and the doctor is happy to offer it to me, and my doctor agrees with my decision to stay on it after the sample runs out - well, who is Consumer Reports to overrule my relationship with my doctor? It’s my money and my suffering, and not CR’s.

Indeed, can you imagine what CR would say if HMOs started forcing patients to give up their preferred medication for a different, but cheaper, one?

Strangely enough, CR’s argument doesn’t seem to apply to their own products. In this very same issue, on page 38, they offer me a "risk-free sample issue" of "CR on Health." If I like it, I have to pay for more.

What makes CR’s offer so much more reasonable than a drug company’s? Isn't it true that free issues are not free? Once CR stops with the expense of giving away free issues, won't the price of a subscription drop? And how come they're not worried that "when my free sample issue runs out I will stick with the same magazine?" After all, there are many less-expensive alternatives than $39 a year.

----

In an article entitled “Make your car last 200,000 miles” (subscription required), CR compares the costs of keeping a new car 15 years, as opposed to replacing your new car every five years. Here are their numbers. The first column is one car over 15 years; the second column is three cars over 15 years.


$19,000 $66,400 Total car price (Honda Civic EX)
------------------------------------
$14,900 $35,500 Depreciation
$18,300 $ 6,200 Maintenance/Repairs
$ 3,100 $ 8,500 Finance/Interest
$ 2,200 $ 4,800 Fees/Taxes
$18,500 $22,500 Insurance
---------------------------
$57,000 $77,500 Total

$20,500 savings
$10,300 potential investment savings
------------------------------------
$30,800 total potential savings


What CR is saying is that, by keeping your Civic for 15 years instead of trading in every five years, you can save $20,500. Plus, if you invest that $20,500, you can earn an extra $10,300 from the interest (at 5% per annum). So the total savings are $30,800.

But there are a couple of things here that don’t make sense. First of all, CR claims that if you pay $19,000 for a Civic, and keep it for 15 years, it will only depreciate $14,900 – which means it’s still worth $4,100. Could that be right? A lot of Civics will rust out over 15 years, and be worthless. Even if similar cars for sale right now are worth $4,100 (or about $2,600 before 15 years’ inflation), there’s a selection bias – most ‘92 Civics are long dead, especially around here with our salty winter roads. (I can tell you for sure that my 8-year-old Sunfire is going to be worth a lot less than 21% of its original price by the time it hits 15.)

Second, where do the investment returns come from? CR is calculating $10,300 in interest on $20,500 principal, which is a 50% return. At 5%, the average dollar of the $20,500 savings would have to compound for 8 years.

But that’s not the case. Over the first five years, both options are absolutely identical, so there’s no savings then. All the savings accrue over the next ten years. If you assume the money is saved equally over each month of those ten years, the average compounding time is five years. For repairs, the savings might slightly skew to the middle five years, since breakdowns are probably more frequent in years 11-15. On the other hand, insurance is lower in years 11-15. Also, the absolute numbers are higher in years 11-15 because of inflation. All these seem to roughly balance out, so let’s assume that our orignal 5-year number is correct.

Five years compounding brings the interest only to about $5,700, far less than the $10,300 claimed.

(Update: this method is an approximation: a better method gives $5,921 instead of $5,700. See comment 5 to this post.)
I could be wrong about all this, of course, but if I’m right, CR is making overstated financial projections that no used-car dealer would be allowed to get away with.

-----

So what’s the deal with CR? They seem to have experts who understand how to deal with testing and evaluating real products, but when it comes to finance and economics, where you have to use logic instead of instruments ... well, if there were a Consumer Reports for Consumer Reports, I think they’d rate a “Not Acceptable.”
-----

UPDATE, 9/12

Here's one more.

In their “Claim Check” feature this month, CR investigates a claim by Tide that you can save “up to $65 a year” in energy costs by using its Tide Coldwater detergent. The savings accrue because the product lets you wash in cold water instead of warm.

CR verifies that Proctor & Gamble are telling the truth: $65 is indeed correct.

Case closed? Nope. CR complains anyway:

“But hold on. With the qualifier 'up to [$65],' the claim is valid even if you save a penny a year.”

That’s literally true, but any reasonable consumer knows what “up to” means. It means "as much as, under the right conditions, but yours will probably be less." Here, the conditions are very reasonable: indeed, if you live in a city with higher-than-average electric rates, you’ll save *more* than $65.

Besides, if the claim was way off, and P&G defended it on the "penny a year" grounds, CR would be all over them, accusing them of misleading the public.

So I would have thought this to be a perfectly legitimate and ethical use of “up to.” Sniping at Tide here is completely inappropriate.

Speaking of energy savings: three pages later in the magazine is an ad for a new book, “Consumer Reports Complete Guide to Reducing Energy Costs.” Why should you buy this book? Because CR says you can

"SAVE UP TO $1000 A YEAR."

Seriously. Three pages after gratuitously bitching about "up to," they use it in an ad for their own product!