Thursday, June 16, 2016

Consumer Reports on Sunscreens

Consumer Reports' July, 2016 article on sunscreens promises to "expose startling truths about product claims and effectiveness."  Their most important claim is this:


"43% of sunscreens in our tests failed to meet the SPF claim on their labels."

Call me skeptical, but my first reaction was to suspect that their testing wasn't quite right. 

CR and I have quite different priors about how companies operate. I believe that companies are generally honest, and dishonest manufacturers and bad products get shunned by consumers quicker than CR thinks, and there's no way 43 percent of manufacturers are outright lying and ripping us off. 

But, sometimes, it seems like CR thinks the manufacturers would cheat us from head to toe if they thought they could get away with it. 

Here's how they tested:


"A standard amount of sunscreen is applied to small areas of our panelists' backs. Then they soak in a tub of water. Afterward, each of those areas is exposed to six intensities of UVB light from a sun simulator for a set time. About a day later, the six spots are examined for redness."

Aha! CR rated sunscreens only on how they performed after water exposure, like swimming and heavy rain. They didn't include "dry" protection, or even protection after sweating. Their accusation that "almost half" the products fail to "live up to the SPF claim on their labels" is based solely on performance after a soaking period. 

It took me a while to figure that out. The article does state that explicitly, but not in the main article. There are only two places that I can see where CR mentions it.

The first is in a full-page sidebar on page 27. But not in the main text. There, CR touts their "troubling" findings and demands FDA action, but doesn't mention they report and rate only on post-immersion performance. You have to look down into the smaller print, beside a couple of pie charts, to find that out. (It's in larger print on their website, which includes only a part of the full magazine article.)

The second is in the ratings. Well, not the ratings themselves -- the charts themselves are headed "Tested SPF" with no mention of water. You have to look at the footnotes.

In fairness to CR, they do note that every product does make claims of water resistance, either for 40 or 80 minutes, which is why they feel justified in testing this way. But ... they never say that the manufacturers guarantee the exact same SPF value after immersion. Couldn't it be that the sunscreens just vary in how water resistant they are?

In other words -- maybe the sunscreens just vary in water resistance! Isn't it possible that the manufacturers are claiming a "dry" SPF, and the products just vary in how well they perform after swimming? 

That was my first guess, but, probably not. I found a website that says the law *does* require the label SPF to be met after immersion. 

But, maybe the immersion tests are weaker than real life, the same way that cars usually get lower mileage than their EPA estimates. According to the website, the test involves submerging an arm in a Jacuzzi, then measuring the SPF afterwards. It could easily be that arms are not the same as backs, or a Jacuzzi is not the same as the tub CR used. 

Who knows, maybe, in their 40 or 80 minutes of soaking, CR's participants leaned their backs against the sides of the tub, and some of the lotion rubbed off? I'm betting CR was too careful to let that happen, but ... in any case, it would be nice if CR would consider that there might be something going on other than fraud. 

If it's true that the FDA does require independent lab testing, it wouldn't just be that 43 percent of manufacturers are lying, it would be that 43 percent of manufacturers are falsifying results. That's a strong accusation to be making, and, to me, not a very plausible one.

------

Even if CR's testing is perfectly accurate, why don't they mention the products' performance in other contexts? Nowhere in the article or sidebars or footnotes does CR mention, even once, how the products did in dry or sweaty conditions. Their accusation that "almost half" the products "fail to live up to the SPF claim on their labels" is based only on performance after soaking. 

I count at least five times where CR accuses the manufacturers of overreporting the SPF, but only one of those times does CR mention "after water immersion" at the same time.

And why don't they report on dry performance at all? In a second test, they report that they "smear sunscreen on plastic plates and pass UV light through and measure the amount of UVA and UVB rays that are absorbed."  So, they actually *do* have test results for UVB effectiveness when dry. 

They report on the dry UVA numbers -- those form the basis of their UVA rating --- but the dry UVB numbers are completely ignored.

(UVB is the part of the spectrum that's relevant to SPF; UVA is a more dangerous component of sunlight that doesn't cause burning or tanning, but still can lead to damage and cancer. CR chose to use a more strenuous test for UVA than the FDA requires. They didn't accuse any products of failing to meet their UVA claims, even those that rated "poor".)

Why don't they tell us how the products performed dry? Is it because, in non-immersive use, they do indeed live up to their claims? Wouldn't it be fair to mention that? 

Why bury the important qualifier that the failures are all related to post-immersion performance? 

-------

It must be that even for the best products, water resistance isn't perfect -- that the performance of all the sunscreens is reduced after a swim. 

In that case, if a product tests at SPF 50 after immersion, doesn't that mean it must have been higher than SPF 50 beforehand? That must be the case, right? Unless some products are so completely, perfectly unaffected by water that they don't degrade at all.

But if products didn't degrade in water ... well, first, wouldn't manufacturers claim more than 80 minutes of wet protection? Second, if natural friction with water isn't sufficient to degrade the layer of sunscreen even a little bit, wouldn't that make it really hard to rub off by other means, like a towel? And, third, if there's no degradation, why are we advised to reapply sunscreen after swimming?

My guess is, if Coppertone Water Babies SPF 50 "meets [its] claim" after 80 minutes of immersion, it must have been a lot more than SPF 50 beforehand. 

Why isn't that a problem for CR, that people are buying what they think is SPF 50, but it's actually SPF 100 when dry? Probably because CR thinks higher is always better. They rate the products by actual (wet) SPF, "not how close a sunscreen comes to meeting its SPF claim."

Which is OK, I guess. The difference between 50 and 100 isn't that big a deal. The SPF is the fraction of UVB the product allows through, so SPF 50 means you're exposed to only 1/50 of what you'd get without it, and SPF 100 means you're exposed to a 1/100 dose. Sure, you get twice as much UVB from SPF 50 as SPF 100, but it's twice a very small amount. The difference, after four hours in the sun, is the equivalent of 2.4 unprotected minutes. 

Fair enough. I can't see anyone saying, "I feel ripped off that my SPF 50 is actually SPF 100. I was counting on those 2.4 extra minutes to tan better."  If you wanted to get a tan, you'd probably use something lower, like SPF 4 or 8 or 15. (That's when you have a problem, that your SPF 4 is actually SPF 10 when dry, which *is* a big difference.)

In any case ... why does CR not advise, explicitly, that higher is better? They imply it, by using a "higher is better" standard in their ratings, but don't actually say, "just get a big number over a small number."

-------

Instead, CR advises you to use only products that are SPF 30 or higher. That "30" doesn't come from any first-hand evidence or argument, but because CR just defers to what the American Association of Dermatology recommends. 

Accordingly, CR didn't test or rate any products labelled lower than SPF 30.

What's strange is ... they don't even *mention the existence* of those other products! More importantly, they don't even mention why lower SPF products exist. I'm guessing it has something to do with tanning but not burning? Except that CR doesn't talk about the difference between a tan and a burn, except that "a burn and the tan are kind of the same response ... you've recieved an injury that can lead to cell mutations that trigger skin cancer."

Nowhere in the article does CR mention what to do if you want to tan as safely as possible.

CR's mission statement says they "empower consumers with the knowledge they need to make better and more informed choices". But, in this case, they don't seem to care about helping us make the choice of whether a tan is worth the risk. For pale caucasians like me, "Stay pasty white in order to be safe!," seems to be only valid choice in their eyes. 

-------

When a product failed to live up to its SPF label after immersion, CR reports the actual SPF they found, like those two kids' sunscreens that tested at 8 instead of 50.

But when a product *did* meet its claim, CR doesn't tell you the actual number. In that column on the chart, they just say, "Meets Claim".

Well, that's strange, isn't it? If an SPF 50 product tested at SPF 60 after immersion, you'd expect CR to say, "hey, this is better than advertised!"

Why don't they? Maybe it's part of their "companies are always out to rip us off" attitude, so they don't want to promote the idea that sometimes we get more than we pay for.

Or, maybe they think that if the manufacturer is only promoting SPF 50, that's all they're obligated to deliver. So, maybe this batch turned out to be SPF 60, but the next batch might not.

That would actually have some logic behind it. But that can't be CR's rationale. If it were, then, for those products, they'd base their ratings just on the label claims. But, instead, they seem to be actually basing their ratings on the full, observed SPF. 

We know they're doing that because: in the lotions chart, "California Baby Super Sensitive SPF 30+" has a "Meets Claim" and a UVB rating of "Good". But, "California Kids #supersensitive SPF 30+" also has a "Meets Claim," but a UVB rating of "Very Good". 

Clearly, they can't be basing the rating on the label "30+" claims, because then the ratings would be identical. So, they must be using the actual, observed SPF.

So, the mystery remains. Why won't they tell us about the products that deliver more than they promise?

-------

A fair bit of the article isn't about specific products, but about sunscreen use in general. And a lot of it doesn't make sense to me. For instance:


"It's not true that sunscreens with higher SPF block double or triple the rays as those with lower ones. They really only provide slightly more protection," [Dr. Mina] Gohara [associate clinical professor of dermatology at Yale School of Medicine] warns. The breakdown: SPF 30 blocks 97 percent of UVB rays, SPF 50 blocks 98 percent, and SPF 100 blocks 100 percent."

OK, sure. But you can recast that like this:


"It IS true that sunscreens with lower SPF allow double or triple the rays as those with higher ones. The breakdown: SPF 30 admits 3 percent of UVB rays, SPF 30 admits 2 percent, and SPF 100 admits only 1 percent."

Put that way, do you still want to say that higher-SPF products only give "slightly more protection?"  

Which perspective makes more sense -- 97 to 100 percent, or 1 to 3 percent? Do you care how much UVB gets rejected, or how much gets through?

Well, you do, in fact, get three times as much harmful UVB using the SPF 30 than you do using the SPF 100. Sure, it's three times a small number, and, sure, you expose yourself to *one hundred* times the harm if you don't use sunscreen at all. So, on the one hand, who cares if the SPF 30 is triple the exposure, if that exposure is small in the first place? 

But, CR isn't about to take that tack, that 1 percent or 3 percent of UVB is nothing to worry about. If they did, they wouldn't stop at SPF 30. Because, they could easily have made the same argument for SPF 20:


"The breakdown: SPF 20 blocks 95 percent of UVB rays, SPF 30 blocks 97 percent, SPF 50 blocks 98 percent, and SPF 100 blocks 100 percent."

But CR won't even consider recommending an SPF 20, even though it takes you 95 percent of the way to safety. It seems kind of arbitrary to insist that (a) 97 percent is acceptable, (b) going up to 99 percent doesn't matter much, but (b) dropping to 95 percent is too dangerous to even talk about.

-------

It's not that hard to quantify the SPF difference in other terms. 

Suppose you spend 8 hours in the sun. With SPF 50, it's like spending 9.6 minutes unprotected, since 8 hours divided by 50 equals 9.6 minutes.

With SPF 100, it's like spending 4.8 minutes unprotected. 

Either way, you're not going to get a sunburn, right? But, maybe 4.8 minutes is still harmful, to a small extent. And maybe that 4.8 minutes worth of damage is cumulative, like smoking. Maybe you don't get cancer from a one-time 4.8-minute session, but expose yourself regularly and your odds get much worse. 

So, is the damage cumulative? CR doesn't tell us. They imply that it is, but their argument doesn't say what they think it does. They hypothetically ask, "If I've never used sunscreen my whole life ... isn't the damage already done?"  And then they respond with a completely irrelevant factoid: 


"Hardly. For years, it was believed that we got the majority of our lifetime dose of UVA and UVB exposure before the age of 18. But now experts know that it's cumulative over your lifetime. By age 40, you've received just 47 percent of your lifetime sun exposure. ... The upshot: It's never too late to start protecting your skin."

That sounds like it's answering the question, but it's not. The "cumulative" here talks only about exposure, not damage.

It's more obvious if you make the same "exposure" argument for car accidents: 


"By age 40, you've driven just 40 percent of your lifetime miles behind the wheel ... so it's never too late to start wearing a seat belt."  

That's true, but that obviously doesn't mean the miles you drove beltless before you were 40 are going to cause you damage in the future.

-------

From the main discussion of sunscreen use in general, here's something that didn't make sense to me:

"Not realizing [the small differences in higher SPF values] may lead people to think that if they use a higher PDF, they don't need to reapply or practice other sun-savvy behaviors, such as seeking the shade or covering up."

Well, I'm "led to think that" because it has to be somewhat true, doesn't it, that if you use a higher SPF, you probably don't have to reapply as much? Not because the higher SPF lotion is more durable, but as it breaks down, there's more protection left over.

The standard advice is to reapply every two hours. But it's not like the protection turns into a pumpkin right at the 2:00 mark. It drops gradually. Suppose, for the sake of argument, that after two hours, you've lost half your sunscreen to friction and sweat, and your SPF 50 is an SPF 25. (This seems reasonable, since CR tells us that if we use half the recommended amount of sunscreen, we get half the SPF.)

If you had used an SPF 100 instead, then after two hours, you're still at SPF 50. That's still pretty good. 

I'd be tempted to get an SPF 100 and reapply every four hours instead of getting an SPF 50 and reapplying every two hours, just for convenience. Would that work? CR doesn't care, or won't tell me. 

But maybe the protection *does* turn into a pumpkin after two hours, after all. Maybe sunscreen drops off bodies at a constant rate, like sand drops through an hourglass -- half is gone after one hour, and the other half after the second hour.* Maybe, when hit by a photon of UVB, each particle of sunscreen absorbs it like a human shield, dying in the attempt, and there are exactly enough particles to last exactly two hours.

(* Actually, I'm pretty sure the second half of the sand in the hourglass drops slower than the first half ... for water, I think I read that the flow rate is proportional to the depth. Maybe it's different for sand. Whatever.  :))

Maybe the deterioration is even faster than constant -- like human aging. If you hire an army of teenagers to protect you, you still have 90 percent of your protection after 40 years. But then your protection gets worse, and by 80 years, your army has almost all died away. 

Would be nice to know, wouldn't it? "Reapply every two hours" sounds like an arbitrary rule, not an attempt to maximize protection given constraints of cost and convenience.

Instead of "reapply every two hours," CR should tell us, "almost all the protection disappears between the second and third hour."  Give us useful information, instead of perplexing advice.

-------

"Won't getting a tan actually protect my skin?" the article asks. And CR answers:


"A tan provides the equivalent of up to an SPF 4, but any darkening is just a sign that your skin is defending itself against a solar assault and attempting to prevent further damage."

So, yes, CR admits, a tan *does* protect my skin. But, CR says, some damage was caused in the process of getting the tan in the first place.

What they don't say is: is the tradeoff worth it? Maybe it takes me 50 hours to tan, but, as a result, my next 4000 hours only count as 1000 hours. In that case, I've saved myself the equivalent of 2950 unprotected hours by doing the extra 50 in the first place!

Of course, it depends. The act of tanning might be especially damaging. I can't get a tan with an hour a week over 50 weeks; it has to be 50 hours over a short period. Maybe those hours are especially dangerous. Or, maybe in the 4000 hours that come later, I'd have gotten a tan anyway. Or, maybe if I didn't get a tan because it was only an hour a day, those hours wouldn't be dangerous at all.

In any case, once I have a tan, I may have an automatic SPF 4. Does that mean if I use an SPF 15, I really have an SPF 60? Because, first the sunscreen lets only 1/15 of UVB get to the surface of the skin, and then the tan admits only 1/4 of what remains. That works out to 1/60. 

It does seem like that should be how it works, based on what CR tells us about how sunscreens provide protection. Doesn't that mean that people with tans can get by with a lower SPF?

Not only does CR not admit the possibility -- they actually go out of their way to implicitly reject the idea, by warning black people to follow the same advice as white people:


"Nor are people of color immune to skin damage from the sun's rays. People of all ethnicities can get sunburn and skin cancer. "There is no circumstance under which dark-skinned people shouldn't be wearing sunscreen when exposed to the sun," Gohara adds."

Well, taking literally, Dr. Gohara's assertion is pure nonsense. Of course there are circumstances in which dark-skinned people -- or people of any other color -- don't need sunscreen. Walking 100 feet to the mailbox, say. I'm pretty sure that even Dr. Gohara herself doesn't slather on the minimum one ounce of SPF 30 to walk from her house to her car before driving to work. 

And what about winter?

This sounds like I'm nitpicking ... after all, I know what she means! But it's not. Because, it's an evasion. CR (and Dr. Gohara) are ducking out of the obligation to admit there's a tradeoff between convenience and safety, and to quantify the tradeoff. To them, wearing sunscreen is a moral obligation, like wearing a bicycle helmet or a seatbelt -- you should do it just because you should do it, because that's what right-thinking people do!

Dr. Gohara isn't just saving words by saying "there is no circumstance."  She, and CR, are trying to bully their way out of acknowledging that the risk is much lower in some situations than others, and that the sun is indeed less dangerous for darker-skinned people than lighter-skinned people. 

I mean, *of course* darker-skinned people can still get skin cancer, just like, *of course* you can still get a concussion if you fall on your head walking without a helmet instead of biking without a helmet. That's not the point. The point is: what's the risk, and is it high enough to justify the trouble and inconvenience?

A quick Google search tells me: white people have *twenty-five times* the risk of skin cancer than African Americans. CR could have, and should have, told us that.

And, given that that's the case, CR should definitely be advocating less stringent requirements for dark-skinned people than light-skinned people. By any cost-benefit calculation under which CR determined that SPF 30 is good enough for white people, the same calculation would come up with a much lower threshold for black people. 

-------

Oh, and by the way: there is no mention of time of day. Ultraviolet radiation, as measured by the "UV index" you see on weather reports, is at its highest around noon -- in the earlier morning, or late afternoon, it looks like it drops to maybe half its peak level. Wouldn't it be OK to wear just SPF 15 if we go out in the morning? Wouldn't a walk in the evening, where the UV index is "low," be one of those non-existent circumstances where dark-skinned (and light-skinned) people can skip the sunscreen? 

I'm pretty sure that if I went to the beach at 7pm, when the shadows are long, and started slathering on the sunscreen, people would look at me like an idiot. But at least Dr. Gohara would be happy with me!

How could CR do an entire article on sunscreens and sunburn and cancer without mentioning the danger varies significantly by time of day?

-------

Are the chemicals in sunscreens toxic? CR says: no. But they hedge their bets. First, they say the risk of getting skin cancer from sunburn trumps any concerns about toxicity. And then, having spent most of the article so far telling us over and over and over that we should use sunscreen, now they tell us, well, maybe it's not a good idea to rely on it too much anyway!

It's a weird non-sequitur. It reads to me like CR admits only reluctantly that sunscreen is safe, because they're vaguely bothered by the idea that people now feel freer to lie on the beach. Because ... well, sunbathing is kind of unwholesome, and even with sunscreen it's not *perfectly* safe.

I bet if you ask CR if smoking causes ingrown toenails, they won't say "no."  They'll say, "no, but it causes lung cancer and emphysema and all kinds of other dangerous conditions, so don't smoke!"  

So, instead of just saying "sunscreen doesn't cause ingrown toenails," CR says "sunscreen doesn't cause ingrown toenails, but it could still be dangerous because you might stay out in the sun longer and not use it properly and wind up getting cancer and even if you do use it you might still get cancer even if you're black and look at all the people who use sunscreen and get cancer anyway!"

Of course, they express it in a higher-class way than that:


"Experts speculate that sunscreen use -- and misuse -- may give some people a false sense of security; people think that no matter how casually they slather the stuff on, they will be protected and therefore they stay out in the sun longer -- often without reapplying. Remember, no sunscreen blocks 100 percent of UV rays, and sun damage is cumulative."

This has absolutely nothing to do with the question of toxicity. They just want to scare us. And, having just finished telling us that sunscreen itself doesn't actually cause cancer in the normal, cause-and-effect way, they proceed to try to scare us into believing it causes cancer statistically. Their statistical arguments are appallingly bad:


"In two European studies, people who used SPF 30 sunscreen sunbathed up to 30 percent longer than those using SPF 10."

Well, duh! Phrase it this way, which means the same thing:


"In two European studies, people who chose to sunbathe longer chose SPF 30 instead of SPF 10."

People are being rational, and making reasonable tradeoffs -- but because they like sunbathing, and CR doesn't, they can't possibly be rational. 

Here's another one:


"... a number of studies have shown a correlation between the use of sunscreen and increased incidence of sunburn."

Well, duh! People who don't go out in the sun (a) don't get sunburn, and (b) don't use sunscreen. CR might as well have written,


"... a number of studies have shown a correlation between the amount of time wearing a seatbelt and increased incidence of car accidents."

This one is just as bad:


"For instance, a Danish study reported that 66 percent of sunburned people had used sunscreen to prolong time spent in the sun."

Well, yes. And probably 66 percent of people who die in car accidents had used seatbelts to prolong time spent in the car.

And, finally:


"On average, your risk for melanoma doubles if you've had more than five sunburns."

Doubles? Meh! That's nothing! You know what? Your chance of induction into the Baseball Hall of Fame probably goes up by twenty-five thousand times if you've had at least one MLB base hit!

The point, of course being: "more than five" includes six, but also includes 50 and 100 and 1000, and people who are so sunburned that they're skin looks dried up and wrinkly and creepy. CR is trying to shoehorn "six sunburn" people into the same classification as "100 sunburn" people, the same way I shoehorned "one base hit" into the same group as "4192 base hits."

Moreover ... your risk doubles compared to what? Compared to zero sunburns? Compared to average? I think it's probably compared to everyone with less than five sunburns, but I'm not sure. So not only is the statement misleading, it's fatally ambiguous.

The American Academy of Dermatology reports that the incidence rate of melanoma, for non-Hispanic Caucasians, is 25 cases per 100,000 population per year. That includes everyone who has had sunburns, as well as those who haven't.

For argument's sake, let's assume that the "five or less" group is the same size as the "more than five" group. In that case, the respective rates must be 16 cases per 100,000, and 32 cases per 100,000, respectively.

So if you're in the latter group, and you're typical of that group -- which probably means substantially more than six sunburns, since six is the minimum -- your excess risk is 16 cases per 100,000 per year. Over 40 years, that's 640 cases cases per 100,000, or 6.4 cases per 1,000. 

In other words, your sunburns have given you an excess risk of six-tenths of a percentage point.

The ADD reports that skin cancer is "highly curable" if detected early, with a 98 percent survival rate. So, the death rate is 50 times smaller than the incidence rate -- 12.8 deaths per 100,000 people over a lifetime of "more than five sunburns."

How risky is that? Well, it's the equivalent fatality risk you get driving an extra 12,800 miles in your lifetime. 

Suppose that the average "more than five" sunburns is 10. That means one sunburn is the equivalent death risk of driving 1,280 miles. 

Coincidentally, I actually make a drive of about that distance, Ottawa to Toronto to Pittsburgh to Ottawa, twice a year. If that's all the risk I get from a sunburn, it seems pretty safe to me. It's a risk I'm certainly willing to take.

-------

Remember when I wrote about Consumer Reports' article on extended automobile warranties? They spent the entire article telling me why I shouldn't buy one, and then, at the end, out of the blue, they said I should!

They did it again here, in the context of "false sense of security":


"Don't think that reapplying sunscreen meticulously allows you to stay in the sun longer than a sunscreen's approximate maximum protection time. After you've exceeded that time, your best option is to cover up, wear a hat, and seek the shade."

Huh???

Both Dr. Gohara and CR spent the first part of the article warning me to reapply every two hours -- but now CR is telling me that reapplying doesn't work!

But reapplying *must* work, right? I mean, the physics of UVB reflection and the laws of chemistry still apply. If the SPF 30 was blocking 97 percent of ultraviolet for the first two hours, it should do the same for the second two hours, no? 

This feels like the kind of tainted, emotion-driven "logic" that a panicky, anxious parent would say. "You've spent two hours in the sun already! That's enough for one day! Come inside! Playing in the sun is dangerous, you have to do it in moderation!"

Sunbathing has a health stigma attached to it, so it feels like a vice, even when the risk is mitigated. There's no way CR would take this kind of position for activities it approves of.
"When your bicycle helmet breaks, don't think that meticulously reapplying a new helmet allows you to ride safely longer than a helmet's maximum protection time. After you've exceeded that time, your best option is to walk, drive, or seek a safer activity."

That's not any more absurd than what CR wrote about sunscreen. 




Labels: ,

Tuesday, September 01, 2015

Consumer Reports on auto insurance, part IV

(Previous posts: part I; part II; part III)


Consumer Reports' biggest complaint is that insurance companies set premiums by including criteria that, according to CR, don't have anything to do with driving. The one that troubles them the most is credit rating:


"We want you to join forces with us to demand that insurers -- and the regulators charged with watching them on ouir behalf -- adopt price-setting practices that are more meaningfully tethered to how you drive, not to who they think you are. ..."

"In the states where insurance companies don't use credit information, the price of car insurance is based mainly on how people actually drive and other factors, not some future risk that a credit score 'predicts'. ..."

"... an unfair side effect of allowing credit scores to be used to set premium prices is that it effectively forces customers to dig deeper into their pockets to pay for accidents that haven't happened and may never happen."

----

Well, you may or may not agree on whether insurers should be allowed to consider credit scores, but, even if CR's conclusion is correct, their argument still doesn't make sense.

First: the whole idea of insurance is EXACTLY what CR complains about:  to "pay for accidents that haven't happened and may never happen." I mean, that's the ENTIRE POINT of how insurance works -- those of us who don't have accidents wind up paying for those of us who do. 

In fact, we all *hope* that we're paying for accidents that don't happen and may never happen! It's better if we don't suffer injuries, and our car stays in good shape, and our premium stays low. 

Well, maybe CR didn't actually mean that literally. What they were *probably* thinking, but were unable or unwilling to articulate explicitly, is that they think credit scores are not actually indicative of car accident risk -- or, at least, not correlated sufficiently to make the pricing differences fair.

But, I'm sure the insurance industry could demonstrate, immediately, that credit history IS a reliable factor in predicting accident risk. If that weren't true, the first insurance company to realize that could steal all the bad-credit customers away by offering them big discounts!

It's possible, I guess, that CR is right and all the insurance companies are wrong. Since it's an empirical question ... well, CR, show us your evidence! Prove to us, using actual data, that bad-credit customers cause no more accidents than their neighbors with excellent credit. If you can't do that, maybe show us that the bad-credit customers aren't as bad as the insurers think they are. Or, at the very, very least, explain how you figured out, from an armchair thought experiment and without any numbers backing you up, that insurance company actuaries are completely wrong, and have been for a long time, despite having the historical records of thousands, or even millions, of their own customers.

------

Just using common sense, and even without data, it IS reasonable that a better credit rating should predict a lower accident rate, holding everything else equal. You get better credit by paying your bills on time, and not overextending your finances -- both habits that demonstrate a certain level of reliability and conscientiousness. And driving safely requires ... conscientiousness. It's no surprise at all, to me, that credit scores are predictive, to some extent, of future accident claims.

And CR's own evidence supports that! As I mentioned, the article lauds USAA as being the cheapest, by far, of the large insurers they surveyed. 

But USAA markets to only a subset of the population. As Brian B. wrote in the comments to a previous post:


"[USAA insurance is available only to] military and families. So their demographics are biased by a subset of hyper responsible and conscientious professionals and their offspring."

Consumer Reports did, in fact, note that USAA limits its customers selectively. But they didn't bother demanding that USAA raise its rates, or stop unfairly judging military families by "what they think they are" -- more conscientious than average.

-----

Not only does CR not bother mentioning the possibility that drivers with bad credit scores might actually be riskier drivers, they don't even hint that it ever crossed their minds. They seem to stick to the argument that nothing can possibly "predict" future risk except previous driving record. They even put "predict" in scare quotes, as if the idea is so obviously ludicrous that this kind of "prediction" must be a form of quackery.

Except when it's not. In the passage I quoted at the beginning of this post, they squeeze in a nod to "other factors" that might legitimately affect accident risk. What might those factors be? From the article, it seems they have no objection to charging more to teenagers. Or, to men. They never once mention the fact that female drivers pay less than males -- which, you'd think, would be the biggest, most obvious non-driving factor there is.

CR demands that I be judged "not by who the insurance companies think I am!" Unless, of course, I'm young and male, in which case, suddenly it's OK.

Why is it not a scandal that I pay more just for being a man? I may not be the aggressive testosterone-fueled danger CR might "think I am."  If I'm actually as meek as the average female, the insurer is going to "profit from accidents I may never have!"

------

I suspect they're approaching the question from a certain moral standard, rather than empirical considerations of the actual risk. It just bugs them that the big, bad insurance companies make you pay more just for having worse credit. On the other hand, men are aggressive, angry, muscle-car loving speeders, and it's morally OK for them to get punished. As well as young people, who are careless risk-takers who text when they drive.

A less charitable interpretation is that CR is just jumping to the conclusion that higher prices are unjustified, even when based on statistical risk, when they affect "oppressed" groups, like the poor -- but OK when they favor "oppressor" groups, like men. (Recall that CR also complained about "good student" discounts because they believe those discounts benefit wealthier families.)

A more charitable interpretation might see CR's thinking as working something like this:

-- It's OK to charge more to a certain group where it's obvious that they generally have a higher risk. Like, teenage drivers, who don't have much experience. Common sense suggests, of course they get into more accidents.

-- Higher rates are like a "punishment," and it's OK, and even good, to punish people who do bad things. People who have at-fault accidents did something bad, so their rates SHOULD go up, to teach them a lesson! As CR says,

"In California, the $1,188 higher average premium our single drivers had to pay because of an accident they caused is a memorable warning to drive more carefully. ... In New York, our singles received less of a slap, only $429, on average."

-- It's OK for men to pay more than women because psychologists have long known that men are more aggressive and prone to take more risks.

-- But it's *not* OK to charge more for someone in a high-risk group when (a) there's no proof that they're actually, individually, a high risk, and (b) the group is a group that CR feels has been unfairly disadvantaged already. Just because someone has bad credit doesn't mean they're a worse driver, even if, statistically, that group has more accidents than others. Because, maybe a certain driver has bad credit because he was conned into buying a house he couldn't afford. First, he was victimized by greedy bankers and unscrupulous developers ... now he's getting conned a second time, by being gouged for his auto policy, even though he's as safe as anyone else!


If CR had actually come out and said this explicitly, and argued for it in a fair and unbiased fashion, maybe I would change my mind and come to see that CR is right. But ... well, that doesn't actually seem to be what CR is arguing. They seem to believe that their plausible examples of bad credit customers with low risk are enough to prove that the overall correlation must be zero!

When a certain model of car requires twice as many repairs as normal, CR recommends not buying it. But when a certain subset of drivers causes twice as many accidents as average, CR not only suggests we ignore the fact -- they even refuse to admit that it's true!

------

Here's a thought experiment to see how serious CR is about considering only driving history.

Suppose an insurer decided to charge more for drivers who don't wear a helmet when riding a bicycle, based on evidence that legitimately shows that people who refuse to wear bicycle helmets are more likely to refuse to wear seatbelts.

But, they note, it's not a perfect correlation. I, for instance, am an exception. I don't wear a bicycle helmet, but I wouldn't dream of driving without a seatbelt (and I might even be scared to drive a car without airbags). 

Would CR come to my defense, demanding that my insurer stop charging me extra?  Would they insist they judge me by how I drive, not by "who they think I am" based on my history of helmetlessness?

I doubt it. I think they'd be happy that I'm being charged more. I think it's about CR judging which groups "deserve" higher or lower premiums, and then rationalizing from there.

(If you want to argue that bicycling is close enough to driving that this analogy doesn't work, just substitute hockey helmets, or life jackets.)

------

I'm not completely unsympathetic to CR's position. They could easily make a decent case.  They could say, "look, we know that drivers with bad credit cause more accidents, as a group, than drivers with good credit. But it seems fundamentally unfair, in too many individual cases, to judge people by the characteristics of their group, and make them pay twice as much without really knowing whether they fit the group average."

I mean, if they said that about race, or religion, we'd all agree, right? We'd say, yeah, it DOES seem unfair that a Jewish driver like Chaim pays less (or more) than a Muslim driver like Mohammed, just because his group is statistically less (or more) risky. 

But, what if it's actually the case that, statistically, one group causes more accidents than the other? We tell the insurance companies, look, it's not actually because of religion that the groups are different. It must be just something that correlates to religion, perhaps by circumstance or even coincidence.  So, stop being so lazy.  Instead of deciding premiums based on religion, get off your butts and figure out what's actually causing the differences! 

Maybe the higher risk is because of what neighborhoods the groups tend to live in, that some neighborhoods have narrower streets and restricted sightlines that lead to more accidents. Shouldn't the insurance company figure that out, so that if they find that Chaim (or Mohammed) actually lives in a safer neighborhood, they can set his premium by his actual circumstances, instead of his group characteristics, which they will now realize don't apply here?  That way, fewer drivers will be stuck paying unfairly high or low premiums because of ignorance of their actual risk factors.

If that works for religion, or race, it should also work for credit score. Can't the insurance companies do a bit more work, and drill down a bit more, to figure out who has bad credit out of irresponsibility, and who has bad credit because of circumstances out of their control?

Yes! And, I'd bet, the insurance companies are already doing that! Their profits depend on getting risk right, and they can't afford to ignore anything that's relevant, lest other insurers figure it out first, and undercut their rates.

And CR actually almost admits that this is happening. Remember, the article tells us that the insurers aren't actually using the customer's standard credit score -- they're tweaking parts of it to create their own internal metric. CR tells us that to complain about it -- it's arbitrary, and secret! -- but it might actually be the way the insurers make premiums more accurate, and therefore fair.  It might be the way insurers make it less likely that a customer will be forced to pay higher premiums for "accidents that may never happen."

-----

But I don't think CR really cares that premiums are mathematically fair. Their notion of fairness seems to be tied to their arbitrary, armchair judgments about who should be paying what. 

I suspect that even if the insurance companies proved that their premiums were absolutely, perfectly correlated with individual driving talent, CR would still object. They don't have a good enough understanding of risk -- or a willingness to figure it out.

A driver's expected accident rate isn't something that's visible and obvious. It's hard for anyone but an actuary to really see that Bob is likely to have an accident every 10 years, while Joe is likely to have an accident every 20. To an outsider, it looks arbitrary, like Bob is getting ripped off, having to pay twice as much as Joe for no reason. 

The thing is: some drivers really ARE double the risk. But, because accidents are so rare, their driving histories look identical, and there doesn't seem to be any reason to choose between them. But, often, there is.

If you do the math, you'll see that a pitcher who retires batters at a 71% rate is at more than double the "risk" of pitching a perfect game than a pitcher with only a 69% rate. But, in their normal, everyday baseball card statistics, they don't look that much different at all -- just a two percentage point difference in opposition OBP.

I think a big part of the problem is just that luck, risk, and human behavior follow rules that CR isn't willing to accept -- or even try to understand.






Labels: ,

Wednesday, August 26, 2015

Consumer Reports on auto insurance, part III

(This is part III.  Part I is here; part II is here.)

------

As part of its auto insurance research, Consumer Reports says they "analyzed more than 2 billion car insurance price quotes."  

That number seems reasonable. CR looked at all 33,419 general US ZIP codes. Multiply that by the 20 demographic groups they considered, then up to 19 different insurance companies per state, and you're up to about 10 million. That leaves around 200 different combinations of variations for each (credit rating, accident history, speeding tickets, etc).

In practical terms, how do you arrange to get two billion quotes? Even if you can get 20 at a time from a website, typing in all that information would take forever. Even at one quote per second, two billion quotes would take 63 years. Or, just one year, if CR had 63 staffers doing the work.

Well, it's much easier than that. CR reports that in most states, insurers have to file their mathematical pricing formulas -- their actuarial trade secrets, by the sound of it -- with state regulators. A private company, Quadrant Information Services, has access to all those formulas, somehow, and sells its services to clients like CR. So, two billion quotes was probably just a matter of contracting with Quadrant, who would just query its database and send along the results.

I always wondered how Progressive was able to get competitors' quotes, in real time, to compare alongside their own. Now I know.

----

CR says those quotes are the real deal, the actual prices policyholders pay:


"Under the state laws that regulate automobile insurance, carriers are required to adhere to the prices generated by their public rate filings. So the premiums we obtained from Quadrant are what each company legally obligates itself to charge consumers."

But ... I'm skeptical. If those quotes are really the actual premiums paid, that would have to contradict some of the issues CR raises elsewhere in the article.

For instance, one thing they're upset about is that some companies practice "price optimization." That's a euphemism for, jacking up the price for certain customers, the ones the company thinks won't complain. For instance, CR says, some insurers might bump your premium if "you're sticking with Verizon FIOS when DirectTV [sic] might be cheaper."

Except ... how can that be possible, if it's all done by formula? When you ask Progressive for quotes, they don't ask you who your TV provider is (or how many iPhones or beers you've purchased, which are other criteria CR mentions). 

Second, CR mentions that each insurer creates their own proprietary credit score, "very different from the FICO score you might be familiar with."  But, again, the formulas can't be taking that into account, can they? CR requested premiums for "poor," "good," and "excellent" credit scores ... but how would they know which was which, without knowing each insurer's proprietary formula?

Third, and also in the context of price optimization, they advise,


"... don't be shy about complaining a little more [to show you're not a pushover for next time]."

But if those formula prices are fixed and non-negotiable, how will complaining help? Unless "number of times having complained" is in the formulas filed with regulators.  

------

So, it doesn't make sense that the entire pricing system is encapsulated in the online (or Quantum) pricing formulas.

So, what's actually going on? 

Maybe what's happening is that the companies aren't obligated to charge EXACTLY those formula prices -- maybe they're obligated to charge those prices OR LESS. 

Kind of like those prices you see on the back of your hotel room door, the maximum that room would ever go for, like (I imagine) Cooperstown on induction weekend. Or, they're like the MSRP on cars, where you can negotiate down from there. Or, maybe they're like car repair estimates, where, if they don't know for sure how much it will cost, they give you the worst-case scenario, because they can lower their estimate much easier than they can raise it.

If that's what's going on, that would easily and plausibly explain the pricing anomalies that CR found.

Take, for instance, the one that surprised me most -- the finding that some companies discriminate against long-term customers. As CR puts it, "some insurers salute your allegiance with a price hike." 

In the state of Washington, the article says, almost half the insurers surveyed didn't offer any discount at all to customers who had been with them for at least 15 years. That doesn't sound right, but, get this: not only did Geico not offer a discount, they actually charged their loyal customers MORE: almost $700 more, according to the article.

That smells a bit fishy to me.  But here's one that smells ... well, I don't have a good metaphor.  Maybe, like a rotting pile of fish carcasses in your driveway?


"Geico Casualty gave us whiplash with its $3,267 loyalty penalty in New Jersey and its $888 discount just across the state line in New York for longtime customers."

Well, that's just not possible, right? Overcharging loyal New Jersey customers THREE THOUSAND DOLLARS A YEAR? That would triple the typical price, wouldn't it? 

When CR came up with that result, didn't any of their staff think, "WTF, can that really be true?" At Consumer Reports, they must have some weird priors. I know they think insurance companies are out to gouge consumers for everything they can, but ... this is too big to make any sense at all, even under CR's own assumptions. 

------

I'd wager that those particular automated quotes aren't at all representative of what those particular customers actually pay. 

Insurance companies don't ask their long-term policyholders to go online and renew anonymously. They send renewal quotes directly. Which they have to, if premiums are tailored for characteristics that aren't used in online applications, like those details from the customer's credit record, like beer purchases.

What CR found could just be a Geico company practice, of not bothering to produce competitive "formula" quotes for established customers who won't be using them anyway. 

I don't know if that's actually the right answer, but, whatever the true explanation is ... well, I'd bet a lot of money that if the magazine surveyed real long-term Geico policyholders in New Jersey, and asked about their premiums, CR would find that "loyalty penalty" doesn't actually exist.  Or at least, not at anything within an order of magnitude of $3,267.

I might be wrong. Feel free to tell me what I'm missing.




(to be continued)





Labels: ,

Thursday, August 20, 2015

Consumer Reports on auto insurance, part II

(Part I is here.)

------

Last post, I linked to an article showing auto insurance profit margins were very low, less than 10 percent of premiums. And, I wondered, if that's the case, how is it possible that CR reports such a large difference in pricing between companies?

In its research, Consumer Reports got quotes for thousands of different drivers -- that is, different combinations of age/sex/and ZIP code -- from five different companies. The average premiums worked out to:  

$1,540 Allstate
$1,414 Progressive
$1,177 Geico
$1,147 State Farm
$  817 USAA

How is it possible that Allstate charged 34% more than Geico, but still made less profit (and only 5.3% profit, at that)? How does USAA stay in business charging a third what the others charge, when those others are barely in the black, as is?

For anyone to take those numbers seriously, Consumer Reports has to explain this apparent impossibility. Otherwise, the only reasonable conclusion is that something went badly wrong with CR's analysis or methodology.

Which I think is what happened. I'm going to take a guess at what's actually going on. I don't know for sure, but I'd be willing to bet it's pretty close.

------

With margins so low, and competition so tight, companies really, really have to get their risk estimates right. If not, they're in trouble.

Let's make some simple assumptions, to keep the analysis clean. First, suppose all customers shop around and always choose the lowest-priced quote. 

Second, suppose that the average teenage driver carries $3,000 in annual risk -- that is, the average teenager will cause $3,000 worth of claims each year. 

Now, we, the blog readers, know the correct number is $3,000 because we just assumed it -- we gave ourselves a God's-eye view. The insurance companies don't have that luxury. They have to estimate it themselves. That's hard work, and they're not going to be perfect, because there's so much randomness involved. (Also, they're all using different datasets.)

Maybe the actuaries at Progressive come up with an estimate of $3,200, while Geico figures it's $2,700. (I'll ignore profit to keep things simple -- if that bothers you, add $100 to every premium and the argument will still work.)

What happens? Every teenager winds up buying insurance from Geico. And Geico loses huge amounts of money: $300 per customer, as the claims start to roll in. Eventually, Geico figures out they got it wrong, and they raise their premiums to $3,000. They're still the cheapest, but now, at least, they're not bleeding cash.

This goes on for a bit, but, of course, Progressive isn't sitting still. They hire some stats guys, do some "Insurance Moneyball," and eventually they make a discovery: good students are better risks than poor students. They find that good students claim $2,500 a year, while the others claim $3,500.

Progressive changes their quotes to correspond to their new knowledge about the "driving talent" of their customers. Instead of charging $3,200 to everyone, they now quote the good students $2,500, and the rest $3,500, to match their risk profiles. That's not because they like the pricing that way, or because they think good students deserve a reward ... it's just what the data shows, the same way it shows that pitchers who strike out a lot of batters have better futures than pitchers who don't.

Now, when the good students shop around, they get quotes of $2,500 (Progressive) and $3,000 (Geico). The rest get quotes of $3,500 (Progressive) and $3,000 (Geico).

So, what happens? The good students go to Progressive, and the rest go to Geico. Progressive makes money, but Geico starts bleeding again: they're charging $3,000 to drivers who cost them $3,500 per year.

Geico quickly figures out that Progressive knows something they don't -- that, somehow, Progressive figured out which teenage customers are lower risk, and stole them all away by undercutting their price. But they don't know how to tell low risk from high risk. They don't know that it has to do with grades. So, Geico can't just follow suit in their own pricing.

So what do they do? They realize they've been "Billy Beaned," and they give up. They raise their price from $3,000 to $3,500. That's the only way they can keep from going bankrupt.

The final result is that, now, when a good student looks for quotes, he sees

$2,500 Progressive
$3,500 Geico

When a bad student looks for quotes, he sees

$3,500 Progressive
$3,500 Geico

Then Consumer Reports comes along, and gets a quote for both. When they average them for their article, they find

$3,000 Progressive
$3,500 Geico

And they triumphantly say, "Look, Progressive is 14 percent cheaper than Geico!"

But it's not ... not really. Because, no good student actually pays the $3,500 Geico quotes them. Since everyone buys from the cheapest provider, Geico's "good student" quote is completely irrelevant. They could quote the good students a price of $10 million, and it wouldn't make any difference at all to what anyone paid.

That's why averaging all the quotes, equally weighted, is not a valid measure of which insurance company gives the best deal.

------

Want a more obvious example?

Company X:
------------------------------
$1,300      25-year-old male
$1,000      30-year-old female
$1 million  16-year-old male

Company Y:
------------------------------
$1,400      25-year-old male
$1,100      30-year-old female
$4,000      16-year-old male

By CR's measure, which is to take the average, company Y is much, much cheaper than company X: $2,166 to $334,100. But in real life, which company is giving its customers greater value? Company X, obviously. NOBODY is actually accepting the $1,000,000 quote. In calculating your average, you have to give it a weight of zero. 

Once you've discarded the irrelevant outlier, you see that, contrary to what the overall average suggested, company X is cheaper than company Y in every (other) case.

------

Want a non-insurance analogy?

"Darget" competes with Target. Their prices are all triple Target's, which is a big ripoff -- except that every product that starts with "D" sells for $1. By a straight average of all items, Darget is almost 300% as expensive as Target. Still, at any given time, Darget has twenty times the number of customers in the store, all crowding the aisles buying diamonds and diapers and DVD players. 

When evaluating Darget, the "300%" is irrelevant. Since everyone buys deodorant at Darget, but nobody buys anti-perspirant at Darget, it makes no sense to average the two equally when calculating a Darget price index.

-----

And I suspect that kind of thing is exactly what's happening in CR's statistics. Allstate *looks* more expensive than USAA because, for some demographics of customer, they haven't studied who's less risky than whom. They just don't know. And so, to avoid getting bled dry, they just quote very high prices, knowing they probably won't get very many customers from those demographics.

I don't know which demographics, but, just to choose a fake example, let's say, I dunno, 75-year-olds. USAA knows how to price seniors, how to figure out the difference between the competent ones and the ones whose hands shake and who forget where they are. Allstate, however, can't tell them apart. 

So, USAA quotes the best ones $1,000, and the worst ones $5,000. Allstate doesn't know how to tell the difference, so they have to quote all seniors $5,000, even the good ones. 

What Allstate is really doing is telling the low-risk seniors, "we are not equipped to recognize that you're a safe driver; you'll have to look elsewhere."  But, I'm guessing, the quote system just returns an uncompetitively high price instead of just saying, "no thank you."

----

Under our assumption that customers always comparison shop, it's actually *impossible* to compare prices in a meaningful way. By analogy, consider -- literally -- apples and oranges, at two different supermarkets.

Store A charges $1 an apple,  and $10 an orange.
Store B charges $2 an orange, and  $5 an apple.

Who's cheaper overall? Neither! Everyone buys their apples at Supermarket A, and their oranges at Supermarket B. There's no basis for an apples-to-apples comparison.

We *can* do a comparison if we relax our assumptions. Instead of assuming that everyone comparison shops, let's assume that 10 percent of customers are so naive that they by all their fruit at a single supermarket. (We'll also assume those naive shoppers eat equal numbers of apples and oranges, and that they're equally likely to shop at either store.)

Overall, combining both the savvy and naive customers, Store A sells 100 Apples and 10 Oranges for a total of $200. Store B sells 100 Oranges and 10 Apples for a total of $250.

Does that mean Store B is more expensive than Store A? No, you still can't compare, because store B sells mostly oranges, and store B sells mostly apples.

To get a meaningful measure, you have to consider only the 10 percent of customers who don't comparison shop. At store A, they spend $11 for one of each fruit. At store B, they spend $7 for one of each fruit.

Now, finally, we see that store B is cheaper than store A!

But:

1. To be able to say that, we had to know that the naive customers are evenly split both on the fruit they buy, and the stores they go to. We (and CR) don't know the equivalent statistics in the auto insurance case.

2. If "Store B is cheaper" it's only for those customers who don't shop around. For the 90 percent who always accept only the lowest price, the question still has no answer. CR wants us to be one of those 90 percent, right? So, their comparison is irrelevant if we follow their advice!

3. All CR's analysis tells us is, if we're completely naive customers, getting one quote at random from one insurance company, then blindly accepting it ... well, in that case, we're best off with USAA.

But, wait, even that's not true! It's only true if we're exactly, equally likely to be any one of CR's thousands of representative customers. Which we're not, since they gave ZIP code 10019 in Manhattan (population 42,870) equal weight with ZIP code 99401 in Alaska (population 273).

------

CR's mistake was to weight the quotes equally, even the absurdly high ones. They should have weighted them by how often they'd actually be accepted. Of course, nobody actually has that information, but you could estimate it, or at least try to. One decent proxy might be: consider only quotes that are within a certain (small) percentage of the cheapest. 

Also, you want to weight by the number of drivers in the particular demographic, not treat each ZIP code equally. You don't want to give a thousand 30-year-old Manhattanites the same total weight as the three 80-year-olds in a rural county of Wyoming.

By adjusting for both those factors, CR would be weighting the products by at least a plausible approximation of how often they're actually bought. 

------

Anyway, because of this problem -- and others that I'll get to in a future post -- most of CR's findings wind up almost meaningless. Which is too bad, because it was a two-year project, and they did generate almost a billion quotes in the effort. And, they're not done yet -- they promise to continue their analysis in the coming months. Hopefully, their coming analysis will be more meaningful.



(to be continued)





Labels: ,