Consumer Reports on auto insurance, part II

(Part I is here.)

------

Last post, I linked to an article showing auto insurance profit margins were very low, less than 10 percent of premiums. And, I wondered, if that's the case, how is it possible that CR reports such a large difference in pricing between companies?

In its research, Consumer Reports got quotes for thousands of different drivers -- that is, different combinations of age/sex/and ZIP code -- from five different companies. The average premiums worked out to:  

$1,540 Allstate
$1,414 Progressive
$1,177 Geico
$1,147 State Farm
$  817 USAA

How is it possible that Allstate charged 34% more than Geico, but still made less profit (and only 5.3% profit, at that)? How does USAA stay in business charging a third what the others charge, when those others are barely in the black, as is?

For anyone to take those numbers seriously, Consumer Reports has to explain this apparent impossibility. Otherwise, the only reasonable conclusion is that something went badly wrong with CR's analysis or methodology.

Which I think is what happened. I'm going to take a guess at what's actually going on. I don't know for sure, but I'd be willing to bet it's pretty close.

------

With margins so low, and competition so tight, companies really, really have to get their risk estimates right. If not, they're in trouble.

Let's make some simple assumptions, to keep the analysis clean. First, suppose all customers shop around and always choose the lowest-priced quote. 

Second, suppose that the average teenage driver carries $3,000 in annual risk -- that is, the average teenager will cause $3,000 worth of claims each year. 

Now, we, the blog readers, know the correct number is $3,000 because we just assumed it -- we gave ourselves a God's-eye view. The insurance companies don't have that luxury. They have to estimate it themselves. That's hard work, and they're not going to be perfect, because there's so much randomness involved. (Also, they're all using different datasets.)

Maybe the actuaries at Progressive come up with an estimate of $3,200, while Geico figures it's $2,700. (I'll ignore profit to keep things simple -- if that bothers you, add $100 to every premium and the argument will still work.)

What happens? Every teenager winds up buying insurance from Geico. And Geico loses huge amounts of money: $300 per customer, as the claims start to roll in. Eventually, Geico figures out they got it wrong, and they raise their premiums to $3,000. They're still the cheapest, but now, at least, they're not bleeding cash.

This goes on for a bit, but, of course, Progressive isn't sitting still. They hire some stats guys, do some "Insurance Moneyball," and eventually they make a discovery: good students are better risks than poor students. They find that good students claim $2,500 a year, while the others claim $3,500.

Progressive changes their quotes to correspond to their new knowledge about the "driving talent" of their customers. Instead of charging $3,200 to everyone, they now quote the good students $2,500, and the rest $3,500, to match their risk profiles. That's not because they like the pricing that way, or because they think good students deserve a reward ... it's just what the data shows, the same way it shows that pitchers who strike out a lot of batters have better futures than pitchers who don't.

Now, when the good students shop around, they get quotes of $2,500 (Progressive) and $3,000 (Geico). The rest get quotes of $3,500 (Progressive) and $3,000 (Geico).

So, what happens? The good students go to Progressive, and the rest go to Geico. Progressive makes money, but Geico starts bleeding again: they're charging $3,000 to drivers who cost them $3,500 per year.

Geico quickly figures out that Progressive knows something they don't -- that, somehow, Progressive figured out which teenage customers are lower risk, and stole them all away by undercutting their price. But they don't know how to tell low risk from high risk. They don't know that it has to do with grades. So, Geico can't just follow suit in their own pricing.

So what do they do? They realize they've been "Billy Beaned," and they give up. They raise their price from $3,000 to $3,500. That's the only way they can keep from going bankrupt.

The final result is that, now, when a good student looks for quotes, he sees

$2,500 Progressive
$3,500 Geico

When a bad student looks for quotes, he sees

$3,500 Progressive
$3,500 Geico

Then Consumer Reports comes along, and gets a quote for both. When they average them for their article, they find

$3,000 Progressive
$3,500 Geico

And they triumphantly say, "Look, Progressive is 14 percent cheaper than Geico!"

But it's not ... not really. Because, no good student actually pays the $3,500 Geico quotes them. Since everyone buys from the cheapest provider, Geico's "good student" quote is completely irrelevant. They could quote the good students a price of $10 million, and it wouldn't make any difference at all to what anyone paid.

That's why averaging all the quotes, equally weighted, is not a valid measure of which insurance company gives the best deal.

------

Want a more obvious example?

Company X:
------------------------------
$1,300      25-year-old male
$1,000      30-year-old female
$1 million  16-year-old male

Company Y:
------------------------------
$1,400      25-year-old male
$1,100      30-year-old female
$4,000      16-year-old male

By CR's measure, which is to take the average, company Y is much, much cheaper than company X: $2,166 to $334,100. But in real life, which company is giving its customers greater value? Company X, obviously. NOBODY is actually accepting the $1,000,000 quote. In calculating your average, you have to give it a weight of zero. 

Once you've discarded the irrelevant outlier, you see that, contrary to what the overall average suggested, company X is cheaper than company Y in every (other) case.

------

Want a non-insurance analogy?

"Darget" competes with Target. Their prices are all triple Target's, which is a big ripoff -- except that every product that starts with "D" sells for $1. By a straight average of all items, Darget is almost 300% as expensive as Target. Still, at any given time, Darget has twenty times the number of customers in the store, all crowding the aisles buying diamonds and diapers and DVD players. 

When evaluating Darget, the "300%" is irrelevant. Since everyone buys deodorant at Darget, but nobody buys anti-perspirant at Darget, it makes no sense to average the two equally when calculating a Darget price index.

-----

And I suspect that kind of thing is exactly what's happening in CR's statistics. Allstate *looks* more expensive than USAA because, for some demographics of customer, they haven't studied who's less risky than whom. They just don't know. And so, to avoid getting bled dry, they just quote very high prices, knowing they probably won't get very many customers from those demographics.

I don't know which demographics, but, just to choose a fake example, let's say, I dunno, 75-year-olds. USAA knows how to price seniors, how to figure out the difference between the competent ones and the ones whose hands shake and who forget where they are. Allstate, however, can't tell them apart. 

So, USAA quotes the best ones $1,000, and the worst ones $5,000. Allstate doesn't know how to tell the difference, so they have to quote all seniors $5,000, even the good ones. 

What Allstate is really doing is telling the low-risk seniors, "we are not equipped to recognize that you're a safe driver; you'll have to look elsewhere."  But, I'm guessing, the quote system just returns an uncompetitively high price instead of just saying, "no thank you."

----

Under our assumption that customers always comparison shop, it's actually *impossible* to compare prices in a meaningful way. By analogy, consider -- literally -- apples and oranges, at two different supermarkets.

Store A charges $1 an apple,  and $10 an orange.
Store B charges $2 an orange, and  $5 an apple.

Who's cheaper overall? Neither! Everyone buys their apples at Supermarket A, and their oranges at Supermarket B. There's no basis for an apples-to-apples comparison.

We *can* do a comparison if we relax our assumptions. Instead of assuming that everyone comparison shops, let's assume that 10 percent of customers are so naive that they by all their fruit at a single supermarket. (We'll also assume those naive shoppers eat equal numbers of apples and oranges, and that they're equally likely to shop at either store.)

Overall, combining both the savvy and naive customers, Store A sells 100 Apples and 10 Oranges for a total of $200. Store B sells 100 Oranges and 10 Apples for a total of $250.

Does that mean Store B is more expensive than Store A? No, you still can't compare, because store B sells mostly oranges, and store B sells mostly apples.

To get a meaningful measure, you have to consider only the 10 percent of customers who don't comparison shop. At store A, they spend $11 for one of each fruit. At store B, they spend $7 for one of each fruit.

Now, finally, we see that store B is cheaper than store A!

But:

1. To be able to say that, we had to know that the naive customers are evenly split both on the fruit they buy, and the stores they go to. We (and CR) don't know the equivalent statistics in the auto insurance case.

2. If "Store B is cheaper" it's only for those customers who don't shop around. For the 90 percent who always accept only the lowest price, the question still has no answer. CR wants us to be one of those 90 percent, right? So, their comparison is irrelevant if we follow their advice!

3. All CR's analysis tells us is, if we're completely naive customers, getting one quote at random from one insurance company, then blindly accepting it ... well, in that case, we're best off with USAA.

But, wait, even that's not true! It's only true if we're exactly, equally likely to be any one of CR's thousands of representative customers. Which we're not, since they gave ZIP code 10019 in Manhattan (population 42,870) equal weight with ZIP code 99401 in Alaska (population 273).

------

CR's mistake was to weight the quotes equally, even the absurdly high ones. They should have weighted them by how often they'd actually be accepted. Of course, nobody actually has that information, but you could estimate it, or at least try to. One decent proxy might be: consider only quotes that are within a certain (small) percentage of the cheapest. 

Also, you want to weight by the number of drivers in the particular demographic, not treat each ZIP code equally. You don't want to give a thousand 30-year-old Manhattanites the same total weight as the three 80-year-olds in a rural county of Wyoming.

By adjusting for both those factors, CR would be weighting the products by at least a plausible approximation of how often they're actually bought. 

------

Anyway, because of this problem -- and others that I'll get to in a future post -- most of CR's findings wind up almost meaningless. Which is too bad, because it was a two-year project, and they did generate almost a billion quotes in the effort. And, they're not done yet -- they promise to continue their analysis in the coming months. Hopefully, their coming analysis will be more meaningful.



(to be continued)