Wednesday, September 16, 2009

You can't forecast outcomes that are random

Predictions are often wrong. In an article in the Wall Street Journal last month, "Numbers Guy" Carl Bialik points out a few that went awry. Two years ago, for instance, a government energy agency predicted that the price of oil would be between $75 and $85 in 2008. In reality, it started out the year close to $100, ran up past $140 in July, and dropped back below $40 by the end of the year. Bialik writes, "winging darts at numbers on a board might have been more accurate."

It's easy to make fun of prognosticators when they get this stuff wrong. But let's not get too hasty. The fact is, the things that are most worth predicting are things that are most unpredictable. If you want a prediction of what time the sun will rise tomorrow morning, you can get 100% accurate predictions from any competent astronomer. But what would be the point?

The price of oil varies so much because there are so many factors that influence it: wars, foreign government policies, consumer behavior, US election results, technological advances, natural disasters, and so on. These things are random. And they are very, very complex, most of them being the result of human thought and action.

Still, shouldn't some people be better skilled at making those predictions than others? Absolutely. Tancred Lidderdale, the economist quoted in Bialik's article, has an excellent understanding of the factors that impact the price of oil, much better than mine. So what's wrong with evaluating his predictions after the fact, to see if he's any good?

The problem is that no matter how much you know about the price of oil, it's random enough that the spread of outcomes is really, really wide: much wider than the effects of any knowledge you bring to the problem.

Suppose that on the basis of Miguel Tejada's career, everyone thinks he should hit .290 next year. But suppose Bob, who's a big fan of Tejada, and follows his plate appearances closely, has noticed something about his performance and thinks differently. Maybe it's some detailed observation that he swings a certain way, and other players with the same swing have declined more in their thirties than average. So Tejada should be only about .286.

That may be absolutely right, and figuring that out was an act of staggering sabermetric genius. Bob's estimate of .286 is correct, and the .290 estimates are all wrong. Bob is literally the only one in the world whose estimate is correct.

But in practice, how do you prove that? The standard deviation of batting average over 500 AB is about 20 points: so even with .286 being correct, there's still a 46% chance that A will hit closer to .290 than .286 next year. There's actually about a 1 in 3 chance that Tejada's average will be below .266 or above .306. For practical purposes, it's impossible to evaluate the two predictions on this one single sample. Even if Bob is omniscient, knowing everything possible about Tejada's talent, health, and diet, it's going to take a lot of evidence to prove that he's a better estimator than the mob, so long as the results of individual at-bats are random.

The problem is the small sample size: over 1000 predictions, or 1,000,000, Bob is going to have a better record than everyone else. But, who makes a million predictions, and who keeps track of them to evaluate them afterwards? And even if we do this a reasonable number of times, like 100, Bob still isn't assured of beating me. If his chance of beating me is 54%, then, if we predict 100 times each, I still have an almost 35% 21% chance of coming out the winner.

That is: an omnisicient expert can beat a reasonably-informed layman only about 65% 79% of the time. And that's after 100 trials each, 100 trials where the predictor actually has a significant edge in knowledge or analysis. In real life, if you get only one trial, and you're not even close to omniscient, and the prediction you're making may not be the one in which you have the most confidence, the public's expectations of you shouldn't be very high. Not because you're ignorant, but because life is just too random.

Of course, this is an arbitrary example, with more randomness (20 points) than knowledge (4 points). But isn't it roughly the same situation for the price of oil? The randomness in the economy is just huge. Part of the reason oil went down last year is because of the recession. The recession happened because of the credit crisis. And very few people foresaw the credit crisis, including people who had thousands, or millions, or billions of dollars on the line. For a government economist to be omniscient, he has to be omniscient about mortgage finance, and on the government's and public's reaction to every crisis that might possibly occur. That's asking a lot, isn't it? To an energy economist, the state of mortgage finance has to be taken as random.

Because life is random, and the price of oil is very sensitive to the randomness of human-caused shocks, you can't expect a single, point estimate of the price of oil to be 95% accurate within $1, or even $5. An estimate that precise is impossible, beyond the scope of human capability, and probably beyond the scope of the most powerful computers that could be imagined. An honest and competent forecaster will tell you that the best he can do is give you *distribution* for the future price of oil: maybe that there's a 60% chance it will be between $60 and $110, and a 10% chance it will be below $60, and maybe a 5% chance it'll go over $200 (if there's a major war in the Middle East, say), and so on. That's not something the newspapers are keen to report on -- it's hard to put in a headline, and it's harder for readers to understand.

What you hope Mr. Lidderdale's agency was probably saying was, "we have our best guess at a probability distribution for what the price of oil will be next year. Its mean is in the $75 to $85 range." If that phrasing makes journalists uncomfortable, fine. But that doesn't change the fact that it's the best anybody can do. And it doesn't change the fact that you can't decide how good a predictor is on the basis of one, two, or even a hundred point estimates. You need a LOT of data. And if an outlier happens, all evaluations are off. I'd bet that anyone who predicted, back in 2007, that oil would jump to $140 and then drop back to $37, is a kook, not an expert. What happened in 2008 was something of an outlier, random, unpredictable, and unknowable. Anyone who came close was probably just drop-dead lucky.



Labels:

12 Comments:

At Wednesday, September 16, 2009 10:43:00 PM, Blogger Tom G said...

People who make their living at making these sort of predictions, either as a writer for the New York Times or, a fund manager picking stocks, are simply selling ideas to their customers. Tell me how you come to your conclusions and I'll decide if they are intelligent or horsecrap

If you have a system for how to pick numbers on the roulette wheel, I'll walk away. If there is strong analysis based past results, with the noise filtered out and explained intelligently, you'll have me as a potential consumer

If someone says "This is what the price of oil will be" or "this baseball player will rise and this one will fall" there is very little of value. If they add "and here's why. . . " they have a chance to win me over. Even if the prediction falls short, the ideas they used still have value

 
At Thursday, September 17, 2009 9:55:00 AM, Anonymous Nate said...

very well reasoned post. it is a situation that is under-reported. the press need simple numbers and simple stories. thus they ask experts for a number. of course that makes for an easy target after the fact, simply because any point estimate will be wrong.

i work in the energy field and any time i'm asked to forecast oil or gas prices, i just point to the futures screen and say "that's my estimate." if they want a distribution for the estimate, i point to the options quotes.

on the sports side, this problem of stats and sample size gets even worse when you listen to play-by-play announcers. in football, if you are 4th and 2 on the opponents 40 yard line the best thing to do (from an expected value of points scores) is go for the 1st down, not punt or attempt a fg. but rest assured that the announcers will jump all over the coach if the team falls short of a first down. just once i'd like to hear an announcer say "yeah, they turn the ball over on downs, but 7 times out of 10 they would have made it - and the gains from those 7 times clearly outweigh the loss of the 3 times the don't.." I'm not holding my breath...

 
At Thursday, September 17, 2009 10:06:00 AM, Blogger Phil Birnbaum said...

Absolutely: the futures and options markets are the only predictions a journalist needs, if he's more interested in getting a good forecast distribution than promoting someone's genius.

Like economists and some of the blogs I read, I love futures markets in sports, weather or politics ... it's a cheap -- FREE! -- way of getting world-class forecasts of important events.

The futures price for December 2015 oil is $87.14. If you want to convince me that I need to rush out and buy a Prius to avoid $6 gasoline, you're going to have a little bit of trouble so long as the futures market ain't agreeing with you.

You're going to have trouble even convincing me that you believe your own estimate, unless you show me that you have a lot of money invested in oil futures.

The market is very good at separating the geniuses from the BSers.

 
At Thursday, September 17, 2009 6:33:00 PM, Anonymous Eddy Elfenbein said...

There's another element to market forecasts that makes them even trickier. The guy who sees Tejada's platonic average falls has zero impact on what Tejada's batting average will be. With the markets, a participant's bias affects the price and vice versa.

 
At Friday, September 18, 2009 10:47:00 AM, Blogger JavaGeek said...

Please note these comments are from my theoretical understanding of the stock market and not first hand experience.

One comment on futures:
"The forward contract is a biased predictor of the future stock price" (Derivative Markets by McDonald)
Forward price = S0*(1+r)
Actual future price = S0*(1+a)
a = true return
r = risk free return

Secondly, the future price should closely follow the present day price as if they were substantially different investors would be inclined to switch:
Eg
present price = 50
future price = 80 (1-year)
Sell future buy present and sell 1-year from now for a 60% return...


Secondly, options will be priced in such away that they can be sold without too much risk to the seller. And in such a way to prevent arbitrage. They also have to make sure they have adequate capital to payout the options they're selling (capital is quite expensive). And don't forget that they make money on the options as well. It doesn't necessarily mean this is a reasonable range for the expected results, but it is probably a good estimate.

 
At Friday, September 18, 2009 12:01:00 PM, Anonymous Anonymous said...

This was a great post, and i agree with it, but i was just curious on how you came with the 35% on the quote below.

"If his chance of beating me is 54%, then, if we predict 100 times each, I still have an almost 35% chance of coming out the winner."

I tried to replicate the 35% using a tree in Excel, and after 100 trials, the results i got is that the expert would win around 79% of the time, which means that the layman would win only 21% of the time.
Is there an easy way (without building the tree) to figure out 35%?

 
At Friday, September 18, 2009 12:08:00 PM, Blogger Phil Birnbaum said...

Oops, I got it wrong!

If you have 100 tries of a binomial (coin) with a 54% chance of heads, the expected number of heads is 54, and the SD of heads is

square root of (.54 * .46 * 100)

Which is almost exactly 5. So to beat you, I need to be 4/5 of an SD above expected, and the probability of that is 20.9%. Which is what you got.

Sorry to make you do all that work! I'll update the post. (The 35% is if you have a 52% chance, which is what I had originally before changing the example.)

 
At Friday, September 18, 2009 12:19:00 PM, Blogger Phil Birnbaum said...

Sorry, I didn't really answer the question. Continuing from my previous comment:

For me to beat you, I need to be in the part of the normal curve that starts at 0.8 SD, and goes to infinity SD. So you win when we're in the part of the normal curve that starts way to the left, goes past the peak, and stops at 0.8 SD.

You can see a picture of that at the top of this table:

http://www.math.unb.ca/~knight/utility/NormTble.htm

And looking up 0.80 in the table shows your probability of winning is .7881. So my probability of winning is 1 minus that, which is .2119.

Hope that helps.

 
At Friday, September 18, 2009 1:08:00 PM, Anonymous Anonymous said...

Thanks. That was helpful.

 
At Friday, September 18, 2009 3:39:00 PM, Anonymous Nate said...

Javageek,

In your first point, what does the SO stand for? I'm not sure I follow the difference between risk-free return and actual return in this example.

In your second point, this analysis is correct for swaps without carry costs. However, commodities require storage which is not always available or cheap (please see henry hub natural gas curve Oct-Jan at the moment...)

In your third point, I disagree. The pricing is such where the seller incurs substantial risk, but is rewarded with cash. There is no guarantee that anyone will make money on options - the market price is simply where 2 counterparties meet.

 
At Friday, September 18, 2009 9:22:00 PM, Blogger JavaGeek said...

The actual math behind my first point would be a little long, but is based on Brownian motion and the assumption that people value +$1 less than -$1.
So two assets prices one year from now might be:
A1 = $105 +/- $0
A2 = $115 +/- $45
A futures contract will be cost $105 now for both assets. Even though the expected price for A2 is 115 and A1 is 105.
Unless oil is risk free the future price today should not be the price tomorrow...
S0 was the initial price.

You're right if the "carry-costs" are extremely high the spot prices could fluctuate substantially between periods as there would be no way to use arbitrage to keep them similar.

My third point can be explained with a simple example:
Imagine a seller of a European put with strike $20 on Crude (let's say it costs 10 cents). Purchasers of this product would buy them in blocks of 1 million. Question: Would you buy this product from someone who had just $100,000 of usable capital? I certainly wouldn't! I'd make sure he had enough capital to pay for what he was selling. If Crude falls to $15 he would be on the hook for $5 million. If he doesn't have it he goes bankrupt and you're left with nothing. In other words part of the expense of providing a cheap put (or call) is the capital to pay out when needed (similar to an insurance company). Even if a seller of this put believed he could make a lot of money selling these items he cannot sell as many as he wants. So each company will be limited by the amount of capital they have.

 
At Tuesday, September 22, 2009 11:56:00 PM, Blogger Hugh said...

You don't have to be right. You just have to be slightly more right than the other guy.

 

Post a Comment

Links to this post:

Create a Link

<< Home