Using crowds to assess people's assesments of crowds

Speaking of net.timesinks ... yesterday a colleague pointed me at the eyeballing game. As timesinks go it's considerably more tasty and nutritious than a lot of stuff out there. Being able to eyeball distances and angles is a useful skill. Finish the game and you can also see what looks like a classic log-normal distribution, long tail and all.

But what also caught my attention was the link to a stock chart game. For ages and ages there has been a raging debate between "chartists" and "fundamentalists" over how best to trade stocks, commodities and such. In broad strokes, the fundamentalists argue from the efficient market hypothesis that prices will tend over time to reflect the intrinsic value of the good being traded. If you have a better idea than of that value than the rest of the world, you can make money by buying when the market price is lower than that value and/or selling when it's higher.

Chartists argue from the efficient market hypothesis that the market will reflect all the information available to everyone and that the odds you know better than the market are pretty low. However, the market is really just a bunch of people placing their bets according to all kinds of none-too-rational measures. This will result in certain patterns of trading, for example masses of people piling into a stock because it's "hot" or bailing out because it's "cold". These can be detected purely by looking at the charts without regard to what the underlying good is worth, or even what it is.

Fundamentalists then scoff that since there are hordes of chartists out there using the same techniques, the odds that you can beat all of them in aggregate are pretty low.

Economists are probably highly amused by all this. They may also point out that the efficient market hypothesis is just that, a hypothesis [or maybe better, an idealized model], and that whatever answer markets might give, there are large error bars around it. But then, economists probably don't have a lot of money riding on the outcome.

I'm neither economist nor trader, but I remember someone saying that the market is a voting machine in the short run and a weighing machine in the long run, and someone else pointing out that in the long run we're all dead. And don't forget transaction costs. The house gets its cut no matter who wins or loses.

But I digress.

The eyeballing guy wants to test this all out empirically, so he's set up a game. In it, you get to see a price history from some unnamed S&P stock over some unspecified time period. You see the prices one by one and at each point you can say "buy" or "sell". At the end you get to see which stock and time period you were trading, and you win if your buys and sells did better than a simple "buy and hold".

By the way, it shouldn't be too hard to game this. There are only 500 or so S&P stocks (including stocks that are no longer in the S&P) and only a few thousand time intervals, so you should be able to get a signature from the first few data points and nail down just which chart you're dealing with. Leaving that aside, there are some things to be careful of when interpreting the results, particularly the difficulty of distinguishing a particular performance from random chance. There's a scam that plays on that:

• Week 1: I send out 16 sets of letters touting my trading prowess. Half say that security X will go up, half say it will go down.
• Week 2: I send out eight sets of letters to the recipients that got the right answer last time. Half say that security Y will go up, half say it will go down.
• Weeks 3-4: Likewise
• Week 5: I send a letter to everyone that's still in, saying "See, I just made five brilliant picks in a row. I'm a genius! Now send me a lot of money."

Likewise, someone playing the chart game is going to win big. That doesn't prove anything. Ideally you'll be able to look at the aggregate and tell whether there are more wins than you would expect by chance. But that only works if the aggregate result is better than chance, presumably because enough people are using a good system. Otherwise, you can just claim that clearly the winners were using the "right" system and the losers were using the "wrong" one. Throw out those pesky losers and hey, charting works! [From a slightly different angle: without any controls, how can you correlate the raw performance data with the method the trader was using?]

My guess is that this particular site won't make that mistake, but it's certainly been made before.