Posted by: Paul Hewitt | December 1, 2009

Measuring Decision Market Accuracy

I came across this post: On Prediction Markets for Climate Change by Rajiv Sethi, an economics professor at Columbia University.  In his post, he makes a very interesting point that I have yet to see in any research paper about prediction markets.  He was commenting on the recent debate between Matt Yglesias and Nate Silver, regarding the use of prediction markets to help guide policy about climate change.  By way of a very brief summary, Matt believes that big business (coal and oil) will manipulate the market to influence the setting (or not) of policies that would be detrimental to their interests.  Nate thinks this is rubbish.  If the markets are broad-based and have sufficient liquidity, attempts to manipulate the market price will not succeed.  Nate thinks the markets would be “efficient”, providing market prices that accurately aggregate available public information.

Compelling Logic?

Here is where it starts to get interesting.  Rajiv comments that the logic of Nate Silver’s position is so compelling, it simply must be true.  That is, broader participation and more liquidity makes for efficient markets that generate more accurate prices.  To his credit (and I might add that he seems to be the only one), Rajiv set out to see whether this holds up in the real world.  He used Intrade and IEM markets about the 2008 election.  The hypothesis was that the IEM markets, with a more limited base and lower trade volumes, should have been less efficient than the Intrade markets.  Instead, he found the opposite!  Compelling, indeed.

How Do You Measure Efficiency?

“First of all, let’s think for a minute about how one might determine which of two markets is aggregating information more efficiently. We can’t just look at events that occurred and examine which of the two markets assigned such events greater probability, because low probability events do indeed sometimes occur.   If we had a very large number of events (as in weather forecasting) then one could construct calibration curves to compare markets, but the number of contracts on IEM is very small and this option is not available. So what do we do?”

This paragraph from Rajiv’s post, summarizes the problem of determining whether a market is “accurate”.  We believe that if a market is well-calibrated, the distribution of its market prices will be “accurate”, reflecting all market information about the outcome.  Consequently, it will be described as “efficient”.  He points out the difficulty (in most cases the impossibility) of measuring the calibration of a market and asks “what do we do?”

Essentially, he comes to the conclusion that it is impossible to measure the efficiency of a market.  However, it is possible to say which market is more efficient.  In other words, we can determine relative efficiency of two markets.  He outlines a cross-market arbitrage mechanism that could be used to eliminate price differentials for identical contracts in different markets.  You can read the approach in his post, cited above.  While he did not actually run the arbitrage experiment, he did perform an informal test to determine which of two markets was more efficient. 

The market with the smaller change in price is the more efficient of the two markets.  Effectively, then, the more efficient market’s price will be a better predictor of the future market price in the other market.  This was how he determined that the IEM markets were more efficient than those in Intrade, despite there having a limited participant pool and lesser liquidity.

So far, we have been able to determine which of two markets is the more efficient, but we don’t know how much more efficient.  Also, we don’t know whether either market is  sufficiently “efficient” for the purpose of determining its accuracy.  Both markets may be “inefficient”, yielding inaccurate or misleading market prices. 

How did IEM do it?

Rajiv gives two possible explanations as to why the IEM markets were more efficient than the Intrade ones.  Neither explanation is good news for Nate Silver’s position.

One explanation has manipulative traders moving into the Intrade markets, in order to influence the prices (odds) quoted in the media and in political blogs.  The argument is that Intrade prices were much more widely cited than those of the IEM markets.  The reasoning goes that temporary dips in market prices can be eliminated through manipulative trading.  A political party may wish to see this done, so as not to upset campaign contributions or to minimize the impact of negative information.  The author argues that the benefit of such manipulative trading could be far in excess of the cost.  Since IEM’s markets were not as widely cited in the media or blogosphere, there was a lesser incentive to manipulate prices there.

Even if we believe the research (limited) on manipulation in prediction markets, it is more than likely that a short term (maybe even a very short term) manipulation could persist long enough to achieve the intended objective.   For example, the price could be manipulated just prior to when news stories are being finalized for the following day’s paper.  Once the paper hits the streets, the manipulated price may have been corrected, but the damage has already been done.  And this is the “best case” scenario regarding prediction market manipulation.  In the worst case, the manipulation is successful as the market is unable to correct the inaccurate price.

I’m not an expert on US campaign finance, but I wonder whether an Intrade market manipulator would need to declare the amount of funds used to implement the price manipulation scheme (or whether such a person or corporation would be considered a donor at all).  If the answer is no, it would provide an additional incentive for parties or candidates to manipulate the markets for political purposes (without having to account for the funds used).  We all know what happens when incentives are strengthened.

The other explanation is that inefficient markets attract higher participation rates and market liquidity, as traders seek to profit from inaccurate prices.  Efficient markets have fewer profit opportunities and less trading is required to keep prices accurate.  As Rajiv explains, Nate Silver is caught in a paradox.  Nate’s attempt to design a market with high participation and strong liquidity, in order to achieve efficiency (and hence, accurate prices), conflicts with Rajiv’s finding that it is the market inefficiency that generates the high participation and liquidity.

The Road Ahead

Despite all of these arguments, Rajiv Sethi believes that prediction markets on climate change topics should be tried.  He suggests that corresponding markets be offered in other marketplaces, such as the IEM, so that market efficiency comparisons can be performed and studied.  I’m sure useful information could be gleaned from this effort. 

We need to keep in mind that some (or most) prediction markets may not work, however.  The objective of prediction markets is to accurately aggregate information held by the market participants.  If those participants do not have the information (or are unable to get it and profit from it), the market will be unable to generate an accurate prediction or there will be too much uncertainty about the prediction, rendering it useless for decision-making.

Personally, I like the idea of decision markets, but I think we will find that our efforts to use these markets to help guide climate change policy will ultimately fail.  There is simply too much information that is needed to accurately predict the important metrics.  It is hopeless to think that, not only will there be “informed” traders, they will be able to counteract the trading of the uninformed traders and the manipulators.  Any useful standard of “informed” traders might result in a mere handful of individuals spread throughout the world.  The impact of manipulators would swamp any efforts of the informed to set the “right” price in the market.  That said, there may be metrics that can be predicted (with reasonable accuracy) by a large number of traders.  Such predictions could be used as inputs into public policy decision models.  As with all prediction markets, the predictions must be accurate and consistently so.


  1. […] prediction markets Written by Chris F. Masse on December 2, 2009 — Leave a Comment Measuring Decision Market Accuracy Posted in Analysis (Market Proposals), Analysis (Meta), Collective Forecasting, Exchanges & […]

  2. […] a matter of prediction market efficiency?  If so, how do we know whether a market is efficient?  Rajiv Sethi provides us with an approach to determining which market is more efficient, but not whether the […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: