Posted by: Paul Hewitt | March 14, 2010

Truth in Advertising – Meet Prediction Markets

Most published papers on prediction markets (there aren’t many) paint a wildly rosy picture of their accuracy.  Perhaps it is because many of these papers are written by researchers having affiliations with prediction market vendors.

Robin Hanson is Chief Scientist at Consensus Point.  I like his ideas about combinatorial markets and market scoring rules, but I think he over-sells the accuracy and usefulness of prediction markets.  His concept of Futarchy is an extreme example of this. Robin loves to cite HP’s prediction markets in his presentations.  Emile Servan-Schreiber (Newsfutures) is mostly level-headed but still a big fan of prediction markets. Crowdcast’s Chief Scientist is Leslie Fine; their Board of Advisors includes Justin Wolfers and Andrew McAfee.  Leslie seems to have a more practical understanding than most, as evidenced by this response to the types of questions that Crowdcast’s prediction markets can answer well: “Questions whose outcomes will be knowable in three months to a year and where there is very dispersed knowledge in your organization tend to do well.”  She gets it that prediction markets aren’t all things to all people.

An Honest Paper

To some extent, all of the researchers over-sell the accuracy and the range of useful questions that may be answered by prediction markets. So, it is refreshing to find an honest article written about the accuracy of prediction markets.  Not too long ago, Sharad Goel, Daniel M. Reeves, Duncan J. Watts, David M. Pennock published Prediction Without Markets.  They compared prediction markets with alternative forecasting methods for three types of public prediction markets: Football and baseball games and movie box office receipts.

They found that prediction markets were just slightly more accurate than alternative methods of forecasting.  As an added bonus, these researchers considered the issue that prediction market accuracy should be judged by its effect on decision-making.  So few researchers have done this!  A very small improvement in accuracy is not considered material (significant), if it doesn’t change the decision that is made with the forecast.  It’s a well-established concept in public auditing, when deciding whether an error is significant and requires correction.  I have discussed this concept before

While they acknowledge that prediction markets may have a distinct advantage over other forecasting methods, in that they can be updated much more quickly and at little additional cost, they rightly suggest that most business applications have little need for instantaneously updated forecasts.  Overall, they conclude that “simple methods of aggregating individual forecasts often work reasonably well relative to more complex combinations (of methods).”

For Extra Credit

When we compare things, it is usually so that we can select the best option.  In the case of prediction markets it is not a safe assumption that the choices are mutually exclusive.  Especially in enterprise applications, prediction markets are heavily dependent on the alternative information aggregation methods as a primary source of market information.  Of course, there are other sources of information and the markets are expected to minimize bias to generate more accurate predictions.  

In the infamous HP prediction markets, the forecasts were eerily close to the company’s internal forecasts.  It wasn’t difficult to see why.  The same people were involved with both predictions!  The General Mills prediction markets showed similar correlations, even when only some of the participants were common to both methods. The implication of these cases is that you cannot replace the existing forecasting system with a prediction market and expect the results to be as accurate.  The two (or more) methods work together. 

Not only do most researchers (Pennock et al, excepted) recommend adoption of prediction markets, based on insignificant improvements in accuracy, they fail to consider the effect (or lack thereof) on decision-making in their cost/benefit analysis.  Even if some do the cost/benefit math, they don’t do it right.   

Where a prediction market is dependent on other forecasting methods, the marginal cost is the total cost of running the market. There is no credit for eliminating the cost of alternative forecasting methods.  The marginal benefit is that expected by choosing a different course of action than the one that would have been taken based on a less accurate prediction.  That is, a slight improvement in prediction accuracy that does not change the course of action has no marginal benefit. 

Using this approach, a prediction market that is only “slightly” more accurate, than those from alternative forecasting approaches, is just not good enough.  So far, there is little, if any, evidence that prediction markets are anything more than “slightly” better than existing methods.  Still, most of our respected researchers continue to tout prediction markets.  Even a technology guru like Andrew McAfee doesn’t get it , in this little PR piece he wrote, shortly after joining Crowdcast’s Board of Advisors.

Is it a big snow job or just wishful thinking?

Advertisements

Responses

  1. […] from Toronto Prediction Market Blog] Posted in All Best Posts Ever, All Guest Authors's Posts, Analysis (Accuracy & Precision), […]

  2. This is a great post: Thanks Paul. Thanks for you interest in our paper and careful reading and analysis. I think most of my coauthors would agree with you, though my own take on our results are a bit different. In our paper, we don’t say anything about cost. In some contexts, markets may be cheaper to run than polls: in fact most markets try to operate at a profit (negative cost). So if some method is both the most accurate (albeit by a small amount) and the least costly, it’s hard to argue against. So to me the message of the paper is to think more carefully about benefit compared to cost.

  3. David, I realize your paper did not discuss the issue of cost. I was attempting to extend the relevance of your conclusion. It is my contention that, in many cases, the accuracy of the prediction market is dependent on information provided by alternative forecasting methods.

    Consequently, in this circumstance, it is not a simple comparison of the net benefits of each method.

    Thank you for your comments. I would appreciate your reviewing some of my other posts, too.

    P.S. Predictalot was operational this morning. It wasn’t working before.

  4. I work in med&lab equipments co.I am sales manager.I need to information about forecast for my company.I study MBA too and need to provide of forecast for my thesis.please contact to me and help me for providing of forecast.

    Best regards
    Ali Didari
    Tehran.Iran

  5. […] Here for the debunking. More. Posted in Collective Forecasting, Collective Intelligence – Wisdom Of Crowds, Exchanges & Markets | Tagged Aljazeera, Aljazeera TV, betting markets, Consensus Point, event derivative markets, event derivatives, InTrade, prediction markets, Robin Hanson, television, TV, video Cancel Reply […]

  6. […] Pennock et al showed that prediction markets were accurate (in the cases they studied), but they were not significantly more accurate than alternative prediction methods.  They concluded that in order for prediction markets to be useful, they must be significantly better than alternative forecasting methods.  In the cases they studied, they found prediction markets were only slightly better than other methods.  In previous posts, I introduced the concept of materiality to the analysis of prediction markets.  Essentially, for a prediction market to be useful, it must be more accurate than the next best predictor, such that the more accurate prediction would make a difference to the decision-maker relying on the forecast.  Then, we need to look at the costs and benefits to determine whether the use of prediction markets is a wise course of action. […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: