Posted by: Paul Hewitt | September 30, 2009

Corporate Prediction Market Success is Elusive

A new study of prediction markets in the corporate world was released, recently.  It’s called Forecasting Consumer Products Using Prediction Markets, by Kai Trepte and Rajaram Narayanaswamy.  Lo and behold, the prediction markets failed to provide any significant improvement in accuracy over that of the traditional corporate forecasting process.  The authors submitted their paper as part of their masters program requirements.  They don’t appear to have been beholden to any software vendor, though they did use the services of Consensus PointToday’s entry will focus on the accuracy and usefulness of the prediction markets that were part of the study.  A subsequent entry will cover other aspects of prediction markets that were discussed by the authors.

The good news is that the authors planned the operation of the markets well, and they used more participants than most studies we have seen.  There appears to have been a conscious effort to maximize the diversity of the participants, but, like most of these studies, many of the prediction market participants also had involvement in the corporate forecasting process.  Consequently, we could pretty much expect that the predictions would be fairly well correlated with the corporate forecasts, and they were.  So, how did they compare? 

The prediction markets weren’t failures, but they weren’t able to do any better than the established corporate forecasting process at General Mills, where 20 prediction markets were put in play.  Despite the efforts of many academics, researchers, vendors and corporations,  the breakthrough success story about enterprise prediction markets remains as elusive as ever. 

FINDINGS & COMMENTARY

Correlation of Predictions and Forecasts

The Mean Absolute Percentage Error (MAPE) of the prediction market and the operations forecast (internal process) were highly correlated.  As mentioned above, this is not surprising, given that many of those involved with the internal forecasting process were also involved with the prediction markets.  Furthermore, the initial probability distribution for the potential outcomes were based on normal distributions around the internally forecasted mean.  That is, the starting point for the prediction market was the corporate forecast.  There were good reasons for doing this, but still, it may have introduced some bias toward the internal forecast.

The authors of the study found that the prediction market forecasts were virtually identical to those of the internal operations forecasting process, as evidenced by their means falling within one standard deviation of each other.  Consequently, we could say that both processes/methods were good aggregators of available information, and any information that was generated internally was also available to the market participants. 

Some Predictions are Better Than Others

The authors included three types of markets:  Volume, Product Category and Promotional markets.  The Volumemarkets were characterized by products that might be considered staples, with fairly stable consumption patterns.  Internal forecasts and market predictions were both able to accurately gauge the future outcome.  Product Category markets were a bit more difficult to predict or forecast, due to the nature of the products and strategies used.  Finally, the Promotional markets, which were characterized by products that had very significant promotions planned, were the most difficult to forecast.  Not even the corporate marketing people were very good at forecasting the effectiveness of the promotional activities.  Again, both the internal forecasts and the market predictions were even less accurate, but still they were basically the same.

It appears that, if it is difficult to analyse data to come up with an accurate forecast, as was the case with the promotional markets, the use of a prediction market will not magically generate the information necessary to make a better prediction.  We have seen this in other studies and examples, where there is a significant amount of uncertainty about the outcome.  This is the information completeness principle that I’ve discussed previously.

Very Short Term Markets

I should note that the prediction markets were in operation for no longer than 10 weeks.  The authors described some of their prediction markets as being “long term”, but in reality, there were anything but.  In our quest for a useful enterprise prediction market, it must be able to generate consistently accurate predictions, sufficiently in advance, so that decision-makers are able to change their tactics, based on the predictions.  In the study’s “longer term” markets, none were able to generate accurate predictions until very near the time when the actual outcome would have been set.  In these cases, management would not have had time to change their tactics or decisions, once the market prediction had become known.  Therefore, even if the prediction had been perfectly accurate, it is completely useless for any decision-making purposes.

Costs vs. Benefits

The authors did not discuss the issue of costs and benefits of prediction markets, but perhaps we should.  Given that both the traditional forecasting process and the prediction markets provided equivalent forecasts, should General Mills’ management scrap their costly forecasting process and adopt these neat new tools?  We can’t know for sure, right now, but if they were to discontinue the internal forecasting process, most of the useful information that needs to be aggregated in the prediction markets would not have been available to the participants.  Accordingly, we would expect the predictions to become very inaccurate. 

It would appear that the accuracy of the prediction markets depends upon the information created by the forecasting process.  If you can’t have prediction markets without the internal forecasting, why would General Mills add prediction markets to the process?  One reason might be to verify the accuracy of the internal forecast, but I’ll bet they already know that, historically, their forecasts are reasonably accurate for their decision-making purposes.  They might consider eliminating the internal aggregation function, while continuing to generate forecasting information.  Prediction markets would be relied upon to perform the aggregation of the information more efficiently.  Finally, prediction markets generate distributions of possible outcomes along with the mean prediction or forecast.  This information can be used to assess the risk and uncertainty surrounding the forecast, enabling management to make better contingency plans.

Filtering Bias

One of the benefits of prediction markets is their ability to filter out bias during the aggregation process.  Consequently, I (and the authors) expected the prediction markets to provide significantly more accurate forecasts than those generated from the internal forecasting process.  The fact that they were not more accurate means, to me, that General Mills’ internal forecasting process performs its function in a reasonably unbiased fashion.  We should be studying why they have been able to minimize bias in their planning!   Another possibility, which I find too scary to contemplate, is that prediction markets aren’t as good at filtering out the bias as we have been led to believe!

Calculating Accuracy

The authors don’t discuss the method of calculating the forecast or prediction error, other than to note that General Mills uses the MAPE (see above) to calculate their own internal forecast errors.  I have a couple of issues with this approach (which was also used in the HP study).  Using the absolute value of the error provides only the magnitude and no information about whether the prediction was an over or under-estimation.  Accordingly, the actual error could be as much as twice the amount of the absolute error quoted.  Also, the authors (and others) use the actual outcome as the denominator in the calculation of the average.  This is incorrect, because it is the forecast (or prediction) value that is being evaluated, rather than the actual outcome.  Management relies upon the prediction in order to make decisions.  They don’t rely on the actual outcome (which isn’t known), when they are making decisions.  Accordingly, the prediction value should be used in the denominator and not the actual outcome.

My next blog entry will cover the authors’ comments about the operation of these prediction markets and how well they appear to aggregate available information.

Advertisements

Responses

  1. Hey, I read a lot of blogs on a daily basis and for the most part, people lack substance but, I just wanted to make a quick comment to say GREAT blog!…..I”ll be checking in on a regularly now….Keep up the good work! 🙂

  2. Great post. I used to sit in on those forecasting meetings when I was a brand manager at Procter & Gamble. I was always amazed at how methodically the forecasting process was run (especially compared to all the bias/postering when we were advocating to corporate for resources to fund our brand initiatives).

    I think looking at big consumer product companies for evidence of the power of prediction markets might set the bar too high. These companies are just too sophisticated in their forecasting processes and prediction markets, arguably, aren’t yet sophisticated enough.

    My company is focused on crowdsourcing demand forecasting in fashion. Here the bar is very low – the forecasters are merchants and, for the most part, they guess through consensus decision-making! In fashion, last year’s sales numbers are much less important than with consumer staples so the margin of error is typically very large.

    I personally don’t believe prediction markets are the end-all be-all. However, I do think crowdsourcing demand, like many innovations before it, will keep growing in sophistication and will become a meaningful compliment if not replacement for current more qualitative methodologies.

  3. […] the pilot markets generated materially more accurate predictions than the official forecasts.  The General Mills prediction markets, using much larger crowds than the HP markets, were no better than the internal […]

  4. […] wasn’t difficult to see why.  The same people were involved with both predictions!  The General Mills prediction markets showed similar correlations, even when only some of the participants were common […]

  5. […] It wasn’t difficult to see why. The same people were involved with both predictions! The General Mills prediction markets showed similar correlations, even when only some of the participants were common […]

  6. […] General Mills trials showed that prediction markets were as good as internal methods, but they were not […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: