A new study of prediction markets in the corporate world was released, recently. It’s called Forecasting Consumer Products Using Prediction Markets, by Kai Trepte and Rajaram Narayanaswamy. Lo and behold, the prediction markets failed to provide any significant improvement in accuracy over that of the traditional corporate forecasting process. The authors submitted their paper as part of their masters program requirements. They don’t appear to have been beholden to any software vendor, though they did use the services of Consensus Point. Today’s entry will focus on the accuracy and usefulness of the prediction markets that were part of the study. A subsequent entry will cover other aspects of prediction markets that were discussed by the authors.
The good news is that the authors planned the operation of the markets well, and they used more participants than most studies we have seen. There appears to have been a conscious effort to maximize the diversity of the participants, but, like most of these studies, many of the prediction market participants also had involvement in the corporate forecasting process. Consequently, we could pretty much expect that the predictions would be fairly well correlated with the corporate forecasts, and they were. So, how did they compare?
The prediction markets weren’t failures, but they weren’t able to do any better than the established corporate forecasting process at General Mills, where 20 prediction markets were put in play. Despite the efforts of many academics, researchers, vendors and corporations, the breakthrough success story about enterprise prediction markets remains as elusive as ever.
FINDINGS & COMMENTARY
Correlation of Predictions and Forecasts
The Mean Absolute Percentage Error (MAPE) of the prediction market and the operations forecast (internal process) were highly correlated. As mentioned above, this is not surprising, given that many of those involved with the internal forecasting process were also involved with the prediction markets. Furthermore, the initial probability distribution for the potential outcomes were based on normal distributions around the internally forecasted mean. That is, the starting point for the prediction market was the corporate forecast. There were good reasons for doing this, but still, it may have introduced some bias toward the internal forecast.
The authors of the study found that the prediction market forecasts were virtually identical to those of the internal operations forecasting process, as evidenced by their means falling within one standard deviation of each other. Consequently, we could say that both processes/methods were good aggregators of available information, and any information that was generated internally was also available to the market participants.
Some Predictions are Better Than Others
The authors included three types of markets: Volume, Product Category and Promotional markets. The Volumemarkets were characterized by products that might be considered staples, with fairly stable consumption patterns. Internal forecasts and market predictions were both able to accurately gauge the future outcome. Product Category markets were a bit more difficult to predict or forecast, due to the nature of the products and strategies used. Finally, the Promotional markets, which were characterized by products that had very significant promotions planned, were the most difficult to forecast. Not even the corporate marketing people were very good at forecasting the effectiveness of the promotional activities. Again, both the internal forecasts and the market predictions were even less accurate, but still they were basically the same.
It appears that, if it is difficult to analyse data to come up with an accurate forecast, as was the case with the promotional markets, the use of a prediction market will not magically generate the information necessary to make a better prediction. We have seen this in other studies and examples, where there is a significant amount of uncertainty about the outcome. This is the information completeness principle that I’ve discussed previously.
Very Short Term Markets
I should note that the prediction markets were in operation for no longer than 10 weeks. The authors described some of their prediction markets as being “long term”, but in reality, there were anything but. In our quest for a useful enterprise prediction market, it must be able to generate consistently accurate predictions, sufficiently in advance, so that decision-makers are able to change their tactics, based on the predictions. In the study’s “longer term” markets, none were able to generate accurate predictions until very near the time when the actual outcome would have been set. In these cases, management would not have had time to change their tactics or decisions, once the market prediction had become known. Therefore, even if the prediction had been perfectly accurate, it is completely useless for any decision-making purposes.
Costs vs. Benefits
The authors did not discuss the issue of costs and benefits of prediction markets, but perhaps we should. Given that both the traditional forecasting process and the prediction markets provided equivalent forecasts, should General Mills’ management scrap their costly forecasting process and adopt these neat new tools? We can’t know for sure, right now, but if they were to discontinue the internal forecasting process, most of the useful information that needs to be aggregated in the prediction markets would not have been available to the participants. Accordingly, we would expect the predictions to become very inaccurate.
It would appear that the accuracy of the prediction markets depends upon the information created by the forecasting process. If you can’t have prediction markets without the internal forecasting, why would General Mills add prediction markets to the process? One reason might be to verify the accuracy of the internal forecast, but I’ll bet they already know that, historically, their forecasts are reasonably accurate for their decision-making purposes. They might consider eliminating the internal aggregation function, while continuing to generate forecasting information. Prediction markets would be relied upon to perform the aggregation of the information more efficiently. Finally, prediction markets generate distributions of possible outcomes along with the mean prediction or forecast. This information can be used to assess the risk and uncertainty surrounding the forecast, enabling management to make better contingency plans.
One of the benefits of prediction markets is their ability to filter out bias during the aggregation process. Consequently, I (and the authors) expected the prediction markets to provide significantly more accurate forecasts than those generated from the internal forecasting process. The fact that they were not more accurate means, to me, that General Mills’ internal forecasting process performs its function in a reasonably unbiased fashion. We should be studying why they have been able to minimize bias in their planning! Another possibility, which I find too scary to contemplate, is that prediction markets aren’t as good at filtering out the bias as we have been led to believe!
The authors don’t discuss the method of calculating the forecast or prediction error, other than to note that General Mills uses the MAPE (see above) to calculate their own internal forecast errors. I have a couple of issues with this approach (which was also used in the HP study). Using the absolute value of the error provides only the magnitude and no information about whether the prediction was an over or under-estimation. Accordingly, the actual error could be as much as twice the amount of the absolute error quoted. Also, the authors (and others) use the actual outcome as the denominator in the calculation of the average. This is incorrect, because it is the forecast (or prediction) value that is being evaluated, rather than the actual outcome. Management relies upon the prediction in order to make decisions. They don’t rely on the actual outcome (which isn’t known), when they are making decisions. Accordingly, the prediction value should be used in the denominator and not the actual outcome.
My next blog entry will cover the authors’ comments about the operation of these prediction markets and how well they appear to aggregate available information.