Recently, there have been several very glaring public prediction market failures, including the IOC site selection and the Economics Nobel Prize markets. Some followers of prediction markets are a bit shocked and concerned, but most, like Chris Masse (Midas Oracle), me, and others are not. These particular types of prediction markets never had a chance to be accurate. Had any of these markets actually managed to “pick” the right outcome, it would have been nothing more than a fluke. Why we continue to waste our time on these types of markets, I’ll never understand.
Jed Christiansen (Mercury’s Blog) is an occasional commenter on Midas Oracle. I may not always agree with him, but I respect his positions in a number of areas. However, in response to these very public failures of prediction markets, Jed provided a number of factors that influence the accuracy of prediction markets. It appears that his comments apply only to outcomes that are determined by a group. Essentially, he means outcomes that are determined through some form of voting or polling, including elections, IOC site selection, Academy Awards, Nobel Prizes, etc… While I applaud his efforts to identify the factors affecting prediction market accuracy, I find some of his comments confusing.
For example, Jed mentions that “more members/voters will be better than fewer” (in terms of improving the accuracy of prediction markets). In these types of markets, the members/voters are determining the actual outcome. This is entirely independent of a prediction market attempting to predict that same outcome. Consequently, having more members involved in determining the actual outcome will have no effect, whatsoever, on the accuracy of any related prediction market. Jed’s comment makes no sense.
Jed is absolutely correct to say that “more objective criteria will be better than less.” However, all this means is that the more objective the determinants of the outcome, the more likely the market participants will be able to figure them out and predict the outcome. The fewer the factors and the less uncertainty surrounding their roles in determining the outcome, the easier it will be to predict the actual outcome. In the extreme, if a condition arises that determines (or causes) the future outcome with a high degree of certainty, the market will be able to predict with uncanny precision. However, if the outcome is this easily predicted, perhaps a simple decision model (If… Then…) would have provided the same “prediction”, without the bother of setting up a prediction market.
Generally, I would agree with Jed that “constrained choices will be better than unconstrained choices.” In keeping with this statement, the fewer the choices, the more likely it is that the outcome will be predictable (only because there are fewer incorrect options)! However, the IOC markets showed that, even with only four choices, the markets failed. The real problem is that these markets did not have the necessary information to choose among even a very small number of alternatives.
Again, I agree with Jed that “voters signalling choices before a vote is better than if they don’t.” Where the outcome is determined by a vote, any prior information about how some or all of the group intends to vote will be important information to be assessed by the market participants. This merely supports the information completeness principle. We see many examples of this type of information being accessed by participants (in the IOWA political prediction markets) where political polling influences the market prices.
Finally, Jed made a curious statement about “secretive and less secretive” committees that make decisions and that “neither will likely be as accurate as traditional open prediction markets.” I have no idea what he means, here! The committees (secretive or not) are the ones determining (creating) the actual outcome. The committee has nothing to do with being “accurate” or predicting the outcome. Traditional markets are expected to predict actual outcomes. Jed is simply wrong to try and compare these two concepts!
Panos Ipeirotis asked if there is a more principled method of capturing the determinants of prediction market accuracy. In response, I would suggest that we look to the first principles of prediction markets. Perhaps the most important of which is that the market possess a sufficient degree of information completeness. In the examples noted, the prediction market participants did not have an adequate level of information completeness to be able to arrive at accurate predictions, because the method of determining the outcome was far too complex and subjective, even when the choices were limited to four.
The only way, to provide the necessary information to the prediction market, in order for it to accurately determine the otucome, would have been to make all (or many) of the outcome voters (committee members) participants in the prediction market. Of course, this would be a needless redundancy. Note that in most of the enterprise prediction markets, many of the participants also take part in the internal forecasting process, effectively including the body of corporate information in the prediction markets. If internal forecasting processes were to be replaced by prediction markets, it is highly doubtful that the markets would be able to provide accurate predictions. The required information to make those accurate predictions would be missing.
These types of markets suffer from a fatal flaw, as well. They are trying to predict a discrete (non-continuous variable) outcome. “Coming close” means being completely wrong. These types of markets are only suitable for betting purposes, and even then, only if they are proven to be “well-calibrated”. It is questionable whether these particular markets were well-calibrated.
I have written fairly extensively on the determinants of prediction market usefulness. I am especially concerned with their accuracy and consistency, for without these, their use in decision-making is not warranted. I draw your attention to the following posts:
To answer Panos, we do have a general, principled model for assessing prediction market accuracy. Now, we need to fill in the details.