This is the second article about predicting Facebook’s closing share price after the IPO. The first article, Not So Intelligent Collective Intelligence, examined an attempt to access the collective intelligence of a crowd, through a poll. The results were remarkably wrong. The average prediction was $54, but the closing share price after the IPO was $38.23. The distribution of guesses ranged from $29 to $87 and had a standard deviation of 11.37. It was a relatively flat distribution, indicating that the participants knew very little about the subject. There appeared to be a significant herding effect, where those without much knowledge follow the guesses of those that appear to know the subject better. The attempt was a complete and utter failure.
So, that experiment got me to thinking. What if they had set up a prediction market instead of a poll? Would the results have been more accurate? Would the prediction have been accurate enough to be useful? This article takes a look at the issues involved in setting up such a prediction market and assesses whether it would have been successful.
There are a variety of considerations that should be addressed when setting up a prediction market. I won’t go into all of them, here, but I will address the crucial issues and the unique aspects of this particular application.
In the poll, there were 2,261 participants. So, I’ll assume that we will have a “crowd”. As we learned in the first article, almost all of the participants were male. It might be a good idea to attract more females to the group. We want to get as diverse and decentralized a crowd as we can. Let’s further assume that we can do this, because we’re most concerned about whether the prediction market model is capable of making an accurate prediction.
What Type of Security Should Be Used?
What is the reason for trying to predict the future share price? Let’s say it is to make a decision about buying or selling Facebook shares. Do we need an exact prediction, or is a range of possible prices adequate? If we need an exact price prediction, the prediction market would probably have to be set up as an indexed security. Participant trades would move the price up or down until the market closed. Unfortunately, this type of security doesn’t give us much information about the distribution of predictions, which would give us an idea about the amount of uncertainty surrounding the market prediction.
The poll used in the initial experiment appears to have used discrete share prices. But, given that the share price could be anything between the discrete dollar amounts, I’ll assume that a price of $54 really means any share price between $54.00 and $54.99. The problem with using such a small range in a prediction market is that there may be too many securities available to the traders. To cover the same range of prices that was covered in the poll, there would have to be 61 securities! Note that to be mutually exclusive and exhaustive, we would need to add a security for $28.99 and under and another for $88.00 and above. Even that setup requires additional thought, because we don’t want to unduly influence the trading by pre-setting the range of securities that are most likely to be true.
How Should We Structure The Market?
A better way to figure this out would be to determine the materiality level for the people who would use the market’s prediction. By this, I mean the size of the error that would cause the decision-makers to change their decisions. For example, I will purchase 1,000 Facebook shares if the predicted price is $40, but I will only purchase 500 if the price is $35 and only 250 if the price is $30. From this, it appears that the materiality level is $5 per share. Therefore, we don’t have to concern ourselves about being too exact in our predictions.
Some research would have to be undertaken to determine the likely minimum and maximum share prices. Let’s say they are $25 and $60. Given our materiality level of $5, we’ll create securities covering the following ranges:
$25 – $29.99
$30 – $34.99
$35 – $39.99
$40 – $44.99
$45 – $49.99
$50 – $54.99
$55 – $59.99
This gives us a reasonable number of securities and covers every possible share price.
Another issue concerns the level of confidence that a typical decision-maker will require before making a decision about whether to invest in Facebook shares. We may need to adjust the range of values for each security to reflect the standard deviation required to achieve a desired level of confidence. After trading begins, the distribution of trades will reveal a market prediction and a standard deviation. The size of the standard deviation will determine the level of confidence that the actual market price will be within the materiality level.
How Will Participants Trade?
Here we have a choice between using an Automated Market Maker (AMM), such as the Logarithmic Market Scoring Rule, or using a Double Auction method (usually continuous). Automated Market Makers are usually used to ensure liquidity in markets that don’t have enough traders. This method allows anyone to make a trade, even if there is no other participant that wishes to take the opposite position.
The Continuous Double Auction (CDA) method maintains an open book of bids (to buy) and of asks (to sell) securities. The highest bids are listed first, as are the lowest
Assuming there are enough participants to make a “crowd”, the better option is to go with the Continuous Double Auction, because it requires a greater consideration of the risks of buying and selling securities. In a CDA market, a trader has to consider that it may not be possible to trade out of a position once it has been taken. There may not be a willing buyer. Contrast this with an AMM market, where all trades are executed. Markets based on an AMM tend to encourage risk-seeking behaviour. Ideally, we would prefer to have risk-neutral investing decisions, which will provide the most unbiased decisions and the greatest accuracy. When the highest bid matches the lowest asking price, a trade takes place.
If we go with the AMM mechanism, we may have a problem with too many uninformed traders. We saw the effects of this in the polling structure (article one), where the chimps swamped the experts. In a CDA based market, we actually want a few chimps (or chumps), because they will provide the liquidity that makes it possible for the better informed traders to effect their trades. So, let’s go with a Continuous Double Auction market.
Incentives to Trade
One of the key functions of a prediction market is that it gives participants incentives to trade on their privately-held information. Searching for information is costly. So, there must be some benefit to entice participants to gather new information. One way is to provide a real monetary reward, if the information is more accurate than that already in the market. Unfortunately, in the U.S., real money prediction markets are not allowed (with some exceptions). So, we have to find another way to compensate traders for seeking out new information.
Studies have shown that play money markets can offer sufficient incentives for participants to gather information and make trades. Many such markets create a leaderboard to rank the best traders. Over time, the traders who acquire the best information will make more profitable trades and acquire more “wealth” relative to the uninformed traders (“chimps”).
Interestingly, in my last article, I mentioned that Ville Miettinen seemed to equate credentials with expertise. Then, he showed that, in the Facebook IPO poll, a lot more non-experts guessed the correct share price than did the “experts”. In prediction markets, we don’t use outside credentials (degrees, job, position, or any other criterion) to determine whether one is an expert. Instead, we let the market identify the experts through their superior trading. Being knowledgeable about social media, for example, does not make you an expert in predicting Facebook’s post IPO share price!
So, we’ll need a leaderboard to keep track of trader performance, and it would be a good idea to have prizes for the top traders. Being the best trader among thousands feeds the ego quite nicely, but being able to take your spouse out to dinner, with your prize, might make it all worthwhile.
Are We There Yet?
Uh, no. If we set the opening odds, or likelihoods, equally among the securities, there will be windfall profits to be taken by those that make the first trades. This is more relevant in markets that use an Automated Market Maker, but there will still be some excessive profit opportunities, using minimal information, in markets using a Continuous Double Auction mechanism. Therefore, we need to set the initial likelihoods based on the best information available.
Since the poll, described in the first article, was set up on an ad hoc basis, we have to assume that this prediction market would have been set up this way, too. That means that everyone who participates will have the same initial wealth from which they can make investments in the market.
One of the functions of prediction markets is to identify the experts and give them more power to move the market than is given to non-experts. This is a natural phenomenon of prediction markets. Those that make the most accurate predictions, earliest, make the most profitable trades and amass the greatest wealth. Those that trade on erroneous or minimal information make losing trades and end up losing their wealth and exit the marketplace. The problem with this being the first market is that everyone has the same wealth. No experts have been identified. Chimps can move the market just as much as the experts can.
What if there are too many chimps in the market? In the polling case, we saw that there was a very wide distribution and evidence of herding behaviour. It’s easy to make the case that the market was dominated by chimps. In a one vote for all poll, the votes of the chimps ameliorated the accurate votes of the experts (whoever they may have been). We have a similar problem, here, because everyone has the same trading power at the beginning.
There is only one solution, and it is an impossible one. There must have been a sufficient number of previous prediction markets, about similar subjects, involving many of the same participants as we have in this market. That is the only way that the “cream” could rise to the top of the leaderboard and the influence of the chimps could be lessened. Unfortunately, at this point, we have no choice but to go with the current market, knowing that it is fatally flawed.
What Would Happen?
Given that this is a one-time market, the leaderboard will be based solely on the results of trading in this market. At least we have eliminated some of the risk-seeking behaviour by going with the CDA mechanism, but there is still likely to be a significant amount of risk-taking among the participants. Those that purchase the outlier securities stand to make the most profit, if they are correct. Go big or go home. If they’re wrong, who really cares?
These are the long-shot trades, similar to picking the long-shot horse in a race. There’s a well-known long-shot bias in horse racing, where more bets are placed on the long-shot horses than are deserving, based on the actual outcome likelihoods. It is likely to be even more pronounced in this market.
It is unlikely that the prediction market will yield a distribution of predictions that is as wide as the one exhibited by the poll. However, it is still likely to reflect a significant amount of uncertainty about the true share price, because the real experts don’t have enough wealth to make the market reflect their information.
The only things that would be positive about this experiment is that we would have an improved leaderboard and we would have some data from which to start assessing the calibration of this type of prediction market. Many more similar markets would have to be run to identify the experts and enable us to measure the calibration of prediction market distributions with the distribution of actual outcomes.
The point of the exercise has been to show that trying to make predictions using ad hoc markets is frivolous. We cannot rely upon these ad hoc markets to deliver accurate predictions. It takes time to operate a sufficient number of similar markets to determine whether they are “accurate”. There are no shortcuts. Rather than try these one-off experiments, we should be looking at long-term solutions.