Posted by: Paul Hewitt | June 28, 2012

SCOTUS Prediction Markets Fail

Today, the Supreme Court of the United States handed down its ruling on whether parts of the Affordable Care Act were constitutional.  Various sites had set up prediction markets to try to predict how the SCOTUS would rule.  I looked at a few of them.  Inkling Markets had a few, the Wisconsin School of Business had one (on Inkling), and Intrade had a real-money market.

The short story is that none of the markets truly got the prediction “right”.  I’ll explain, below, what I mean by “right”.  Most of the markets were wildly wrong, in fact.  What went wrong?

Wisconsin School of Business Market

This prediction market was only concerned with the individual mandate (IM) component of the Healthcare Reform Act.  The options were to delay the decision, rule the IM unconstitutional and not severable from the remainder of the Act, rule the IM unconstitutional and severable, and rule that the IM is constitutional.

This market was unique, because trading ceased after April 27, 2012, two months prior to the outcome.  All other markets that I looked at either traded right up to the outcome or continue to be traded into the future.

This was the only market that got it “right”, in the sense that if we had relied on the market, we would have predicted that the individual mandate would be found constitutional.  Still, with only a 42.27% likelihood, we wouldn’t have been very certain of our decision.

Here are a few interesting observations.  There was very little trading in this market.  Only 220 trades in total, with most of them taking place on April 4, when the market resoundingly favoured the “unconstitutional and not severable option”.  There were a few small trades for the remainder of the trading period, which moved the likelihoods in the right direction.

Usually, prediction markets don’t start to become accurate until 30 days or so before the outcome.  This market appears to have arrived at a more accurate prediction a full two months before the outcome.  I have a theory as to why, which I will share with you, below.  Here is the trading activity and pricing:

Inkling Markets

Inkling runs a public marketplace, which included a similar, play money prediction market.  Here are the results for that market:

This market closed immediately prior to the outcome being revealed.  There were only 68 trades.  Again, very thinly traded.  The participants were way off in their predictions.  Medicaid Expansion was the one area that was iffy, in terms of its constitutionality, and it was predicted to be least likely to be struck down.  This market got everything wrong – way wrong.  Based on this, I’m thinking a bit like George Costanza on Seinfeld.  Do the opposite of everything you would normally do.

What about the trading activity?  Here’s how the market prices moved while the market was open.

When the market launched, there was a flurry of trading which heavily favoured the IM mandate being struck down.  This option increased in likelihood during the last week or so before the outcome.  The likelihood of the Premature Challenge option jumped a couple of weeks prior to the close.

A couple of observations are notable.  Apart from the Premature Challenge bump, the likelihoods for all of the options didn’t change all that much after the initial trades were made.  Obviously, there were very few trades after the initial activity, which kept the likelihoods about the same.  Could this be an indication that there was no new information available on which to trade?  Possibly, and this would be a good thing, because there was no new information, about the outcome, available to the market participants.  More likely, however, is that traders forgot about the market for a long period of time, until the issue was discussed in the media.

I think the media had a lot to do with the predictions of these markets.  There was no information leaking from the SCOTUS, at all.  There were, however, numerous pundits discussing the issue, and it was a highly politicized conversation.  Even CNN pundits were largely of the opinion that the Individual Mandate would be struck down.  Could it be that the participants relied on the pundits’ “information” in forming their predictions?  I think so.  I also think that predictions were, to a large extent, made based on the outcome the participants wanted to happen.  In other words, the “home team bias” appears to be quite evident in these markets.

The “home team bias” is a bias toward the outcome one wants to occur, not the one that is necessarily likely to occur.  Prediction markets are supposed to eliminate biases, such as this one.  Perhaps political persuasion trumps rational thought.  Lots of evidence of that!

Intrade

Intrade ran the only real money prediction market that I reviewed, yet if fared no better than the others.  Here is a chart of the trading activity.

Again, not a lot of trading, given that this issue is one of the most anticipated events to occur in a long time.  This chart shows trading over the last 17 months.  Once again, there really wasn’t any new information made available to the participants, other than the models they may have used to interpret existing data.  Yet, we see significant swings in the market price throughout the trading period.  How can this be?  Rather than new models or new information, it appears that the only thing “new” in this market is the dissemination of pundit views, which were eagerly lapped up by the participants.

Given the fact that there was very little relevant, new information about the subject, if any, perhaps trading should have been halted before the talking heads began their assault on our senses!

The following chart shows trading for the day leading up to the outcome (June 28, 2012):

The first thing to note is that this market runs until December 31, 2012.  Right up to the time when the SCOTUS decision was handed down, the market predicted that the Healthcare Act would be considered unconstitutional.  It appears that some traders bid the price up, when the decision was released (CNN initially reported that the IM had been declared unconstitutional).  Then, the price plummeted, once the outcome was known with more certainty.  All this shows is that the market can react to new information.  Prior to the release of real, new information, the market had no information to incorporate into the market price.

If we take these two charts together, we see that in the latter chart, the market was able to quickly incorporate new information into the market price, but the price change looks eerily similar to those in the former chart, where there was no new information available to the traders.  In other words, the trading charts look the same regardless of whether the participants are trading on new information or not.  That, is not how prediction markets are supposed to work!

How do we Know the Markets Failed?

Just because a market predicts one thing and something else occurs, doesn’t mean that the market is inaccurate.  Of course, it doesn’t mean the market is accurate, either!

When prediction markets try to predict binary events or discrete outcomes, there is no way to be almost right.  Relying on the market to make an accurate prediction, you will either be bang on or dead wrong.  In the case of these markets, except for the Wisconsin market, the predictions were dead wrong.  We would have made disastrous decisions relying on these markets.

It is difficult to compare these three markets, because they were asking different questions about the same subject.  However, there certainly seems to be a disconnect between the three groups of traders.  We do know one thing.  All of the participants had access to the same information, which was very little.  The Wisconsin market predicted, with a likelihood of 51%, that the IM would be unconstitutional.  Inkling predicted this with a likelihood of 91%!  How can the difference be 40%?  While the Intrade market was looking at any part of the Healthcare Act being unconstitutional, for the most part the issue concerned the IM.  Its prediction was about 70% likely!  Three markets, three wildly different predictions!

Given this wide range of predictions about the same issue, we can honestly say that at least two of them are not “accurate” and probably all three.  One of the requirements of accuracy is that the market be calibrated with the likelihood of actual outcomes.  This is an empirical test, which is impossible to perform on such issues and markets.  If we were able to have the Supreme Court issue numerous rulings on the same subject, we could, theoretically, obtain a distribution of ruling outcomes.  Ideally, we could compare the distribution with those from multiple prediction markets on the same subject.  If every 40% prediction came true 40% of the time, every 50% prediction came true 50% of the time, etc…, we would be able to say that the prediction market was well-calibrated.  We could rely upon the prediction market to accurately predict the likelihood of the various outcomes.

Still, the markets could get it wrong for any single prediction.  The market may have predicted that the IM would be struck down, with a 40% likelihood, but the actual outcome may be the opposite.  This may have been one of the 60% of the times that it would be ruled constitutional.  To further explain this point, if we only considered a market’s prediction to be “accurate” if it did, in fact, come true, the probability would have to be 100%, not 40% or 50% or some other likelihood!

Is There Any Usefulness in These Markets?

There is only one way that any of these markets might be useful to decision-makers.  If we could test for calibration to ensure that the likelihoods provided by the prediction market were reasonably accurate, we could use those predictions to perform a risk analysis.  The distribution of likelihoods given by the market represent the uncertainty surrounding the outcome from the SCOTUS.  Each outcome would have related consequences (costs and benefits), which could be quantified.  The risk for each outcome is the net cost multiplied by the uncertainty of the event occurring.  Contingency plans would be sought for any significant risks.

No one ever talks about this implication, but it is one of the few beneficial features of accurate prediction markets.

Other than that, there is little value in running prediction markets involving discrete outcomes.  This is especially true for markets designed to predict the outcome of a panel of experts, such as the nine judges that make up the SCOTUS, or the panel that chooses future Olympic cities, or the panel that selects the winner of Britain’s Got Talent, or…

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: