“Our forecasters are simply the best!  (That’s not just our opinion:  in the early days of the tournament, the Good Judgment Team’s aggregate forecasts have proven to be more accurate than those of any other research team participating in the IARPA tournament.)”

This got me to thinking.  How is the IARPA determining which Team is more accurate in their predictions?  I’ve posed the question to my team, but haven’t received a response, yet.  So, let’s make a few educated guesses.

Each Team has a large number of participants.  On our Team, there are a number of groups, presumably with some common characteristics, that are each predicting future events.  We took a variety of tests before joining the team, to measure or describe how we make decisions, process information, etc…

Almost all of the questions about future events are  binary.  They will either happen or not, by a specific date.  Our Team uses a modified prediction market to generate a likelihood of each event occurring (more information here).  Now, this is where it gets interesting.  I’m guessing that most, if not all, of the Teams predicted the correct outcomes for most of the questions.  If our Team got one or two more correct than the other teams, does that really mean that we are “simply the best“?

Could it be that our collective likelihoods of the events that occurred were higher than those for the other Teams?  In other words, when an event did happen, our Team gave the event a higher likelihood of occurring.  Jeez, I hope not, for a number of reasons.  Remember, these are binary events.  Just because a likelihood is higher doesn’t mean that it is more correct than a lower likelihood prediction!  What we really want to compare is the calibration of the market predictions with market outcomes.  Unfortunately, there isn’t enough data, yet, to determine whether our predictions are better calibrated than any other Team’s.

These markets are kept open for trading until a day or so before the outcome is revealed, unless the outcome is determined prior to the anticipated closing.  Uncertainty surrounding the outcome decreases as time marches toward the market closing.  Consequently, at the market close, all that should remain is the irreducible uncertainty (random events that affect the outcome).  Accordingly, most market should converge on a likelihood close to 100% for one of the binary outcomes, and there shouldn’t be very much variability among the Teams.

Could it be that accuracy is being determined at various points in time prior to the market close?  It’s a better basis, but again, we can’t prove calibration.  So, this isn’t likely the answer.  Maybe it’s the speed of adjusting predictions, given new information?  I doubt this one, too.  In some cases, information will lead one forecaster to conclude the event is more likely and another to conclude the opposite.  It would be impossible to determine whether the market was incorporating new information in every case.

Maybe our Team won more money.  Nope.  Basically, with an Automated Market Maker, except for the seed capital, it’s a zero sum game.  All teams would do equally well, with the same system.

Conclusion

Let’s forget for a minute that these predictions are pretty useless, if they’re only “accurate” immediately before the outcome being revealed.  How many times have I spouted on about this issue?  Also, they’re predicting binary events.  There’s no such thing as being almost right in a binary market.  So, even though it isn’t theoretically correct, I’m going to guess that the IARPA thinks a higher likelihood prediction is more accurate than a lower likelihood one, when the event does, in fact, come true.  Maybe that’s the best they can do, until they figure out the calibration issue.

This may change, but our present aggregation mechanism is an odd variant of a prediction market with an automated market maker.  Let me explain.  During the first two months, just about every question has been binary (either it will happen or it won’t).  Apparently, there may be some questions that have up to five derivative shares in a winner-take-all market.  All markets involve an automated market maker.

Participants can place trades (up to $1,000) in any market, for the event to happen or not, by a given date.  As trades are filled, the market price changes.  So far, so good.  The twist is that trades can be rescinded at any time up until the market closes or the event becomes known.  When you rescind a trade, you get back all of the money that was originally invested.  Huh?  That’s right, there’s almost no risk of selecting the wrong outcome!  But, part of what makes markets “accurate” is that there is a consequence for being wrong.  Not so here.  In a traditional prediction market, selling out of a position would net you the current market price (not your original purchase price).

At least you can’t take positions in both sides of a binary market!  The market mechanism encourages you to bet the maximum, usually at the beginning of the market.  This will allow you to double your investment (if you are correct).  In some cases, the likelihood will fall and you can generate a higher profit by investing at that point.  Usually, you will want to maximize your bet when you first enter the market, because if you try to revise your bet later, you will receive the new payoff on your entire investment (if correct).

If the odds for the outcome you selected start to fall, but you still wish to hold that investment, you need to continually revise your investment, to obtain the most favorable odds.

The other quirk is that the maximum bet is $1,000 (previously $500).  That’s a minor point, but it does potentially hinder someone with “perfect” information from placing a bet that would move the market to the appropriate likelihood.  Recall that part of the rationale for prediction markets is that it helps identify the best forecasters (they have the most funds).  When you combine this with the failure to penalize poor guesses (by allowing traders to rescind investments without penalty), I’m wondering whether this particular prediction market mechanism will be as accurate as it might otherwise be.

The Search for Something Better

A few years ago, this got me to thinking.  If prediction markets might be better than alternative prediction methods, could there be an even better model?  And so, I scoured the literature in search of just such a model.  I thought I had found one a couple of years ago, and set out to prove the case for its replacement of prediction markets.

In making my assessment of “better”, in terms of predictions, I considered the calibration of the predictions with the actual outcomes and how far in advance the calibration was reasonably accurate.  I chose to consider the latter characteristic, because prediction markets are notoriously poor at being able to predict anything but very short-term outcomes.

I am pleased to report that my alternative prediction model appears to be better than prediction markets in most respects!  My model was able to match the calibration of prediction markets in every case, but the real benefit was how far in advance my model was able to predict the outcome, with equal or better calibration than prediction markets!  In all cases, my model was very well-calibrated with the outcomes a full two years prior to the outcome being revealed!   To my knowledge, no prediction market has ever been well-calibrated two years prior to the outcome.

Not only that, but my model was able to achieve this level of accuracy for the most difficult to predict outcomes.  Unfortunately, however, my model was not able to forecast so-called “easier to predict” outcomes with the same level of accuracy.

A Model Prediction Model

I’m sure I have kept you in suspense long enough.  My model involves a hand, a wrist and a coin.  Who knew that a simple coin toss might be as good, or better, a predictor of future events than a prediction market?  Very difficult-to-predict binary events have a likelihood near 50%.  If a prediction market for such an event indicates a 50.1% likelihood of occurrence, the decision-maker would predict that the event was going to occur, and he’d be right about 50% of the time.  Same thing with the coin toss, but we can toss the coin two years before the event and get an equally well-calibrated prediction.  For these really-hard-to-predict events, prediction markets, typically, fluctuate all over the map before settling on the safer 50% likelihood.

In discussing the principles that help ensure prediction markets provide accurate predictions, the author makes reference to “information completeness”, in the following passage:

At the end of the day, a prediction market must have sufficient “information completeness” even if the individuals interacting in the market do not, to accurately predict outcomes with a reasonable degree of certainty.

Here is the passage where I introduced the concept of “information completeness”:

Prediction markets must have sufficient information completeness to accurately predict outcomes with a reasonable degree of certainty.

I added the bold italic parts to show the exact same words in each paper.  I’m still flattered, just a bit miffed.

Galton’s Ox Revisited

One other interesting point in the paper concerned a reference to a recent test in the Netherlands that tried to replicate Galton’s ox experiment (James Surowiecki, The Wisdom Of Crowds).  Using 1,400 guessers (oops again, I mean participants), the average estimate of a cow’s weight was 552Kg, but the actual weight was 740Kg.  The guessers were off by a full 25%!  How could this happen?

The average guess of Francis Galton’s townspeople was remarkably accurate (1,197lbs vs. 1,198lbs).   Clearly, the townspeople were a bit more knowledgeable about the likely weight range of a butchered ox than the Netherlands guessers were about the weight range of a cow.  The author of the Forrester paper calls this “perspective“, which is a good word for it.

I called it  having a minimal level of information about the subject in order to make a prediction.  If you think about the problem, logically, when the townsfolk made their estimates, there was a fairly narrow range of possible weights from which to choose.  We would expect a normal distribution of guesses that would centre around the true weight, given reasonably small estimation errors (which cancel).

The cow guessers didn’t have a narrow range of possible weights (they actually guessed between 108 and 4,500 Kg.)!  The errors would have been much more significant, on average, and much less likely to cancel out when aggregated.

Interestingly, there must have been a few knowledgeable cow weight estimators among the 1,400.  Would a prediction market have provided a more accurate number than the simple aggregation of estimates?  That would have been an interesting follow-up experiment.

On a humourous note, this research paper is the first I’ve read on prediction markets that does NOT mention Robin Hanson.  How can this be?

Well, in this week’s The Economist, Growth figures:  Six years into a lost decade, there is ample proof of my claim.  The U.S.  Bureau of Economic Analysis (BEA) has revised it’s growth numbers for the 4th quarter 2008.  Initially, it was estimated to contract 3.8%.  This was revised a year later to indicate a much more serious decline of 6.8%.  Now, it has revised the estimate downward still, to 8.9%.

The inaccuracy is blamed on a piecemeal and slow collection of survey data, which gets fed into a national economic model.  Revisions to past estimates are made but once a year.

Perhaps the BEA needs a better model to estimate economic growth!  Maybe take a walk down Main Street and see how many storefronts are for lease.  Measure the length of unemployment lines.  Actually talk to real people about their spending plans.

In March, 2009, I estimated that growth would be down at least 10% compared with government estimates of -1% to -4%.  Seems that my noggin houses a better economic model than the that of the Bureau of Economic Analysis.

So, knowing that, at best, the Oscar prediction markets are mildly amusing diversions, I present a few interesting observations.

When we use prediction markets to make decisions, we usually make a decision based on the most likely possible outcome in the market.  Consequently, in Oscar prediction markets, when we rely on the markets, we select the actor/movie that the market gives the highest likelihood of winning.  As I have written before, in discrete markets, you will be disappointed using prediction markets.

The Good

Prediction markets at Inkling and HSX had a few amazing successes!  Yes, once again, prediction markets have proven to be remarkably accurate predictors of slam-dunk outcomes.  We can now say, at least anecdotally, that if an Oscar prediction market gives an outcome at least a 70% chance of occurring, we can rely on the market to pick the correct outcome.

Here are the markets that predicted an outcome with a 70%+ probability of occurring:

• The King’s Speech wins Best Movie (71.28% on hsx)
• Colin Firth wins Best Leading Actor (89.36% on hsx)
• Christian Bale wins Best Supporting Actor (77.92% on hsx)
• Natalie Portman wins Best Leading Actress (81.04% on hsx)
• Toy Story 3 wins Best Animated Feature Film (94.82% on Inkling)
• The Social Network wins Best Film Editing (76.29% on Inkling)
• The Wolfman wins Best Makeup (70.74% on Inkling)
• Inception wins Best Sound Editing (76.83% on Inkling)
• Inception wins Best Sound Mixing (77.53% on Inkling)
• Inception wins Best Visual Effects (93.51% on Inkling)
• The Social Network wins Best Adapted Screenplay (74.16% on Inkling)
• The King’s Speech wins Best Original Screenplay (71.52% on Inkling)
• The King’s Speech wins the Most Oscars (70.1% on Inkling)

The Bad

There were a few “upsets”:

• Alice in Wonderland won for Best Art Direction (18.04% on Inkling), even though The King’s Speech (favourite at 38.25%) and Inception (26.68%) were more likely to win.
• True Grit was favoured to win for Best Art Cinematography (65.19%), but Inception (11.53%) did win.
• Alice in Wonderland won for Best Costume Design (31.27%), but The King’s Speech was favoured at 46.67%.
• The Inside Job won for Best Documentary Feature (30.78%), but Exit Through The Gift Shop was favoured (51.34%).
• Biutiful (34.94%) got beat out by In a Better World (24.98%) for Best Foreign Language Film.
• The Lost Thing (6.95%) pulls off a major upset against The Gruffalo (42.09%) and Day & Night (36.89%) to win Best Animated Short Film.
• The God of Love (12.08%) wins the Best Short Film, beating out front runners, Wish 143 (39.34%) and Na Wewe (27.13%).

There was another possible upset.  The King’s Speech won the Oscar for Best Directing.  Was it an upset?  On the HSX, it was a bit of an upset.  The Social Network was favoured at 54.44%, but The King’s Speech won with 33.48%.  On Inkling, however, the two films each had an identical likelihood of winning, at 43.68%.

Getting Better All The Time?

In most prediction markets, we expect the forecast to get more and more accurate the closer it gets to the outcome being revealed.  In the Best Directing Oscar markets (HSX), we saw the exact opposite!  Basically, it was a two-horse race between The Social Network and The King’s Speech.  The King’s Speech had been steadily becoming less likely to win over the last three weeks of trading.  In normal markets this type of trend would require a steady diet of negative information.  Logically, we would expect sudden jumps in likelihoods, when (if) significant information comes to light about which way Academy voters are likely to vote.  I suppose it is possible for there to be a gradual revelation of information (say one voter/day discloses his vote), it isn’t likely.  The Academy likes to keep these things secret until the show.

At any rate, the market was right, but trending wrong.  Maybe there was some information that came to light, resulting in more uncertainty about the outcome.  Then again, maybe the predictors were really just guessers, and the markets are simply aggregating “garbage information”.  Garbage in, garbage out.

While this may not have been an upset, it does bring up another important issue.  Two prediction markets  trying to predict the same thing, unfortunately, the markets predicted significantly different likelihoods.  There were many examples, here are but a few:

For the Best Original Screenplay, The King’s Speech had a likelihood of winning of 71.52% on HSX but only 53.99% on Inkling.  That’s a difference of almost 18%.  Seems quite high to me.  The same thing happened with the Best Adapted Screenplay, where The Social Network won.  This time Inkling predicted it with a likelihood of 88.93%, while HSX gave it a likelihood of only 74.16% (about a 15% difference).

Suffice it to say, the prediction market “industry” must find out why this happens and how it can be corrected.  Otherwise, these types of markets should be abandoned for serious prediction purposes.  What am I saying?  These aren’t serious prediction markets!  Okay, the industry needs to get to the bottom of this issue, so these types of markets can be used as fair betting markets.

There are several possible reasons for the different likelihoods, and none of them help the case for prediction market accuracy or usefulness (for these types of markets).  I’ve discussed these issues in previous posts (too many to link to), so I won’t do so here.  If you took the time to read The Wisdom of Crowds, surely, you can spend a couple of hours reading this blog to learn the reasons.

Something Doesn’t Add Up

Inkling’s prediction markets consider each award as a separate market, with each nominee being a separate “share” within the market.  Accordingly, the sum of all of the likelihoods of the possible shares always add up to one (1.0 or 100%).  However, on HSX, each nominee is a separate market.  All of the markets (nominees) for a particular category are aggregated to show the results the same way Inkling does, but the sum of the likelihoods did not always add up to one.  In fact, they were often significantly different.

For examples (Award, sum of likelihoods),

• Best Picture, 93%
• Leading Actor, 109%
• Supporting Actor, 110%
• Leading Actress, 111%
• Best Directing, 106%

Even though this is a phenomenon created by the structure of the markets, it still begs the question – why?  Shouldn’t the markets have been arbitraged back to a total likelihood of around 100%?  Not only did these discrepancies occur, they persisted!  While I didn’t continuously monitor these markets, I did take snapshots at various times and the sum of the nominee markets rarely added up to 100%.  If I start getting into all of the reasons why this might have happened, this would turn into a book. 

The Ugly

No one told us the writers had gone on strike, again!  A mere eight minutes in and we had barely cracked a smile.  When we did, it wasn’t for anything either of the hosts said, it was for the wink that Anne Hathaway directed at Colin Firth (as the King) in the opening film vignette.  Other than that, there was a lot of odd (not funny) banter between presenters and little to keep us occupied until the next Anne Hathaway appearance.  Their writers were pathetic, but her makeup person seemed to be on his or her game.  Note to the Academy:  hire Randy Newman to write next year’s script.  Either that or put Ricky Gervais on speed dial.

Final Words

For the second year in a row, my picks (from the prediction markets) were better than my wife’s.  All that’s left to be determined is my prize for this feat.

For a bit of comic relief, I present this video news report.  It pokes fun at Torontonians, who seem to have acquired a reputation for being, well, shall we say, a bit sensitive when confronted by inconveniences (or even Acts of God for that matter).  Enjoy.  Being a prediction market blog, I should note that no prediction market could have seen this storm coming.  We were caught completely by surprise.

Gartner Hype Cycle Social Software 2010

As we can see from the Gartner Hype Cycle Graph for Social Software, Prediction Markets are now on the downside of the dreaded “Trough of Disillusionment” (2010). Last year, it was just entering this phase, and in 2008 it was at the most-hyped “Peak of Inflated Expectations”.  The object of this paper is to examine the current status of the prediction market “industry”, discuss several troubling issues that are holding back enterprise prediction market adoption, and look at the prospects for the future.  Even if you get really sleepy reading this paper, keep going to the very end, where I will reveal a very, very long-term prediction!  Can you guess what it is about?

You’re probably already familiar with the following graph showing the Prediction Market growth trend.  It’s the one that appears in many presentations on prediction markets.  As far as I know, the graph hasn’t been updated since 2006.  It sure did look like the market was going to experience explosive growth!  Did it?

Prediction Market Growth Trend 1997-2006, Source: Newsfutures.

According to a McKinsey Global Survey of Web 2.0 adoption, enterprise prediction market “adoption” grew from less than 1% in 2007 to 8% in 2009.  This is how Consensus Point disclosed this results of the McKinsey report.  I looked at the actual McKinsey interactive graphs and found that prediction market adoption was 9% in 2008.  Does this mean that prediction market adoption had already peaked in 2008?  I thought we were just getting started!   If the survey is correct, prediction markets experienced more than an eight-fold increase in usage in the last two years.  Based on what we can see, there appears to be something wrong with the definitions of “adoptions” and  ”prediction markets”.  Alternatively, prediction market adoption is taking place behind closed doors or it isn’t really happening at all.

If the adoption rate is correct, why aren’t we seeing a significant spike in reported success stories?  There has been very little reporting of any prediction market results – good or bad.  I suspect the companies that have “adopted” prediction markets have done so in very limited pilot studies.  Here’s another possibility.  A quick review of several vendor websites indicates that many of the success stories involve idea pageant (or idea market) “prediction markets”.  I’m willing to bet that the companies that implemented these “markets” were included in those “adopting” prediction markets.  While this type of market does involve collective intelligence, it isn’t really a prediction market.

To start the review of the current status of prediction markets, let’s check in with Jed Christiansen, who recently posted his take on the industry.

There was nothing new in Jed Christiansen’s Prediction Market Review for 2010. His comments are correct, but he didn’t provide much commentary about the reasons for the developments over the past year.  Essentially, his summary was as follows:

1. Real money betting sites are booming
2. Free public prediction markets cannot survive without monetizing site traffic
3. Software vendors are providing more consulting services to their clients

He sees the PM industry as “maturing”.  Existing vendors will continue to establish themselves, “as more companies experiment with new management tools and techniques.”  The problem with the industry is that the product is still in its infancy.  I don’t think you can call a market “maturing”, when the majority of the clients are merely experimenting with the concept of prediction markets and the “product” is, basically, still a concept.  As we saw at the beginning of this post, prediction markets appear to be firmly entrenched in the trough of disillusionment.  Furthermore, Gartner estimates that mainstream adoption is 5-10 years away, the same estimate they gave in 2009 and 2008.

Not only is the industry mired in the trough of disillusionment, I think the primary researchers are stuck in one too (with one notable exception)!  Over the last few years, there have been no important new research studies, no significant published prediction market trials, and no major prediction market issues resolved.  It is as if the researchers don’t want to look too closely at the issues for fear that some of them may seriously undermine the usefulness or potential of prediction markets.

I exclude one researcher (and his team) from the list of disillusioned researchers.  During the year, David Pennock and his group at Yahoo! Research, launched Predictalot to showcase a fairly complex example of a combinatorial prediction market.  So far, it has been used to predict the winners of the NCAA March Madness basketball tournament and the World Cup.  On a humorous note, Predictalot and its developers received the Best Prediction Market Development of the Year for 2010. I’ll have more to say about the significance of this development, below.

Let’s look at the reasons behind Jed’s industry developments, which will lead into a discussion of the issues holding back the adoption of prediction markets and the future prospects for the industry.

Betting Markets

Real-money prediction markets are booming and expected to continue to boom, not because they are good predictors, but because betting is booming.  The major players are Betfair and Intrade, neither of which spout on about the predictive abilities of their markets.

Discrete outcome markets (like horse races) are perfect for betting but not nearly as useful for making predictions and the decisions based upon them.  Most of the markets generate predictions that are too general or too public to be useful.  The value of information depends on having it before someone else and being able to act upon it.  Since these markets are ill-suited for useful predictions, their success will depend almost entirely on the public’s desire for betting opportunities.  Personally, I think these types of markets should be excluded from the definition of prediction markets.  Horse race odds are considered to be pretty good predictors of the race outcome, but we don’t consider horse race betting pools to be prediction markets.

Public Prediction Markets

Most public prediction markets are not very useful, at all.  Even if they were proven to be accurate, no one would pay for information that is already publicly available.  With few ways to generate revenue, growth prospects are bleak. Hubdub ceased operations during the last year.  While it was fun to play on their prediction markets, participants became disinterested as the novelty of “betting” on trivial outcomes wore off.

No amount of explaining will convince participants that it was a good thing that Susan Boyle lost Britain’s Got Talent, even though she had a 78% chance of winning.  Once we’re done explaining that, we can take a stab at explaining why there was such a wide variance between Hubdub’s (78%) and Intrade’s (49%) likelihoods of her winning.  Personally, I think she should have won!

HSX and IEM run somewhat more useful markets, but neither is very good at accurately forecast long-term outcomes.  Forecasting short-term outcomes is not particularly useful.  Unless HSX can be turned into a real-money market, the prospects for any commercial success are minimal.  However, this and other public markets are still valuable for research purposes.

Don’t expect any growth in this sector.

Vendor Consulting Services

This is a growth area, because their clients are ill prepared to create useful prediction markets without guidance.  Failed trials mean the client companies will stop experimenting with prediction markets.  Vendors help their clients achieve reasonable prediction results.  None of the existing vendors can survive on software sales, alone.  Vendors should try to get as many trials as possible and investigate the unresolved prediction market issues (see below).

There will be few new vendors, because the prospects for enterprise prediction markets are not very rosy (more about this, below).

WHAT IS HOLDING BACK ENTERPRISE PREDICTION MARKETS?

It’s no secret that prediction markets have not taken off in the corporate world.  Don’t corporate decision-makers know a good thing when they see it or is there something wrong with the product?

Since getting involved with prediction markets, I have maintained a list of issues that remain unresolved.  In my opinion, not resolving these issues is the reason enterprise prediction markets have failed to take hold in the marketplace.  Despite several researchers – especially Robin Hanson as the most published adherent – stating that prediction markets are at least as accurate as other forecasting methods, the case has not really been made (at least not to my satisfaction).

As we will see, prediction markets are unable to accurately predict long-term outcomes, and they have poor records for accuracy and reliability, all of which are crucial for enterprise adoption.  I haven’t mentioned the issues of market design, participant training, number of participants, etc…, because these things are easily solvable.  It makes little sense to tackle these issues, unless the important issues are resolved first.

“Just in Time” is Not Timely Enough

Prediction markets need to be able to forecast long-term events.  In order to make long-term decisions, we need information about conditions, events and outcomes that will occur far off in the future.  Well, at least longer than a month or two!  While there have been several long-term prediction markets (public ones), not one has provided an accurate prediction of the future outcome, until very close to the time when the outcome was revealed.  Such predictions, no matter how accurate, are not actionable.  In other words, these markets have been wholly inadequate for management decision-making purposes.  The use of prediction markets to forecast any long-term outcome is questionable, if not down-right dangerous.

The following two graphs of historical prices in two long-term (14 year) prediction markets are  from Ideosphere.  In both of these markets, the predictions only became reasonably accurate during the last year before the outcomes were revealed.  Of course, some prediction market advocates will argue that the markets were accurate throughout the trading period.  The market price, at any point in time, accurately reflects all available information in the market at that time.  Consequently, the markets are considered “accurate”.  However, they aren’t accurate, if our purpose is to rely on them to make decisions about outcomes in the long-future.

Unfortunately, even if these long-term markets are “accurate” several years away from the outcome, we have no way of knowing whether they can be relied upon.  It is impossible to verify the calibration of these markets (though it has been claimed that they are – 30 days before the market close).  It is difficult to imagine that these markets were calibrated back in 1998, where the market prices were approximately 75% – 80%, yet the eventual closing prices were 0%.  It’s possible, but highly unlikely.  It is much more likely that these markets were reflecting a significant amount of uncertainty about the outcome.

Ideosphere 14 year Cancer Cure Market

Ideosphere 14 year Earthquake Market

The longer the trading period of the market, the more sources of uncertainty there will be.  The steady march of time gradually reduced the uncertainty in these markets.  It is as simple as that.  Even if it were possible to acquire enough information to reduce the uncertainty surrounding the outcome, it is highly unlikely that the incentives would be enough to cover the search costs.

I don’t have the answers as to why these markets have not worked, but here are a few possibilities:

• Traders are not patient enough to bet on long-term events.  They want to make a trade and quickly find out whether they have won.
• The longer the time period between the prediction and the outcome, the more likely it is that there will be more random, intervening events that affect the outcome, increasing uncertainty.
• Intervening events that have a complex influence on the outcome will increase uncertainty around the prediction AND increase the likelihood of a wrong prediction.  Such outcomes may not be predictable by any method.

As the markets move closer to the outcome, uncertainty about intervening events decreases.  Generally, about 30 days before the outcome, the markets become reasonably accurate.  In fact, for most of the period the prediction markets were in operation, the predictions were wildly inaccurate! The question is whether there this is enough advance notice for the prediction to be acted upon, making them useful.

Here is an example from IEM, used to show how even fairly heavily traded markets are unable to make actionable predictions until very near the market close.

IEM 2006 US Congressional Control Market

Note in the Congressional Control Market for 2006, the market prediction was inaccurate until a few days before the election.  For decisions that need to know which way the election would go, the prediction would likely be too late.  Most long-term markets exhibit this characteristic.

Accuracy

The Hewlett Packard pilot was one of the first studies of enterprise prediction markets (my commentary, here).  Even though it is over 10 years old, it is still the most often cited case!  This pilot study found that 6 of 8 markets outperformed the company’s internal forecasts.  That’s pretty good, except that the “better” predictions were only slightly better and three of the predictions were really poor (greater than 25% error).  One of the study’s authors commented:   “The accuracy improvement was not high enough to be adopted,” says Chen. “You need to be a lot more accurate before it’s worth it to implement a new process.”

We can say that these markets were effective aggregators of participant information.  When you consider that the participants in the prediction market trials were also involved in making the internal forecasts, it is not difficult to understand why the prediction markets were better at predicting the internal forecasts than they were at predicting the actual outcomes!  Unfortunately, prediction markets need to be good at predicting the future outcomes.

The General Mills trials showed that prediction markets were as good as internal methods, but they were not significantly better and some of the internal forecasters were also participants in the prediction markets.  It should be kept in mind that these were very short-term predictions, such that it would have been almost impossible to act upon the predictions.

Pennock et al showed that prediction markets were accurate (in the cases they studied), but they were not significantly more accurate than alternative prediction methods.  They concluded that in order for prediction markets to be useful, they must be significantly better than alternative forecasting methods.  In the cases they studied, they found prediction markets were only slightly better than other methods.  In previous posts, I introduced the concept of materiality to the analysis of prediction markets.  Essentially, for a prediction market to be useful, it must be more accurate than the next best predictor, such that the more accurate prediction would make a difference to the decision-maker relying on the forecast.  Then, we need to look at the costs and benefits to determine whether the use of prediction markets is a wise course of action.

One of the measures of accuracy is calibration.  We can be fairly sure that horse race odds are well-calibrated with race outcomes, because we can analyse thousands of homogeneous races to prove the claim.  Unfortunately, we are hard pressed to find more than a handful of similar PMs from which we might test the PM’s calibration with the outcomes.  Yet claims are made that PMs are reasonably well-calibrated and “therefore, they are accurate.”

Given the above comments about long-term PMs, we have to ask, when is a PM “well-calibrated”?  Is it when the market closes?  If so, the prediction is useless, because it cannot be acted upon, even though it may be quite accurate.  Is it 30 days before the outcome of a long-term PM?  If so, this is a bit better, but still pretty useless.  Is it near when the market opens and continuously until the market closes?  This would be ideal, but it is highly unlikely to be the case.

Galton’s ox and the missing submarine stories are examples of collective intelligence, not prediction markets, yet they are frequently cited as proof that prediction markets are accurate.

Reliability

In order to be useful in an enterprise setting, prediction markets must reliably provide accurate predictions of future outcomes.  Furthermore, they must be at least as accurate and timely as other traditional forecasting methods, and hopefully, make predictions at a lesser cost.  Here, reliability means consistency.  The same type of prediction market must consistently provide more accurate forecasts than other available means.

In the discussion about long-term markets (above), we found that PMs were very unreliable until close to the time the outcome is revealed.  This brings up a couple of crucial questions.  How far in advance can prediction markets make accurate predictions?  How will we know the point in time when a prediction is “accurate”?

Recall the Susan Boyle Britain’s Got Talent markets.  Why are there wildly different predictions of the same outcome in different prediction markets?  How do we know which market is accurate?  Is it a matter of prediction market efficiency?  If so, how do we know whether a market is efficient?  Rajiv Sethi provides us with an approach to determining which market is more efficient, but not whether the market is sufficiently efficient.  Are there differences in participant information in the two markets?  Is there a lack of diversity in one of the markets?  Evidence of Cascading?  Herding?  Are there inadequate incentives to acquire and reveal information in the markets?  Does sufficient information exist in one or both of the markets?  If not, both markets may be aggregating guesses rather than informed opinions.

Prediction markets are touted as being excellent information aggregation methods, and by all accounts, they probably are very good at this.  It almost seems too obvious to mention, but I will anyway.  In order for the markets to provide accurate, reliable predictions, there must be a sufficient amount of information available to be aggregated.  No one is really looking at this issue, yet it is crucial to success of prediction markets.  This is the issue of information completeness.

THE FUTURE OF PREDICTION MARKETS

Where to from here?  Despite the significant unresolved issues, I still believe prediction markets have potential (though not as much as we all once thought).

Can PMs ever replace traditional forecasting processes?

Probably not. As discussed, the HP and General Mills prediction markets used individuals involved in the internal forecasting process.  Accordingly, the HP predictions were closer to the internal forecasts than they were to the actual outcome.  At General Mills, both the predictions and the internal forecasts were very close.

The nagging question is, if the internal forecasting processes had not been in place, would the prediction markets have been as accurate as they were?  We may never know, because I doubt there are any companies willing to test this proposition.  My intuition tells me that stand alone prediction markets would be less accurate than internal forecasts as well as PMs in conjunction with internal forecasts.

I’m not arguing that prediction markets are poor aggregators of information.  The reason for the lesser accuracy of stand-alone prediction markets is that there is much less information to aggregate (without the internal processes to search for information).

Is there a place for PMs to supplement traditional forecasting methods?

Yes.

Prediction markets involve a relatively small marginal cost.  So, it is relatively painless to implement key prediction markets to supplement traditional forecasting methods.  Some of the benefits are:  the ability to quickly check the internal forecast for significant deviations from the prediction (which can be investigated), more information by incentivizing participants to search for more information, and a reduction of forecasting bias.

The real benefit, in my opinion, is that prediction markets provide a better measurement of uncertainty around the outcome than do traditional forecasting methods.  It does this in the form of a distribution of predictions, which can be seen visually and measured by the standard deviation.  The information can be used to identify the need for further information and can be used in risk management and contingency planning.  In addition, management can measure the reduction of uncertainty over time as new information is revealed or possible sources of uncertainty are removed.

One of the most promising applications is in project management.  Task and project completion forecasts involve the most bias, and prediction markets have the potential to significantly decrease this bias.  While long-term predictions are not particularly useful, short-term ones appear to be reasonably accurate and prediction markets have been shown to quickly aggregate known information.  In managing projects, it is important to obtain very short-term forecasts for task completion, so that corrective action may be taken.  Prediction markets appear to be particularly well suited to this task.

Projects can be separated into tasks along the critical path, and PMs can be put in place to predict completion dates for these tasks.  Because completion dates are continuous variables, coming close to the actual outcome will often be good enough, even if the prediction market is not a perfect predictor.

An interesting avenue of research would be to create a combinatorial prediction market in which all of the critical tasks are linked to the total project completion date.  (See additional comments below).

IDEA PAGEANTS

While they are not really prediction markets – they’re more like weighted opinion polls or high-tech suggestion boxes – they are usually counted as being “prediction markets”.  Oddly, these types of information markets make up the majority of “prediction markets” in use.  They also have the greatest growth potential.

Idea pageants generate ideas quickly, at a very low cost.  They are relatively easy to understand and implement.  These applications don’t need a high level of accuracy to be useful – companies can investigate the top 10 ideas vs. needing to know the best one.  Management doesn’t have to delegate all authority to the market.  Weak or impractical ideas are quickly filtered out, but decision-makers are free to investigate all ideas, not just those that have high probabilities of success.

Based on the knowledge that the further away from the outcome, the greater the possible number of events occurring that would affect the outcome, predictions will be inaccurate and/or widely dispersed, until near the time the outcome becomes known.  These intervening events are random, but the likelihoods are not (in most cases).  Another possible application is to create markets, similar to idea markets, except that they would identify possible future events that might affect the outcome that we are trying to predict.  This information, combined with prediction markets to estimate the likelihoods of these events occurring would add useful information to the market predicting the outcome of interest.

For example, we could predict the likelihood of a truckers strike during the third quarter, which could be used to make a better prediction of third quarter revenue (the outcome of another prediction market).  Eventually, it might be possible to link the potential intervening events to the outcome in a combinatorial prediction market.

COMBINATORIAL PREDICTION MARKETS

Continuing with the previous example, we might apply Robin Hanson approach.  Much of his work in the area of combinatorial prediction markets focuses on conditional probabilities.  He might run two prediction markets.  The first would predict 3rd Quarter revenue given a truckers strike.  The Second market would predict 3rd Quarter revenue, given no strike.  The difference between the two predictions would be the forecast cost of a trucker strike (in terms of revenue lost).  Robin calls these decision markets, and they form the backbone of his futarchy concept.  Decision markets represent one form of a combinatorial prediction market.

With great fanfare, Crowdcast released their innovative trading platform designed to make trading more intuitive.  Essentially, it is a mechanism to allow traders to bet on user-defined spreads. For example, revenues will fall between $1.2m and $1.4m or $1.85m and $2.12m.  It allows traders to make combination bets for any range they choose.  While I think this innovation has potential, there may be a number of tricky issues regarding the effects of assumptions required to make this platform work.  Still, it is a promising development.

Combinatorial prediction markets make an awful lot of sense, if they can be practically implemented.  The above types of combinatorial prediction markets are relatively easy to implement. Perhaps the most difficult to design and implement is the type of combinatorial prediction market developed by David Pennock and his group.  While it is used for sports betting (play market), the concepts may be applied to enterprise prediction markets.

Predictalot provides a working example of a fairly complex combinatorial prediction market, which involves combinatorial betting on the NCAA March Madness and the World Cup.  For example, if Duke is predicted to win the championship, this automatically increases the likelihood of Duke winning in all of the rounds leading up to the final.  Also, if Duke is predicted to win in the first round, this increases the likelihood of Duke winning the championship.  This platform allows bettors to bet only on those things that they have knowledge.

The same combinatorial prediction market concept could be applied to project management.  It is difficult to predict the completion date of a complex project (Predictalot Champion).  Some participants will have specialized knowledge of the task (Predictalot Team) they are working on, but little knowledge of other tasks along the critical path.  A combinatorial market would allow participants to trade on those outcomes in which they have knowledge.  The market structure will implicitly incorporate the predictions of tasks into the prediction of the overall project completion date.  Similarly, the prediction of the overall project completion date will influence the predictions of the various tasks along the critical path.

This is an important development, because traders may have specific or local knowledge about one or more components of an outcome, though they  have little knowledge about the eventual outcome itself.  A single prediction market for the project outcome may fail, because there is not enough information about the outcome to generate an accurate prediction.

While it is true that a project outcome could be split into several prediction markets to predict the required tasks.  The problem is that each prediction market may be too thinly traded to generate an accurate prediction.  Also, there is no automatic inclusion of the task predictions in the project outcome prediction.  A combinatorial prediction market has the potential to solve this problem and generate better predictions of the outcome.

Looking at a more generalized application, many outcomes are dependent (or conditional) on other events, actions or conditions.  In order to better predict an outcome, we would like to know the factors that will have an effect on the outcome (discussed in the Idea Pageant section, above), and we would like to know how likely these factors are to arise.  We could set up a series of separate prediction markets to predict the likely effects of each of the factors that will affect the outcome.  The results of these markets would be available to the traders predicting the outcome of interest. While this is better than existing prediction models, it’s not ideal.   Alternatively, the factors can be combined with the outcome in a combinatorial prediction market, allowing the likely effects of the factors to be automatically incorporated in the outcome prediction.

Certainly food for thought, and it is the reason that I selected Predictalot as the most important development in the area of prediction markets for 2010.

YOUR REWARD FOR READING THIS FAR!

No discussion of the future of prediction markets would be complete without commenting on the most comprehensive system of prediction markets ever conceived.  Of course, I’m talking about Futarchy , one of the New York Times buzzwords for 2008.  Sadly, for Robin Hanson, its creator, Futarchy has failed to take hold, anywhere.  If the concept had any merit, Surely, at the very least, it would have been implemented in some small, South Pacific island nation by now (it hasn’t happened).  About a year ago, I commented on the Future of Futarchy, where I dismissed the concept.  Despite this, I see that in December 2010 Robin Hanson is still trying to promote the idea!  While I disagree with Futarchy, I do heartily endorse his use of decision markets.

If there were a long-term prediction market on whether Futarchy would be implemented anywhere in the world in Robin Hanson’s lifetime, the price would be flat-lining on $0.00.  Occasionally, the market price it would jump up to $0.50 (reflecting Robin’s trades), only to be smacked down by Mencius Moldbug’s trades.  I suspect there will be a smirk on Robin’s face each time the market corrects his attempt to manipulate the market.

This market illustrates another key aspect of prediction markets.  The outcome must be clearly defined.  In this market, “Robin Hanson’s lifetime” is defined to mean his lifetime in his current body.  It’s no secret that Robin wishes to have his head lopped off (when he dies, not before) and cryogenically frozen, to be thawed at some time in the future when bodies will be more “durable” or when brains can be downloaded into some robot-like “life” form.  No word, yet, about whether the good professor’s wife will be similarly decapitated.  Without this clear definition of the outcome, we wouldn’t be able to collect our bets, and it is likely that if brain cloning is possible, so is Futarchy!

So, my forecast is that Futarchy will never come to fruition and it should be cryogenically frozen now, too.

FINAL THOUGHTS

It has been quite an undertaking putting this paper together.  Undoubtedly, I have missed a few key items, for which I apologize.  As always, your comments are appreciated.  While there have been few new developments, there are still many tasks to be completed, if enterprise prediction markets are to gain traction in the market.  In writing this paper, it became evident how most of the major issues remain unresolved.  I hope that some of the researchers will get over their disillusionment and ascend the slope of enlightenment!  If so, I promise to get out of my own trough of disillusionment with respect to prediction markets!

Commenting after the awards ceremony, Mr. Pennock said that, “it is a great honour and I’m proud of the entire team that brought this important concept to fruition.  Frankly, it’s a bit ironic – we just didn’t see this award coming at all!”

He was talking about the InTrade market on Health Care Reform.  In theory, the market price in such a derivative market should equal the expectation of the underlying event coming true.  However, Paul Krugman (and many others) forgot one of the most basic assumptions of the market model!  Transaction costs.

When the market price is over 95, InTrade charges a transaction fee of 3 cents per contract (real money).  While market prices are quoted in percentages, the payoff for a winning ticket is $10 (real money).  Therefore, the transaction fee is 0.3% of the winning payoff.  In addition, InTrade charges 10 cents per contract on expiry (if you “win”).  That’s another 1.0%. 

So, when the market was quoting 97.8% likelihood of the HCR bill passing before June 2010, this didn’t really mean that there was a 2.2% chance of the bill not passing.  A winning ticket would be subject to 1.3% transaction fees.  The real likelihood of failure was 0.9% – approximating the uncertainty that Obama would be “hit by a bus” before signing the bill into law. 

No rational investor would wish to purchase a share for more than 98.7, given the transaction costs.  In a sense, this is the market’s “100%”.  Interestingly, at 1:49pm GMT today (March 23), there are 695 bids at 99.1 and 413 asks at 99.2.  Clearly, some traders are not subject to the full transaction fees at InTrade.  More about that here.

I love Paul Krugman, but this time, he made a silly little mistake.  Of course, all of this assumes the market price is accurate in the first place!