The Good Judgment Project

I have been participating in The Good Judgment Project, one of five teams in a US government sponsored, four year, forecasting tournament.  Each team develops its own methods for forecasting world events.  Our team is based in the University of Pennsylvania and the University of California Berkeley.  I gather each team will be using some form of collective intelligence to make predictions.

This may change, but our present aggregation mechanism is an odd variant of a prediction market with an automated market maker.  Let me explain.  During the first two months, just about every question has been binary (either it will happen or it won’t).  Apparently, there may be some questions that have up to five derivative shares in a winner-take-all market.  All markets involve an automated market maker.

Participants can place trades (up to $1,000) in any market, for the event to happen or not, by a given date.  As trades are filled, the market price changes.  So far, so good.  The twist is that trades can be rescinded at any time up until the market closes or the event becomes known.  When you rescind a trade, you get back all of the money that was originally invested.  Huh?  That’s right, there’s almost no risk of selecting the wrong outcome!  But, part of what makes markets “accurate” is that there is a consequence for being wrong.  Not so here.  In a traditional prediction market, selling out of a position would net you the current market price (not your original purchase price).

At least you can’t take positions in both sides of a binary market!  The market mechanism encourages you to bet the maximum, usually at the beginning of the market.  This will allow you to double your investment (if you are correct).  In some cases, the likelihood will fall and you can generate a higher profit by investing at that point.  Usually, you will want to maximize your bet when you first enter the market, because if you try to revise your bet later, you will receive the new payoff on your entire investment (if correct).

If the odds for the outcome you selected start to fall, but you still wish to hold that investment, you need to continually revise your investment, to obtain the most favorable odds.

The other quirk is that the maximum bet is $1,000 (previously $500).  That’s a minor point, but it does potentially hinder someone with “perfect” information from placing a bet that would move the market to the appropriate likelihood.  Recall that part of the rationale for prediction markets is that it helps identify the best forecasters (they have the most funds).  When you combine this with the failure to penalize poor guesses (by allowing traders to rescind investments without penalty), I’m wondering whether this particular prediction market mechanism will be as accurate as it might otherwise be.