Calibration Games

We expect experts to offer qualified advice. We rarely ask them to qualify their confidence in their own predictions (the “confidence interval” (CI).

A 90% CI is a range of values that is 90% likely to contain the correct value. Unfortunately, most experts are over confident in their own predictions and offer misleading advice. On the other hand, many of us are underconfident in our own capabilities and are afraid to act decisively. For example in some studies, the true value lay in subjects’ 90% CIs only 50% of the time! 

Equivalent Bet Test

These subjects were overconfident. For a well-calibrated estimator, the true value will lie in her 90% CI roughly 90% of the time.

Suppose you’re asked to give a 90% CI for predicting your team’s  sales activity next quarter, and you can win $1,000 bonus in one of two ways:

  1. You win $1,000 if your prediction falls within your 90% CI. Otherwise, you win nothing.
  2. Alternatively, you can spin a dial divided into two “pie slices,” one covering 10% of the dial, and the other covering 90%. If it lands on the big slice, you win $1,000, if not don’t win anything.

If you find yourself preferring option #2, then you must think spinning the dial has a higher chance of winning you $1,000 than option #1. Your brain is trying to tell you that your originally stated 90% CI is overconfident.

If instead you find yourself preferring option #1, then you must think there is more than a 90% chance your stated 90% CI contains the true value. By preferring option #1, your brain is trying to tell you that your original 90% CI is under confident.

To make a better estimate, adjust your 90% CI until option #1 and option #2 seem equally good to you. Research suggests that even pretending to bet money in this way will improve your calibration.

Calibration Games

Improving mental calibration relies on repetition and feedback. Several Web resources offer exercises for learning to calibrate your opinions. Examples include :  

CFAR’s Calibration Game.

Courtesy of Luke Muehlhauser at LessWrong.com