calibration training exercises
(I originally created this for OPTIC; reposting here; written in late 2023/early 2024, so pretty likely to be out of date)
Summary
Here’s the Wikipedia explanation of calibration training.
Calibrated probability assessments are subjective probabilities assigned by individuals who have been trained to assess probabilities in a way that historically represents their uncertainty. For example, when a person has calibrated a situation and says they are "80% confident" in each of 100 predictions they made, they will get about 80% of them correct. Likewise, they will be right 90% of the time they say they are 90% certain, and so on.
Calibration training improves subjective probabilities because most people are either "overconfident" or "under-confident" (usually the former). By practicing with a series of trivia questions, it is possible for subjects to fine-tune their ability to assess probabilities.
Calibration training is a great way to improve your forecasting skills. We’ve collated some calibration training exercises that you can use for practice in your group.
Most of the below were gathered from this).
Recommended calibration training exercises
Quantified Intuitions: Calibration
Pros:
- extremely intuitive UI
- easy to set up & use
- helps get intuition behind confidence intervals
Cons:
- some of the questions aren’t great (i.e. they’re unclear, or ask questions in a way that’s difficult or annoying to answer, etc)
- picking confidence intervals can be confusing/annoying (though probably good in the long term)
Things to note:
- each of the “decks” contain different calibration questions — make sure to select a deck that matches your attendees’ interests
- make sure participants know & understand what a “confidence interval” is before they train on Quantified Intuitions
Clearer Thinking & Open Philanthropy: Calibration Training
Description:
- This is the original, robust calibration training program, including questions on various topics and a calibration chart.
Pros:
- extremely in-depth
- tons and tons of questions
- lets you pick among several different types of game modes & topics
- has clear data on how you’ve performed
- pretty user-friendly
Cons:
- not always incredibly user-friendly (generally fine, but sometimes a bit annoying)
Note: Open Philanthropy is one of several financial sponsors of OPTIC.
Other calibration training exercises
Here are some additional calibration training exercises — they might serve slightly different purposes, or not be kept as up-to-date. You might find some of them useful, but note that we haven’t gone through & checked these so deeply as the ones above.
Metaculus Tutorials
Description:
- The Metaculus Tutorials are a set of three tutorials aimed to provide a basic understanding of calibration, forecasting, and the Metaculus platform.
Pros:
- helps people understand the Metaculus platform
- allows people to test their calibration on non-binary questions (i.e. provide distributions, instead of binary events)
Cons:
- limited number of questions (~50)
- doesn’t clearly give a “calibration chart” at the end — gives the user a number of points, which is substantially less informative
Note: Metaculus is one of several financial sponsors of OPTIC.
Confidence Success Equation
Description:
- a simple calibration quiz, with about 50 questions. answer “true” or “false,” then give your credence in your answer.
Pros:
- quite simple, intuitive
- covers a lot of different subjects
Cons:
- not very long (only 50 questions total)
- doesn’t let you pick your own confidence; forces you to choose between 55%/65%/75%/85%/95%
Bayes-Up!
Description:
- multiple-choice test, where you put your confidence (as a percentage) in each answer, instead of picking one answer — sort of like this proposal
Pros:
- it works!
Cons:
- not a great UI
- doesn’t have intuitive explanations
Things to note:
- you can run a similar sort of thing using a google form & an attached google sheet with some formulas — if you feel comfortable setting this up yourself, it’d probably work a bit better
Andrew Critch: Overconfidence quiz & Calibration practice
Description:
- a simple, “pset”-style worksheet for testing and training your calibration
Pros:
- easily printable
- easily recognizable format
Cons:
- doesn’t work online well (PDF)
- relatively short — only about 25 questions
- doesn’t give you a calibration chart/etc
Things to note:
- this might work well as extra “practice problems” or something — but would probably be a bit less fun than an interactive game. also, notably not repeatable
Scout Mindset Calibration Practice
Description:
- a simple calibration quiz, with about 50 questions. answer “true” or “false,” or between two options, then give your credence in your answer.
Pros:
- cool, nice UI
Cons:
- doesn’t show a calibration chart at the end
- doesn’t let you pick your own confidence; forces you to choose between 55%/65%/75%/85%/95%
Peter Attia: Confidence Intervals
Description:
- a simple, 20 question quiz that asks you to give your 95% confidence interval on a variety of trivia questions.
Pros:
- simple
- gets across the idea of a “95% confidence interval” quite clearly
Cons:
- only 20 questions
- doesn’t make a calibration chart or anything
Others
Android App: The Calibration Game
Outside the Asylum: Probability Calibration (the questions are only on the game Magic: The Gathering)