Improving Your Forecasting Ability by Tracking Your Predictions
Improving Your Forecasting Ability by Tracking Your Predictions
Improving your decision making skills is hard, because it is difficult to keep track of the results of your decisions. One way to quantify decisions is by using brier scores and keeping a decision-making journal.
Brier Scores, developed by Glenn W. Brier in 1950, provide a numerical evaluation of the accuracy of probabilistic predictions. They measure the mean squared difference between the predicted probabilities and actual outcomes, making it easy to see how well your predictions align with reality.
By tracking your forecasts and calculating your Brier Scores, you can identify patterns and biases in your thinking, ultimately leading to better decision-making.
Calculating Brier Scores
Formula
brier score = (probability - outcome)^2 + (probability - outcome)^2 / N
Where outcome = 1 if win / correct or 0 f loss / incorrect
N = number of items
Example calculation
For example an 81% win forecast versus a 19% loss where the underdog wins
(win probability – 0)^2 + (loss probability – 1)^2
(.81 – 0)^2 + (.19-1)^2 / 2 = .66
If the result would have been a win for the favorite, then your brier score would be:
(.81 – 1)^2 + (.19-0)^2 / 2 = .036
A lower Brier Score indicates more accurate predictions, while a higher score suggests room for improvement.
A Decision Making Journal
A prerequisite to tracking and improving your decisions is to start using a decision-making journal. The journal will help you keep track of your decisions and their outcomes and by incorporating Brier Scores into your journal, you can quantify the accuracy of your forecasts.
Here's an example of what a decision making journal could look like:
Date: Record the date of the prediction.
Prediction: Describe the event or decision you're forecasting.
Probability: Assign a probability to your prediction (from 0 to 1).
Reasoning: Briefly explain the rationale behind your prediction.
Outcome: Record the actual outcome of the event or decision (1 for true, 0 for false).
Brier Score: Calculate the squared difference between the predicted probability and the actual outcome.
Brier Score Avg: After accumulating multiple predictions, calculate the mean of all squared differences to get your average Brier Score.
An example entry in a decision-making journal could look like this:
By consistently updating your decision-making journal and calculating your Brier Scores, you can identify patterns, biases, and areas for improvement in your decision-making process. Over time, you'll become more aware of your forecasting strengths and weaknesses, leading to better-informed decisions and more accurate predictions.