Products · Calibration Check
Is your AI's 90% really 90%?
Check whether a model's confidence can be trusted. Give it labeled examples and the model's probabilities; get the calibration error, a fix tested on held-out cases, the cut-off that costs least, and the mistakes worth reading. Works for decision models that answer with probabilities, classifiers and AI judges. It runs in your browser, and your file never leaves this page.
Real data: Laya, an open decision model, read 1,795 Amazon reviews of insulated drinkware and gave each a probability that the reviewer is happy. The right answer is the reviewer's own stars (4-5 happy, 1-2 unhappy). Data: Amazon Reviews 2023 (McAuley Lab, UC San Diego); model: Laya (Apache 2.0).
A CSV or tab-separated file with a header row. It is read here, in your browser.
What the file needs
- Yes/no models: a
labelcolumn (1/0, yes/no or true/false) and apcolumn: the model's probability of "yes". - Multiple-choice models (for example a decision model's "pick one of these options"):
labelholds the right option, and onep_<option>column per option, such asp_billing,p_shipping,p_other. - Optional:
id, andtext(shown in the list of mistakes). - At least 20 rows; a few hundred labeled cases give steady numbers.
The six parts of this check
Does its confidence mean what it says?
Each dot is a band of cases: how sure the model said it was (across) and how often it was right (up). On the dashed line, 80% sure means right 80% of the time.
Where to set the cut-off
Let the model act when it is sure enough, and send the rest to a person. Put in your own costs.
Error analysis
Speed and cost
Use the fix
The fitted temperature, ready for your code. Isotonic works better here only if its step curve is saved and applied the same way.
How it works
- Everything is measured on the option the model picked and how sure it was. For yes/no models that matches the usual Brier score.
- Calibration error: 10 equal bands of confidence; in each band, the gap between how sure the model said it was and how often it was right, weighted by the number of cases.
- The rows are split at random into two halves (fixed seed, so the same file gives the same answer). Both fixes are fitted on the first half and judged only on the second.
- Temperature: p becomes p1/T, rescaled so the options add up to 1. T above 1 softens the probabilities, below 1 sharpens them. T is searched from 0.05 to 10 in steps of 0.05 for the lowest log loss.
- Isotonic: a step curve that only goes up, fitted by pooling neighbors that break the order, read between points with straight lines.
- The fix shown is the one with the lower Brier score on the test half; the 90% ranges come from 300 resamples of the test half.
- The math is checked against scikit-learn, and on simulated data with a known temperature, which the fit gets back.