How to Check if an AI Model's Confidence Can Be Trusted

The question

Many AI tools give a confidence score with each answer. The score might be 90%. That number feels like a promise. But is it one?

A model is well calibrated when its confidence matches how often it is right. If it says 90% on a large set of answers, about 90% of those answers should be correct. If only a much smaller share is correct, the model is overconfident. If nearly all of them are correct, it is underconfident.

This matters when you use the score to make choices. Maybe you only send low-confidence cases to a person. Maybe you act on high-confidence ones without checking. Either way, you are trusting that number, so test it before you lean on it.

How to check it

Step 1: Collect real cases. You need answers where the model gave a confidence score and where you later learned the true answer. Use cases that look like your real work, not just easy examples.

Step 2: Group the answers by confidence. Put answers with similar scores together. For example, all answers near 90% go in one group.

Step 3: Count the hits in each group. In each group, find the share of answers that were actually right.

Step 4: Compare the two numbers. For each group, put the stated confidence next to the real hit rate. If they are close, the score is honest for that range. If the hit rate is lower, the model claims more than it delivers there.

Step 5: Look at every range, not just the average. A model can look fine overall and still be off at the high end, which is often where people trust it most.

Step 6: Watch the group sizes. A group with only a few answers can look good or bad by chance. Trust the groups with many cases more than the thin ones.

Step 7: Test on later data. A model that was calibrated on old data can drift. Check again on newer cases, after the period the model was built on.

Check it yourself

1. Pull a list of past predictions. For each one, keep the confidence score and whether the answer turned out right or wrong.

2. Open my Calibration Check. It runs in your browser.

3. Load your list and look at each confidence range. Ask one plain question: when the model said 90%, was it right about 90% of the time?

4. Write down the ranges where the gap is large. Those are the places to add a human check, or to stop treating the score as a real chance.

Where this fits

Calibration is one part of a fair test. The other big part is checking the model on data it has not seen, from a later time. My case study testing an alert model on later data walks through that kind of test, and looks at the AI that explains its alerts.