When Models Know When They Do Not Know: Calibration, Cascading, and Cleaning

When a model knows when it does not know, many possibilities emerge. The first question is how to enable a model to recognize that it does not know. A promising approach is to use confidence, computed from the model’s internal signals, to re- flect its ignorance. Prior work in specific domains has shown that calibration can provide reliable confidence estimates. In this work, we propose a simple, effective, and universal training-free method that applies to both vision and language mod- els, performing model calibration, cascading, and data cleaning to better exploit a model’s ability to recognize when it does not know. We first highlight two key empirical observations: higher confidence corresponds to higher accuracy within a single model, and models calibrated on the validation set remain calibrated on a held-out test set. These findings empirically establish the reliability and compara- bility of calibrated confidence. Building on this, we introduce two applications: 1. Model cascading with calibrated advantage routing and 2. Data cleaning based on model ensemble. Using the routing signal derived from the comparability of cali- brated confidences, we cascade large and small models to improve efficiency with almost no compromise in accuracy, and we further cascade two models of com- parable scale to achieve performance beyond either model alone (e.g., combining two models with around 82% accuracy yields over 90% accuracy). Leveraging multiple experts and their calibrated confidences, we design a simple yet effec- tive data-cleaning method that balances precision and detection rate to identify mislabeled samples in ImageNet and Massive Multitask Language Understand- ing (MMLU) datasets. Our results demonstrate that enabling models to recognize when they do not know is a practical step toward more efficient, reliable, and trustworthy AI.