- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper calibrates pretrained vision and language models. Its router uses the models' average confidence difference within bins to decide when to call the second model. Its cleaning rule flags labels rejected by several models whose predictions fall in their highest confidence bins. Experiments evaluate routing and label detection on vision and language benchmarks.
Critical strengths
- Calibrated confidence is comparable across models, so the difference between two of them serves as a routing signal for cascading and cleaning with almost no extra training.
- The cascading result goes beyond common practice. The usual setting has a large model cover a small one, and here two models of comparable scale are cascaded and beat either one alone. Cleaning reports precision and detection rate together on ImageNet and MMLU.
Critical weaknesses
- The paper sets out to show that calibration improves routing. Algorithm 1, however, decides from the average confidence difference between the two models inside a bin, and the validation set that builds the bins carries labels, so the accuracy difference inside the same bin can be computed without any calibration and could provide an alternative routing rule. The two rules are never compared, so the experiments cannot say how much of the gain comes from calibration.
- The cleaning experiments do not isolate what calibration contributes. Algorithm 2's high-confidence disagreement rule selects samples using each model's top K confidence bins. If calibration preserves cross-sample confidence rankings and predictions, the equal-count bins and flagged set remain unchanged under consistent tie handling. However, this invariance is not guaranteed for multiclass temperature scaling. Figure 4 varies the detection rate without a matched before-and-after calibration comparison, so the independent contribution of calibration remains unclear.
Questions for authors
- Inside the bins Algorithm 1 uses, how does the accuracy difference between the two models compare with the confidence difference near the threshold? Would routing change where the two disagree?
- With models, bin count and K fixed, how many flagged samples change after calibration? Is precision at the same detection rate different?
- Does the routing experiment report the fraction of inputs sent to the large model before and after calibration, and if that fraction barely moves, where does the accuracy gain come from?
Thanks for the new experiments. They do answer some of my questions, but the answers make me more cautious about the main claim. On routing, calibration seems to matter mainly when labeled data is limited. On cleaning, raw and calibrated confidence give very similar results. So the role of calibration is smaller than the paper presents, and I think fixing this needs more than adding tables. I will keep my score for now.