- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
I have very serious concerns with the paper. Among these, two concern me the most. The first one is obviously clarity. The writing is so obscure I could in no reasonable time provide a constructive feedback. Infact, the authors make no effort whatsoever in comunicating clearly the problem they are trying to solve nor their intents. This makes most of the paper so hard to follow. Nonetheless, I want to mention a second concern of mine. Indeed, although I see many other problems with the paper, related for instance to notation and positioning with related literature, what concerned me the most is the relevance of the contribution. Again, even if one could grasp what the authors objective with this manuscript was, which is not straightforward at all, there is no motivation for what they are doing. I cannot figure one practical use case of the paper contribution nor one context where one would even resort to the same anlaysis they try to propose: determining claibration error when moving from evaluations with candidates to evluations with candidates. If calibration was so important, and one could not do without more candidates, why not just calibrating the new confidence when more candidates are present? I simply do not undertsand what the practical use case of the analysis, which is the whole point of the paper, would be. IAnd I just cannot ignore that the trivial notion, that calibration may not hold when increasing the number of candidates, seems to me all one would ever need to know.
Critical strengths
see summary
Critical weaknesses
see summary
Questions for authors
see summary
5 comments
I thank the authors for their detailed response and substantial revisions. The presentation has improved, and I now have a clearer understanding of the problem being investigated and its intended contributions.
However, I remain unconvinced of the practical significance of the proposed approach. In particular, I would have liked to see a direct comparison between the rank-aware method and established post-hoc calibration techniques applied at the target selection budget, such as isotonic regression or temperature scaling. I understand that the authors aim to avoid collecting additional labeled data and recalibrating at each budget. Nevertheless, such a comparison would help establish whether the rank-aware approach offers meaningful advantages over simpler alternatives, or at least quantify the trade-offs involved. In particular, understanding how calibration performance varies with the amount of additional labeled data required by conventional methods would provide valuable practical guidance and substantially strengthen my assessment of the contribution.
While I acknowledge the theoretical contributions and the improvements made during rebuttal, my reservations regarding the overall significance of the work remain. I therefore maintain my original score at this stage.
Thank you for the follow-up and we are glad the revision made the problem and our contributions clearer. Following your suggestion, we compared the rank-aware predictor, which is fitted once at and uses no labels at the target budget, with standard post-hoc calibration methods applied at the target budget, and measured how their performance changes with the number of additional labels.
Compare the rank-aware method with established post-hoc calibration applied at the target budget, such as isotonic regression or temperature scaling, and show how calibration varies with the amount of additional labeled data.
Setup. We use Weaver, in which Llama-3.1 8B and 70B each generate 100 candidate answers per prompt, every candidate is labeled for correctness and scored by 96 verifiers, and the prompts come from MATH500 and MMLU-Pro. We follow the config of our Weaver experiment (details in App. G.6).
and are fitted once at . At each target , we fit 4 post-hoc calibration methods, temperature scaling (TS), Platt scaling, isotonic regression, and histogram binning, on top of with additional labels. One label is the correctness of the selected Best-of- answer for one prompt in the calibration set, which is disjoint from the training and test sets. The calibrators therefore use all the labels used to train , together with additional labels at the target budget.
ECE(%) at the target budget. In each cell, the four values correspond to , , , and target labels. Medians over the 96 verifiers, and each averaged over 10 splits.
| method | |||
|---|---|---|---|
| , no target labels | 4.24 | 6.73 | 8.34 |
| , no target labels | 2.70 | 2.96 | 3.90 |
| TS | 4.58 / 3.72 / 3.14 / 2.99 | 5.00 / 3.92 / 3.31 / 3.05 | 5.37 / 4.22 / 3.66 / 3.24 |
| Platt | 6.94 / 4.36 / 2.86 / 1.97 | 6.92 / 4.53 / 3.13 / 2.19 | 7.04 / 4.82 / 3.21 / 2.36 |
| isotonic | 10.79 / 7.48 / 5.11 / 3.39 | 9.98 / 7.07 / 4.95 / 3.24 | 9.33 / 6.58 / 4.56 / 3.02 |
| hist. binning | 7.70 / 7.29 / 6.87 / 5.34 | 7.30 / 6.93 / 6.22 / 4.88 | 7.30 / 6.43 / 5.19 / 3.89 |
Brier score / log loss at the target budget, with target labels for each calibrator.
| method | |||
|---|---|---|---|
| , no target labels | 16.43 / 49.72 | 15.92 / 48.67 | 15.65 / 51.53 |
| , no target labels | 15.65 / 47.35 | 14.91 / 46.46 | 14.91 / 46.92 |
| TS | 16.38 / 49.44 | 15.39 / 47.24 | 14.91 / 47.45 |
| Platt | 16.24 / 49.10 | 15.28 / 47.30 | 15.06 / 47.32 |
| isotonic | 16.53 / 55.51 | 15.52 / 52.53 | 15.10 / 50.03 |
| hist. binning | 17.29 / 62.34 | 15.99 / 57.01 | 15.42 / 53.11 |
Analysis.
- With few labels, recalibration does not reach the rank-aware predictor. With , every calibrator has higher ECE than rank-aware at every budget.
- Achieving comparable performance takes 100–200 labels for each budget. At , TS and Platt scaling need 100 labels to reach an ECE comparable to , and isotonic regression and histogram binning need 200. At and , only Platt scaling with 200 labels achieves comparable performance.
- In Brier score and log loss, rank-aware is better at every budget / label we tested. The calibrators themselves are fitted by minimizing these two metrics. TS and Platt scaling minimize log loss, and isotonic regression and histogram binning minimize Brier score. Even when every calibrator uses 200 target labels, has lower log loss than all of them at every budget, and lower or equal Brier score. With fewer labels, the gap is larger. This holds for all 96 verifiers at and , and for 74–95 of the all 96 verifiers at .
- A calibrator fitted at one budget does not transfer to other budgets, so its labels must be collected again whenever changes. After fitting at with 200 labels, TS, Platt scaling, and isotonic regression have ECE 3.83, 9.43, and 12.75 at . Here Platt scaling and isotonic regression are even worse than without recalibration (4.24). The rank-aware , trained once at , has ECE 2.70 at and 3.90 at without any target labels.
Thanks for encouraging us to improve the clarity of our writing. We have revised the draft to state the problem, the setup, and the analysis more directly. We hope these changes make the paper more accessible.
Regarding the relevance of the contribution, we would like to clarify. Our work is a theoretical analysis of how calibration changes as the selection budget N grows. We characterize how much calibration error can grow when a predictor calibrated at small budgets is deployed at a larger N. Our main result shows that as N grows, the guarantee from calibration at small budgets becomes progressively weaker, and beyond a certain scale it provides no useful constraint on calibration error at all. Moreover, we show there exist natural predictors whose calibration error reaches this worst case.
We further analyze the case with realistic, noisy estimates, where the useful range becomes even shorter. Our analysis tells a practitioner how far existing calibration can be trusted and when recalibration becomes necessary. As the reviewer mentions, simply recalibrating at the new N is one option. We never deny the value of recalibration — in fact, our analysis explains why it is necessary in many real scenarios when deploying at a larger N.
Beyond this, we also analyze a practical alternative: incorporating within-prompt rank. We prove that this approach removes the limitation on the extrapolation range. In experiments, rank-aware calibration g(S,U^), trained once, achieves accuracy comparable to recalibration at the target N (details in Appendix G.3), without requiring new labeled data or retraining required by recalibration. It needs only unlabeled references at inference time. In summary, our contribution goes beyond the qualitative observation that calibration may not hold at larger N. We provide analysis of when and how much it degrades, and a theoretically grounded solution that works in practice.
We recognize that the presentation in our original submission may not have conveyed these contributions as intended. We have revised the writing and added new results that validate the theoretical predictions on real datasets. We would be grateful if you could refer to the updated draft for further details, and we are happy to provide additional information that would be helpful for your review.