- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies confidence calibration for answers with the help of predecessor model, labeled source data, and unlabeled target inputs. The main motivation is that scores from predecessor model may retain useful distinctions. The proposed method, A-PAIR, constructs a predecessor-derived confidence anchor, audits candidate stable regions, preserves anchor confidence where supported, and applies importance-weighted multi-calibration updates elsewhere. The method is evaluated using Qwen2.5-1.5B and its instruction-tuned version on MMLU-Pro and SciQ under controlled distribution shifts, comparing against raw confidence, CPCS, and DACA.
Critical strengths
-
The paper clearly distinguishes score-only information limits, observable audit guarantees, and the transfer assumptions required for target calibration.
-
The evaluation separates construction, auditing, and testing, demonstrates complementary predecessor information.
Critical weaknesses
- Missing ablations leave the benefit of selective inheritance unclear. Without all-repair, anchor-only, and simple pair-aware recalibration baselines, the experiments do not isolate the contribution of selective inheritance. On MMLU-Pro, repair and TCE evaluation share subject indicators, so the improvement may reflect subgroup correction. The mixed ECE results and weaker SciQ calibration further limit evidence for a consistent benefit.
Questions for authors
-
Can you report results for all-repair, anchor-only, and a simple importance-weighted pair-aware recalibrator on both benchmarks? Comparisons using the same source-label budget and paired features would help distinguish selective inheritance from subgroup repair and additional information access.
-
With an all-zero initializer, the reference Brier loss equals the source correctness rate, potentially making the screen permissive in a high-accuracy setting such as SciQ. How often does the screen reject candidate iterates, and why use this reference rather than the fine-tuned confidence or anchor?
Thank you for the rebuttal. The additional evaluations have addressed most of my original concerns. In particular, the label-budget experiments provide useful evidence that selective inheritance can improve calibration when local supervision is scarce. Overall, I find the rebuttal satisfactory and will maintain my original score of Weak Acceptance.