- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Ranking-Aware Calibration (RAC), a framework for supervising verbalized answer confidence during multimodal reinforcement learning. It combines a within-prompt ranking objective, which assigns higher confidence to correct than incorrect rollouts, with a clean–corrupted comparison objective that encourages confidence to decrease with visual degradation. These objectives are converted into sequence-level reward shaping terms and optimized with GRPO. Experiments cover four backbones, six multiple-choice benchmarks, and several corruption severities, with component ablations, hyperparameter studies, and post-hoc calibration comparisons. The reported results suggest that the combined objectives improve ECE and Brier score relative to Vanilla-RL, although accuracy gains and the contributions of individual components vary across backbones.
Critical strengths
-
The paper addresses an important reliability problem through a practical training intervention. Confidence that responds to answer quality and visual degradation is useful for multimodal systems. The two comparison axes are understandable and require no additional human confidence annotations. Using verifier outcomes from sampled rollout groups makes the approach compatible with existing group-based RL workflows. Table 1 reports calibration improvements over Vanilla-RL across all four backbones, providing encouraging evidence that the intervention is not confined to a single model family.
-
The experiments expose meaningful trade-offs rather than presenting a uniformly favorable picture. The separate +Rank and +Pair variants help distinguish accuracy improvements from calibration improvements. For example, on Qwen2.5-VL-7B, +Rank improves average accuracy from 0.617 to 0.635 while worsening ECE, whereas the combined method improves ECE with a small accuracy decrease relative to Vanilla-RL. The severity-transfer study identifies a detrimental effect of excessive training corruption, and Table 2 explicitly includes strong post-hoc controls. These analyses are informative and provide a useful basis for understanding when each signal helps.
Critical weaknesses
W1. The relationship between the proposed objectives and calibrated confidence is not sufficiently justified, and their implementation is incompletely specified.
- The margin objectives in Equations (5) and (8) constrain relative confidence differences but do not determine an appropriate absolute probability scale. Multiple confidence assignments can satisfy the same comparisons while having different calibration. Moreover, the argument in Remark 1 that binary correctness is not a suitable probability target is unconvincing: binary outcomes can supervise calibrated probabilities through a proper scoring loss in expectation. The paper should distinguish the limitations of a particular optimization procedure from the statistical validity of such supervision. The existing RLCR comparison is useful, but does not fully isolate relative supervision from a proper-loss alternative under the same paired training setup.
- Equation (8) requires a severity-dependent confidence drop for independently sampled clean and corrupted answers without conditioning on their correctness. The clean rollout can be wrong while the corrupted rollout is correct, and some perturbations may leave the relevant evidence intact. A universal margin can therefore conflict with answer-level confidence calibration. The paper does not quantify these cases or show whether a correctness-aware or evidence-preserving version changes the outcome. The deterioration under stronger training corruption makes this assumption particularly relevant.
- Equations (6) and (9) define aggregate losses, but Equation (11) does not explicitly define the per-rollout rewards derived from them. “Accumulating” violations does not specify all signs, denominators, or how a clean–corrupted penalty is allocated between branches. These choices matter when advantages are normalized separately within each prompt group: a term shared by all rollouts cancels under centering. Exact reward formulas and the confidence-token treatment are needed to assess the actual learning signal.
W2. The evaluation does not yet establish the additional value of RAC for useful confidence discrimination and robustness.
- Table 2 shows that Vanilla-RL with post-hoc calibration already substantially outperforms raw RAC in ECE and Brier score. RAC plus Platt scaling is better still, which is promising, but Table 3 reports very small fitted RAC slopes, approximately 0.032–0.071. These transformations strongly compress the probability range while preserving within-fold ordering. Their low ECE therefore does not by itself establish useful discrimination between correct and incorrect predictions. The paper lacks within-prompt ranking accuracy, error-detection AUROC, or risk–coverage results, despite motivating confidence through abstention, reranking, and routing. These measurements, together with a constant-confidence control, would clarify what the learned signal contributes beyond aggregate recalibration.
- Training and evaluation use the same eight corruption operators (Section 4.1 and Appendix E). The severity-transfer experiment is useful, but it does not test transfer to unseen corruption families or distinguish task-relevant evidence loss from a learned response to visible corruption. Evaluation on held-out operators, together with confidence and correctness changes stratified by corruption type, would directly test the central evidence-quality claim.
- The reported comparisons have no uncertainty estimates, and it is unclear whether Vanilla-RL and RLCR receive the same clean/corrupted exposure and rollout budget. This matters for both attribution and small differences, such as the 0.006 accuracy gain on Qwen3.5-4B. Dataset-level results, repeated-run variability, and explicit matched-budget settings are needed to judge the consistency and practical size of the gains.
Questions for authors
-
Calibration objective: How do you justify the criticism of binary correctness targets in Remark 1 given their use with proper scoring losses? Under identical paired training and compute, how does RAC compare with directly supervising confidence using Brier or Bernoulli log loss?
-
Clean–corrupted comparisons: How often is the clean answer incorrect while the paired corrupted answer is correct, or both answers remain correct? How does confidence behave in these cases, and does conditioning the margin on answer correctness materially change the results?
-
Reward implementation: Please provide explicit formulas for the per-rollout ranking and corruption rewards in Equation (11), including pair aggregation, normalization, branch allocation, and handling of invalid confidence reports. Which output tokens receive the shaped GRPO advantage?
-
Useful discrimination: Can you report within-prompt ranking accuracy, error-detection AUROC, and risk–coverage curves for Vanilla-RL, RAC, and their post-hoc calibrated variants? How do these compare with a constant-confidence predictor, especially after the strong compression produced by Platt scaling?
-
Robustness and comparison protocol: Are training exposure and rollout budgets matched across Vanilla-RL, RLCR, and RAC? Please provide per-benchmark results, uncertainty estimates, and performance on corruption operators excluded from training. For the cross-fitting in Appendix B, are all corrupted versions of an original question kept in the same fold?
-
Backbone identity: Table 1 and Section 4.1 specify InternVL-3.5-8B, whereas Appendix D names and links InternVL2.5-8B. Which checkpoint was actually used?
5 comments
Rebuttal by Authors (2/4)
W1. Calibration objectives and reward construction
W1.3/Q3. Rollout-level reward construction and confidence-specific optimization
RAC implements both relative calibration objectives through explicit rollout-level reward allocation and confidence-specific policy-gradient optimization.
Ranking reward. Following Eqs. (4–6), let () denote the valid comparisons involving rollout (i) and rollouts of different correctness. The ranking reward is defined as
with zero assigned when no valid comparison exists. The per-rollout normalization provides confidence-specific credit assignment by aggregating the margin violations associated with each confidence decision. Both participants in a valid comparison receive their respective violation-based signals.
Clean–corrupted reward. For each valid clean–corrupted pair (k), Eq. (8) defines the severity-dependent violation
The corresponding rollout-level rewards are assigned to both participating branches:
with zero assigned to invalid comparisons. This construction provides joint supervision over the paired confidence decisions, encouraging confidence to reflect changes in visual evidence quality.
Optimization and normalization. The task objective follows standard GRPO, with advantages normalized separately within clean and corrupted prompt groups, as specified in Sec. 3.2 and Algorithm 1. The ranking and corruption rewards are applied directly to numerical confidence tokens through dedicated, uncentered policy-gradient terms with token-specific masking and normalization. This preserves the pairwise calibration signals during optimization, including when both participating rollouts receive identical rewards. The resulting confidence-specific updates optimize the corresponding relative supervision objectives.
Equation (11) summarizes the unified task and calibration reward components, while Algorithm 1 describes the overall group-based training procedure. We will provide a more detailed specification of the rollout-level reward allocation, confidence-specific policy-gradient terms, and token-level optimization in the camera-ready version, making the implementation of the unified training framework fully explicit.
W2. Confidence discrimination, robustness, and experimental reliability
W2.1/Q4. Training-time calibration and confidence discrimination
RAC improves the learned policy and its confidence estimates directly, while post-hoc calibration provides complementary refinement.
The distinction between these two mechanisms is central to our contribution. Temperature scaling and Platt scaling transform confidence estimates from a frozen policy, leaving the generated answers and their accuracy unchanged. RAC incorporates correctness-based ranking and visual-evidence sensitivity into RL training, directly optimizing the model's confidence behavior alongside reasoning performance.
The improvements of RAC are already evident before any post-hoc calibration. Table 1 shows that RAC reduces raw ECE and Brier across all four backbones. On Qwen3.5-4B, ECE decreases from 0.362 to 0.248 and Brier from 0.369 to 0.292. On InternVL-3.5-8B, ranking supervision additionally improves accuracy from 0.438 to 0.557. These results demonstrate substantial benefits from training-time supervision independently of confidence rescaling.
Table 2 further demonstrates the complementarity of post-hoc calibration. RAC+Platt achieves ECE 0.017 and Brier 0.238, compared with 0.062 and 0.243 for Vanilla-RL+Platt, while retaining higher accuracy (0.603 vs. 0.597).
Regarding the fitted Platt parameters, the positive slopes preserve confidence ordering within each calibration fold. The transformation changes the numerical confidence scale, while the underlying ordering is determined by the learned policy.
We further evaluate the discrimination of raw confidence estimates using error-detection AUROC (e-AUROC). This metric directly measures how effectively confidence distinguishes correct from incorrect predictions, complementing the probabilistic calibration metrics.
| Backbone | Vanilla-RL e-AUROC ↑ | RAC e-AUROC ↑ | Improvement |
|---|---|---|---|
| Qwen2.5-VL-3B | 0.581 | 0.667 | +0.086 |
| Qwen3.5-4B | 0.607 | 0.709 | +0.102 |
| InternVL-3.5-8B | 0.596 | 0.686 | +0.090 |
| Qwen2.5-VL-7B | 0.618 | 0.691 | +0.073 |
These discrimination metrics provide a direct evaluation of the ranking information carried by raw confidence estimates, independently of post-hoc confidence transformations.
Rebuttal by Authors (3/4)
W2. Confidence discrimination, robustness, and experimental reliability
W2.2. RAC generalizes across diverse corruption mechanisms, including operators excluded from training.
Our original evaluation covers four model backbones, six multimodal reasoning benchmarks, eight corruption operators, and six test severity levels, yielding 36 benchmark–severity subsets. The selected operators span noise, atmospheric effects, geometric transformations, and digital distortions.
We further evaluate the remaining seven standard ImageNet-C operators excluded from training: Defocus Blur, Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, and Pixelate. The following results are averaged over the seven held-out operators and six benchmarks:
| Backbone | Method | Accuracy ↑ | ECE ↓ | Brier ↓ |
|---|---|---|---|---|
| InternVL-3.5-8B | Vanilla-RL | 0.423 | 0.371 | 0.389 |
| RAC | 0.517 | 0.327 | 0.311 | |
| Qwen2.5-VL-3B | Vanilla-RL | 0.499 | 0.342 | 0.341 |
| RAC | 0.574 | 0.297 | 0.322 | |
| Qwen2.5-VL-7B | Vanilla-RL | 0.603 | 0.361 | 0.329 |
| RAC | 0.599 | 0.310 | 0.328 | |
| Qwen3.5-4B | Vanilla-RL | 0.587 | 0.378 | 0.380 |
| RAC | 0.589 | 0.259 | 0.302 |
The held-out results show that RAC maintains its calibration improvements across all four backbones, with largely preserved or improved reasoning accuracy. These results are consistent with the original evaluation and further support generalization to unseen corruption operators.
The six-benchmark evaluation already covers diverse reasoning domains and visual conditions. The additional held-out experiments strengthen the evidence for robustness across corruption families.
W2.3/Q5. Dataset-level and repeated-run evaluations provide further evidence of consistent performance.
We provide an additional per-dataset breakdown for Qwen3.5-4B, covering six reasoning benchmarks. Accuracy, ECE, and Brier are reported as averages over the evaluated corruption settings.
| Dataset | Vanilla Acc ↑ | Vanilla ECE ↓ | Vanilla Brier ↓ | RAC Acc ↑ | RAC ECE ↓ | RAC Brier ↓ |
|---|---|---|---|---|---|---|
| M3CoT | 0.663 | 0.287 | 0.305 | 0.662 | 0.176 | 0.264 |
| MathVerse | 0.520 | 0.441 | 0.446 | 0.520 | 0.257 | 0.330 |
| MathVision | 0.339 | 0.588 | 0.583 | 0.338 | 0.424 | 0.403 |
| MMMU | 0.578 | 0.359 | 0.374 | 0.590 | 0.177 | 0.289 |
| ScienceQA | 0.936 | 0.093 | 0.094 | 0.927 | 0.260 | 0.167 |
| WeMath | 0.546 | 0.404 | 0.412 | 0.580 | 0.195 | 0.299 |
| Macro Avg | 0.597 | 0.362 | 0.369 | 0.603 | 0.248 | 0.292 |
The dataset-level analysis provides a more granular perspective on the aggregate results. RAC achieves lower ECE and Brier on five of the six benchmarks, with particularly substantial calibration improvements on MathVerse, MathVision, MMMU, and WeMath. Average accuracy improves at the macro level, while the relative performance varies across individual datasets.Expanded dataset-level evaluations will be included in the camera-ready version.
We additionally conduct three independent training runs for Vanilla-RL and RAC on all four backbones. The results are reported as mean ± standard deviation:
| Backbone | Method | Accuracy ↑ | ECE ↓ | Brier ↓ |
|---|---|---|---|---|
| InternVL-3.5-8B | Vanilla-RL | 0.441 ± 0.0047 | 0.360 ± 0.0006 | 0.385 ± 0.0041 |
| RAC | 0.543 ± 0.0015 | 0.314 ± 0.0024 | 0.319 ± 0.0048 | |
| Qwen2.5-VL-3B | Vanilla-RL | 0.516 ± 0.0032 | 0.330 ± 0.0042 | 0.351 ± 0.0066 |
| RAC | 0.582 ± 0.0042 | 0.295 ± 0.0021 | 0.317 ± 0.0008 | |
| Qwen2.5-VL-7B | Vanilla-RL | 0.618 ± 0.0004 | 0.324 ± 0.0165 | 0.338 ± 0.0038 |
| RAC | 0.614 ± 0.0014 | 0.301 ± 0.0015 | 0.313 ± 0.0004 | |
| Qwen3.5-4B | Vanilla-RL | 0.594 ± 0.0035 | 0.365 ± 0.0011 | 0.369 ± 0.0029 |
| RAC | 0.600 ± 0.0037 | 0.249 ± 0.0040 | 0.298 ± 0.0036 |
The observed accuracy variations remain limited, while the improvements in ECE and Brier persist across the four backbones. This further supports the consistency of RAC's calibration gains under independent training runs.
All main experiments follow the GRPO configuration described in Sec. 4.1 and equal budgets, including a global batch size of 512 sequences, identical training steps and response lengths etc..
Rebuttal by Authors (1/4)
We thank the reviewer for recognizing the importance of reliable confidence estimation, the practical design of RAC, and the breadth of our experimental evaluation. We address the concerns below, with additional evidence on calibration objectives, corruption robustness, and experimental consistency.
W1. Calibration objectives and reward construction
W1.1/Q1. Relative supervision provides effective calibration signals, supported by the main experiments and additional controlled comparisons.
As discussed in Sec. 3.2 and Remark 1, RAC exploits two complementary relationships naturally available during online RL: correct reasoning trajectories should receive higher confidence than incorrect ones, and confidence should reflect the quality of supporting visual evidence. Proper scoring rules provide statistically valid objectives for probabilistic calibration in expectation. RAC investigates the additional value of structured relative supervision within RL optimization.
Table 1 demonstrates consistent ECE and Brier improvements over Vanilla-RL on all four backbones. On Qwen3.5-4B, RAC achieves relative reductions of 31.5% in ECE and 20.9% in Brier.
RLCR is a strong and directly relevant training-time calibration baseline. Damani et al. develop RLCR specifically to address confidence calibration during reinforcement learning. Their method incorporates a proper scoring rule into the RL reward, provides a theoretical analysis of calibration-aware optimization, and demonstrates substantial calibration improvements across in-domain and out-of-domain evaluations. It therefore represents a substantive comparison against an established calibration-aware RL approach, rather than relying exclusively on correctness-only RL baselines.
Our results compare favorably with this baseline. On Qwen3.5-4B, RAC outperforms RLCR across Accuracy (0.603 vs. 0.578), ECE (0.248 vs. 0.254), and Brier (0.292 vs. 0.294). On Qwen2.5-VL-7B, RAC matches RLCR's ECE (0.302), while achieving higher accuracy (0.615 vs. 0.605) and lower Brier (0.315 vs. 0.322).
We additionally compare direct Bernoulli and Brier confidence supervision under the paired training setting:
Both direct-loss methods improve calibration relative to Vanilla-RL, while their accuracy decreases and their calibration improvements remain below those of RLCR and RAC. These comparisons further support the effectiveness of incorporating relative supervision into multimodal RL.
W1.2/Q2. The corruption objective supervises evidence sensitivity while preserving correctness-based optimization.
Equation (8) introduces a directional relationship between confidence and visual evidence quality. The standard correctness reward remains active on both clean and corrupted branches, ensuring that correct reasoning continues to be rewarded regardless of corruption.
A corrupted rollout can occasionally be correct when its clean counterpart is incorrect, particularly under stochastic generation. Such cases arise naturally during multimodal inference. RAC maintains correctness-based policy optimization, allowing the model to solve corrupted inputs successfully.
We additionally measure the frequency of clean-incorrect / corrupted-correct pairs at a fixed corruption severity, averaged over the test set:
The frequencies are comparable between RAC and Vanilla-RL across all four backbones, with differences of at most 1.1 percentage points. This suggests that the relative frequency of such stochastic correctness inversions remains largely unchanged under RAC training.
Explicitly encouraging corrupted inputs to produce incorrect answers would introduce an undesirable optimization shortcut, associating degradation with failure regardless of the available evidence. RAC instead retains the task reward as the primary supervision for answer correctness.
Furthermore, Fig. 3 already studies the influence of corruption severity. Mild corruption produces the strongest calibration transfer, while excessive corruption degrades both accuracy and calibration. Our training mixture (50% T0.2, 40% T0.4, 10% T0.6) balances informative visual degradation with preservation of task-relevant evidence.