acceptodds
Under review as a conference paper at ICLR 2027

Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learning

Abstract

Reinforcement learning (RL) post-training has substantially improved the reasoning accuracy of vision-language models, yet the resulting policies remain poorly calibrated. Terminal correctness rewards provide no gradient that penalizes confident errors more than uncertain ones, and no signal that ties confidence to the quality of the supporting visual evidence. This mismatch becomes especially severe under corrupted or ambiguous visual inputs, where models continue to report high confidence on wrong answers. We introduce **Ranking-Aware Calibration (RAC)**, a training-time framework that supervises confidence using two comparison signals that group-based RL already produces at no additional labeling cost. The *ranking-aware group loss* enforces that a better rollout receives higher confidence than a worse one within the same prompt. The *clean–corrupted pairwise loss* enforces that confidence attenuates as visual evidence degrades. The ranking signal requires the policy to distinguish between correct and incorrect rollouts and improves task accuracy on most evaluated backbones. Both losses require no external confidence annotations and integrate naturally with group-based RL post-training. We instantiate RAC on Qwen2.5-VL, Qwen3.5, and InternVL-3.5 backbones and evaluate on six multimodal reasoning benchmarks under clean and corrupted inputs. Empirical results show that the ranking-aware loss improves task accuracy on most evaluated backbones, while the pairwise corruption loss reduces calibration error. Their combination improves ECE and Brier score over Vanilla-RL on all four backbones and achieves best or joint-best calibration among the compared methods across all four, while improving accuracy on three of the four backbones. Our code is available at <https://anonymous.4open.science/r/RAC>.

Primary area:
reinforcement learning
Keywords:
Calibration, Vision-Language Models, Reinforcement Learning, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VGH5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Ranking-Aware Calibration (RAC), a framework for supervising verbalized answer confidence during multimodal reinforcement learning. It combines a within-prompt ranking objective, which assigns higher confidence to correct than incorrect rollouts, with a clean–corrupted comparison objective that encourages confidence to decrease with visual degradation. These objectives are converted into sequence-level reward shaping terms and optimized with GRPO. Experiments cover four backbones, six multiple-choice benchmarks, and several corruption severities, with component ablations, hyperparameter studies, and post-hoc calibration comparisons. The reported results suggest that the combined objectives improve ECE and Brier score relative to Vanilla-RL, although accuracy gains and the contributions of individual components vary across backbones.

Critical strengths
  1. The paper addresses an important reliability problem through a practical training intervention. Confidence that responds to answer quality and visual degradation is useful for multimodal systems. The two comparison axes are understandable and require no additional human confidence annotations. Using verifier outcomes from sampled rollout groups makes the approach compatible with existing group-based RL workflows. Table 1 reports calibration improvements over Vanilla-RL across all four backbones, providing encouraging evidence that the intervention is not confined to a single model family.

  2. The experiments expose meaningful trade-offs rather than presenting a uniformly favorable picture. The separate +Rank and +Pair variants help distinguish accuracy improvements from calibration improvements. For example, on Qwen2.5-VL-7B, +Rank improves average accuracy from 0.617 to 0.635 while worsening ECE, whereas the combined method improves ECE with a small accuracy decrease relative to Vanilla-RL. The severity-transfer study identifies a detrimental effect of excessive training corruption, and Table 2 explicitly includes strong post-hoc controls. These analyses are informative and provide a useful basis for understanding when each signal helps.

Critical weaknesses

W1. The relationship between the proposed objectives and calibrated confidence is not sufficiently justified, and their implementation is incompletely specified.

  • The margin objectives in Equations (5) and (8) constrain relative confidence differences but do not determine an appropriate absolute probability scale. Multiple confidence assignments can satisfy the same comparisons while having different calibration. Moreover, the argument in Remark 1 that binary correctness is not a suitable probability target is unconvincing: binary outcomes can supervise calibrated probabilities through a proper scoring loss in expectation. The paper should distinguish the limitations of a particular optimization procedure from the statistical validity of such supervision. The existing RLCR comparison is useful, but does not fully isolate relative supervision from a proper-loss alternative under the same paired training setup.
  • Equation (8) requires a severity-dependent confidence drop for independently sampled clean and corrupted answers without conditioning on their correctness. The clean rollout can be wrong while the corrupted rollout is correct, and some perturbations may leave the relevant evidence intact. A universal margin can therefore conflict with answer-level confidence calibration. The paper does not quantify these cases or show whether a correctness-aware or evidence-preserving version changes the outcome. The deterioration under stronger training corruption makes this assumption particularly relevant.
  • Equations (6) and (9) define aggregate losses, but Equation (11) does not explicitly define the per-rollout rewards derived from them. “Accumulating” violations does not specify all signs, denominators, or how a clean–corrupted penalty is allocated between branches. These choices matter when advantages are normalized separately within each prompt group: a term shared by all rollouts cancels under centering. Exact reward formulas and the confidence-token treatment are needed to assess the actual learning signal.

W2. The evaluation does not yet establish the additional value of RAC for useful confidence discrimination and robustness.

  • Table 2 shows that Vanilla-RL with post-hoc calibration already substantially outperforms raw RAC in ECE and Brier score. RAC plus Platt scaling is better still, which is promising, but Table 3 reports very small fitted RAC slopes, approximately 0.032–0.071. These transformations strongly compress the probability range while preserving within-fold ordering. Their low ECE therefore does not by itself establish useful discrimination between correct and incorrect predictions. The paper lacks within-prompt ranking accuracy, error-detection AUROC, or risk–coverage results, despite motivating confidence through abstention, reranking, and routing. These measurements, together with a constant-confidence control, would clarify what the learned signal contributes beyond aggregate recalibration.
  • Training and evaluation use the same eight corruption operators (Section 4.1 and Appendix E). The severity-transfer experiment is useful, but it does not test transfer to unseen corruption families or distinguish task-relevant evidence loss from a learned response to visible corruption. Evaluation on held-out operators, together with confidence and correctness changes stratified by corruption type, would directly test the central evidence-quality claim.
  • The reported comparisons have no uncertainty estimates, and it is unclear whether Vanilla-RL and RLCR receive the same clean/corrupted exposure and rollout budget. This matters for both attribution and small differences, such as the 0.006 accuracy gain on Qwen3.5-4B. Dataset-level results, repeated-run variability, and explicit matched-budget settings are needed to judge the consistency and practical size of the gains.
Questions for authors
  1. Calibration objective: How do you justify the criticism of binary correctness targets in Remark 1 given their use with proper scoring losses? Under identical paired training and compute, how does RAC compare with directly supervising confidence using Brier or Bernoulli log loss?

  2. Clean–corrupted comparisons: How often is the clean answer incorrect while the paired corrupted answer is correct, or both answers remain correct? How does confidence behave in these cases, and does conditioning the margin on answer correctness materially change the results?

  3. Reward implementation: Please provide explicit formulas for the per-rollout ranking and corruption rewards in Equation (11), including pair aggregation, normalization, branch allocation, and handling of invalid confidence reports. Which output tokens receive the shaped GRPO advantage?

  4. Useful discrimination: Can you report within-prompt ranking accuracy, error-detection AUROC, and risk–coverage curves for Vanilla-RL, RAC, and their post-hoc calibrated variants? How do these compare with a constant-confidence predictor, especially after the strong compression produced by Platt scaling?

  5. Robustness and comparison protocol: Are training exposure and rollout budgets matched across Vanilla-RL, RLCR, and RAC? Please provide per-benchmark results, uncertainty estimates, and performance on corruption operators excluded from training. For the cross-fitting in Appendix B, are all corrupted versions of an original question kept in the same fold?

  6. Backbone identity: Table 1 and Section 4.1 specify InternVL-3.5-8B, whereas Appendix D names and links InternVL2.5-8B. Which checkpoint was actually used?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

We thank the reviewer for recognizing the importance of reliable confidence estimation, the practical design of RAC, and the breadth of our experimental evaluation. We address the concerns below, with additional evidence on calibration objectives, corruption robustness, and experimental consistency.

W1. Calibration objectives and reward construction

W1.1/Q1. Relative supervision provides effective calibration signals, supported by the main experiments and additional controlled comparisons.

As discussed in Sec. 3.2 and Remark 1, RAC exploits two complementary relationships naturally available during online RL: correct reasoning trajectories should receive higher confidence than incorrect ones, and confidence should reflect the quality of supporting visual evidence. Proper scoring rules provide statistically valid objectives for probabilistic calibration in expectation. RAC investigates the additional value of structured relative supervision within RL optimization.

Table 1 demonstrates consistent ECE and Brier improvements over Vanilla-RL on all four backbones. On Qwen3.5-4B, RAC achieves relative reductions of 31.5% in ECE and 20.9% in Brier.

RLCR is a strong and directly relevant training-time calibration baseline. Damani et al. develop RLCR specifically to address confidence calibration during reinforcement learning. Their method incorporates a proper scoring rule into the RL reward, provides a theoretical analysis of calibration-aware optimization, and demonstrates substantial calibration improvements across in-domain and out-of-domain evaluations. It therefore represents a substantive comparison against an established calibration-aware RL approach, rather than relying exclusively on correctness-only RL baselines.

Our results compare favorably with this baseline. On Qwen3.5-4B, RAC outperforms RLCR across Accuracy (0.603 vs. 0.578), ECE (0.248 vs. 0.254), and Brier (0.292 vs. 0.294). On Qwen2.5-VL-7B, RAC matches RLCR's ECE (0.302), while achieving higher accuracy (0.615 vs. 0.605) and lower Brier (0.315 vs. 0.322).

We additionally compare direct Bernoulli and Brier confidence supervision under the paired training setting:

BackboneMethodAccuracy ↑ECE ↓Brier ↓
Qwen2.5-VL-7BVanilla-RL0.6170.3220.334
+Brier Loss0.5900.3130.332
+Bernoulli Loss0.6020.3180.331
RLCR0.6050.3020.322
RAC0.6150.3020.315
Qwen3.5-4BVanilla-RL0.5970.3620.369
+Brier Loss0.5600.3240.307
+Bernoulli Loss0.5420.3230.301
RLCR0.5780.2540.294
RAC0.6030.2480.292

Both direct-loss methods improve calibration relative to Vanilla-RL, while their accuracy decreases and their calibration improvements remain below those of RLCR and RAC. These comparisons further support the effectiveness of incorporating relative supervision into multimodal RL.

W1.2/Q2. The corruption objective supervises evidence sensitivity while preserving correctness-based optimization.

Equation (8) introduces a directional relationship between confidence and visual evidence quality. The standard correctness reward remains active on both clean and corrupted branches, ensuring that correct reasoning continues to be rewarded regardless of corruption.

A corrupted rollout can occasionally be correct when its clean counterpart is incorrect, particularly under stochastic generation. Such cases arise naturally during multimodal inference. RAC maintains correctness-based policy optimization, allowing the model to solve corrupted inputs successfully.

We additionally measure the frequency of clean-incorrect / corrupted-correct pairs at a fixed corruption severity, averaged over the test set:

BackboneRACVanilla-RL
Qwen2.5-VL-3B14.0%13.7%
Qwen3.5-4B4.8%5.1%
InternVL-3.5-8B8.8%7.7%
Qwen2.5-VL-7B9.9%9.3%

The frequencies are comparable between RAC and Vanilla-RL across all four backbones, with differences of at most 1.1 percentage points. This suggests that the relative frequency of such stochastic correctness inversions remains largely unchanged under RAC training.

Explicitly encouraging corrupted inputs to produce incorrect answers would introduce an undesirable optimization shortcut, associating degradation with failure regardless of the available evidence. RAC instead retains the task reward as the primary supervision for answer correctness.

Furthermore, Fig. 3 already studies the influence of corruption severity. Mild corruption produces the strongest calibration transfer, while excessive corruption degrades both accuracy and calibration. Our training mixture (50% T0.2, 40% T0.4, 10% T0.6) balances informative visual degradation with preservation of task-relevant evidence.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

W1. Calibration objectives and reward construction

W1.3/Q3. Rollout-level reward construction and confidence-specific optimization

RAC implements both relative calibration objectives through explicit rollout-level reward allocation and confidence-specific policy-gradient optimization.

Ranking reward. Following Eqs. (4–6), let (Vi\mathcal V_i) denote the valid comparisons involving rollout (i) and rollouts of different correctness. The ranking reward is defined as

rirank=−1∥Vi∥∑j∈Vi[mrank−sgn⁡(ai−aj)(ci−cj)]+,r_i^{\mathrm{rank}}=-\frac{1}{\|\mathcal V_i\|}\sum_{j\in\mathcal V_i}\left[m_{\mathrm{rank}}-\operatorname{sgn}(a_i-a_j)(c_i-c_j)\right]_+,

with zero assigned when no valid comparison exists. The per-rollout normalization provides confidence-specific credit assignment by aggregating the margin violations associated with each confidence decision. Both participants in a valid comparison receive their respective violation-based signals.

Clean–corrupted reward. For each valid clean–corrupted pair (k), Eq. (8) defines the severity-dependent violation

ℓkcorr=[ck(s)−ck(0)+mcorr+αs]+.\ell_k^{\mathrm{corr}}=\left[c_k^{(s)}-c_k^{(0)}+m_{\mathrm{corr}}+\alpha s\right]_+.

The corresponding rollout-level rewards are assigned to both participating branches:

rkcorr,(0)=rkcorr,(s)=−ℓkcorr,r_k^{\mathrm{corr},(0)}=r_k^{\mathrm{corr},(s)}=-\ell_k^{\mathrm{corr}},

with zero assigned to invalid comparisons. This construction provides joint supervision over the paired confidence decisions, encouraging confidence to reflect changes in visual evidence quality.

Optimization and normalization. The task objective follows standard GRPO, with advantages normalized separately within clean and corrupted prompt groups, as specified in Sec. 3.2 and Algorithm 1. The ranking and corruption rewards are applied directly to numerical confidence tokens through dedicated, uncentered policy-gradient terms with token-specific masking and normalization. This preserves the pairwise calibration signals during optimization, including when both participating rollouts receive identical rewards. The resulting confidence-specific updates optimize the corresponding relative supervision objectives.

Equation (11) summarizes the unified task and calibration reward components, while Algorithm 1 describes the overall group-based training procedure. We will provide a more detailed specification of the rollout-level reward allocation, confidence-specific policy-gradient terms, and token-level optimization in the camera-ready version, making the implementation of the unified training framework fully explicit.

W2. Confidence discrimination, robustness, and experimental reliability

W2.1/Q4. Training-time calibration and confidence discrimination

RAC improves the learned policy and its confidence estimates directly, while post-hoc calibration provides complementary refinement.

The distinction between these two mechanisms is central to our contribution. Temperature scaling and Platt scaling transform confidence estimates from a frozen policy, leaving the generated answers and their accuracy unchanged. RAC incorporates correctness-based ranking and visual-evidence sensitivity into RL training, directly optimizing the model's confidence behavior alongside reasoning performance.

The improvements of RAC are already evident before any post-hoc calibration. Table 1 shows that RAC reduces raw ECE and Brier across all four backbones. On Qwen3.5-4B, ECE decreases from 0.362 to 0.248 and Brier from 0.369 to 0.292. On InternVL-3.5-8B, ranking supervision additionally improves accuracy from 0.438 to 0.557. These results demonstrate substantial benefits from training-time supervision independently of confidence rescaling.

Table 2 further demonstrates the complementarity of post-hoc calibration. RAC+Platt achieves ECE 0.017 and Brier 0.238, compared with 0.062 and 0.243 for Vanilla-RL+Platt, while retaining higher accuracy (0.603 vs. 0.597).

Regarding the fitted Platt parameters, the positive slopes preserve confidence ordering within each calibration fold. The transformation changes the numerical confidence scale, while the underlying ordering is determined by the learned policy.

We further evaluate the discrimination of raw confidence estimates using error-detection AUROC (e-AUROC). This metric directly measures how effectively confidence distinguishes correct from incorrect predictions, complementing the probabilistic calibration metrics.

BackboneVanilla-RL e-AUROC ↑RAC e-AUROC ↑Improvement
Qwen2.5-VL-3B0.5810.667+0.086
Qwen3.5-4B0.6070.709+0.102
InternVL-3.5-8B0.5960.686+0.090
Qwen2.5-VL-7B0.6180.691+0.073

These discrimination metrics provide a direct evaluation of the ranking information carried by raw confidence estimates, independently of post-hoc confidence transformations.

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

W2. Confidence discrimination, robustness, and experimental reliability

W2.2. RAC generalizes across diverse corruption mechanisms, including operators excluded from training.

Our original evaluation covers four model backbones, six multimodal reasoning benchmarks, eight corruption operators, and six test severity levels, yielding 36 benchmark–severity subsets. The selected operators span noise, atmospheric effects, geometric transformations, and digital distortions.

We further evaluate the remaining seven standard ImageNet-C operators excluded from training: Defocus Blur, Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, and Pixelate. The following results are averaged over the seven held-out operators and six benchmarks:

BackboneMethodAccuracy ↑ECE ↓Brier ↓
InternVL-3.5-8BVanilla-RL0.4230.3710.389
RAC0.5170.3270.311
Qwen2.5-VL-3BVanilla-RL0.4990.3420.341
RAC0.5740.2970.322
Qwen2.5-VL-7BVanilla-RL0.6030.3610.329
RAC0.5990.3100.328
Qwen3.5-4BVanilla-RL0.5870.3780.380
RAC0.5890.2590.302

The held-out results show that RAC maintains its calibration improvements across all four backbones, with largely preserved or improved reasoning accuracy. These results are consistent with the original evaluation and further support generalization to unseen corruption operators.

The six-benchmark evaluation already covers diverse reasoning domains and visual conditions. The additional held-out experiments strengthen the evidence for robustness across corruption families.

W2.3/Q5. Dataset-level and repeated-run evaluations provide further evidence of consistent performance.

We provide an additional per-dataset breakdown for Qwen3.5-4B, covering six reasoning benchmarks. Accuracy, ECE, and Brier are reported as averages over the evaluated corruption settings.

DatasetVanilla Acc ↑Vanilla ECE ↓Vanilla Brier ↓RAC Acc ↑RAC ECE ↓RAC Brier ↓
M3CoT0.6630.2870.3050.6620.1760.264
MathVerse0.5200.4410.4460.5200.2570.330
MathVision0.3390.5880.5830.3380.4240.403
MMMU0.5780.3590.3740.5900.1770.289
ScienceQA0.9360.0930.0940.9270.2600.167
WeMath0.5460.4040.4120.5800.1950.299
Macro Avg0.5970.3620.3690.6030.2480.292

The dataset-level analysis provides a more granular perspective on the aggregate results. RAC achieves lower ECE and Brier on five of the six benchmarks, with particularly substantial calibration improvements on MathVerse, MathVision, MMMU, and WeMath. Average accuracy improves at the macro level, while the relative performance varies across individual datasets.Expanded dataset-level evaluations will be included in the camera-ready version.

We additionally conduct three independent training runs for Vanilla-RL and RAC on all four backbones. The results are reported as mean ± standard deviation:

BackboneMethodAccuracy ↑ECE ↓Brier ↓
InternVL-3.5-8BVanilla-RL0.441 ± 0.00470.360 ± 0.00060.385 ± 0.0041
RAC0.543 ± 0.00150.314 ± 0.00240.319 ± 0.0048
Qwen2.5-VL-3BVanilla-RL0.516 ± 0.00320.330 ± 0.00420.351 ± 0.0066
RAC0.582 ± 0.00420.295 ± 0.00210.317 ± 0.0008
Qwen2.5-VL-7BVanilla-RL0.618 ± 0.00040.324 ± 0.01650.338 ± 0.0038
RAC0.614 ± 0.00140.301 ± 0.00150.313 ± 0.0004
Qwen3.5-4BVanilla-RL0.594 ± 0.00350.365 ± 0.00110.369 ± 0.0029
RAC0.600 ± 0.00370.249 ± 0.00400.298 ± 0.0036

The observed accuracy variations remain limited, while the improvements in ECE and Brier persist across the four backbones. This further supports the consistency of RAC's calibration gains under independent training runs.

All main experiments follow the GRPO configuration described in Sec. 4.1 and equal budgets, including a global batch size of 512 sequences, identical training steps and response lengths etc..

Reviewer_U3u6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ranking-Aware Calibration (RAC), a reward-shaping framework for multimodal reinforcement learning. RAC combines a within-prompt ranking loss that encourages higher confidence for correct rollouts than for incorrect ones, with a clean–corrupted pairwise loss that encourages confidence to decrease with increasing corruption severity. Experiments on three model families across six multiple-choice benchmarks show overall improved performance compared to vanilla GRPO.

Critical strengths
  • The problem is well motivated: correctness-only rewards do not explicitly supervise reported confidence. Synthetic image corruption provides a lightweight way to examine how confidence responds to degraded visual evidence.

  • Evaluation across multiple model families provides evidence of the method’s applicability across different multimodal architectures.

Critical weaknesses
  • Although six benchmarks are evaluated, only the macro average is reported. It would be more convincing to provide per-benchmark accuracy to assess the consistency of improvements across benchmarks.

  • In Table 1, +Rank worsens both ECE and Brier score on three of four backbones. It would be better to provide some discussion or clarification for this degradation.

  • Degrading visual input is a common technique in multimodal post-training. Even if direct empirical comparisons are not appropriate, the paper should clarify how RAC differs from prior approaches [1–3] in its use of degraded visual inputs.

[1] Liu, Xiangyan, et al. "Noisyrollout: Reinforcing visual reasoning with data augmentation." NeurIPS 2025.

[2] Liu, Chenxi, et al. "Modality-balancing preference optimization of large multimodal models by adversarial negative mining." arXiv preprint arXiv:2506.08022 (2025).

[3] Wang, Tianfu, et al. "All-Weather VLM: Enhancing Vision Language Models' Robustness Under Adverse Imaging Conditions." Third Conference on Language Modeling.

Questions for authors
  • Why are RLCR results omitted for InternVL-3.5 and Qwen2.5-VL-3B in Table 1?

  • What is RAC’s computational overhead relative to vanilla GRPO, and how does the comparison ensure relatively fair computational budget?

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank the reviewer for recognizing the motivation of RAC and its applicability across different multimodal architectures. We address the concerns below with additional experimental results and clarifications.

W1. Per-benchmark results

The calibration improvements extend across diverse benchmarks, with the complete macro-level evaluation already reported in Table 1.

Our evaluation covers six multimodal reasoning benchmarks across different domains, four model backbones, and six corruption severity levels. We report macro-averaged results in Table 1 to provide a concise comparison across model architectures and corruption conditions.

To provide a finer-grained view, we include the following per-benchmark results for Qwen3.5-4B:

DatasetVanilla Acc ↑Vanilla ECE ↓Vanilla Brier ↓RAC Acc ↑RAC ECE ↓RAC Brier ↓
M3CoT0.6630.2870.3050.6620.1760.264
MathVerse0.5200.4410.4460.5200.2570.330
MathVision0.3390.5880.5830.3380.4240.403
MMMU0.5780.3590.3740.5900.1770.289
ScienceQA0.9360.0930.0940.9270.2600.167
WeMath0.5460.4040.4120.5800.1950.299
Macro Avg0.5970.3620.3690.6030.2480.292

RAC improves ECE and Brier on five of the six benchmarks, with substantial gains on MathVerse, MathVision, MMMU, and WeMath. Average accuracy is maintained or improved on several benchmarks, with the overall macro-average increasing from 0.597 to 0.603. These findings further illustrate the effectiveness of RAC across heterogeneous reasoning tasks.

We will include the expanded per-benchmark results for the remaining backbones in the camera-ready version.

W2. Understanding the accuracy–calibration trade-off of +Rank

The distinct behaviors of +Rank and +Pair demonstrate the complementary roles of the two supervision signals.

The ranking objective directly supervises confidence discrimination between correct and incorrect reasoning trajectories. By encouraging the policy to distinguish successful reasoning from unsuccessful alternatives, it provides an additional learning signal beyond terminal correctness rewards. Accordingly, +Rank improves accuracy on three of the four backbones, including a substantial improvement from 0.438 to 0.557 on InternVL-3.5-8B.

However, stronger discrimination can also increase confidence separation and amplify overconfidence, especially when the absolute confidence scale is insufficiently constrained. This produces the observed trade-off between accuracy and calibration.

This phenomenon is consistent with the independent findings of Ma et al. (Decoupling Reasoning and Confidence, ICML 2026), which identify an optimization conflict between reasoning accuracy and probabilistic calibration during RL with verifiable rewards.

RAC addresses this trade-off through the complementary corruption objective. The clean–corrupted pairwise loss provides direct supervision over confidence sensitivity to visual evidence quality. Table 1 clearly demonstrates the resulting division of roles:

  • +Rank: Improves accuracy on three of four backbones, while calibration improvements depend on the model.

  • +Pair: Reduces both ECE and Brier on all four backbones.

  • +Pair+Rank: Improves both ECE and Brier over Vanilla-RL on all four backbones, while improving accuracy on three.

For example, on Qwen2.5-VL-7B, +Rank improves accuracy from 0.617 to 0.635, while ECE increases from 0.322 to 0.333. Introducing the corruption objective reduces ECE to 0.302 and Brier to 0.315, achieving a substantially better accuracy–calibration balance.

These results directly support the motivation for combining the two objectives: ranking supervision strengthens reasoning discrimination, while evidence-sensitive supervision improves confidence reliability.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W3. Distinction from prior visual degradation methods

RAC uses visual degradation to construct explicit confidence supervision within RL, addressing a distinct learning objective from existing augmentation-based approaches.

The three cited works employ degraded visual inputs for different purposes:

  • NoisyRollout [1] mixes clean and distorted trajectories to improve exploration diversity and visual reasoning robustness. Its contribution lies in modifying the rollout distribution and exploration process.

  • MBPO [2] constructs adversarial negative examples to mitigate modality imbalance and improve preference-based multimodal alignment, emphasizing the model's reliance on visual information.

  • All-Weather VLM [3] enhances reasoning under adverse imaging conditions through degradation-aware learning and preference optimization, targeting robust visual understanding.

RAC introduces a different training signal. Given clean and corrupted views of the same question, it explicitly supervises the relationship between their reported confidence values using a severity-dependent margin. Combined with within-prompt correctness-based ranking, this provides two complementary axes of confidence supervision directly within group-based RL.

The distinction lies in what visual degradation teaches the model. Existing approaches primarily exploit degraded images to improve reasoning robustness, exploration, or visual grounding. RAC additionally uses the relationship between clean and corrupted evidence to optimize confidence behavior itself, evaluated through ECE and Brier score.

This formulation requires no additional human confidence annotations and integrates into the existing RL training process through lightweight reward shaping.

Q1. Additional RLCR results

RLCR is a strong and directly relevant training-time calibration baseline, and the additional comparisons further support RAC's effectiveness.

RLCR explicitly incorporates a proper scoring rule into RL training and provides theoretical and empirical evidence for improved confidence calibration. It therefore represents a substantive comparison against an established calibration-aware RL approach.

Table 1 initially presents RLCR comparisons on Qwen2.5-VL-7B and Qwen3.5-4B alongside the full component ablations. We additionally provide RLCR results on InternVL-3.5-8B and Qwen2.5-VL-3B:

BackboneMethodAccuracy ↑ECE ↓Brier ↓
InternVL-3.5-8BVanilla-RL0.4380.3590.386
RLCR0.5080.3290.347
RAC0.5440.3110.316
Qwen2.5-VL-3BVanilla-RL0.5140.3290.349
RLCR0.5390.3090.327
RAC0.5800.2940.314
Qwen2.5-VL-7BVanilla-RL0.6170.3220.334
RLCR0.6050.3020.322
RAC0.6150.3020.315
Qwen3.5-4BVanilla-RL0.5970.3620.369
RLCR0.5780.2540.294
RAC0.6030.2480.292

These comparisons provide additional evidence that RAC achieves competitive or superior calibration against a strong training-time baseline while maintaining favorable reasoning accuracy.

Q2. Computational overhead and training-budget fairness

RAC introduces only lightweight scalar reward computations on top of the existing group-based RL workflow.

Both Vanilla-RL and RAC use the same clean–corrupted training mixture and matched rollout budgets. The comparison isolates the contribution of RAC's calibration objectives under identical data exposure.

Both objectives operate directly on quantities already available from sampled rollouts: verifier-based correctness, parsed confidence, and corruption severity. They require no auxiliary reward model, additional critic, or modification to the underlying model architecture. Thus, the additional computational complexity of RAC is concentrated in inexpensive reward shaping, making the method straightforward to incorporate into existing multimodal RL pipelines.

Dear Reviewer U3u6,

As the discussion period is drawing to a close, we wanted to kindly follow up and ask whether our response has addressed your concerns. If there are any remaining questions or points that you feel have not been sufficiently addressed, we would sincerely appreciate the opportunity to clarify them before the discussion period ends.

Thank you again for your time, thoughtful feedback, and consideration.

Reviewer_iEMUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ranking-Aware Calibration (RAC), which improves confidence in multimodal RL through two forms of relative supervision: correct responses should receive higher confidence than incorrect responses within the same prompt, and confidence should decrease as visual evidence degrades across clean–corrupted input pairs. Both objectives are converted into sequence-level rewards and optimized using GRPO. Experiments across four backbones and six benchmarks demonstrate improved calibration and accuracy gains on some models.

Critical strengths
  • An important problem and an easily integrated method. The paper addresses the gap between accuracy improvements and confidence reliability in multimodal RL. It constructs supervision from existing correctness labels and synthetic corruptions, without additional confidence annotations or changes to the model architecture.

  • Broad experimental coverage with explicit trade-offs. Experiments cover multiple models, benchmarks, and corruption levels, comparing individual losses, combined objectives, and post-hoc calibration. The results also clearly acknowledge that calibration gains do not always accompany accuracy improvements.

Critical weaknesses
  • The connection between relative ranking and probabilistic calibration is insufficiently established. The two objectives constrain the relative ordering and differences of confidence values, whereas calibration requires confidence to match empirical correctness rates. Satisfying ranking constraints does not guarantee calibration, and fixed margins may introduce unnecessary shifts.

  • The conversion from calibration losses to rollout-level rewards is unclear. Equation (11) introduces (r_i^{\mathrm{rank}}) and (r_i^{\mathrm{corr}}), but the main text does not fully specify their signs, allocation rules, or normalization. This is crucial because GRPO centers rewards within each prompt group, canceling penalties shared across the group. How rewards are assigned to the correct and incorrect responses in ranking pairs, and to both responses in clean–corrupted pairs, determines whether optimization actually encourages the intended confidence changes.

  • The benefits beyond simple confidence adjustment and additional training data have not been sufficiently isolated. In Table 2, post-hoc-calibrated Vanilla-RL already substantially outperforms uncalibrated RAC, while RAC with temperature scaling does not achieve lower ECE than the corresponding baseline. More comprehensive comparisons are therefore needed to establish the additional value of training-time calibration over simpler calibration approaches.

Questions for authors

Please refer to the Weakness part.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank the reviewer for recognizing the importance of reliable confidence estimation in multimodal RL and the practical applicability of RAC. We address the three concerns below.

W1. Relative supervision and probabilistic calibration

RAC introduces two complementary supervision signals, whose effectiveness is directly established by probabilistic calibration metrics.

Our motivation, detailed in Sec. 3.2 and Remark 1, is that conventional outcome-based RL provides limited supervision over confidence behavior. RAC addresses two essential relationships: (i) correct reasoning should receive higher confidence than incorrect reasoning under the same prompt, and (ii) confidence should respond to the quality of visual evidence. These relationships provide informative supervision directly from RL rollouts, avoiding the need to assign absolute confidence targets to individual trajectories.

The empirical results demonstrate consistent improvements in probabilistic calibration. As shown in Table 1, RAC reduces average ECE from 0.359 to 0.311 (InternVL-3.5-8B), 0.329 to 0.294 (Qwen2.5-VL-3B), 0.322 to 0.302 (Qwen2.5-VL-7B), and 0.362 to 0.248 (Qwen3.5-4B). Brier scores improve across all four backbones, with relative reductions reaching 31.5% in ECE and 20.9% in Brier.

Importantly, RAC also compares favorably with RLCR, which directly optimizes uncertainty-related training signals. On Qwen3.5-4B, RAC outperforms RLCR across Accuracy (0.603 vs. 0.578), ECE (0.248 vs. 0.254), and Brier (0.292 vs. 0.294). On Qwen2.5-VL-7B, RAC achieves equal ECE (0.302), higher accuracy (0.615 vs. 0.605), and lower Brier (0.315 vs. 0.322).

The sensitivity analysis in Fig. 4 further examines the influence of reward weights and demonstrates the effectiveness of appropriately balanced relative supervision. Together, these results substantiate our central finding: the two relative supervision signals provide effective training objectives for improving multimodal confidence calibration.

W2. Rollout-level rewards and GRPO optimization

RAC implements the two relative calibration objectives through explicit rollout-level rewards and confidence-specific policy-gradient optimization.

Ranking reward. Following Eqs. (4–6), let (Vi\mathcal V_i) denote the valid comparisons involving rollout (i) and rollouts of different correctness. The ranking reward is

rirank=−1∥Vi∥∑j∈Vi[mrank−sgn⁡(ai−aj)(ci−cj)]+,r_i^{\mathrm{rank}}=-\frac{1}{\|\mathcal V_i\|}\sum_{j\in\mathcal V_i}\left[m_{\mathrm{rank}}-\operatorname{sgn}(a_i-a_j)(c_i-c_j)\right]_+,

with zero assigned when no valid comparison exists.

The per-rollout normalization provides confidence-specific credit assignment, aggregating the valid margin violations associated with each confidence decision. Both participants in a valid comparison receive their respective violation-based signals, enabling the ranking objective to supervise confidence ordering within each prompt group.

Clean–corrupted reward. For each valid clean–corrupted pair (k), Eq. (8) defines the severity-dependent violation

ℓkcorr=[ck(s)−ck(0)+mcorr+αs]+.\ell_k^{\mathrm{corr}}=\left[c_k^{(s)}-c_k^{(0)}+m_{\mathrm{corr}}+\alpha s\right]_+.

The corresponding rollout-level rewards are

rkcorr,(0)=rkcorr,(s)=−ℓkcorr,r_k^{\mathrm{corr},(0)}=r_k^{\mathrm{corr},(s)}=-\ell_k^{\mathrm{corr}},

with zero assigned to invalid comparisons. Both participating confidence decisions receive the same pairwise supervision signal, encouraging confidence to respond appropriately to degraded visual evidence.

Optimization and normalization. The standard task objective uses GRPO advantages normalized separately within clean and corrupted prompt groups. The ranking and corruption rewards are applied to numerical confidence tokens through dedicated, uncentered policy-gradient terms, with token-specific masking and normalization. This design preserves the pairwise calibration signals throughout optimization, including when participating rollouts receive identical rewards.

Equation (11) summarizes the unified task and calibration reward components, while Algorithm 1 presents the overall group-based training procedure. We will provide a more detailed specification of the confidence-specific policy-gradient objectives, rollout-level reward allocation, and token-level normalization in the final version, further clarifying the implementation of RAC within the unified RL training framework.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W3. Advantages of training-time calibration over post-hoc methods

Post-hoc calibration effectively adjusts confidence estimates, but its benefits are limited to confidence rescaling. Temperature scaling and Platt scaling operate on a frozen policy, leaving generated answers and task accuracy unchanged. They also require a separate labeled validation set to fit calibration parameters. Consequently, improvements in ECE and Brier obtained through post-hoc adjustment do not necessarily translate into improved reasoning performance or confidence discrimination.

Training-time calibration enables joint improvements in reasoning accuracy and confidence reliability. RAC incorporates correctness-based ranking and visual-evidence sensitivity directly into RL training, allowing relative calibration supervision to influence the learned policy. Table 1 demonstrates these benefits across multiple backbones: RAC improves InternVL-3.5-8B accuracy from 0.438 to 0.544 and Qwen2.5-VL-3B accuracy from 0.514 to 0.580, while simultaneously reducing ECE and Brier. On Qwen3.5-4B, RAC reduces raw ECE from 0.362 to 0.248 and Brier from 0.369 to 0.292, alongside an accuracy improvement from 0.597 to 0.603.

RAC also complements post-hoc calibration, achieving competitive or superior calibration in most reported comparisons. Across temperature scaling and Platt scaling in Table 2, RAC combined with post-hoc calibration improves or matches the corresponding Vanilla-RL pipeline on three of four ECE/Brier comparisons. In particular, RAC+Platt achieves ECE 0.017 and Brier 0.238, outperforming Vanilla-RL+Platt (0.062 and 0.243), while retaining higher accuracy (0.603 vs. 0.597). Under temperature scaling, RAC matches Vanilla-RL in Brier (0.242), although its ECE is higher (0.079 vs. 0.063).

Furthermore, RAC outperforms the strong training-time RLCR baseline on the evaluated backbones. On Qwen3.5-4B, RAC achieves higher accuracy (0.603 vs. 0.578), lower ECE (0.248 vs. 0.254), and lower Brier (0.292 vs. 0.294). On Qwen2.5-VL-7B, RAC also achieves higher accuracy (0.615 vs. 0.605) and lower Brier (0.315 vs. 0.322), with identical ECE (0.302).

Together, these results demonstrate the value of integrating relative calibration supervision into RL training, yielding improvements in reasoning performance and raw confidence quality, with further gains available through complementary post-hoc calibration.

Dear Reviewer iEMU,

As the discussion period is drawing to a close, we wanted to kindly follow up and ask whether our response has addressed your concerns. If there are any remaining questions or points that you feel have not been sufficiently addressed, we would sincerely appreciate the opportunity to clarify them before the discussion period ends.

Thank you again for your time, thoughtful feedback, and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary by Authors

Dear Reviewers,

Thank you for your constructive feedback and valuable suggestions. We have conducted additional experiments and provided further methodological clarifications to address the main concerns raised during review. A summary of our responses is provided below:

  1. Additional Calibration Baselines (@iEMU, @U3u6, @VGH5): We extend RLCR comparisons to all four backbones and introduce direct Brier and Bernoulli loss baselines. The results demonstrate RAC's competitive advantages over existing training-time calibration methods.

  2. Confidence Discrimination (@iEMU, @VGH5): We evaluate error-detection AUROC across all four backbones, confirming that RAC improves the ability to distinguish correct from incorrect predictions beyond reducing calibration errors.

  3. Robustness to Unseen Corruptions (@VGH5): We evaluate seven additional corruption operators excluded from training. RAC maintains improved calibration across all backbones, demonstrating generalization to unseen visual degradations.

  4. Statistical Reliability and Dataset-Level Evaluation (@U3u6, @VGH5): We conduct three independent training runs across all four backbones and provide per-benchmark results. These evaluations confirm the consistency of RAC's calibration improvements across training runs and diverse reasoning tasks.

  5. Reward Construction and Optimization (@iEMU, @VGH5): We clarify the rollout-level reward allocation for both objectives and the confidence-specific policy-gradient optimization, explaining how pairwise supervision is preserved alongside standard GRPO training.

  6. Calibration Behavior and Experimental Protocol (@iEMU, @U3u6, @VGH5): We explain the complementary roles of ranking and corruption supervision, analyze correctness inversions under visual degradation, and clarify matched training budgets and question-level grouped cross-fitting.

We sincerely thank all reviewers and the AC for their thoughtful feedback. We hope these additional experiments and clarifications address the main concerns and support a favorable reassessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.