acceptodds
Under review as a conference paper at ICLR 2027

Confidence Inflation from RL Post-Training in Video Language Models

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves answer correctness in video multimodal LLMs, but what it does to confidence is unmeasured. We compare two checkpoints item by item based on the model’s option-letter distribution, which serves as the confidence readout, and limit the analysis to items that both answer incorrectly so that confidence changes are separated from changes in correctness. On that set, RL post-training systematically increases confidence on unchanged wrong answers, and we call this excess the calibration tax. The magnitude of the calibration tax varies across question types in a pattern that is associated with the post-training recipe and is not explained by item difficulty, and this effect replicates across three benchmarks and three model scales. At the item level, the tax appears as a scalar change in the readout that leaves the predicted option unchanged but can substantially alter the numerical confidence assigned to it. At a fixed confidence cutoff of 0.8, items whose predictions did not change are accepted for automatic handling 59% more often, and the error rate among them rises from 27.6% to 36.2%. We then ask where the tax can be controlled. Reward-side interventions can move the readout, but their effect is unstable across training runs. Because the tax also varies from item to item, a single global post-hoc correction removes only part of it, which motivates controlling confidence directly during RL. Our method, Reference Margin Anchoring (RMA), adds a two-sided penalty during RL that anchors the readout margin to the frozen reference already used by the KL term. In a 1200-step run, it removes 95% of the RL-induced calibration tax, at a cost of 0.4 accuracy points that fall within run-to-run variation, and returns the threshold curves close to their pre-RL level. Our results show that RL post-training can introduce substantial confidence inflation beyond its effect on answer correctness and that this increment can be controlled directly at the readout.

Primary area:
reinforcement learning
Keywords:
vision-language model, reinforcement learning with verifiable rewards, calibration, post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gQj2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates confidence inflation following reinforcement learning with verifiable rewards in video language models. It defines a paired “calibration tax” as the increase in confidence on examples answered incorrectly by both the SFT and RL checkpoints. Confidence is measured using a restricted distribution over option-letter tokens without chain-of-thought generation. The authors find positive confidence shifts across several benchmarks and model sizes, frequently without changes in the predicted option. A decomposition attributes much of this shift to changes in the magnitude of the option-logit vector. Motivated by this observation, the paper proposes reference-matched post-hoc rescaling and Reference Margin Anchoring (RMA), which penalizes deviations from the frozen reference model’s top-two logit margin during RL. On HERBench, the reported 1200-step RMA run reduces the tax from 3.26 to 0.16 percentage points, with a 0.40-point accuracy cost. Additional experiments examine transfer to TVBench, reward-side interventions, and training variability.

Critical strengths
  1. The empirical diagnosis is carefully developed and extends beyond a single checkpoint comparison. Examining shared failures—and then identical wrong predictions—helps demonstrate that the observed confidence increase cannot be explained solely by changes in answer correctness. Replication across benchmarks and InternVL model sizes strengthens this observation. The investigation of the HERBench meta-option confound is particularly valuable: the authors identify an initially misleading task-level pattern and qualify their interpretation accordingly.
  2. RMA is a simple intervention with encouraging evidence of controlling the measured effect. The method uses an existing frozen reference and a straightforward margin penalty. The one-sided versus two-sided comparison provides a useful design ablation. At 1200 steps, RMA improves ECE and multiclass Brier score relative to untreated RL, alongside its reduction of the paired tax. The reported 91% tax reduction on TVBench also suggests that the effect extends beyond HERBench.
Critical weaknesses
  1. The connection between the main confidence probe and confidence in actual model outputs remains insufficiently established. The main experiments use space-prefixed option tokens even though the models normally emit unspaced letters. Appendix E reports only 83–85% agreement between the predictions under these readouts, which is a substantial discrepancy. The positive tax under emitted-letter scoring supports the existence of the phenomenon, but the appendix does not establish that RMA’s reported 95% reduction transfers to that readout. This distinction matters because RMA directly optimizes the same probe used for evaluation.

    Furthermore, the measurements omit the reasoning process used by these models. The authors appropriately acknowledge this limitation, but confidence saturation after chain-of-thought does not establish that the alternative probe measures uncertainty in the generated reasoning-based answer. The deployment interpretation therefore needs stronger validation.

    The explanation in Section 2.5 also conflates two different rankings. Positive logit scaling preserves the ordering of options within an example, but item-dependent scaling can change confidence rankings across examples. Consequently, it does not imply invariance of AUROC or risk–coverage curves. Their observed stability is an empirical result, not a consequence of the proposed decomposition.

Questions for authors
  1. Readout validity: For the 1200-step untreated and RMA checkpoints, what are the tax, ECE, Brier score, accuracy, and threshold-based error rates using the actual unspaced option tokens? Is there evidence that RMA improves confidence estimates associated with answers generated using the normal reasoning procedure?
  2. Comparative calibration: At 1200 steps, how does RMA compare with properly tuned temperature scaling and the adapted Brier objective on full-set ECE, Brier score, and NLL? Please specify the Brier coefficient, its selection procedure, and whether calibration parameters are selected independently of the reported evaluation data.
  3. Replication and accuracy: Do matched-seed repetitions of the complete 1200-step experiment support both the reported tax reduction and the accuracy-preservation claim? Please provide uncertainty estimates for the accuracy differences and clarify how much of the RL accuracy gain RMA retains.
  4. Implemented objective: Appendix A.1 states that RMA includes an additional term with weight 0.10. What is its exact formula, and does an ablation establish that its contribution is negligible? Activation on approximately 1% of training steps alone does not determine its effect on optimization.
3 comments
AuthorsOpenReview ↗

(1/2) Official Comment by Authors

(1/2) Dear Reviewer gQj2,

Thank you for a precise review. We ran additional analyses in response to your questions and summarize the relevant results below.

W1 and Q1. Readout validity. On the emitted unspaced letters at 1200 steps, the untreated tax is +3.93 pp, and the four RMA runs remove 61, 92, 61 and 88%. Across the four runs, ECE drops from 0.210 (untreated RL) to 0.176–0.190, Brier from 0.811 to 0.792–0.803, and NLL from 1.705 to 1.638–1.673. Accuracy is 0.350 for untreated RL and 0.343 to 0.349 for the four RMA runs, with no significant difference from untreated RL (all p ≥ 0.1). RMA was trained only on the spaced probe, so this is transfer. At τ = 0.8 on the unspaced readout, SFT accepts 219 items (35.2% error), untreated RL 314 (41.7%), and the RMA runs 212 to 251 (35.8 to 37.8%). At every threshold from 0.6 to 0.9 and on both readouts, all four runs accept fewer items with lower error than untreated RL (full table in our response to Reviewer otsy).

W2 and Q1. Reasoning-based answers. We measured confidence as agreement over k = 5 CoT samples. RMA removes 32 to 47% of the raw tax. After label-based recalibration of both models, untreated RL keeps +3.06 ± 0.26 pp while RMA keeps +0.44 to +1.25 pp. RMA therefore also reduces inflation in final answers produced after reasoning, where post-hoc correction leaves a residual. This is confidence in the reasoning-conditioned answer, not uncertainty within the trace.

W3. Section 2.5. You are right. Per-item scaling preserves the order of options within an item but can reorder confidence across items, so the decomposition alone does not guarantee invariant AUROC or risk-coverage. We have now verified this directly. AURC stays between 0.512 and 0.519 for every checkpoint, and the revised Section 2.5 will present it as a verified empirical result.

Q2. Tuned TS and the Brier objective. TS uses 2-fold cross-fitting, so each item is scored with a temperature fit on the other fold, never on its own label. This experiment directly tests our scale account. TS removes about 88% of the tax, as a scale mechanism predicts, and attains the lowest ECE, Brier and NLL, after which all checkpoints are nearly indistinguishable on these aggregate metrics. The paired tax still resolves what they miss (general response, item 3). With TS applied to both SFT and RL, untreated RL keeps +0.39 ± 0.06 pp and RLCR +0.37 ± 0.08 pp, while RMA gives −0.04, +0.20, −0.11 and −0.12 pp. RMA is thus the training-time complement to post-hoc calibration, not a substitute for it. Our RLCR baseline is an RLCR-style Brier term adapted to our readout, not a reproduction. Its coefficient (λ = 0.45) was set once before training to match the magnitude of the RMA penalty on stored logits, and not tuned on any reported metric.

Q3. Replication and accuracy. Every RMA run removes at least 58% of the tax (95, 58, 58 and 79%, mean 73%), and we will report all four in the revision. These are independent RMA repetitions, not matched-seed comparisons against untreated RL, because rollout seeds were not fixed across conditions. They show the reduction holds in every run. At 300 steps, all seven two-sided RMA runs show a lower tax than both untreated runs (permutation p ≈ 1/36). On accuracy, the RL gain over SFT without CoT is itself small and not significant at 1200 steps (+0.85 pp, 95% CI −0.20 to +1.90). No RMA run differs significantly from SFT. Two keep part of the RL gain (0.2 and 0.4 pp below untreated RL), and two are at or slightly below the SFT level (1.1 and 1.4 pp below untreated RL). With CoT, accuracy is 0.299 for SFT, 0.325 for untreated RL and 0.296 to 0.307 for RMA. We will report this cost in the limitations. A second untreated run is in progress, and we will post its result here as a follow-up.

Q4. The additional term in Appendix A.1. With m the top-1 minus top-2 option-logit margin, the full objective is

L = L_GRPO + β1 · 1[ref correct] · |m_θ − m_ref| + β2 · 1[ref correct and policy wrong] · CE(π_θ, y)

with β1 = 0.45, β2 = 0.10, and CE computed over the softmax of the option-letter logits. We agree that its activation rate alone does not show that its effect is negligible, so we retrained RMA without it using the same recipe for 1200 steps. This single-term run removes 66% of the tax on the spaced readout, 75% on the unspaced readout and 43% on TVBench, all within the corresponding ranges of the four full-objective runs (58 to 95%, 61 to 92% and 23 to 92%). Under CoT it removes 52%, slightly above the full-objective range of 32 to 47%. Its accuracy is within noise of untreated RL. The margin anchor in Eq. 4 alone therefore accounts for most of the reduction on the spaced, unspaced and TVBench evaluations, and the CoT result is comparable. As this ablation is a single run, we do not rule out a small contribution from the additional term. We will add the formula and this ablation to Appendix A.1.

AuthorsOpenReview ↗

(2/2) Official Comment by Authors

(2/2) Full-set Brier and NLL for Q2. You asked how RMA compares with tuned TS and the adapted Brier objective on full-set ECE, Brier and NLL. ECE and the paired tax are in the general response (item 3). The table below adds Brier and NLL on all 2,000 HERBench items at 1200 steps (spaced readout). TS is fit with 2-fold cross-fitting, so each item is scored with a temperature fit on the other half. Lower is better for both metrics.

\text{Model} & \text{Accuracy} & \text{Brier raw} & \text{Brier after TS} & \text{NLL raw} & \text{NLL after TS} \\\\ \hline \text{SFT} & \text{0.346} & \text{0.776} & \text{0.745} & \text{1.602} & \text{1.496} \\\\ \text{Untreated RL} & \text{0.355} & \text{0.789} & \text{0.744} & \text{1.646} & \text{1.493} \\\\ \text{RLCR-style Brier} & \text{0.353} & \text{0.790} & \text{0.744} & \text{1.645} & \text{1.494} \\\\ \mathbf{RMA\\ (4\\ runs)} & \text{0.341–0.353} & \text{0.777–0.785} & \text{0.745–0.747} & \text{1.604–1.629} & \text{1.495–1.500} \end{array}$$ These results show three things. - **Without post-hoc calibration, every RMA run improves Brier and NLL over untreated RL**, and brings both close to the SFT level. The RLCR-style Brier objective leaves both essentially unchanged at 1200 steps (Brier 0.790 vs 0.789 for untreated RL, NLL 1.645 vs 1.646). - **TS largely corrects the common scale in every checkpoint.** Brier (0.744 to 0.747) and NLL (1.493 to 1.500) then barely distinguish them, so these aggregate metrics cannot show what remains. - **The paired estimator still detects residual differences after TS.** It is +0.39 pp for untreated RL and +0.37 pp for RLCR, but −0.12 to +0.20 pp for RMA (general response, item 3). This is the item-dependent component that aggregate metrics average away. The unspaced readout shows the same pattern. Raw Brier is 0.811 for untreated RL and 0.792 to 0.803 for RMA, and after TS all checkpoints fall between 0.744 and 0.746. We will include this table in the revision. **TVBench across four runs.** Your summary cites a 91% tax reduction on TVBench from the single run in the paper. Across the four 1200-step RMA runs, the reductions are 92, 63, 23 and 77% (mean 64%), and we will report all four. **Per-run accuracy uncertainty for Q3.** Paired differences on the same 2,000 HERBench items without CoT (spaced readout), with normal-approximation 95% CIs and unadjusted McNemar p-values. $$\begin{array}{l|c|c|c} \text{Model} & \text{Accuracy} & \Delta \text{ vs SFT (pp)} & \Delta \text{ vs untreated RL (pp)} \\\\ \hline \text{Untreated RL} & \text{0.355} & \text{+0.85 (see Q3)} & \text{} \\\\ \text{RMA run 1} & \text{0.351} & \text{+0.45 [-0.38, +1.28], p = 0.34} & \text{-0.40 [-1.46, +0.66], p = 0.52} \\\\ \text{RMA run 2} & \text{0.353} & \text{+0.65 [-0.11, +1.41], p = 0.12} & \text{-0.20 [-1.20, +0.80], p = 0.77} \\\\ \text{RMA run 3} & \text{0.341} & \text{-0.55 [-1.26, +0.16], p = 0.17} & \text{-1.40 [-2.44, -0.36], p = 0.010} \\\\ \text{RMA run 4} & \text{0.344} & \text{-0.25 [-1.03, +0.53], p = 0.62} & \text{-1.10 [-2.13, -0.07], p = 0.045} \end{array}$$ No run differs significantly from SFT. Runs 3 and 4 show nominally lower accuracy than untreated RL (unadjusted p = 0.010 and 0.045), whereas runs 1 and 2 do not. We report this variability as a limitation. We would be glad to discuss any remaining questions.
AuthorsOpenReview ↗

Additional Official Comment by Authors

Follow-up on Q3. Second untreated run. As promised, we trained a second untreated run with the same recipe for 1200 steps. Its tax is +4.42 pp on the spaced readout, +5.30 pp on the emitted readout, and +1.35 pp on TVBench, against +3.26, +3.93 and +1.05 pp for the first run. Every RMA run (+0.16 to +1.38 pp spaced, +0.30 to +1.55 pp emitted, +0.09 to +0.81 pp on TVBench) stays below both untreated runs on all three evaluations. Measured against the mean of the two untreated runs, the four RMA runs remove 64 to 96% of the spaced tax.

On the main spaced readout, the second untreated run reaches essentially the same accuracy as the first (0.356 against 0.355, McNemar p = 1.0), so the accuracy comparisons in the paper are unchanged. On the emitted readout, it reaches 0.362, 1.2 pp above the first run, which places the four RMA runs 1.3 to 1.9 pp below it on that readout. Both untreated runs are reported in Appendices F and G of the revised manuscript, and this variability is discussed in the limitations.

Reviewer_otsyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether RL with verifiable rewards (RLVR) makes video-language models overconfident on incorrect answers. The authors introduce a paired estimator that compares two checkpoints only on items that both answer incorrectly, thereby separating confidence changes from changes in correctness. On HERBench, confidence on shared failures increases by 3.74 percentage points after RL; in 88% of these cases, the model selects the same wrong option but assigns it higher confidence. This shifts the operating point of fixed-confidence automation: at a threshold of 0.8, more items are accepted automatically while the error rate among accepted items increases. The authors further show that much of the change can be described as a rescaling of the option-logit readout rather than a change in option ranking. They propose Reference Margin Anchoring (RMA), which constrains the top-1 readout margin during RL training toward the frozen reference model. In a 1200-step experiment, RMA reduces the paired calibration tax from +3.26 to +0.16 percentage points (reported as 95% removal), at an accuracy cost of 0.4 points. The paper also evaluates reward-side penalties and post-training rescaling; reward-side interventions are unstable across runs, while global rescaling only partially removes the effect.

Critical strengths
  1. The paired estimator and same-prediction threshold analysis are clearly motivated and make a useful distinction between answer changes and confidence changes. The paper also connects the phenomenon to a concrete deployment consequence.

  2. RMA is closely aligned with the diagnosed logit-margin mechanism. The paper provides a coherent comparison between training-time control, post-hoc correction, and reward-side interventions, and the reported 1200-step result is practically compelling.

Critical weaknesses
  1. The confidence construct is narrow: it is measured without chain-of-thought as the maximum softmax probability over five option-letter tokens, rather than the model's generated confidence. The conclusions therefore mainly apply to multiple-choice video QA and this particular readout; they do not yet establish the same effect for chain-of-thought or open-ended answers.

  2. The paired estimator conditions on shared failures, and many deployment analyses further restrict to items with unchanged predictions. This is a useful diagnostic, but it does not fully characterize confidence changes on corrected answers or newly introduced errors. In addition, the explanatory value of the per-item rescaling decomposition would be more convincing with stronger random or statistical null baselines.

Questions for authors
  1. Does RMA transfer to the emitted/unspaced option-token readout, to confidence measured after chain-of-thought, and to held-out benchmarks? For the post-hoc rescaling, can the scale be estimated on a held-out set or transferred across benchmark distributions rather than estimated from the evaluation items themselves?

  2. Can you report full selective-risk/coverage curves including items whose predictions change, with separate analysis of corrected errors and newly introduced errors?

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer otsy,

Thank you for the clear and constructive review. We address your two questions and the related concerns below.

One clarification regarding your summary. The 95% removal came from a single run. Across four runs, every RMA run removes at least 58% of the untreated tax (mean 73%). Without CoT, the accuracy gain of RL over SFT is itself small and not statistically significant (+0.85 pp, 95% CI −0.20 to +1.90), and the accuracy of all four RMA runs is statistically indistinguishable from SFT (general response, item 1).

W1 and Q1. Confidence construct and transfer beyond the main readout. Open-ended answers have no single option-level readout, so we will state the multiple-choice scope in the limitations. Within it, we tested three further settings.

  • Emitted unspaced tokens. On the unspaced option letters the model actually emits, the untreated tax is +3.93 pp. Every RMA run removes at least 61% (mean 75%), with no significant accuracy change. RMA was trained on the space-prefixed readout only.
  • After chain-of-thought. With agreement among k = 5 sampled reasoning chains as confidence, the untreated tax is +11.51 ± 0.75 pp, and RMA reduces it by 32 to 47%. Label-based recalibration alone leaves a residual of +3.06 pp for untreated RL. RMA further reduces this residual to +0.44 to +1.25 pp, a 59 to 86% reduction.
  • Held-out benchmark. On TVBench, the untreated tax is +1.05 pp. The four RMA runs, trained on HERBench, reduce it by 92, 63, 23 and 77% (mean 64%).
  • Post-hoc scale. Estimated on a held-out split, the reference-matched scale gives nearly the same correction as the in-sample estimate. Transferred from HERBench to TVBench, it over-corrects, so the scale should be estimated per benchmark.

W2 and Q2. Items whose predictions change. Groups compare untreated RL with SFT on the same items (fixed by RL means wrong under SFT and right under RL). Cells give the confidence change vs SFT (spaced readout), and the last column is the range over four RMA runs.

\text{Group vs SFT} & \text{Items} & \text{Untreated RL (pp)} & \text{RMA range (pp)} \\\\ \hline \text{Both correct} & \text{647} & \text{+4.01} & \text{+0.06 to +1.40} \\\\ \text{Both wrong} & \text{1,244} & \text{+3.26} & \text{+0.16 to +1.38} \\\\ \text{Fixed by RL} & \text{63} & \text{+1.11} & \text{-0.27 to +1.01} \\\\ \text{Newly broken by RL} & \text{46} & \text{+1.33} & \text{-0.80 to +0.94} \end{array}$$ RL increases confidence in all four groups, including items on which it introduces new errors. RMA attenuates these increases in every group. The threshold-wise selective-risk analysis below includes all items, changed predictions included. Each cell gives items accepted / error among accepted (%). $$\begin{array}{l|l|c|c|c|c} \text{Readout} & \text{Model} & \tau = 0.6 & \tau = 0.7 & \tau = 0.8 & \tau = 0.9 \\\\ \hline \text{Spaced} & \text{SFT} & \text{447 / 44.3} & \text{261 / 36.4} & \text{161 / 30.4} & \text{81 / 19.8} \\\\ & \text{Untreated} & \text{588 / 48.3} & \text{375 / 42.7} & \text{223 / 37.2} & \text{117 / 26.5} \\\\ & \mathbf{RMA\\ (4\\ runs)} & \text{448–488 / 44.4–45.7} & \text{256–303 / 36.3–40.6} & \text{157–176 / 28.7–32.4} & \text{80–91 / 16.2–19.8} \\\\ \hline \text{Unspaced} & \text{SFT} & \text{579 / 48.0} & \text{354 / 42.9} & \text{219 / 35.2} & \text{119 / 21.8} \\\\ & \text{Untreated} & \text{743 / 51.7} & \text{498 / 47.0} & \text{314 / 41.7} & \text{166 / 31.9} \\\\ & \mathbf{RMA\\ (4\\ runs)} & \text{578–649 / 49.8–51.0} & \text{357–410 / 42.9–46.7} & \text{212–251 / 35.8–37.8} & \text{110–132 / 18.2–26.8} \end{array}$$ In all 32 comparisons, every RMA run accepts fewer items than untreated RL with a lower error rate among them. AURC stays at 0.512 to 0.519 for every checkpoint, so RL mainly shifts the operating point of a fixed threshold rather than ranking quality. This is an empirical finding, and we will state it as such in the revised Section 2.5. **W2. Null baselines for the decomposition.** Per-item scales reproduce 67% of the shared-failure shift on the spaced readout, 80% on the unspaced readout and 102% on TVBench (slightly overshooting the small TVBench shift). As a null, we permute the estimated scales across items, which gives 58 ± 1.5%, 76 ± 1.1% and 47 ± 7%. No permuted baseline matched the observed values. A single global scale reproduces 63%, 81% and 72%. Capping the logit norm at the reference value removes 91 and 97% of the tax, a third independent check. These results suggest that much of the shift comes from a common logit scale, which global temperature scaling can correct. The remaining item-dependent component, most pronounced on TVBench, cannot be captured by a single temperature, and it is the component RMA is designed to address. We hope these analyses address your concerns.
Reviewer_znyjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how reinforcement learning with verifiable rewards (RLVR) affects the confidence calibration of video multimodal LLMs. Using an item-matched evaluation restricted to joint failures (items where both SFT and RL models answer incorrectly), the authors show that RLVR systematically inflates confidence on wrong answers, a phenomenon termed "calibration tax." Logit decomposition indicates that this inflation is primarily driven by a uniform scalar rescaling of option logits rather than changes in their relative ordering. Based on this observation, the authors propose two mitigations: Reference Margin Anchoring (RMA), which adds an L1 penalty to the GRPO objective to keep the top-1/top-2 option logit margin close to that of the frozen reference model, and a post-hoc Reference-Matched Rescaling method that divides the output projection weights by the estimated mean rescaling factor. Experiments on multiple-choice video QA benchmarks across Qwen2.5-VL and InternVL3.5 models show that both methods reduce confidence inflation while largely preserving accuracy gains.

Critical strengths
  1. The empirical diagnostic is clean and well-controlled. Restricting the paired confidence comparison to jointly failed items removes item-difficulty survivorship bias, and decomposing option logit shifts into scalar rescaling versus directional residuals clearly illustrates the geometry of RL-induced overconfidence.

  2. The paper provides a useful methodological observation regarding RL training variance: reward-side calibration penalties suffer from high run-to-run instability across rollout seeds, highlighting the necessity of seed-matched ablations in RLVR studies.

  3. The empirical characterization is consistent across multiple video QA benchmarks (HERBench, TVBench) and model scales (Qwen2.5-VL and InternVL3.5 from 2B to 14B).

Critical weaknesses
  1. The core phenomenon investigated in this paper, that policy-gradient post-training (RLHF and RLVR) causes output distribution sharpening, entropy collapse, and severe calibration degradation, is already a well-established conclusion in the text LLM literature (prominently documented in the GPT-4 Technical Report and subsequent RLHF/RLVR calibration studies). This paper essentially takes this known RL-sharpening phenomenon and performs a fine-grained paired statistical decomposition on multiple-choice Video-LLM benchmarks, without uncovering any mechanism that is fundamentally unique to video or multimodal representations.

  2. The methodological novelty of the two proposed solutions is limited. Given that confidence inflation is driven by scalar logit rescaling, the post-hoc Reference-Matched Rescaling method (dividing the output projection matrix by a scalar factor t-bar) is mathematically equivalent to classic Temperature Scaling (Guo et al., 2017), differing only in that the temperature is estimated against the SFT reference logits rather than a labeled validation set. Similarly, Reference Margin Anchoring (RMA) is a straightforward L1 distance regularizer on the top-1/top-2 logit margin relative to the reference model, closely paralleling standard logit-level distillation and confidence penalties.

  3. Both the diagnostic decomposition and the RMA regularizer are restricted to multiple-choice QA setups where confidence is read out from four discrete option-letter logits (A/B/C/D). The paper does not demonstrate how the margin definition or the rescaling decomposition can be applied to open-ended video generation or reasoning tasks where answers span multi-token sequences.

  4. The experimental evaluation of the post-hoc rescaling method omits standard post-hoc calibration baselines such as validation-set Temperature Scaling or Platt Scaling, which are the default lightweight solutions when scalar logit inflation occurs.

Questions for authors
  1. What aspect of the observed confidence inflation or scalar logit rescaling is specific to Video-LLMs, as opposed to the standard distribution sharpening and calibration degradation already widely observed in text-only RLHF and RLVR?

  2. How does the proposed post-hoc Reference-Matched Rescaling compare against standard Temperature Scaling fitted on a small held-out validation set?

  3. Since RMA relies on computing the top-1 minus top-2 logit margin over discrete multiple-choice option tokens and requires conditioning on whether the reference model answered correctly, how can this regularizer be generalized to free-form generative tasks?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer znyj,

Thank you for the positive assessment and for recognizing our controlled diagnostic, the seed-level analysis and the cross-model evaluation. The new analyses build on each of these. The paired diagnostic now shows a confidence increase in all four item groups, including errors newly introduced by RL. Unlike the reward-side penalties whose run-to-run instability you highlighted, on HERBench RMA removes at least 58% of the tax in each of the four runs. The consistency across settings now also covers the emitted readout, TVBench and chain-of-thought answers. We answer your questions below.

W1 and Q1. What is specific to video. We agree that RL-induced overconfidence is known in text, and we will cite the GPT-4 Technical Report alongside Bereket & Leskovec (2025). We do not claim that logit-scale sharpening is unique to video models. Our contribution is to measure it precisely in video reasoning, identify its mechanism, and show that where it lands depends on the video reward.

  1. Measurement. The paired estimator separates confidence change from correctness change, which aggregate ECE cannot, and shows that RL keeps the same wrong option in 88% of shared failures.
  2. Mechanism. Entropy reduction alone cannot tell a uniform rescaling of the logits from a change in which option the model prefers. Our decomposition can, and it identifies growth of the logit scale as the primary driver (Appendix B.1). This is supported by temperature scaling, which removes about 88% of the tax, and by a logit-norm cap, which removes 91 and 97%. Growing logit norm is a known cause of overconfidence in supervised training (LogitNorm, Wei et al., 2022), and we show it is the main driver under RLVR.
  3. Video reward design. Under T-GRPO, which rewards temporal order, the tax on action-counting items is 2.5 times the model's overall tax. Under Open-o3, which rewards grounded spatial evidence, the tax on false-object-memory items is 2.3 times the model's overall tax. The distribution of the tax across error types thus depends on the video-specific reward signal, an aspect not captured by prior text-only analyses.

W2, W4 and Q2. Post-hoc rescaling, temperature scaling and distillation. You are right that reference-matched rescaling is temperature scaling with the temperature taken from the SFT reference instead of labels. We use it as a label-free diagnostic of the scale mechanism, not as our method. Table 7 of the submission already compares global, per-type, vector and matrix scaling at 300 steps (vector and matrix scaling are the multiclass extensions of Platt scaling in Guo et al., 2017). We now add tuned held-out TS at 1200 steps (general response, item 3), which will also appear in the revision.

This comparison supports our account rather than replacing RMA. TS removes about 88% of the tax, as a scale mechanism predicts, and reaches the lowest aggregate calibration error. Yet our paired estimator still detects an item-dependent residual of +0.39 pp, which a single global temperature cannot correct and which RMA largely removes. With chain-of-thought, post-hoc correction leaves +3.06 pp, and RMA brings it to +0.44 to +1.25 pp. A post-hoc temperature therefore does not replace RMA, especially where the model reasons before answering.

RMA also differs from logit distillation in two ways. It constrains one scalar, the top-1 minus top-2 margin, and leaves the rest of the distribution free. It acts only on items the reference answers correctly. The full objective also has a small auxiliary CE term (weight 0.10), and removing it leaves the reduction within the range of the full runs on the spaced, unspaced and TVBench evaluations (see our reply to Reviewer gQj2). Our GRPO objective already includes a KL penalty to the reference (coefficient 0.04), yet the tax still appears. The KL penalty does not explicitly preserve the option-level margin, whereas RMA anchors it directly.

W3 and Q3. Free-form generation. Our CoT experiment measures confidence after the model generates its own reasoning, though the final answer is still multiple-choice. For free-form tasks with a verifiable reward, RMA could be extended by grouping sampled answers by meaning, taking the margin as the gap between the shares of the two most frequent groups, and judging reference correctness with the same verifier used for RL. We have not evaluated this extension, and we will state the multiple-choice scope and this direction in the limitations.

Thank you again for the constructive review.

Thanks to the authors for the honest reply and the new baseline numbers.

I appreciate your clarification that logit-scale sharpening is not unique to video models, and that reference-matched rescaling is used mainly as a diagnostic tool. Your new results also show that standard post-hoc Temperature Scaling (removing 88% of the tax with the lowest overall ECE) and LogitNorm (removing 91%-97%) already solve most of this issue in the direct setting, and RMA is still limited to multiple-choice tasks, though I see that RMA leaves a smaller residual in the CoT setting (+0.44 to +1.25 pp vs +3.06 pp).

Overall, while the algorithmic novelty of RMA is modest, the paired diagnostic analysis and seed-controlled RLVR ablations are solid and useful, so I will keep my score at 3 (Weak acceptance).

Dear Reviewer znyj,

Thank you for reading our responses and for confirming your score. We appreciate your view that the paired diagnostic and the seed-controlled ablations are solid and useful.

Thank you again for your constructive review.

The Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response to all reviewers

Dear Reviewers,

Thank you for your careful reviews and constructive feedback. Your questions led to new analyses that make our central finding stronger and its mechanism clearer. We summarize the shared results here and answer each of your points in the individual responses. A revised manuscript with these results will follow shortly.

In short, RL post-training raises confidence in answers that stay wrong, and in 88% of these cases the model keeps the same wrong option. The effect holds on both readouts, on TVBench, under chain-of-thought (CoT) sampling, and in every group of items, including errors that RL newly introduces. Its main cause is a growth of the logit scale, confirmed in three independent ways, and it appears even with the KL anchor to the reference (coefficient 0.04). A single temperature removes most of the tax, which confirms the scale account. The item-dependent remainder is invisible to ECE, Brier, and NLL, and RMA removes most of it. Under CoT, post-hoc correction leaves more behind, and RMA again reduces it (item 4).

1. Four RMA runs (HERBench, 1200 steps). Here untreated RL means the same RL recipe without RMA, and pp means percentage points. Every run removes at least 58% of the untreated tax of +3.26 ± 0.14 pp (95, 58, 58 and 79%, mean 73%). The RLCR-style objective removes 4%. Without CoT, the accuracy gain of RL over SFT is itself small and not significant (+0.85 pp, 95% CI −0.20 to +1.90). All four RMA runs are statistically indistinguishable from SFT, at 0.2 to 1.4 pp below untreated RL. We will replace the single-run number in the paper with all four runs.

2. Emitted readout and deployment. Our main readout uses space-prefixed option letters. On the unspaced letters the model actually emits, every run removes at least 61% of the tax (mean 75%), and ECE, Brier and NLL all improve. At every fixed threshold from 0.6 to 0.9, on both readouts, all four RMA runs accept fewer items than untreated RL with lower error among the accepted items (32 of 32 comparisons), while AURC (area under the risk-coverage curve) is unchanged (0.512 to 0.519).

3. Tuned temperature scaling (TS). We fit TS on held-out data with 2-fold cross-fitting. Lower ECE is better, and a paired tax closer to 0 is better, since 0 means the same confidence on shared failures as the SFT reference.

\text{Model (HERBench)} & \text{ECE raw} & \text{ECE after TS} & \text{Paired tax raw (pp)} & \text{Paired tax, TS on both (pp)} \\\\ \hline \text{Untreated RL} & \text{0.166} & \text{0.023} & \text{+3.26 } \pm \text{ 0.14} & \text{+0.39 } \pm \text{ 0.06} \\\\ \text{RLCR} & \text{0.165} & \text{0.028} & \text{+3.12 } \pm \text{ 0.19} & \text{+0.37 } \pm \text{ 0.08} \\\\ \text{RMA (4 runs)} & \text{0.138 to 0.159} & \text{0.026 to 0.035} & \text{+0.16 to +1.38} & \text{-0.12 to +0.20} \end{array}$$ This experiment is a direct test of our scale account. If the tax were mainly a change in which option the model prefers, no single temperature could remove it. TS removes about 88% of it, as the account predicts. The same experiment also shows what aggregate metrics miss. After TS, every checkpoint reaches nearly the same ECE, Brier, and NLL, yet our paired estimator still detects a residual tax of +0.39 pp in untreated RL. This residual is the item-dependent component that no global temperature can reach, and it is largely what RMA removes, to within noise of zero in three of four runs and by half in the fourth. RMA is therefore the training-time complement to post-hoc calibration rather than a substitute for it, and RMA with TS brings the tax within 0.2 pp of zero in all four runs. **4. Chain-of-thought.** With k = 5 sampled reasoning chains on 2,000 items, we take the share of samples that agree with the majority answer as the confidence. On this measure the untreated tax is +11.51 ± 0.75 pp. Here, post-hoc correction leaves much more behind. Label-based recalibration on both models still leaves +3.06 ± 0.26 pp. RMA brings this residual to +0.44 to +1.25 pp, removing 59 to 86% of what post-hoc correction leaves. This comes with an accuracy cost under CoT (accuracy 0.299 for SFT, 0.325 for untreated RL and 0.296 to 0.307 for RMA), which we will report in the limitations. **5. Changes to the paper.** We will add all four runs, the unspaced, TS, CoT and threshold tables, a revised Section 2.5, and the accuracy trade-off in the limitations. Together, these results give a more complete test of the calibration tax across readouts, benchmarks and reasoning modes, a clearer account of its mechanism, and stronger evidence for what RMA adds beyond post-hoc calibration. We hope these responses and the upcoming revision address your main concerns, and we would be glad to discuss any remaining questions.
AuthorsOpenReview ↗

Revised manuscript uploaded and summary of changes

Dear Reviewers,

We appreciate your time and consideration. We have uploaded a revised manuscript in which all changes are marked in blue. Below is where each of your questions is addressed.

  • Four RMA runs and a second untreated run (gQj2 Q3, all). Abstract, Section 4.1, Table 2 and Appendix F. Every RMA run stays below both untreated runs on all three evaluations.
  • Emitted readout (gQj2 W1/Q1, otsy Q1). Appendix G.
  • Tuned temperature scaling and the Brier objective (gQj2 Q2, znyj W4/Q2). Section 4.3 and Table 8.
  • Chain-of-thought (gQj2 W2, otsy W1). Appendix H.
  • Auxiliary term (gQj2 Q4). Appendix A.1 with Algorithm 1, and the ablation in Appendix J.
  • Section 2.5 and null baselines (gQj2 W3, otsy W2). Section 2.5.
  • Changed predictions and risk-coverage curves (otsy Q2). Appendix I.
  • Related work and scope (znyj Q1/Q3). Section 5 and the limitations.

For the CoT analysis, we recomputed the recalibrated residuals with a fully specified procedure (Appendix H), which gives +3.11 pp for untreated RL and −0.29 to +0.95 pp for RMA, slightly different from our earlier reply and with the same conclusion.

Before the discussion period closes on November 18, we would be grateful for any further feedback or remaining concerns. Thank you again for your thoughtful reviews.

Best regards, The Authors

AuthorsOpenReview ↗

Brief reminder before the discussion closes

Dear Reviewers,

As the discussion period closes today, we would like to briefly note that the revised manuscript (changes in blue) and our responses address each of your questions. This includes four RMA runs, a second untreated run, the emitted readout, tuned temperature scaling, chain-of-thought, and the auxiliary-term ablation.

We hope the revision and our responses have addressed your concerns. If any concern remains, we would be glad to clarify it before the deadline, and we would welcome any final thoughts.

Thank you again for your time.

The Authors

AuthorsOpenReview ↗

Summary for the Area Chair

Dear Area Chair,

We thank you for handling our submission. Reviewer znyj replied and kept their weak-accept score, noting that the paired diagnostic and the seed-controlled RLVR ablations are solid and useful. We have not received replies from Reviewers gQj2 and otsy, so we briefly summarize how the revision addresses each concern. All changes are marked in blue in the revised PDF.

Replication and seed variance (gQj2 Q3). We now report four independent 1200-step RMA runs and a second untreated run. RMA removes 73% of the calibration tax on average and at least 58% in every run (Table 2, Appendix F). Every RMA run has a lower tax than both untreated runs on all three evaluations.

Readout validity (gQj2 W1/Q1, otsy Q1). On the unspaced letters the model actually emits, RMA removes 61 to 92% of the tax and improves ECE, Brier and NLL (Appendix G). It also transfers to the held-out TVBench and to confidence after chain-of-thought.

Comparison with temperature scaling (gQj2 Q2, znyj W4/Q2). Tuned temperature scaling with 2-fold cross-fitting removes most of the tax, which confirms our scale account. It leaves an item-dependent residual (+0.39 pp) that ECE, Brier and NLL cannot detect, and RMA removes it to −0.12 to +0.20 pp (Section 4.3, Table 8). After chain-of-thought, post-hoc recalibration leaves +3.11 pp, and RMA reduces this to −0.29 to +0.95 pp (Appendix H). The logit-norm cap mentioned by Reviewer znyj uses the reference model's logit norm, so it is a diagnostic of the scale mechanism rather than a deployable correction. RMA is therefore a training-time complement to post-hoc calibration.

Auxiliary term (gQj2 Q4). The exact objective is in Appendix A.1 with Algorithm 1. Removing the auxiliary term keeps the reduction within the range of the full runs (Appendix J), so the margin anchor accounts for most of the effect.

Section 2.5 and null baselines (gQj2 W3, otsy W2). The risk-coverage stability is now stated as a verified empirical result (AURC 0.512 to 0.519), and the scale decomposition exceeds all 200 permutation nulls (p < 0.005).

Changed predictions (otsy Q2). Appendix I adds risk-coverage curves over all items and separate analyses of corrected errors and newly introduced errors. RL raises confidence in every group, and RMA attenuates it in every group.

Novelty and scope (znyj W1/Q1/Q3). Our contributions are a paired estimator that separates confidence change from correctness change, a mechanism test showing that where the tax lands depends on the video-specific reward, and a training-time method that removes the component post-hoc calibration cannot reach. The multiple-choice scope and a path to free-form tasks are stated in Section 5 and the limitations.

Accuracy. Without chain-of-thought, RL's accuracy gain over SFT is itself small and not significant (+0.85 pp, 95% CI −0.17 to +1.87). RMA trades 0.2 to 1.4 pp of this accuracy for its calibration gain, which we report in the limitations.

We hope this summary helps your assessment. Thank you for your time.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.