- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates confidence inflation following reinforcement learning with verifiable rewards in video language models. It defines a paired “calibration tax” as the increase in confidence on examples answered incorrectly by both the SFT and RL checkpoints. Confidence is measured using a restricted distribution over option-letter tokens without chain-of-thought generation. The authors find positive confidence shifts across several benchmarks and model sizes, frequently without changes in the predicted option. A decomposition attributes much of this shift to changes in the magnitude of the option-logit vector. Motivated by this observation, the paper proposes reference-matched post-hoc rescaling and Reference Margin Anchoring (RMA), which penalizes deviations from the frozen reference model’s top-two logit margin during RL. On HERBench, the reported 1200-step RMA run reduces the tax from 3.26 to 0.16 percentage points, with a 0.40-point accuracy cost. Additional experiments examine transfer to TVBench, reward-side interventions, and training variability.
Critical strengths
- The empirical diagnosis is carefully developed and extends beyond a single checkpoint comparison. Examining shared failures—and then identical wrong predictions—helps demonstrate that the observed confidence increase cannot be explained solely by changes in answer correctness. Replication across benchmarks and InternVL model sizes strengthens this observation. The investigation of the HERBench meta-option confound is particularly valuable: the authors identify an initially misleading task-level pattern and qualify their interpretation accordingly.
- RMA is a simple intervention with encouraging evidence of controlling the measured effect. The method uses an existing frozen reference and a straightforward margin penalty. The one-sided versus two-sided comparison provides a useful design ablation. At 1200 steps, RMA improves ECE and multiclass Brier score relative to untreated RL, alongside its reduction of the paired tax. The reported 91% tax reduction on TVBench also suggests that the effect extends beyond HERBench.
Critical weaknesses
-
The connection between the main confidence probe and confidence in actual model outputs remains insufficiently established. The main experiments use space-prefixed option tokens even though the models normally emit unspaced letters. Appendix E reports only 83–85% agreement between the predictions under these readouts, which is a substantial discrepancy. The positive tax under emitted-letter scoring supports the existence of the phenomenon, but the appendix does not establish that RMA’s reported 95% reduction transfers to that readout. This distinction matters because RMA directly optimizes the same probe used for evaluation.
Furthermore, the measurements omit the reasoning process used by these models. The authors appropriately acknowledge this limitation, but confidence saturation after chain-of-thought does not establish that the alternative probe measures uncertainty in the generated reasoning-based answer. The deployment interpretation therefore needs stronger validation.
The explanation in Section 2.5 also conflates two different rankings. Positive logit scaling preserves the ordering of options within an example, but item-dependent scaling can change confidence rankings across examples. Consequently, it does not imply invariance of AUROC or risk–coverage curves. Their observed stability is an empirical result, not a consequence of the proposed decomposition.
Questions for authors
- Readout validity: For the 1200-step untreated and RMA checkpoints, what are the tax, ECE, Brier score, accuracy, and threshold-based error rates using the actual unspaced option tokens? Is there evidence that RMA improves confidence estimates associated with answers generated using the normal reasoning procedure?
- Comparative calibration: At 1200 steps, how does RMA compare with properly tuned temperature scaling and the adapted Brier objective on full-set ECE, Brier score, and NLL? Please specify the Brier coefficient, its selection procedure, and whether calibration parameters are selected independently of the reported evaluation data.
- Replication and accuracy: Do matched-seed repetitions of the complete 1200-step experiment support both the reported tax reduction and the accuracy-preservation claim? Please provide uncertainty estimates for the accuracy differences and clarify how much of the RL accuracy gain RMA retains.
- Implemented objective: Appendix A.1 states that RMA includes an additional term with weight 0.10. What is its exact formula, and does an ablation establish that its contribution is negligible? Activation on approximately 1% of training steps alone does not determine its effect on optimization.
3 comments
(2/2) Official Comment by Authors
(2/2) Full-set Brier and NLL for Q2. You asked how RMA compares with tuned TS and the adapted Brier objective on full-set ECE, Brier and NLL. ECE and the paired tax are in the general response (item 3). The table below adds Brier and NLL on all 2,000 HERBench items at 1200 steps (spaced readout). TS is fit with 2-fold cross-fitting, so each item is scored with a temperature fit on the other half. Lower is better for both metrics.
\text{Model} & \text{Accuracy} & \text{Brier raw} & \text{Brier after TS} & \text{NLL raw} & \text{NLL after TS} \\\\ \hline \text{SFT} & \text{0.346} & \text{0.776} & \text{0.745} & \text{1.602} & \text{1.496} \\\\ \text{Untreated RL} & \text{0.355} & \text{0.789} & \text{0.744} & \text{1.646} & \text{1.493} \\\\ \text{RLCR-style Brier} & \text{0.353} & \text{0.790} & \text{0.744} & \text{1.645} & \text{1.494} \\\\ \mathbf{RMA\\ (4\\ runs)} & \text{0.341–0.353} & \text{0.777–0.785} & \text{0.745–0.747} & \text{1.604–1.629} & \text{1.495–1.500} \end{array}$$ These results show three things. - **Without post-hoc calibration, every RMA run improves Brier and NLL over untreated RL**, and brings both close to the SFT level. The RLCR-style Brier objective leaves both essentially unchanged at 1200 steps (Brier 0.790 vs 0.789 for untreated RL, NLL 1.645 vs 1.646). - **TS largely corrects the common scale in every checkpoint.** Brier (0.744 to 0.747) and NLL (1.493 to 1.500) then barely distinguish them, so these aggregate metrics cannot show what remains. - **The paired estimator still detects residual differences after TS.** It is +0.39 pp for untreated RL and +0.37 pp for RLCR, but −0.12 to +0.20 pp for RMA (general response, item 3). This is the item-dependent component that aggregate metrics average away. The unspaced readout shows the same pattern. Raw Brier is 0.811 for untreated RL and 0.792 to 0.803 for RMA, and after TS all checkpoints fall between 0.744 and 0.746. We will include this table in the revision. **TVBench across four runs.** Your summary cites a 91% tax reduction on TVBench from the single run in the paper. Across the four 1200-step RMA runs, the reductions are 92, 63, 23 and 77% (mean 64%), and we will report all four. **Per-run accuracy uncertainty for Q3.** Paired differences on the same 2,000 HERBench items without CoT (spaced readout), with normal-approximation 95% CIs and unadjusted McNemar p-values. $$\begin{array}{l|c|c|c} \text{Model} & \text{Accuracy} & \Delta \text{ vs SFT (pp)} & \Delta \text{ vs untreated RL (pp)} \\\\ \hline \text{Untreated RL} & \text{0.355} & \text{+0.85 (see Q3)} & \text{} \\\\ \text{RMA run 1} & \text{0.351} & \text{+0.45 [-0.38, +1.28], p = 0.34} & \text{-0.40 [-1.46, +0.66], p = 0.52} \\\\ \text{RMA run 2} & \text{0.353} & \text{+0.65 [-0.11, +1.41], p = 0.12} & \text{-0.20 [-1.20, +0.80], p = 0.77} \\\\ \text{RMA run 3} & \text{0.341} & \text{-0.55 [-1.26, +0.16], p = 0.17} & \text{-1.40 [-2.44, -0.36], p = 0.010} \\\\ \text{RMA run 4} & \text{0.344} & \text{-0.25 [-1.03, +0.53], p = 0.62} & \text{-1.10 [-2.13, -0.07], p = 0.045} \end{array}$$ No run differs significantly from SFT. Runs 3 and 4 show nominally lower accuracy than untreated RL (unadjusted p = 0.010 and 0.045), whereas runs 1 and 2 do not. We report this variability as a limitation. We would be glad to discuss any remaining questions.Additional Official Comment by Authors
Follow-up on Q3. Second untreated run. As promised, we trained a second untreated run with the same recipe for 1200 steps. Its tax is +4.42 pp on the spaced readout, +5.30 pp on the emitted readout, and +1.35 pp on TVBench, against +3.26, +3.93 and +1.05 pp for the first run. Every RMA run (+0.16 to +1.38 pp spaced, +0.30 to +1.55 pp emitted, +0.09 to +0.81 pp on TVBench) stays below both untreated runs on all three evaluations. Measured against the mean of the two untreated runs, the four RMA runs remove 64 to 96% of the spaced tax.
On the main spaced readout, the second untreated run reaches essentially the same accuracy as the first (0.356 against 0.355, McNemar p = 1.0), so the accuracy comparisons in the paper are unchanged. On the emitted readout, it reaches 0.362, 1.2 pp above the first run, which places the four RMA runs 1.3 to 1.9 pp below it on that readout. Both untreated runs are reported in Appendices F and G of the revised manuscript, and this variability is discussed in the limitations.
(1/2) Official Comment by Authors
(1/2) Dear Reviewer gQj2,
Thank you for a precise review. We ran additional analyses in response to your questions and summarize the relevant results below.
W1 and Q1. Readout validity. On the emitted unspaced letters at 1200 steps, the untreated tax is +3.93 pp, and the four RMA runs remove 61, 92, 61 and 88%. Across the four runs, ECE drops from 0.210 (untreated RL) to 0.176–0.190, Brier from 0.811 to 0.792–0.803, and NLL from 1.705 to 1.638–1.673. Accuracy is 0.350 for untreated RL and 0.343 to 0.349 for the four RMA runs, with no significant difference from untreated RL (all p ≥ 0.1). RMA was trained only on the spaced probe, so this is transfer. At τ = 0.8 on the unspaced readout, SFT accepts 219 items (35.2% error), untreated RL 314 (41.7%), and the RMA runs 212 to 251 (35.8 to 37.8%). At every threshold from 0.6 to 0.9 and on both readouts, all four runs accept fewer items with lower error than untreated RL (full table in our response to Reviewer otsy).
W2 and Q1. Reasoning-based answers. We measured confidence as agreement over k = 5 CoT samples. RMA removes 32 to 47% of the raw tax. After label-based recalibration of both models, untreated RL keeps +3.06 ± 0.26 pp while RMA keeps +0.44 to +1.25 pp. RMA therefore also reduces inflation in final answers produced after reasoning, where post-hoc correction leaves a residual. This is confidence in the reasoning-conditioned answer, not uncertainty within the trace.
W3. Section 2.5. You are right. Per-item scaling preserves the order of options within an item but can reorder confidence across items, so the decomposition alone does not guarantee invariant AUROC or risk-coverage. We have now verified this directly. AURC stays between 0.512 and 0.519 for every checkpoint, and the revised Section 2.5 will present it as a verified empirical result.
Q2. Tuned TS and the Brier objective. TS uses 2-fold cross-fitting, so each item is scored with a temperature fit on the other fold, never on its own label. This experiment directly tests our scale account. TS removes about 88% of the tax, as a scale mechanism predicts, and attains the lowest ECE, Brier and NLL, after which all checkpoints are nearly indistinguishable on these aggregate metrics. The paired tax still resolves what they miss (general response, item 3). With TS applied to both SFT and RL, untreated RL keeps +0.39 ± 0.06 pp and RLCR +0.37 ± 0.08 pp, while RMA gives −0.04, +0.20, −0.11 and −0.12 pp. RMA is thus the training-time complement to post-hoc calibration, not a substitute for it. Our RLCR baseline is an RLCR-style Brier term adapted to our readout, not a reproduction. Its coefficient (λ = 0.45) was set once before training to match the magnitude of the RMA penalty on stored logits, and not tuned on any reported metric.
Q3. Replication and accuracy. Every RMA run removes at least 58% of the tax (95, 58, 58 and 79%, mean 73%), and we will report all four in the revision. These are independent RMA repetitions, not matched-seed comparisons against untreated RL, because rollout seeds were not fixed across conditions. They show the reduction holds in every run. At 300 steps, all seven two-sided RMA runs show a lower tax than both untreated runs (permutation p ≈ 1/36). On accuracy, the RL gain over SFT without CoT is itself small and not significant at 1200 steps (+0.85 pp, 95% CI −0.20 to +1.90). No RMA run differs significantly from SFT. Two keep part of the RL gain (0.2 and 0.4 pp below untreated RL), and two are at or slightly below the SFT level (1.1 and 1.4 pp below untreated RL). With CoT, accuracy is 0.299 for SFT, 0.325 for untreated RL and 0.296 to 0.307 for RMA. We will report this cost in the limitations. A second untreated run is in progress, and we will post its result here as a follow-up.
Q4. The additional term in Appendix A.1. With m the top-1 minus top-2 option-logit margin, the full objective is
L = L_GRPO + β1 · 1[ref correct] · |m_θ − m_ref| + β2 · 1[ref correct and policy wrong] · CE(π_θ, y)with β1 = 0.45, β2 = 0.10, and CE computed over the softmax of the option-letter logits. We agree that its activation rate alone does not show that its effect is negligible, so we retrained RMA without it using the same recipe for 1200 steps. This single-term run removes 66% of the tax on the spaced readout, 75% on the unspaced readout and 43% on TVBench, all within the corresponding ranges of the four full-objective runs (58 to 95%, 61 to 92% and 23 to 92%). Under CoT it removes 52%, slightly above the full-objective range of 32 to 47%. Its accuracy is within noise of untreated RL. The margin anchor in Eq. 4 alone therefore accounts for most of the reduction on the spaced, unspaced and TVBench evaluations, and the CoT result is comparable. As this ablation is a single run, we do not rule out a small contribution from the additional term. We will add the formula and this ablation to Appendix A.1.