acceptodds
Under review as a conference paper at ICLR 2027

Rankings Survive, Scales Drift: Hidden-State Error Estimation Across Post-Training

Abstract

Hidden-state probes offer an efficient way to estimate whether a vision-language model (VLM) will answer correctly. Yet post-training changes both the representations a probe reads and the errors it must predict. We ask whether a probe remains useful as the model evolves, separating error ranking from probability calibration. Our central finding is that a frozen probe can retain useful rankings even when its probability estimates become unreliable. We characterize ranking stability in terms of score perturbations and establish that updated error probabilities are not identifiable from unlabeled scores alone. Under a common score shift that preserves the conditional error relation, we derive a finite-sample calibration guarantee for re-centering a calibrated base probe using only unlabeled inputs. Changes in this relation call for labeled recalibration; deteriorating rankings motivate refitting. We further find that sampled disagreement can supervise transferable rankings without correctness labels, and that hidden-state probes remain informative when reasoning precedes the answer and first-token uncertainty is less predictive. Experiments across VLM post-training methods and answer formats support these findings. These results separate the cost of recalibrating a probe from that of relearning its error signal.

Primary area:
reinforcement learning
Keywords:
Post-Training, Reinforcement Learning, Confidence Calibration, Internal Representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nptROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether hidden-state error probes remain useful after VLM post-training. It separates ranking stability from probability calibration and proposes unlabeled score re-centering under a common-shift assumption, evaluated across SFT, GRPO, and on-policy distillation.

Critical strengths

Useful probe-maintenance problem.

Detailed task-level results and failure cases.

Critical weaknesses

Thanks for submitting this paper. The distinction between maintaining an error ranking and maintaining calibrated probabilities is useful, and I appreciate the explicit discussion of failure cases. However, I still have several major concerns.

First, the theoretical contribution provides limited explanation of the actual transfer setting. Proposition 1 establishes invariance under increasing transformations, Proposition 2 applies a local smoothness bound, and Proposition 3 constructs different conditional label distributions with the same score marginal. These results clarify the setup, but the ranking bound holds for fixed error labels and prefixes, whereas post-training changes both. The paper reports that 18% of inputs change error status, and SLAKE-zh provides a substantial failure: frozen AUROC drops from 0.932 to approximately 0.46. Can you provide a quantitative analysis that incorporates changing error labels and predicts when transfer succeeds, rather than bounding only one component of the problem?

Second, the calibration guarantee assumes the crucial property that needs to be established in practice. Proposition 4 requires both an exact score translation and preservation of the conditional error relation after translation. Matching score means or preserving the overall error rate cannot establish the latter, as the paper correctly acknowledges. This matters empirically: short-answer re-centering still leaves ECE around 0.194–0.201, and Table 16 shows SLAKE-zh ECE worsening from 0.488 to 0.627 after SFT. How would you decide whether to apply re-centering, labeled recalibration, or refitting before observing the updated model’s correctness labels? The proposed maintenance workflow needs an evaluated decision procedure and its annotation cost.

Third, the update diversity is narrower than the headline experimental scale suggests. All updates use rank-16 LoRA, and Appendix B.1 reports that GRPO and OPD increments are only 5.6–8.8% of the preceding SFT update. The strongest unlabeled calibration results also come from just two Qwen3-VL models on ScienceQA and GSM8K. These experiments support reuse under the tested adapter updates, but provide limited evidence about a probe’s lifetime across continued model evolution. Would the same common-shift behavior survive stronger updates, full-parameter fine-tuning, or several sequential updates? At least one such stress test is needed to establish the scope of the central finding.

Questions for authors

How can you determine whether the conditional score–error relation is preserved after an update, and choose between re-centering, recalibration, and refitting? What labeled audit budget does this decision require?

Does the common-shift behavior persist under full-parameter fine-tuning, stronger updates, or repeated sequential updates beyond the small GRPO/OPD increments evaluated here?

1 comment
AuthorsOpenReview ↗

Response to Reviewer nptR

Thank you for the constructive review. "Part" refers to the general response. SA is short answers and TR is reasoning traces.

W1 (a quantitative account of changing labels and when transfer succeeds). T1 extends Corollary 1 to changing labels. For the error probe, ranking loss comes mostly from label turnover and less from score drift (Part 4). Across updates, the ranking weakens as the update raises the error rate (Part 3). A label-free rank-agreement check flags updates that need a refit, including updates trained after its threshold was fixed (Part 5).

W2 (the guarantee assumes the property it needs). We now test this property in each cell. The conditional assumption of Proposition 4 is rejected in 362 of the 696 SA cells. T2 replaces it with an identity. The residual after re-centering is mainly the error-rate change (slope 1.03, r = 0.93). On SLAKE-zh, the error rate changes by 0.60 and the residual is 0.60. Labels are needed to account for this change (Parts 4–5).

W3/Q2 (stronger, full-parameter, or sequential updates). Parts 1–3 add full-parameter SFT, GRPO, and OPD, training on all prompts, a 5-stage chain, the sweep repeated on Qwen3.5-9B, released 2B–14B pipelines, one GRPO run tracked across training, and a learning-rate sweep that shows where transfer stops. The reasoning-trace protocol now also covers Qwen3.5-4B, Qwen3.5-9B, and the released pipelines.

Q1 (decision procedure and labeling cost). Part 5 gives the rule, the labels it uses, and the ECE at each budget. A label-free rank-agreement check determines whether to refit. Labels set the scale. With 32 labels, we close 55% of the ECE gap to a full refit. With 256 labels, intercept recalibration closes 71% of the gap and the decision rule closes 86%.

Reviewer_VnSMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether hidden-states based error probe can still be useful after model post-training. The authors sepeprate error risk ranking from probability calibration and propose estimating mean value recentering through unlabeled inputs. The paper also investigates the signal provided by sampled answer disagreement and the usefulness of hidden-states across formats. Experiments cover five models and eight short answer tasks. The paper proposes to retain the probe, repair its probability scale, and use labeled checks to determine whether recalibration is needed for models after post-training.

Critical strengths

The fixed-text control experiments help isolate the parameter update effects from changes in the generated answers. The disaggregated results in Section 4.2 and the explicitly reported failures in Section 4.3 make the limitations of transfer inspectable. These are some interesting results for studying and understanding the error probe after post-training.

Critical weaknesses
  1. The experimental scope is too limited to establish broadly applicable empirical principles. All experiments use low-rank without full tuning, grpo and opd only on 512 prompts. This setting is significantly different from many practical post-training setting scales. Thus, the observations in this paper cannot be reliably extended to practical scenarios.

  2. The theoretical analysis relies on strong conditions that may not hold in practical situations. The rank analysis fixes error labels but post-training changes which answers are wrong. Proposition 4 requires a common score shift and preserves the score and error conditional relationship, but the overall error rate stability cannot guarantee these conditions. The results on SLAKE-zh illustrate the consequences of updates outside the assumed conditions. The suggested small-label audit is also not validated.

  3. The technical depth and incremental contribution are limited. The analysis mainly formalizes ranking invariance, the non-identifiability of calibration, and the recovery of probability calibration under the assumed conditions. The proposed method estimates a mean shift. It provides limited insights into which structure holds the condition and when it fails. The introduction section also doesn't clearly present the advances beyond existing probe and calibration techniques. These restricted observations are not sufficiently developed into a deeper explanation or a more capable probe maintenance method.

  4. Straightforward unlabeled baselines are missing. Under a common score shift, differences in means and corresponding quantiles estimate the same population shift. Thus, quantile alignment and resetting a quantile threshold are direct baseline methods. The paper does not establish why mean recentering should be preferred.

Questions for authors
  1. What evidence supports extending the observations beyond the current small-scale, low-rank updates? Do the findings remain consistent with larger training datasets or full-parameter fine-tuning?

  2. How are the common shift and conditional score error invariance assumptions checked in the experiments? How does the proposed small label audit detect violations, and what evidence supports its reliability?

  3. What is the main technical advance beyond known ranking and calibration properties and a mean shift correction? Which results explain when the required structure should hold or fail?

  4. Under the same unlabeled sample budget, how does mean recentering compare with quantile alignment in calibration accuracy and robustness? What evidence supports preferring the mean-based method?

2 comments
AuthorsOpenReview ↗

Response to Reviewer VnSM (1/2)

Thank you for the review and for suggesting the quantile baselines, which we added in Part 6. "Part" refers to the general response. SA is short answers and TR is reasoning traces.

W1/Q1 (rank-16 LoRA, 512 prompts). We tested beyond this setting with LoRA ranks 64 and 256, 3 epochs, full-parameter SFT/GRPO/OPD, GRPO/OPD on all 5,629 prompts, a 5-stage chain, the sweep repeated on Qwen3.5-9B, and seven released 2B–14B pipelines. Across these, the frozen probe's ranking stays close to that under the submission recipe. On SA with Qwen3-VL-2B, AUROC is 0.67–0.70 against 0.65–0.67 (Part 1). It weakens as an update raises the error rate. On Qwen3.5-4B, it falls from 0.79 under LoRA to 0.74–0.75 under full-parameter training and 0.51–0.55 at 20× the learning rate (Part 3). We will state this in Limitations.

W2 (the rank analysis fixes the error labels). T1 extends Corollary 1 to changing labels. For the error probe, ranking loss comes mostly from label turnover (-0.083), less from score drift (-0.036). Turnover is bounded by the answer-change rate, which needs no labels (Part 4).

W2/Q2 (conditions of Proposition 4, and how they are checked). Tests reject the conditional assumption in 362 of the 696 SA cells. T2 replaces it with an identity. The residual after re-centering is mainly the error-rate change (slope 1.03, r = 0.93). On SLAKE-zh, the error rate changes by 0.60 and the residual is 0.60 (Part 4).

W2/Q2 (validation of small-label calibration). We evaluated the decision rule on held-out and prospective updates, with 16–256 labels (Part 5). With 32 labels, we close 55% of the calibration gap to a full refit. A small labeled sample does not reliably tell which action is needed, so the refit decision uses a label-free rank-agreement check. Its detection AUROC is 0.75 on SA and 0.87 on TR.

AuthorsOpenReview ↗

Response to Reviewer VnSM (2/2)

W3/Q3 (technical depth and when the structure holds or fails). Our contribution goes beyond formalizing known properties. We find that post-training affects a frozen probe's ranking and scale differently. The ranking stays close to that under the submission recipe while the scale shifts. On released checkpoints, one common shift explains 79% of the score change (Parts 1–2). T1 and T2 make the conditions measurable. For the error probe, ranking loss comes mostly from label turnover (-0.083), less from score drift (-0.036). After re-centering, the calibration residual is mainly the change in error rate (slope 1.03, r = 0.93). These results also show where transfer fails. The ranking weakens as an update raises the error rate (Part 3), and re-centering falls short by about the change in error rate. We also provide a decision rule and its labeling cost, tested on held-out updates and on updates trained after its thresholds were fixed (Part 5).

W4/Q4 (quantile alignment and quantile thresholds). We added both under the same unlabeled budget (Part 6). The comparison does not support preferring the mean shift. Median re-centering gives the lowest ECE and Brier score among the tested methods, so we adopt it. Resetting the threshold by quantile (τ = 0.3) leaves a 0.103 gap between realized and claimed risk, compared to 0.114 for the re-centered probability and 0.088 after labeled recalibration.

Presentation. The revised paper will add a contributions paragraph and a notation table, use a single name for each probe, and state the decision rule in the main text.

Reviewer_JdESOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether hidden-state error probes remain useful after vision-language models undergo post-training. It distinguishes error ranking from probability calibration, finding that frozen probes often retain useful ranking ability while their probability estimates become severely miscalibrated. The authors propose re-centering the probe scores using unlabeled inputs and establish a calibration guarantee under a common score shift and an invariant conditional error relation. Experiments cover several models and tasks involving both short answers and reasoning traces. Re-centering substantially improves calibration in several post-training settings, while sampled disagreement provides supervision for transferable error rankings. However, ranking transfer fails on some tasks, and unlabeled repair cannot generally address changes in the conditional error relation.

Critical strengths

A key strength is the paper’s clear separation between ranking and calibration. The paper addresses the maintenance cost of error monitors after model updates, an important issue that static probe evaluations can overlook. In the reasoning experiments, frozen probes substantially overestimate the error rate while retaining useful ranking ability. This demonstrates that AUROC alone is insufficient to establish continued reliability and motivates maintaining the risk signal and its probability mapping separately.

Critical weaknesses

All updates use low-rank adaptation, and the GRPO and OPD increments beyond SFT are relatively small compared with the initial SFT update. The experiments therefore do not establish whether ranking retention and approximately common score shifts generalize to full-parameter fine-tuning, larger updates, or repeated sequential updates.

The proposed workflow recommends re-centering, auditing with a small labeled sample, and recalibrating or refitting when necessary. Although budget experiments and a practical maintenance recipe are provided, the evaluation does not show whether a small labeled audit can reliably distinguish cases that require recalibration from those that require refitting. This matters because frozen-probe ranking falls below chance on one evaluated task, and re-centering cannot repair the ranking. Thus, the results demonstrate the utility of individual maintenance operations but do not establish that the proposed audit can reliably identify the appropriate operation.

Questions for authors
  1. The experiments use low-rank adaptation, with relatively small GRPO and OPD increments beyond SFT. Do ranking retention and the calibration benefits of re-centering persist under larger or repeated sequential updates? Can the authors provide results across update strengths to clarify the method’s applicability?

  2. Can a small labeled audit reliably distinguish when recalibration is sufficient from when the probe must be refitted? In particular, can the authors provide an appropriate decision rule and the audit budget needed to make this distinction?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JdES

Thank you for the review and for recognizing the distinction between ranking and calibration. "Part" refers to the general response. SA is short answers and TR is reasoning traces.

W1/Q1 (full-parameter, larger and repeated sequential updates). Parts 1–2 add full-parameter SFT, GRPO, and OPD, training on all prompts, a 5-stage chain, the sweep repeated on Qwen3.5-9B, seven released pipelines, and one GRPO run followed over an 88× range of steps. Across these, the frozen probe's ranking stays close to that under the submission recipe while the scale shifts. This supports reusing the ranking and re-estimating the scale.

Q1 (results across update strengths). The ranking weakens as an update raises the error rate (Part 3). On Qwen3.5-4B SA, the frozen AUROC is 0.79 under LoRA, 0.74–0.75 under full-parameter training at the default learning rate, and 0.51–0.55 at 20× that rate, where the error rate has risen by 0.30–0.36. A refit still reaches 0.71–0.74 there.

W2/Q2 (can a small labeled sample tell recalibration from refitting, and at what budget?). It cannot tell reliably, so we use separate criteria for the two decisions (Part 5). Labels set the scale. With 32 labels, we close 55% of the calibration gap to a full refit. A rank-agreement check on unlabeled inputs determines whether to refit. Updates that pass it lose on average 0.07 (SA) and 0.05 (TR) AUROC against a refit. Flagged updates lose 0.13 and 0.15. The pattern holds on updates trained after its threshold was fixed. The rule and the ECE at each budget are in Part 5.

Thank you for the detailed response and additional experiments. My main concerns have been addressed, and I will maintain my positive score.

AuthorsOpenReview ↗

Thank you, Reviewer JdES

Dear Reviewer JdES,

Thank you very much for the prompt follow-up and for maintaining your positive assessment. We are glad our additional experiments resolved your main concerns, and we truly appreciate your constructive suggestions in helping us improve the paper.

Best regards,
The Authors

Comments on the paper

9 comments
AuthorsOpenReview ↗

General response to reviewers and chairs

We thank the reviewers for their careful feedback. Their questions focus on whether ranking retention with a common score shift holds under realistic post-training, when label-free re-centering is sufficient, and how many labels are needed when it is not.

  1. Ranking. Under stronger, full-parameter and sequential updates, and across released post-training pipelines, the frozen probe's ranking stays close to that under the submission recipe. It weakens as an update raises the error rate (Parts 1–3).
  2. Scale. Re-centering removes the common score shift without labels. T2 accounts for the residual, which is mainly the change in error rate in our experiments. This needs labels to estimate. With 32 labels, we close 55% of the calibration gap to a full refit (Parts 4–5).
  3. When to refit. A label-free rank-agreement check flags the updates that need a refit, including updates trained after its threshold was fixed (Part 5).
PartTopicAddresses
1Scale I: stronger, full-parameter, sequential updatesVnSM W1/Q1 · JdES W1/Q1 · nptR W3/Q2
2Scale II: sweep on Qwen3.5-9B, released 2B–14B pipelines, one run over timeVnSM W1/Q1 · JdES W1/Q1 · nptR W3/Q2
3LoRA vs. full-parameter updates and transfer limitsVnSM W1 · JdES Q1 · nptR W1
4Theory: T1 and T2VnSM W2/W3 · nptR W1/W2
5What labels are for, and when to refitVnSM W2 · JdES W2/Q2 · nptR Q1
6Label-free baselines, incl. quantile alignmentVnSM W4/Q4

Notation. SA is the short-answer protocol and TR is reasoning traces at the answer end (both Section 4.1). A cell a→b gives the AUROC of the frozen probe (a) and of a probe refit on the updated model (b), unless noted. The revised paper will add a contributions paragraph, a notation table, unified probe names, the decision rule in the main text, and the full tables in the appendix.

AuthorsOpenReview ↗

General response (1/6). Scale I: stronger, full-parameter and sequential updates

We vary the submission recipe one factor at a time on Qwen3-VL-2B (2B) and Qwen3.5-4B (4B). The appendix will include all 19 updates.

Update2B SA4B SA2B TR4B TR
Submission recipe0.65–0.67→0.800.79→0.840.74–0.79→0.80–0.820.65–0.68→0.78
LoRA r2560.70→0.830.79→0.840.67→0.760.59→0.83
LoRA, 3 epochs0.68→0.810.79→0.840.77→0.810.64→0.73
Full-param. SFT0.68→0.830.71→0.830.73→0.79—
Full-param. SFT, 3 epochs0.69→0.800.70→0.810.44→0.83—
GRPO, all 5,629 prompts0.68→0.800.78→0.840.76→0.800.66→0.80
OPD, all 5,629 prompts0.68→0.810.78→0.850.77→0.810.71→0.80
Full-param. GRPO0.69→0.820.75→0.830.75→0.810.65→0.82
Full-param. OPD0.67→0.810.74→0.850.75→0.800.60→0.83
Full-param. GRPO, all prompts0.67→0.820.74→0.850.77→0.810.70→0.85
Full-param. OPD, all prompts0.69→0.820.73→0.840.76→0.810.73→0.86
5-stage chain SFT→GRPO→OPD→GRPO→SFT0.68→0.800.80→0.840.76→0.790.68→0.81

Ranking. On SA, frozen AUROC is close to that under the submission recipe: 0.67–0.70 versus 0.65–0.67 on 2B, and 0.78–0.80 versus 0.79 for LoRA variants on 4B. Full-parameter training on 4B gives 0.70–0.75 . On TR, it is 0.67–0.77 on 2B and 0.59–0.73 on 4B, versus 0.74–0.79 and 0.65–0.68. Three epochs of full-parameter SFT on 2B is the only setting below chance.

Scale. On TR, re-centering reduces ECE from 0.05–0.77 to 0.03–0.06 in 15 of 20 cells, as in the paper. In the other five , the error rate rises most: 0.09–0.30 above the submission recipe, compared with at most 0.04 elsewhere. On SA, every update lowers the error rate by 0.08–0.15. No unlabeled statistic identifies this change. Re-centering leaves ECE at 0.15–0.24, down from 0.36–0.57. T2 explains this residual through the error-rate change . Part 5 reports how much labels recover.

AuthorsOpenReview ↗

General response (2/6). Scale II: larger backbones, released pipelines, one run over time

We repeat the Part 1 sweep on Qwen3.5-9B, one of the paper's five models:

Qwen3.5-9BSATR
Submission recipe0.76→0.84–0.850.71–0.78→0.83–0.87
LoRA r2560.76→0.860.70→0.88
LoRA, 3 epochs0.74→0.850.76→0.77
LoRA r64, 3 ep., lr 2e-40.75→0.850.73→0.83
GRPO, all prompts0.77→0.840.74→0.80

We also test seven released multi-stage pipelines. The probe is fitted on the first released stage and applied frozen to the last. Entries give frozen→refit AUROC at the last stage.

Released pipeline (first → last stage)SATRΔweights vs. our SFT
InternVL3.5-2B0.79→0.820.79→0.800.19×
InternVL3.5-4B0.80→0.840.77→0.860.24×
InternVL3.5-8B0.83→0.860.77→0.850.27×
InternVL3.5-14B0.82→0.840.82→0.750.30×
Qwen2.5-VL-7B0.83→0.840.80→0.830.29×
Tulu-3-8Btext only0.65→0.790.14×
OLMo-2-7Btext only0.71→0.770.30×

Δweights is the relative change ‖ΔW‖/‖W‖ of the LM attention weights over the whole pipeline, normalized by the change under our LoRA SFT (0.70%). In every pipeline, this change is smaller than under our LoRA SFT. On InternVL3.5-14B TR, the refit is below the frozen probe.

We also track one GRPO run on all prompts throughout training. The table gives frozen AUROC and the TR score shift in s.d. The refit AUROC stays at 0.79–0.84.

GRPO step2B SA2B TR2B shift4B SA4B TR4B shift
320.660.80+3.10.790.62+0.0
2560.680.81+2.40.790.67+0.2
10240.680.78+0.90.790.65+0.5
28160.680.77+0.20.800.67+0.4

The ranking is stable over an 88× range of steps while the score scale changes. On the released checkpoints, fixed-text replay attributes the shift to weight changes. One common shift explains 79% of the score change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.