- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether hidden-state error probes remain useful after VLM post-training. It separates ranking stability from probability calibration and proposes unlabeled score re-centering under a common-shift assumption, evaluated across SFT, GRPO, and on-policy distillation.
Critical strengths
Useful probe-maintenance problem.
Detailed task-level results and failure cases.
Critical weaknesses
Thanks for submitting this paper. The distinction between maintaining an error ranking and maintaining calibrated probabilities is useful, and I appreciate the explicit discussion of failure cases. However, I still have several major concerns.
First, the theoretical contribution provides limited explanation of the actual transfer setting. Proposition 1 establishes invariance under increasing transformations, Proposition 2 applies a local smoothness bound, and Proposition 3 constructs different conditional label distributions with the same score marginal. These results clarify the setup, but the ranking bound holds for fixed error labels and prefixes, whereas post-training changes both. The paper reports that 18% of inputs change error status, and SLAKE-zh provides a substantial failure: frozen AUROC drops from 0.932 to approximately 0.46. Can you provide a quantitative analysis that incorporates changing error labels and predicts when transfer succeeds, rather than bounding only one component of the problem?
Second, the calibration guarantee assumes the crucial property that needs to be established in practice. Proposition 4 requires both an exact score translation and preservation of the conditional error relation after translation. Matching score means or preserving the overall error rate cannot establish the latter, as the paper correctly acknowledges. This matters empirically: short-answer re-centering still leaves ECE around 0.194–0.201, and Table 16 shows SLAKE-zh ECE worsening from 0.488 to 0.627 after SFT. How would you decide whether to apply re-centering, labeled recalibration, or refitting before observing the updated model’s correctness labels? The proposed maintenance workflow needs an evaluated decision procedure and its annotation cost.
Third, the update diversity is narrower than the headline experimental scale suggests. All updates use rank-16 LoRA, and Appendix B.1 reports that GRPO and OPD increments are only 5.6–8.8% of the preceding SFT update. The strongest unlabeled calibration results also come from just two Qwen3-VL models on ScienceQA and GSM8K. These experiments support reuse under the tested adapter updates, but provide limited evidence about a probe’s lifetime across continued model evolution. Would the same common-shift behavior survive stronger updates, full-parameter fine-tuning, or several sequential updates? At least one such stress test is needed to establish the scope of the central finding.
Questions for authors
How can you determine whether the conditional score–error relation is preserved after an update, and choose between re-centering, recalibration, and refitting? What labeled audit budget does this decision require?
Does the common-shift behavior persist under full-parameter fine-tuning, stronger updates, or repeated sequential updates beyond the small GRPO/OPD increments evaluated here?
Response to Reviewer nptR
Thank you for the constructive review. "Part" refers to the general response. SA is short answers and TR is reasoning traces.
W1 (a quantitative account of changing labels and when transfer succeeds). T1 extends Corollary 1 to changing labels. For the error probe, ranking loss comes mostly from label turnover and less from score drift (Part 4). Across updates, the ranking weakens as the update raises the error rate (Part 3). A label-free rank-agreement check flags updates that need a refit, including updates trained after its threshold was fixed (Part 5).
W2 (the guarantee assumes the property it needs). We now test this property in each cell. The conditional assumption of Proposition 4 is rejected in 362 of the 696 SA cells. T2 replaces it with an identity. The residual after re-centering is mainly the error-rate change (slope 1.03, r = 0.93). On SLAKE-zh, the error rate changes by 0.60 and the residual is 0.60. Labels are needed to account for this change (Parts 4–5).
W3/Q2 (stronger, full-parameter, or sequential updates). Parts 1–3 add full-parameter SFT, GRPO, and OPD, training on all prompts, a 5-stage chain, the sweep repeated on Qwen3.5-9B, released 2B–14B pipelines, one GRPO run tracked across training, and a learning-rate sweep that shows where transfer stops. The reasoning-trace protocol now also covers Qwen3.5-4B, Qwen3.5-9B, and the released pipelines.
Q1 (decision procedure and labeling cost). Part 5 gives the rule, the labels it uses, and the ECE at each budget. A label-free rank-agreement check determines whether to refit. Labels set the scale. With 32 labels, we close 55% of the ECE gap to a full refit. With 256 labels, intercept recalibration closes 71% of the gap and the decision rule closes 86%.