acceptodds
Withdrawn from ICLR 2027

S2R: Selective-Stability Repair in Frozen Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision–language models can be unstable when an answer should stay the same and insensitive when its visual support is weakened. We call the desired behavior *selective stability*: agreement across answer-preserving text and image changes, coupled with reduced confidence when proposed visual evidence is attenuated. **S2R** turns this criterion into a test-time repair signal. It reads answers at several points along the base reasoning trajectory and fits a temporary, hard-sparse edit of visual-token states in an otherwise frozen model. Hyperparameters and gate thresholds are selected on separate calibration data. On Qwen3-VL-8B-Thinking, S2R raises macro-average accuracy from 72.4% to 75.5%, while S2R-Full reaches 75.7%. The improvement transfers to two other frozen 8B backbones, and ablations show that preserving views, the directional evidence constraint, and localized support contribute complementary gains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, VLM repairing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YpMMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces “selective stability”: a VLM’s answer should stay the same under answer-preserving changes (question paraphrases, perturbing an unimportant image region), but its confidence should drop when the region the model itself relies on is perturbed. S2R turns this into a test-time objective. For each input it builds paraphrases and matched image pairs (the same perturbation applied to a high- and a low-relevance region), reads answer distributions at several points of the base reasoning trace, and optimizes a sparse additive edit on 128 visual-token coordinates of a frozen model. The answer is then regenerated from the edited states. On six benchmarks, Qwen3-VL-8B-Thinking improves from 72.4% to 75.5% (75.7% for S2R-Full), about one point above the best baseline (VisRef, 74.5%), and similar gains are reported for Qwen3-VL-8B-Instruct and InternVL3.5-8B.

Critical strengths

The selective-stability idea is clear and well motivated. Most test-time consistency methods only ask for agreement, which can reinforce an answer that ignores the image. Adding a directional requirement on the evidence region is a sensible step, and the design (matched pairs, one edit shared across views, discarding the probe trace before regenerating) follows naturally from it. The results are broad and consistent: gains on all six benchmarks with clustered CIs above zero, transfer to two other backbones, and support-corruption and region-swap experiments that behave as expected. The paper is also well documented, with all hyperparameters, a latency breakdown and a full worked example.

Critical weaknesses
  1. It is not clear that the gain comes from repairing visual evidence. In the worked example (Appendix E), the base model reads all bar values correctly and the mistake is in the reasoning; four of the six views already give the right answer before repair, and most of the edit sits on the May bar, which supports the wrong answer. This looks more like the edit passing on the consensus of the views, or simply perturbing the greedy decode, than fixing perception. Two simple controls would help: majority voting over full generations for the same paraphrases and low-relevance views, and an un-optimized edit of the same norm followed by regeneration. The case for the directional term also relies mostly on the sparse-symmetric control, but that control additionally forces invariance to the evidence region, which should hurt by itself; the cleaner estimate (74.7 → 75.3 in Table 3b) is only +0.6, without an interval. The trajectory readouts are listed as a contribution but are never ablated, and the two transfer backbones gain about as much without them.
  2. The evaluation setup may affect the margin over the baselines. The Qwen3-VL-8B-Thinking model card recommends sampling (temperature 1.0, top-p 0.95) with an output length of 40,960 tokens, while the paper uses greedy decoding with a 4,096-token cap. Regenerating from a perturbed context could fix some greedy failures, such as repetition or truncation, regardless of the objective. Self-consistency with eight samples gaining only +0.4 also seems low for math benchmarks with a thinking model, and S2R tunes many more hyperparameters than the baselines on the same calibration pool. Since the margin over VisRef is 1.0 point at 34% more latency and 49% more memory, these points matter.
Questions for authors
  1. For the Thinking model, do the view ensemble and TTAug run full reasoning for each view, or only average answer logits?
  2. What fraction of outputs reach the 4,096-token limit or contain no extractable answer, before and after repair?
  3. Among the wrong-to-correct cases, how many already had at least one base view giving the correct answer? What happens on errors that are consistent across all views?
  4. Are the Table 4 residuals measured on the same views used during optimization? Results on held-out views would be more convincing. Minor: r is used for both the reasoning trace and the pair index in Section 4.2. Section 6.1 refers to Table D.3 for default S2R, but D.3 reports S2R-Full. Please also state where the labels for the calibration pool drawn from the MMMU test split come from.
Reviewer_rTHDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces S2R, a test-time repair method for frozen vision–language models based on selective stability: predictions should remain consistent under answer-preserving changes while becoming less confident when proposed visual evidence is weakened. S2R constructs question variants and matched image perturbations, then optimizes a temporary edit to 128 visual-state coordinates using trajectory-conditioned answer distributions. The model subsequently regenerates its answer and discards the edit. Evaluation across six visual-reasoning benchmarks and three 8B backbones shows consistent accuracy gains. On Qwen3-VL-8B-Thinking, macro-average accuracy increases from 72.4% to 75.5%, reaching 75.7% with S2R-Full. Ablations, support-corruption tests, and cost analyses examine the method’s components and practical trade-offs.

Critical strengths
  1. Consistent empirical improvements across benchmarks and backbones. On the primary backbone, S2R improves macro-average accuracy from 72.4% to 75.5%, while S2R-Full reaches 75.7%. Default S2R exceeds the strongest evaluated external baseline, VisRef, by approximately 1.0 percentage point, with a reported 95% confidence interval of [+0.6, +1.4]. Improvements also transfer to two additional backbones. Clustering related MathVerse and DynaMath variants in the statistical analysis, together with reporting DynaMath worst-case accuracy, strengthens the empirical evidence. This is the main consideration supporting acceptance.

  2. The ablations meaningfully investigate the repair mechanism. The paper compares forward aggregation, dense editing, and sparse symmetric consistency, and examines objective components, editable layers, support size, and optimization depth. The gain from the directional evidence term and the degradation under support corruption provide complementary evidence for the proposed design. These controls help establish that the contribution extends beyond simply spending additional inference compute.

Critical weaknesses
  1. The repair signal lacks independent validation. Both region types are identified by the original model and may be unreliable when its answer is wrong. Matching perturbation size and strength does not verify their intended effects, and the paper does not clearly evaluate stability on views excluded from optimization. Thus, the evidence supports accuracy gains more strongly than improved visual grounding.

  2. Key implementation details remain unclear. The paper does not fully specify gradient–attention score computation and normalization or the proximal update rule. The sampling procedure behind the random-support overlap of 0.08 in Table D.2 also needs clarification. These omissions limit reproducibility and confidence in the support-quality analysis.

Questions for authors
  1. Have you independently audited the semantic validity of the generated paraphrases and matched image interventions? In particular, when the original answer is incorrect, how often does the low-relevance perturbation preserve the information needed to answer the question? Do the improvements in Table 4 also hold for perturbation instances or operator families excluded from optimization?

  2. Please specify the exact computation and normalization of (g_t) and (a_t), the aggregation of attention across layers and heads, the proximal update rule, and the sampling space used for random support. How is the random-support overlap of 0.08 in Table D.2 obtained?

  3. What is the contribution of trajectory-conditioned readouts? How does using only (\tau=0) compare with using all three positions under the same update budget? For free-form tasks, how frequently does the fixed candidate set contain the correct answer, and how effective is repair when it does not? These results would clarify the dependence of the gains on the optimization proxy.

Reviewer_iF32OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces S2R, a test-time repair method for frozen VLMs based on selective stability: predictions should remain consistent under answer-preserving changes while confidence should decrease when supporting visual evidence is weakened. The method uses constrained question paraphrases, matched image perturbations, and answer readouts along the original reasoning trajectory to edit visual-token states before regenerating the answer. Across six visual-reasoning benchmarks, S2R improves Qwen3-VL-8B-Thinking’s macro-average accuracy from 72.4% to 75.5%, with S2R-Full reaching 75.7%.

Critical strengths

The problem is well motivated. The paper aims to control the model's answer and confidence under different input perturbations.

Critical weaknesses
  • The paper is poorly presented and very hard to follow. Many notations and terms are used without being defined. For example, Sec 4.1 uses "branch" and introduces the edit, but does not define "branch" or any notation. The writing is superficial; the meaning of Sec 3 and 4 are not clear. There is no algorithm to formally explain how the method works. None of the objectives are explained. What does it even mean by "transfer" in Sec 6.2?
  • The connection between the objective and accuracy is not clear. Why do we expect that consistency would improve performance? Also the improvement in Table 1 is not very significant.
  • There are already many works improving consistency under perturbation. The paper does not discuss how this method is different from prior work.
Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.