- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces “selective stability”: a VLM’s answer should stay the same under answer-preserving changes (question paraphrases, perturbing an unimportant image region), but its confidence should drop when the region the model itself relies on is perturbed. S2R turns this into a test-time objective. For each input it builds paraphrases and matched image pairs (the same perturbation applied to a high- and a low-relevance region), reads answer distributions at several points of the base reasoning trace, and optimizes a sparse additive edit on 128 visual-token coordinates of a frozen model. The answer is then regenerated from the edited states. On six benchmarks, Qwen3-VL-8B-Thinking improves from 72.4% to 75.5% (75.7% for S2R-Full), about one point above the best baseline (VisRef, 74.5%), and similar gains are reported for Qwen3-VL-8B-Instruct and InternVL3.5-8B.
Critical strengths
The selective-stability idea is clear and well motivated. Most test-time consistency methods only ask for agreement, which can reinforce an answer that ignores the image. Adding a directional requirement on the evidence region is a sensible step, and the design (matched pairs, one edit shared across views, discarding the probe trace before regenerating) follows naturally from it. The results are broad and consistent: gains on all six benchmarks with clustered CIs above zero, transfer to two other backbones, and support-corruption and region-swap experiments that behave as expected. The paper is also well documented, with all hyperparameters, a latency breakdown and a full worked example.
Critical weaknesses
- It is not clear that the gain comes from repairing visual evidence. In the worked example (Appendix E), the base model reads all bar values correctly and the mistake is in the reasoning; four of the six views already give the right answer before repair, and most of the edit sits on the May bar, which supports the wrong answer. This looks more like the edit passing on the consensus of the views, or simply perturbing the greedy decode, than fixing perception. Two simple controls would help: majority voting over full generations for the same paraphrases and low-relevance views, and an un-optimized edit of the same norm followed by regeneration. The case for the directional term also relies mostly on the sparse-symmetric control, but that control additionally forces invariance to the evidence region, which should hurt by itself; the cleaner estimate (74.7 → 75.3 in Table 3b) is only +0.6, without an interval. The trajectory readouts are listed as a contribution but are never ablated, and the two transfer backbones gain about as much without them.
- The evaluation setup may affect the margin over the baselines. The Qwen3-VL-8B-Thinking model card recommends sampling (temperature 1.0, top-p 0.95) with an output length of 40,960 tokens, while the paper uses greedy decoding with a 4,096-token cap. Regenerating from a perturbed context could fix some greedy failures, such as repetition or truncation, regardless of the objective. Self-consistency with eight samples gaining only +0.4 also seems low for math benchmarks with a thinking model, and S2R tunes many more hyperparameters than the baselines on the same calibration pool. Since the margin over VisRef is 1.0 point at 34% more latency and 49% more memory, these points matter.
Questions for authors
- For the Thinking model, do the view ensemble and TTAug run full reasoning for each view, or only average answer logits?
- What fraction of outputs reach the 4,096-token limit or contain no extractable answer, before and after repair?
- Among the wrong-to-correct cases, how many already had at least one base view giving the correct answer? What happens on errors that are consistent across all views?
- Are the Table 4 residuals measured on the same views used during optimization? Results on held-out views would be more convincing. Minor: r is used for both the reasoning trace and the pair index in Section 4.2. Section 6.1 refers to Table D.3 for default S2R, but D.3 reports S2R-Full. Please also state where the labels for the calibration pool drawn from the MMMU test split come from.