- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates prediction instability in VLMs under subtle visual variations. The authors propose FlipDir, which extracts a low-rank activation subspace from answer-flipping image pairs and applies margin-gated steering during decoding. They also introduce VisFlip to evaluate prediction recovery and preservation, reporting improvements on the proposed metric across 18 settings.
Critical strengths
- Meaningful problem. The paper addresses an important issue concerning the stability of VLM predictions under subtle visual variations.
- Simple and effective method. The proposed method does not require model fine-tuning, and the experiments demonstrate consistent improvements on the proposed recovery–preservation metric.
Critical weaknesses
-
Limited generalizability and potentially unfair comparisons. FlipDir requires separate subspace extraction and parameter tuning for each model, dataset, and perturbation type. Even its transfer experiments still require target-specific tuning. In contrast, VTI and LEAD are not similarly tuned, making it unclear whether the improvements come from FlipDir itself or its additional calibration.
-
Insufficient discussion of answer flips. Similar prediction instability has been studied in prior work, but the paper does not clearly explain what is new about its formulation. The calculation and perturbation settings behind the flip rates in Table 1 could also be explained more clearly. More extensive experiments on larger models would help demonstrate how widespread and important this problem is.
-
Unclear connection between motivation and method. The paper emphasizes error propagation during long-horizon reasoning, but FlipDir does not specifically address long reasoning processes. It is also unclear why flip-subspace suppression and margin-based gating are better suited to this problem than existing steering and uncertainty-based decoding methods.
Questions for authors
See Weakness
Thank you for taking the time to review our paper.
We believe that our responses and additional experiments have addressed the concerns you raised. We would appreciate it if you could review these updates and let us know if any concerns remain unaddressed or if you have further questions.
We would be grateful if you could take these updates into account in your final assessment.