acceptodds
Under review as a conference paper at ICLR 2027

Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoning

Abstract

Vision-language models (VLMs) achieve strong visual reasoning performance, yet subtle changes from routine image capture and processing can alter their reasoning trajectories even when images appear nearly identical. In long-horizon generation, the resulting activation shifts may accumulate across decoding steps, progressively altering reasoning tokens and ultimately changing the final answer, a phenomenon referred to as answer flips. To address this instability, we propose FLIPDIR (Flip-Direction Steering), a training-free inference-time method that estimates a low-rank flip-inducing activation subspace from contrastive pairs of original and answer-flipping inputs and selectively steers hidden states during decoding. A margin-based gate limits subspace attenuation to uncertain decoding steps, recovering original predictions while preserving stable ones. To evaluate robustness beyond accuracy or consistency on fixed test sets, we introduce VISFLIP, a benchmark framework that constructs evaluation groups for a target model and visual variation setting to separately assess recovery of original predictions and preservation of stable ones. VISFLIP spans nine dataset–variation combinations across scientific reasoning, robot-scene understanding, and medical VQA, covering subtle visual variations common in each domain. Experiments across 18 settings demonstrate that FLIPDIR consistently outperforms existing methods on the combined recovery and preservation metric. We will make our code publicly available.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Visual Reasoning, Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KxY4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates prediction instability in VLMs under subtle visual variations. The authors propose FlipDir, which extracts a low-rank activation subspace from answer-flipping image pairs and applies margin-gated steering during decoding. They also introduce VisFlip to evaluate prediction recovery and preservation, reporting improvements on the proposed metric across 18 settings.

Critical strengths
  1. Meaningful problem. The paper addresses an important issue concerning the stability of VLM predictions under subtle visual variations.
  2. Simple and effective method. The proposed method does not require model fine-tuning, and the experiments demonstrate consistent improvements on the proposed recovery–preservation metric.
Critical weaknesses
  1. Limited generalizability and potentially unfair comparisons. FlipDir requires separate subspace extraction and parameter tuning for each model, dataset, and perturbation type. Even its transfer experiments still require target-specific tuning. In contrast, VTI and LEAD are not similarly tuned, making it unclear whether the improvements come from FlipDir itself or its additional calibration.

  2. Insufficient discussion of answer flips. Similar prediction instability has been studied in prior work, but the paper does not clearly explain what is new about its formulation. The calculation and perturbation settings behind the flip rates in Table 1 could also be explained more clearly. More extensive experiments on larger models would help demonstrate how widespread and important this problem is.

  3. Unclear connection between motivation and method. The paper emphasizes error propagation during long-horizon reasoning, but FlipDir does not specifically address long reasoning processes. It is also unclear why flip-subspace suppression and margin-based gating are better suited to this problem than existing steering and uncertainty-based decoding methods.

Questions for authors

See Weakness

Reviewer_LABuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates answer instability in vision-language models (VLMs) under subtle visual perturbations and proposes FlipDir, a training-free inference-time intervention method. FlipDir estimates a low-rank subspace from activation differences between clean and answer-flipping image pairs using SVD, and selectively suppresses the corresponding hidden-state components during decoding through a margin-based gate. The paper also introduces VisFlip, a benchmark that evaluates prediction recovery and preservation across different visual perturbations. Experiments on two VLMs across three task domains and 18 evaluation settings demonstrate consistent improvements over existing baselines, with an average gain of 2.8 percentage points on the combined metric. Ablation studies further examine the effects of subspace estimation, steering strength, and margin-based gating.

Critical strengths
  • Comprehensive framework design and evaluation. The paper presents a complete framework integrating flip-subspace estimation, selective inference-time intervention, and the VisFlip benchmark. The evaluation covers multiple models, tasks, and perturbation types, providing a systematic assessment of prediction recovery and preservation.

  • Consistent empirical improvements. FlipDir achieves the best combined recovery–preservation performance across all 18 evaluated settings. The ablation studies further demonstrate the effectiveness of margin-based gating, supporting the practical value of the proposed approach.

Critical weaknesses
  • Limited generalization. FlipDir requires separate subspace estimation and hyperparameter calibration for each model, dataset, and perturbation type. Even its transfer experiments retain target-specific intervention parameters, leaving generalization to unseen settings unverified.

  • High calibration cost and dependence on prior knowledge. The method assumes known perturbation types and requires extensive VLM inference to collect answer-flipping pairs, followed by activation extraction and validation-based calibration. This limits its practicality in real-world scenarios with unknown or changing perturbations.

  • Limited methodological novelty. Contrastive activation differences, SVD-based low-rank subspaces, and inference-time steering have been extensively explored. Activation-Space Personality Steering[1] already combines these techniques, while methods such as VCD avoid setting-specific subspace estimation. The paper insufficiently distinguishes its technical contributions and advantages from existing approaches.

  • Heuristic design and insufficient justification. The method depends on many hyperparameters that are selected manually. The central assumption that high-variance activation differences should be suppressed during decoding lacks sufficient justification. SVD does not establish causal relevance, and suppressing these directions may remove useful information. The transition from prefill-derived subspaces to decoding-time intervention also lacks adequate validation.

[1] Activation-space personality steering: Hybrid layer selection for stable trait control in LLMs

Questions for authors
  1. Generalization: How does FlipDir perform when the subspace, intervention layer, and steering strength are transferred together to unseen tasks or perturbation types, without target-specific calibration?

  2. Intervention justification: What evidence supports the assumption that dominant activation differences identified during prefill should be suppressed during decoding? How can the authors distinguish causally relevant flip directions from general perturbation-related or task-relevant representations?

For other questions, please see Weakness.

1 comment

Thank you for taking the time to review our paper.

We believe that our responses and additional experiments have addressed the concerns you raised. We would appreciate it if you could review these updates and let us know if any concerns remain unaddressed or if you have further questions.

We would be grateful if you could take these updates into account in your final assessment.

Reviewer_ZXQcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates answer flips in vision-language models under subtle visual perturbations. It proposes FlipDir, which estimates a low-rank subspace from clean–flip activation differences at the final prompt position, then selectively attenuates this subspace during decoding using a next-token margin gate. It also introduces VisFlip, a model-specific framework that separately measures recovery of clean predictions and preservation on clean and non-flipping inputs. Across two backbones and nine dataset–perturbation combinations, FlipDir achieves the highest reported harmonic mean of recovery and preservation, with an average gain of 2.8 percentage points over the strongest baseline in each setting. Ablations examine subspace construction, gating, rank, and calibration size.

Critical strengths
  1. FlipDir achieves a consistent improvement in the reported recovery–preservation metric across all 18 settings, covering two backbones and three application domains. The component ablations provide supporting evidence: margin gating improves H in all six tested comparisons, and flip-conditioned activation differences outperform the alternative constructions in Table 5.

  2. VisFlip makes the trade-off between repairing flips and changing stable predictions explicit. The manuscript clearly documents model-specific construction, disjoint calibration and evaluation pools, and evaluation denominators. It also acknowledges important limitations, including recovery of incorrect clean predictions and the target-specific calibration retained in transfer experiments.

Critical weaknesses
  1. The comparisons do not establish an advantage under matched tuning conditions.

Appendix C.4 states that LEAD and VTI use recommended configurations without task-specific retuning, whereas FlipDir selects its intervention layer and strength on validation data for each setting. The reported advantage may therefore reflect both the intervention and unequal tuning opportunities. This matters given the average improvement of 2.8 H points and smaller margins in several settings. The recovery–preservation plots compare individual configurations rather than tuned frontiers. Comparable validation access and tuning budgets are needed to establish the strength of the advantage over these baselines.

  1. The evaluation leaves the net benefit and reliability of the intervention insufficiently established.

The authors explicitly target preservation of clean predictions rather than accuracy, which is a legitimate diagnostic objective. However, H weights the three groups equally without accounting for how frequently flips occur under independently sampled perturbations. Higher H therefore does not necessarily imply better overall agreement with clean predictions. Collateral changes can be substantial: Gemma/M3CoT exposure retains only 70.2% of clean predictions and 69.1% of non-flip predictions. Aggregate post-intervention accuracy would also help establish whether recovery preserves correct decisions or reinstates errors.

Uncertainty in the reported gains remains insufficiently characterized. Appendix C.5 demonstrates exact reproduction with saved subspaces but also notes that numerically close re-estimated subspaces can change H by several points. Paired uncertainty estimates and calibration-resampling results are needed to assess whether the smaller improvements are robust.

Questions for authors
  1. Do the advantages over LEAD and VTI persist when these baselines receive comparable validation access and task-specific tuning budgets? A comparison of their tuned recovery–preservation trade-offs would help isolate the contribution of FlipDir.

  2. On perturbations sampled independently of flip status, how does FlipDir affect overall agreement with clean predictions and ground-truth accuracy? Please distinguish cases where the clean prediction is correct from those where it is incorrect, so that useful recovery and harmful changes can be assessed separately.

  3. How stable are the gains under paired resampling of test parents and re-estimation of the calibration subspace? In light of the numerical sensitivity reported in Appendix C.5, can you quantify variation across re-extractions and calibration samples, particularly for settings with small improvements over baselines?

  4. Table 5 supports flip-conditioned subspace estimation in two settings. Do these findings extend to additional settings, and how does the method compare with matched-size generic-perturbation subspaces or rank-matched random controls under the same gating procedure? This would strengthen the interpretation that the estimated subspace captures behavior specifically associated with answer flips.

2 comments

Thank you for taking the time to review our paper.

We believe that our responses and additional experiments have addressed the concerns you raised. We would appreciate it if you could review these updates and let us know if any concerns remain unaddressed or if you have further questions.

We would be grateful if you could take these updates into account in your final assessment.

Thank you for the rebuttal and the additional experiments. The matched-budget tuning addresses my main concern about the baseline comparison. FlipDir still achieves the highest reported H in all 18 settings. The added subspace controls strengthen the ablation, and the two evaluations with independently sampled perturbations show improved agreement with the original predictions. In both settings, the best baseline achieves slightly higher ground-truth accuracy.

My remaining concern is the reliability of the smaller gains. Appendix J.1 reports the three test subsets with the largest observed gains, which makes the resampling results optimistic. The calibration resampling also illustrates the uncertainty: on Gemma/GMAI rotation, FlipDir’s mean H across the three re-estimated subspaces is 63.03 (SD 2.20), compared with 63.51 for the strongest tuned baseline. The rebuttal strengthens the comparison, but the evidence that the smaller gains persist across test and calibration samples remains limited. My final rating is 2 (Weak Reject).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.