acceptodds
Withdrawn from ICLR 2027

Counterfactual Interventions for Latent Visual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent visual reasoning offers intermediate computation between direct answering and explicit textual chain-of-thought. Its value depends on whether the answer uses the evidence carried by continuous states. Answer supervision and representation alignment can leave this dependence weak: a decoder may answer from the image while bypassing the latent. We observe that counterfactual image pairs address this gap by providing both a meaningful replacement state and the answer that should follow its transfer. Unlike arbitrary perturbations, evidence-changing edits supply known targets for latent interventions. Continuous substitution passes answer gradients through the transferred state, while paired vector differences provide directions for exploring evidence-related variations. We introduce CILVR, which uses this relation in supervised latent learning and policy optimization. Supervised training combines alignment with differentiable swaps targeting the donor image's answer. Its reinforcement stage, CA-GRPO, concentrates exploration along paired latent differences and rewards correctness and consistency under transfer. An automated editing and verification pipeline supplies CFVR, a dataset of verified counterfactual image pairs with reasoning traces on both sides. CILVR-8B achieves competitive performance across ten vision-centric and reasoning benchmarks. Ablations compare paired data, training objectives, and sensitivity to latent interventions. Code and data will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Reasoning, Latent Reasoning, Counterfactual Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VW7XOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CILVR, a latent visual reasoning framework based on counterfactual image pairs. CILVR encourages the model to actually use its intermediate latent states. The authors swap the latent representations between paired images and train the model to predict the answer associated with the transferred latent. The paper further introduces CA-GRPO for latent-space policy optimization and constructs the CFVR dataset with 304K verified counterfactual image pairs.

Critical strengths
  1. The paper targets the latent-bypass problem: the possibility that a model produces meaningful-looking latent states while the decoder largely ignores them.

  2. The latent swap objective is a well-motivated formulation, as it directly links changes in latent representations to changes in model predictions, rather than relying only on representation alignment.

Critical weaknesses
  1. The current experiments mainly show that replacing or perturbing latent states affects model outputs (sec 4.4), but provide limited analysis of what information is actually encoded in these latent representations and how consistently this information supports downstream reasoning.

  2. A substantial part of the comparison still relies on models such as GPT-4o, which makes it difficult to assess the competitiveness of CILVR against the current SOTA in multimodal reasoning. Although a few newer proprietary models are included, the comparison would benefit from a more systematic set of recent baselines such as GPT-5.6sol / Qwen3.5/ Claude Fable/ opus.

  3. The technical notation is overly dense. The method introduces a large number of notations, objectives, training stages, intervention terms, and RL-specific quantities with insufficient motivation or intuitive explanation. The transition from counterfactual latent swapping to the direction-conditioned Gaussian policy is particularly abrupt, making the method substantially harder to understand than necessary.

  4. Basic presentation and formatting issues remain throughout the paper; for example, multiple table captions are placed below the tables rather than above them. I think the author should be more careful about the manuscript formatting.

Questions for authors

Alongside the weaknesses, I have the following questions:

  1. How sensitive is CILVR to the quality of the counterfactual edits and verification pipeline?

  2. How does performance change when using fewer counterfactual pairs or substantially less RL compute?

  3. Can the authors clarify whether the gains are consistent across different edit types in CFVR?

  4. Why does GPT -5 achieve better performance yet not bold in Table 1?

I hope the authors will successfully address my concerns and I may raise my scores. I hope the authors could put more attention on the manuscript; I don't think the formatting problem should have shown up on submission.

Reviewer_FbBUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CILVR, a method that uses counterfactual image pairs in VLMs training to address an issue, where VLMs can ignore reasoning traces and answer to questions to images directly from images. The method uses automatically generated counterfactual edits with respect to the question asked, so that the answer changes. The training pipeline itself is complex and includes many distinct stages. Experiments demonstrate performance gains from the proposed method with respect to the base model.

Critical strengths

The idea of using counterfactual image edits for augmenting VLM training data is reasonable and makes sense.

Critical weaknesses
  1. My biggest issue is clarity. The paper is difficult to parse. For example, Section 3.1 opens with formal notation, and then describes the method in plain language without referring back to just-introduced variables. The writing itself is ambiguous, e.g. "The image path remains available." What is image path? Available to what? Or "Correctness on the unmodified inputs anchors this behavior to the visual task."

Notation is confusing and overloaded. The relationship between f_{\theta}, p_{\theta}, and \pi_{\theta} is never stated. \mu denotes the anchor-span mean and the per-step latent mean, and the number of gradient updates per rollout batch.

The warmup stage is unclear. It is described as next-token supervision on written reasoning and answers, but it is unclear what the model exactly outputs and what and how supervises the latents.

  1. Design decisions are not well motivated and explained. For example, in Equation 1 the swap loss trains the model to output a' for (I,q) when given z' originally received from (I', q). In other words, it trains the model to give the wrong answer for the image it is looking at. I see why this would increase reliance on the latent, however, it does so by explicitly teaching the model to ignore image information that contradicts the latent.The auxiliary intervention penalty in CA-GRPO has a similar problem. It penalizes answers are correct when the latent is zeroed or random.

In self-distillation part the teacher and student share the same weights, and the teacher is given the written reasoning. The student learns to reproduce the teacher’s hidden states at step boundaries, starting from latent tokens alone. The paper does not explain why this indirection is needed rather than supervising the reasoning tokens directly, or what the latents capture beyond a compressed copy of the trace. It is also unclear whether the teacher is the frozen warm-up model or the current student weights (Fig. 2 labels it M_warm-up, while the text says “the same network without gradient.”).

Questions for authors
  1. What motivates the design of Equation 1? Why train the model to output the donor answer, which is incorrect for the given image-question pair, rather than using an alternative such as an special “incompatible” answer for mismatched image-latent pairs? Can you quantitatively evaluate how much the model relies on the image vs. the latent before and after this stage?

  2. What is the purpose of self-distillation? Why can’t the reasoning traces, which are already available, be used directly? Is the teacher the frozen warm-up model or the current student?

  3. What is the inference/training cost of CILVR vs. the text-CoT controls.

Reviewer_MFM8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets "latent bypass" in Latent Visual Reasoning (LVR), where the decoder ignores continuous latent states and relies directly on the raw image. The authors propose CILVR, combining multi-stage SFT (with differentiable latent swaps) and Counterfactual-Anchored GRPO (CA-GRPO) trained on CFVR, a curated dataset of 304K counterfactual image pairs. Experiments across ten benchmarks validate the approach.

Critical strengths

Crucial Problem & Intuitive Design: Addressing latent bypass is indeed vital for LVR. Using counterfactual image pairs to supervise swapped latent states and guiding RL exploration via paired latent differences in CA-GRPO are intuitive and well-designed. Editing only a small image region may fail to support other advanced visual reasoning tasks that require global semantics, but this is not a significant issue.

Solid Dataset Contribution: The four-pass verification pipeline for building the 304K CFVR dataset is rigorous and ensures high counterfactual quality.

Comprehensive Evaluation & Ablations: The method is thoroughly tested across ten diverse benchmarks and multiple reasoning formats (No CoT, Text CoT, Latent). The causal intervention experiments (zeroing, random, and cross-question replacement in Figure 4) effectively probe the true causal utility of the learned latents.

Critical weaknesses
  1. [Major Concern] Unfair Backbone Comparison in Main Tables: In Tables 1–3, all external latent baselines (LVR, Monet, SkiLa, SkiLa-V) use Qwen2.5-VL-7B, whereas CILVR uses the much stronger Qwen3-VL-8B. In fact, zero-shot Qwen3-VL-8B alone already outperforms all Qwen2.5-VL-7B latent baselines on almost every benchmark. Without evaluating CILVR on Qwen2.5-VL-7B or re-implementing baselines on Qwen3-VL-8B, the comparison is unfair.

  2. High Training Overhead & Reproducibility Barriers: The multi-stage training pipeline (3 SFT stages + CA-GRPO) is overly complex and computationally heavy. CA-GRPO requires numerous forward/decoding passes per pair (anchor rollouts, 2G2G group rollouts, 2G2G swapped greedy decodings, and multiple intervention passes) alongside over a dozen hyperparameters. This complexity severely hinders strict reproducibility for other researchers. A quantitative cost comparison against vanilla GRPO is needed.

  3. Limited Effectiveness of SFT Stages (Figure 4): Figure 4 shows that perturbing latents (Z/R/Q) after SFT (S1–S3) has minimal impact on accuracy (most predictions remain "Still correct"), and the number of initially correct answers in S2 and S3 is even lower than in S1. Although RL improves performance later, the inability of SFT to prevent latent bypass remains a notable limitation.

  4. Poor Readability & Overly Dense Presentation: The text, especially in Section 3, is overly dense, convoluted, and difficult to follow. With too many loss terms, training stages, and notations packed tightly together without clear intuitive transitions, reading the methodology requires excessive effort. A thorough revision to streamline the presentation and improve readability is strongly recommended.

Questions for authors
  1. Separate Models in SFT Stage 2 vs. Self-Distillation (e.g., CODI): Why does SFT Stage 2 train separate models/passes rather than adopting an end-to-end joint self-distillation paradigm like CODI, which typically yields stronger gains with a simpler pipeline?

  2. Outlier Gain on MME-RealWorld-Lite: Why does CILVR achieve an unusually large boost on MME-RealWorld-Lite (>10 points) compared to modest gains (1–2 points) on most other benchmarks?

  3. Concrete Open-Source Plan: Do the authors have a concrete open-source plan? Please provide an explicit checklist and timeline of what will be released (e.g., CFVR dataset, data generation pipeline, SFT/CA-GRPO training code, and model checkpoints), rather than leaving a placeholder "coming soon" repository as many previous projects conducted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.