Counterfactual Interventions for Latent Visual Reasoning

Latent visual reasoning offers intermediate computation between direct answering and explicit textual chain-of-thought. Its value depends on whether the answer uses the evidence carried by continuous states. Answer supervision and representation alignment can leave this dependence weak: a decoder may answer from the image while bypassing the latent. We observe that counterfactual image pairs address this gap by providing both a meaningful replacement state and the answer that should follow its transfer. Unlike arbitrary perturbations, evidence-changing edits supply known targets for latent interventions. Continuous substitution passes answer gradients through the transferred state, while paired vector differences provide directions for exploring evidence-related variations. We introduce CILVR, which uses this relation in supervised latent learning and policy optimization. Supervised training combines alignment with differentiable swaps targeting the donor image's answer. Its reinforcement stage, CA-GRPO, concentrates exploration along paired latent differences and rewards correctness and consistency under transfer. An automated editing and verification pipeline supplies CFVR, a dataset of verified counterfactual image pairs with reasoning traces on both sides. CILVR-8B achieves competitive performance across ten vision-centric and reasoning benchmarks. Ablations compare paired data, training objectives, and sensitivity to latent interventions. Code and data will be released upon acceptance.