acceptodds
Under review as a conference paper at ICLR 2027

Reason Through the Latent! Making Latent Visual Reasoning Necessary

Abstract

Latent visual reasoning aims to perform multimodal reasoning through hidden-state computation rather than explicit textual chains of thought. However, visual information being present in a latent state does not imply that the model actually relies on that state when producing its answer, especially when alternative image-conditioned paths remain available. We introduce Causal Visual Recurrent Reasoning (CVRR), which preserves pretrained visual competence while making recurrent computation the required image-conditioned path to prediction. CVRR initializes recurrence from the question hidden state after the pretrained vision-language model has incorporated the image, then repeatedly updates this state while re-reading the same fixed visual evidence. Before decoding, visual states and the original multimodal KV cache are removed so that only the final recurrent state carries image-conditioned information to the answer. Across the , MMVP, BLINK, and MME-RealWorld-Lite benchmarks, CVRR retains strong performance under this strict interface, while compatible latent reasoners fail to recover comparable visual competence even when retrained under the same constraint. Causal interventions further show that predictions remain sensitive to recurrent content when the question is held fixed, and that persistent visual evidence causally revises the recurrent trajectory. These results distinguish latent informativeness from latent computation that is actually used for prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Latent Visual Reasoning, Multimodal Reasoning, Vision-Language Models, Recurrent Reasoning, Causal Interventions, Mechanistic Interpretability, Visual Question Answering, Hidden-State Reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_UbSmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the “bypass” problem in implicit visual reasoning, where models may answer through original visual tokens or multimodal KV caches without relying on their latent states. It introduces CVRR, which frames latent-state reliance as a causal path-design problem and uses image-conditioned initialization, learned recurrent updates, and repeated visual rereading to make the recurrent latent state necessary for prediction while preserving pretrained visual competence. Experiments on V*, MMVP, BLINK, and MME-RealWorld-Lite, together with no-bypass baseline comparisons, component ablations, and causal interventions, show that CVRR retains relatively strong performance under this constraint and that its predictions depend on recurrent states continually refined by visual evidence.

Critical strengths
  1. Novel problem perspective and sound methodological design. The paper distinguishes between the informativeness of hidden states and pathway necessity. By eliminating the visual bypass via CVRR, it preserves pretrained visual capabilities while making the recurrent hidden state a necessary channel for prediction.

  2. Comprehensive validation framework. Beyond multi-benchmark comparisons, the paper employs component ablation, state replacement, same-question cross-image swapping, and visual region intervention, adequately demonstrating that recurrent states and continuous visual re-reading indeed influence predictions.

Critical weaknesses
  1. While the paper demonstrates causal utilization and visual refinement, it has not yet fully substantiated multi-step implicit reasoning in the stronger sense; moreover, some necessity conclusions are directly guaranteed by the architecture itself. This limits the paper’s claim from demonstrating multi-step latent reasoning to demonstrating causal latent-state utilization and iterative visual refinement.

  2. Although the no-bypass baselines were uniformly retrained, issues remain regarding interface adaptation, fairness in hyperparameter tuning, and incomplete comparison coverage. Therefore, the magnitude of CVRR’s advantage over prior latent reasoning methods remains uncertain.

Questions for authors
  1. Current experiments show that CVRR's predictions depend on recurrent states and that these states continuously read visual information, but is this sufficient to distinguish "genuine multi-step implicit reasoning" from "repeated feature refinement or information compression over the same visual representation"? Could the authors provide more direct controls, for example, a parameter- and compute-matched non-recurrent bottleneck, or interventions that selectively disable visual re-reading at individual recurrent steps?

  2. Could the authors report the method-specific hyperparameter search, training budget, and multi-seed results for the no-bypass baselines, and clarify how much of their performance degradation may arise from interface mismatch rather than bypass dependence?

4 comments
AuthorsOpenReview ↗

Response to Reviewer UbSm (Part 1/3)

We sincerely appreciate the reviewer's constructive evaluation and recognition of our methodological contributions, particularly the distinction between latent-state informativeness and causal path necessity. We conducted additional experiments to address the concerns raised, as detailed below.


Current experiments show that CVRR's predictions depend on recurrent states and that these states continuously read visual information, but is this sufficient to distinguish "genuine multi-step implicit reasoning" from "repeated feature refinement or information compression over the same visual representation"? Could the authors provide more direct controls, for example, a parameter- and compute-matched non-recurrent bottleneck, or interventions that selectively disable visual re-reading at individual recurrent steps?

We appreciate the reviewer's insightful suggestion. To further distinguish sequential recurrent computation from independent feature refinement, we conducted both proposed controls: a Non-recurrent Bottleneck comparison and a Step-wise Visual Re-reading Intervention.

1. Non-recurrent Bottleneck Comparison

We introduced a Non-recurrent Bottleneck using the same native initialization h1h_1, fixed visual evidence vv, frozen decoder layer, and strict no-bypass interface as CVRR. While CVRR sequentially updates its recurrent state three times, the control processes the same h1h_1 and vv through three independent branches and aggregates their outputs. Each branch employs independently trained LoRA parameters with distinct scaling factors (α=8,16,32\alpha=8,16,32), allowing the branches to learn different visual transformations rather than sharing identical updates. The control was trained under the same recipe as CVRR. This enables a comparison of sequential state refinement against independent visual processing under comparable computational budgets, with the non-recurrent control using more trainable parameters.

The generation-based results are presented below.

MetricNon-recurrent (%)CVRR (%)Δ\Delta (pp) [95% CI]
V∗V^* Overall77.081.2+4.2 [1.1, 7.3]
Direct Attributes79.182.6+3.5 [0.0, 7.8]
Relative Position73.779.0+5.3 [1.3, 10.5]
MMVP Item70.774.7+4.0 [1.3, 6.7]
MMVP Pair46.052.7+6.7 [2.0, 11.3]

Here, Δ\Delta denotes the accuracy improvement of CVRR in percentage points, with 95% CIs computed using paired bootstrap resampling (image-pair-clustered for MMVP).

CVRR outperforms the Non-recurrent Bottleneck across all metrics, achieving improvements of 4.2 percentage points on V∗V^* Overall and 6.7 percentage points on MMVP Pair Accuracy. The confidence intervals exclude zero for all metrics except Direct Attributes, providing evidence that sequential recurrent refinement offers additional benefits beyond independent visual processing.

2. Step-wise Visual Re-reading Ablation

We further examined the contribution of individual recurrent steps by selectively disabling question-to-visual attention at each recurrent update and independently retraining each ablation under the same T=4T=4 training configuration. Step kk refers to the kk-th recurrent update, while native multimodal initialization h1h_1 and the recurrent transition architecture are preserved across all conditions. All experiments used generation-based evaluation.

ConditionV∗V^* Acc. (%)Δ\Delta [95% CI] (pp)MMVP Pair (%)Δ\Delta [95% CI] (pp)
Full CVRR81.2–52.7–
w/o Step 177.53.7 [1.1, 6.8]45.37.3 [3.3, 12.0]
w/o Step 276.44.7 [2.1, 7.9]42.710.0 [5.3, 15.3]
w/o Step 378.03.1 [1.1, 5.8]46.06.7 [2.7, 10.7]
All-off69.112.1 [7.3, 16.8]18.034.7 [27.3, 42.7]

Here, Δ\Delta denotes the accuracy reduction relative to Full CVRR, with differences and paired bootstrap 95% CIs computed from unrounded sample-level results (pair-clustered resampling for MMVP). The All-off condition corresponds to the w/o Visual Re-read ablation in Table 2.

Despite independent retraining, disabling visual re-reading at any individual step reduces accuracy by 3.1–4.7 percentage points on V∗V^* and 6.7–10.0 percentage points on MMVP Pair Accuracy, with all confidence intervals excluding zero. Disabling visual access at all recurrent steps produces larger reductions of 12.1 and 34.7 percentage points, respectively.

Together with Figure 5, these results demonstrate the importance of visual access across multiple recurrent steps, while the Non-recurrent Bottleneck comparison supports the benefit of sequential state refinement. As the reviewer noted, path necessity is enforced by construction; our analyses focus on the computation performed along this required path. We will clarify our primary contribution as causal latent-state utilization and prediction-relevant iterative visual refinement.

AuthorsOpenReview ↗

Response to Reviewer UbSm (Part 2/3)

Could the authors report the method-specific hyperparameter search, training budget, and multi-seed results for the no-bypass baselines, and clarify how much of their performance degradation may arise from interface mismatch rather than bypass dependence?

We appreciate the reviewer's concern regarding baseline fairness. Our comparison preserves each method's original latent mechanism and training objectives while imposing a common no-bypass constraint, allowing us to evaluate visual competence retention under matched training conditions.

Specifically, each baseline generates latent representations through its original image-question forward pass. We preserve all KK latent tokens and their residual states across all 28 decoder layers, including method-specific boundary tokens and hybrid text–latent reasoning mechanisms. These representations construct the answer-side latent KV states without additional projection or pooling. As in CVRR, the original multimodal KV caches are removed, and question tokens use image-free KV caches from a text-only pass, ensuring that image-conditioned information reaches answer decoding through the preserved latent pathway.

The evaluated latent configurations and training objectives are summarized below.

MethodLatent RepresentationLatent Tokens (KK)Training Objectives
LVRLast-position Hidden State8Answer CE + Visual Reconstruction MSE
MonetFinal-layer Hidden State10Answer CE + Layer-wise Cosine Alignment Loss
SkiLaSketch Hidden State27Answer CE + Latent Sketch Reconstruction Loss
LaserLast-position Hidden StateUp to 8Answer CE + Dynamic Windowed Alignment Learning
UniVLRLast-position Hidden State12Answer CE + Latent Alignment
HyLaRCanvas Hidden StateUp to 32Answer CE + Canvas Prediction MSE

Each baseline was initialized from its publicly released checkpoint rather than trained from scratch. All six baselines were further fine-tuned on the same Visual CoT dataset for three epochs using the same SFT configuration as CVRR: learning rate 2×10−52 \times 10^{-5}, AdamW, cosine scheduler, warmup ratio 0.05, effective batch size 128, and BF16 precision. The common SFT settings were standardized while preserving each method's original objectives, auxiliary losses, and loss weights. Evaluation used greedy decoding with each method's original generation configuration. We did not perform method-specific hyperparameter searches. Instead, we standardized the common SFT hyperparameters across all baselines and CVRR to ensure a fair and controlled comparison. Complete per-method settings will be provided in the revised appendix.

Under the same LoRA configuration (r=32,α=16r=32,\alpha=16), each baseline uses 80.74M trainable LoRA parameters across 28 decoder blocks, compared with 2.88M in CVRR's single shared block. Thus, the baselines were not disadvantaged by a smaller trainable parameter budget.

We further examined the training loss, including method-specific auxiliary objectives where applicable, to assess optimization progress under the shared three-epoch schedule.

ModelEpoch 1Epoch 2Epoch 3
LVR3.05581.69151.6818
Monet3.02951.28841.0246
SkiLa8.39515.45342.6955
Laser6.70412.65991.4224
UniVLR6.82381.97570.8424
HyLaR9.10486.64452.6363
CVRR6.17021.72990.9307

All methods exhibit substantial reductions in training loss under the same training budget. For most methods, including CVRR, the loss reduction becomes smaller in the final epoch, suggesting that optimization is approaching stabilization.

AuthorsOpenReview ↗

Response to Reviewer UbSm (Part 3/3)

We additionally evaluated the no-bypass baselines across three independent training seeds (0, 45, 100) to examine the consistency of their performance.

ModelV∗V^* OverallMMVP ItemMMVP PairBLINK OverallMME-RW-Lite
LVR-7B35.4 ± 1.349.9 ± 0.50.7 ± 0.737.8 ± 1.126.2 ± 1.1
Monet-7B36.5 ± 1.650.6 ± 0.82.0 ± 0.738.5 ± 1.233.2 ± 1.2
SkiLa-7B37.5 ± 1.329.7 ± 1.90.2 ± 0.438.5 ± 1.215.6 ± 1.4
Laser-7B37.5 ± 1.150.0 ± 0.31.1 ± 0.838.1 ± 1.031.8 ± 0.9
UniVLR-7B38.4 ± 0.849.9 ± 0.50.7 ± 0.738.9 ± 0.727.4 ± 1.3
HyLaR-7B39.3 ± 1.942.2 ± 1.82.4 ± 1.033.5 ± 1.432.4 ± 1.5
CVRR-7B81.5 ± 0.674.7 ± 0.752.7 ± 0.755.6 ± 0.446.9 ± 0.2

All values represent generation-based accuracy (%) as mean ± standard deviation across training seeds. These results are consistent with the single-run findings in Table 1, demonstrating that the limited visual competence retained by existing baselines under the strict interface is not specific to a single training seed. Moreover, most baselines obtain approximately 50% MMVP Item and near-zero MMVP Pair accuracy. Because the two images in each MMVP pair share the same question but have opposite answers, a decoder that receives no usable image information gives the same answer to both and therefore obtains exactly 50% Item and 0% Pair accuracy. Since the interface passes image-conditioned latent states to the decoder at every layer, this pattern suggests that the baselines may have difficulty effectively utilizing image-specific information from their latent representations under the strict interface. In contrast, CVRR achieves 81.5 ± 0.6% on V∗V^* Overall and 52.7 ± 0.7% on MMVP Pair Accuracy across the same three seeds, showing that strong visual competence can be retained under the same strict constraint.

Importantly, the weak latent-state reliance observed in Figure 1(a) was established under the original multimodal inference pathways, before any no-bypass modifications. Thus, this observation cannot be attributed to interface adaptation. Table 1 provides complementary evidence by evaluating visual competence retention after removing alternative image-conditioned pathways and retraining under the same constraint.

Together, these findings support our central observation that latent-state informativeness alone does not guarantee prediction reliance or visual competence retention. We will clarify the implementation details in the revised manuscript. PEARL was excluded because imposing the same constraint would require modifying its original predictive latent mechanism rather than merely removing an alternative multimodal pathway.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer UbSm

Thank you again for your thoughtful review and valuable feedback. As the discussion period is approaching its end, we would like to kindly ask whether our responses and the additional experiments in the revised manuscript have addressed your concerns. If you have any remaining questions or concerns, we would be happy to provide further clarification.

Reviewer_rCWCOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a model uses the visual information encoded in its latent states when predicting an answer. CVRR initializes a question state at a causally selected layer of a pretrained VLM. One shared LoRA-adapted decoder layer repeatedly updates this state while re-reading fixed visual evidence. Before answer decoding, visual states and multimodal KV caches are removed. The experiments compare ordinary and no-bypass latent reasoners, ablate recurrent components, and intervene on states and visual connections. They also examine other backbones, larger models, and inference cost.

Critical strengths
  1. The paper tests whether latent visual information affects predictions through interventions, beyond representation probes. Crossed-state experiments enable or block visual connections, and image swaps hold the question fixed. Together with the strict decoder interface, these experiments support a role for recurrent content and visual re-reading. The paper also distinguishes making a path necessary by construction from showing useful computation along that path.

  2. The controls are carefully chosen. No-bypass baselines and component ablations are retrained, and a full-multimodal SFT control uses the same data. CVRR reaches 81.2% on V*, 52.7% MMVP pair accuracy, and 55.2% BLINK overall under the strict interface. Appendices E-F add cross-backbone results and latency/FLOP measurements. The paper acknowledges that increasing depth is not consistently beneficial and that some settings lose performance.

Critical weaknesses
  1. The benefit of repeated updates is less clear than the benefit of the architecture as a whole. Table 2 includes a T=2 control with one adapted visual re-read. Moving to T=4 adds 2.7 points on V* and 2.0 points in MMVP pair accuracy, roughly five examples and three pairs. These comparisons use one training seed and lack paired uncertainty estimates. T=1 also changes whether a recurrent adapter is trained, so that comparison cannot separate repeated computation from adaptation. The multi-step claim needs to be qualified accordingly.

  2. The no-bypass baselines lose substantial performance, but Appendix B.2 gives only a generic account of their adaptation. For each method, it is unclear exactly which latent output reaches the decoder, which caches are removed, how the model is initialized, and which auxiliary losses are retained. These details matter when judging whether each baseline has a viable interface. There is also a mismatch in evaluation protocols: most mechanistic experiments use a restricted first-option-token readout, while the main results use generation with each implementation's default token budget. It remains unclear whether the intervention effects persist under that generation protocol.

Questions for authors
  1. Could you report paired confidence intervals for T=4 versus T=2 and the main component ablations, with pair-clustered resampling for MMVP as in Appendix B.3? If feasible, do the small gains from greater depth persist across training seeds?

  2. Please give the strict decoder interface, initialization, trainable parameters, retained objectives/weights, and generation settings for each baseline, along with evidence of optimization convergence.

  3. Do the same-question swap and crossed-state visual re-read effects persist in generation-based evaluation under the main benchmark settings? This would connect the intervention results to the reported task performance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer rCWC (Part 1/3)

We sincerely appreciate the reviewer's constructive evaluation and recognition of our methodological and empirical contributions. We have conducted additional experiments and analyses to address the concerns raised, as detailed below.


Could you report paired confidence intervals for T=4 versus T=2 and the main component ablations, with pair-clustered resampling for MMVP as in Appendix B.3? If feasible, do the small gains from greater depth persist across training seeds?

We appreciate the reviewer's suggestion regarding statistical reliability. To address this concern, we conducted additional paired bootstrap analyses and multi-seed experiments.

1. Paired Bootstrap Analysis

We performed 10,000 paired bootstrap iterations over 191 V∗V^* examples and 150 MMVP image pairs, using pair-clustered resampling for MMVP. The following table reports accuracy differences in percentage points and their 95% confidence intervals.

ComparisonV∗V^* Overall Δ\Delta [95% CI]MMVP Item Δ\Delta [95% CI]MMVP Pair Δ\Delta [95% CI]
T=4T=4 vs. T=2T=2+2.7 [0.5, 5.2]+0.7 [-0.7, 2.0]+2.0 [0.0, 4.7]
T=4T=4 vs. T=1T=1+5.8 [2.6, 9.4]+14.0 [10.0, 18.0]+30.0 [22.7, 37.3]
w/o Visual Re-read+12.1 [7.3, 16.8]+18.0 [14.3, 22.0]+34.7 [27.3, 42.7]
w/o Native MM Init+44.0 [37.2, 50.8]+24.0 [19.7, 28.3]+50.7 [42.7, 58.7]
w/o Learned Transition+4.8 [2.1, 7.9]+9.0 [5.7, 12.7]+20.0 [14.0, 26.7]
w/o Entire Visual Path+44.0 [37.2, 50.8]+24.7 [20.3, 29.0]+52.7 [44.7, 60.7]

Here, Δ\Delta denotes the accuracy of Full CVRR (T=4T=4) minus that of the compared variant, following the rounded differences in Table 2, whereas the confidence intervals were computed from unrounded paired differences.

The T=4T=4 model improves over T=2T=2 by 2.7 percentage points on V∗V^* and 2.0 percentage points on MMVP Pair Accuracy. As the reviewer noted, since T=1T=1 directly decodes h1h_1 without a trained recurrent adapter, its comparison with T=4T=4 reflects both learned adaptation and repeated computation. We therefore focus on T=2T=2 versus T=4T=4 to assess the contribution of increased recurrent depth. While the MMVP Pair confidence interval includes zero, removing visual re-reading or the learned transition produces substantially larger reductions, with confidence intervals excluding zero. Together, these findings provide complementary evidence that learned recurrent transitions and persistent visual re-reading contribute meaningfully to CVRR's predictive performance. We will describe the role of recurrent depth more carefully in the revised manuscript and incorporate the reviewer's feedback accordingly.

2. Multi-seed Training

To examine training variability, we independently trained T=2T=2 and T=4T=4 models using three seeds (0, 45, 100) under identical training and evaluation settings.

MetricT=2T=2 Mean ± SDT=4T=4 Mean ± SD
V∗V^* Overall78.5 ± 2.181.5 ± 0.6
Direct Attributes80.9 ± 2.382.9 ± 0.5
Relative Position75.0 ± 2.379.4 ± 0.8
MMVP Item72.3 ± 1.574.7 ± 0.7
MMVP Pair47.4 ± 3.152.7 ± 0.7

Across all three seeds, T=4T=4 consistently outperforms T=2T=2 on V∗V^* Overall and MMVP Pair Accuracy, with mean improvements of 3.0 and 5.3 percentage points, respectively. The T=4T=4 model also exhibits lower observed variability. Together with the paired bootstrap results, these findings provide additional evidence that the benefits of recurrent refinement are not limited to a single training seed.

AuthorsOpenReview ↗

Response to Reviewer rCWC (Part 2/3)

Please give the strict decoder interface, initialization, trainable parameters, retained objectives/weights, and generation settings for each baseline, along with evidence of optimization convergence.

We appreciate the reviewer's concern regarding baseline fairness and reproducibility. Our no-bypass evaluation preserves each method's native latent generation and reasoning mechanism while imposing a common constraint that prevents original visual information from bypassing latent computation.

Specifically, each baseline generates latent representations through its original image-question forward pass. We preserve the complete sequence of KK latent tokens and their residual states across all 28 decoder layers. Method-specific boundary tokens and hybrid text–latent reasoning structures are preserved, without introducing additional projection or pooling operations. During answer decoding, these layer-wise representations reconstruct the answer-side latent KV states without retaining the original visual states or multimodal KV caches. The question tokens use image-free KV caches computed from a text-only pass, as in CVRR, so that the latent tokens are the only image-conditioned input to the answer decoder. Since these latent tokens are generated with the image in context, this path remains viable for image-conditioned information to reach the answer.

The evaluated latent configurations and training objectives are summarized below.

MethodLatent RepresentationLatent Tokens (KK)Training Objectives
LVRLast-position Hidden State8Answer CE + Visual Reconstruction MSE
MonetFinal-layer Hidden State10Answer CE + Layer-wise Cosine Alignment Loss
SkiLaSketch Hidden State27Answer CE + Latent Sketch Reconstruction Loss
LaserLast-position Hidden StateUp to 8Answer CE + Dynamic Windowed Alignment Learning
UniVLRLast-position Hidden State12Answer CE + Latent Alignment
HyLaRCanvas Hidden StateUp to 32Answer CE + Canvas Prediction MSE

Each baseline was initialized from its publicly released checkpoint rather than trained from scratch. All six baselines were further fine-tuned on the same Visual CoT dataset for three epochs, using the same SFT settings as CVRR: learning rate 2×10−52 \times 10^{-5}, AdamW, cosine scheduler, warmup ratio 0.05, effective batch size 128, and BF16 precision. Method-specific objectives, auxiliary losses, and their original weights were preserved. Evaluation used greedy decoding with each method's original generation configuration. We will report the complete per-method settings, including loss weights and token budgets, in the appendix of the revised manuscript.

Under the same LoRA configuration (r=32r=32, α=16\alpha=16), each no-bypass baseline uses 80.74M trainable LoRA parameters across 28 decoder blocks, compared with 2.88M in CVRR's single shared block. Thus, the baselines were not disadvantaged by a smaller trainable parameter budget.

We further examined the training loss, including method-specific auxiliary objectives where applicable, across epochs to assess optimization progress.

ModelEpoch 1Epoch 2Epoch 3
LVR3.05581.69151.6818
Monet3.02951.28841.0246
SkiLa8.39515.45342.6955
Laser6.70412.65991.4224
UniVLR6.82381.97570.8424
HyLaR9.10486.64452.6363
CVRR6.17021.72990.9307

All methods exhibit substantial reductions in training loss under the shared three-epoch schedule. For most methods, including CVRR, the loss reduction becomes smaller in the final epoch, suggesting that optimization is approaching stabilization under the same training budget.

Importantly, Figure 1(a) establishes weak latent-state reliance under the original multimodal inference pathways, before any no-bypass modification. Table 1 then evaluates visual competence retention after imposing the strict constraint and retraining. These complementary experiments support our central finding that latent-state informativeness alone does not ensure prediction reliance or competence retention. We will further clarify the implementation details to improve reproducibility.

AuthorsOpenReview ↗

Response to Reviewer rCWC (Part 3/3)

Do the same-question swap and crossed-state visual re-read effects persist in generation-based evaluation under the main benchmark settings? This would connect the intervention results to the reported task performance.

We appreciate the reviewer's suggestion regarding evaluation consistency. We repeated both the Same-question Swap and Crossed-state Visual Re-reading interventions on MMVP using the main benchmark's generation-based evaluation protocol, while retaining the original CVRR checkpoint and strict no-bypass interface.

1. Generation-based Same-question Swap

ConditionAccuracy (%)Drop (pp) [95% CI]
Clean74.7–
Whole-state Swap23.651.1 [44.6, 57.6]
Same-Q Residual Swap25.349.3 [40.7, 58.0]
Text-only Anchor50.024.7 [20.3, 29.0]

The following table compares the original restricted first-token readout in Section 5.3 with generation-based evaluation under the main benchmark protocol.

ConditionFirst-token Readout (%)Generation-based (%)
Clean73.374.7
Whole-state Swap23.623.6
Same-Q Residual Swap26.725.3
Text-only Anchor50.050.0

The generation-based results closely reproduce the intervention patterns observed in Section 5.3. Whole-state Swap yields the same accuracy (23.6%) under both protocols, while Same-Q Residual Swap yields similarly low accuracies (26.7% vs. 25.3%). These consistent results demonstrate that causal reliance on image-conditioned recurrent states persists in actual answer generation, rather than being specific to restricted first-token readout.

2. Generation-based Crossed-state Visual Re-reading

We further evaluated how enabling or blocking question-to-visual connections affects generation under crossed initial states and visual evidence.

InterventionGeneration-based (pp) [95% CI]First-token Readout (pp) [95% CI]
Rescue (ON−OFF)+12.7 [9.3, 16.3]+14.8 [10.6, 19.1]
Corruption (ON-OFF)-12.7 [-16.3, -9.3]-14.8 [−19.1, −10.6]

Providing clean visual evidence to an incorrect h1h_1 yields a 12.7 percentage point Rescue Effect, while incorrect visual evidence provided to a clean h1h_1 produces a corresponding Corruption Effect. The equal magnitudes and sign-mirrored confidence intervals reflect the symmetric construction of the paired crossed-state comparisons. Both effects confirm that visual re-reading causally influences generated predictions.

Together, these results demonstrate that recurrent-state reliance and persistent visual re-reading remain effective under generation-based evaluation, consistent with our original mechanistic analysis.

Reviewer_eTg3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an important research question, whether a latent visual reasoning model actually use the latent reasoning, or just bypass these tokens and answer based on the original inputs. The authors propose Casual Visual Recurrent Reasoning (CVPR), which repeatedly updates the status while re-reading the visual inputs, and remove the original input before the answer decoding. This guaranttees a good performance when the origianl visual inputs are absent and the model has no shortcuts to answer the question.

Critical strengths
  • This paper focus on a very important research question. I always think latent visual reasoning methods would rely more on the original clear visual inputs. Even injecting certain noice into the latent tokens would not affect the performance. This paper provide solid evidence for this and give a good lens, showing all previous methods rely on these shortcuts of different levels.
Critical weaknesses
  • One concern is that, in real senarios, the visual inputs should always be visible. The reasoning trajectory, including the latent visual reasoning, should provide extra evidence to help answer the questions that can not directly be inferred from the raw inputs. I acknowledge the value of the findings. But from my perspective, a good latent visual reasoning policy should teach the model to use the latent visual tokens to solve certain tasks beyond original inputs, not without original inputs. I am wondering when provided with full original inputs, whether CVPR could acheive better performance compared with previous baselines.
Questions for authors
  • In Table 4, CVPR is even worse than Zero-shot performance in a lot benchmarks on larger models. Could authors provide some analysis on this?
  • It would help understanding if the authors could provide results of CVPR reasoning together with the original visual inputs.
2 comments
AuthorsOpenReview ↗

Response to Reviewer eTg3 (Part 1/2)

We sincerely appreciate the reviewer's thoughtful evaluation and recognition of our investigation into bypass pathways in latent visual reasoning. We conducted additional experiments to address the concerns raised, as detailed below.


In Table 4, CVPR is even worse than Zero-shot performance in a lot benchmarks on larger models. Could authors provide some analysis on this?

We appreciate the reviewer's concern regarding performance degradation on larger backbones. As shown in Table 4, similar degradation occurs in the Full Multimodal SFT models trained on the same Visual CoT dataset, even though they retain the standard multimodal answer path. The SFT control falls below its zero-shot reference in 9 of the 16 backbone–benchmark settings, including MMVP Pair Accuracy on three of the four backbones. This suggests that the degradation relative to the zero-shot reference is not unique to CVRR. One possible contributing factor is that, to ensure a controlled comparison, the larger backbones were fine-tuned using the same SFT configuration as the 7B model without backbone-specific tuning, which may not be equally suitable across model scales.

Importantly, the remaining performance gap does not necessarily indicate that CVRR's learned recurrence causes the degradation; instead, recurrent refinement may partially mitigate it. To examine this distinction, we compared the native initialization h1h_1 and the final recurrent state hTh_T under the same strict decoder interface (T=4T=4, β=0.5\beta=0.5). Since h1h_1 is computed by the frozen pretrained layers without recurrent updates, this comparison directly measures the additional contribution of learned recurrence under the strict interface.

BackboneV∗V^* h1→hTh_1 \rightarrow h_TMMVP Pair h1→hTh_1 \rightarrow h_TBLINK h1→hTh_1 \rightarrow h_TMME-RW-Lite h1→hTh_1 \rightarrow h_T
Qwen2.5-VL-32B80.1→80.680.1 \rightarrow 80.6$ 45.3 \rightarrow 47.3$55.4→56.055.4 \rightarrow 56.046.6→46.646.6 \rightarrow 46.6
Gemma-4-31B60.2→63.960.2 \rightarrow 63.961.3→62.061.3 \rightarrow 62.065.8→66.965.8 \rightarrow 66.948.7→51.248.7 \rightarrow 51.2
Gemma-3-27B49.7→51.849.7 \rightarrow 51.824.0→27.324.0 \rightarrow 27.351.9→51.751.9 \rightarrow 51.743.3→44.243.3 \rightarrow 44.2
InternVL3-38B77.5→78.577.5 \rightarrow 78.562.7→66.762.7 \rightarrow 66.760.5→61.260.5 \rightarrow 61.253.8→54.853.8 \rightarrow 54.8

All values represent generation-based accuracy (%).

Recurrent refinement improves accuracy in 14 of the 16 settings, including all V∗V^* and MMVP Pair comparisons. These improvements suggest that the remaining gap to zero-shot performance is more closely associated with the challenge of preserving pretrained visual competence under the strict latent prediction pathway, rather than with learned recurrence itself. Importantly, the learned recurrence helps recover part of this gap by progressively refining the image-conditioned question representation. We will incorporate this analysis into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer eTg3 (Part 2/2)

One concern is that, in real senarios, the visual inputs should always be visible. The reasoning trajectory, including the latent visual reasoning, should provide extra evidence to help answer the questions that can not directly be inferred from the raw inputs. I acknowledge the value of the findings. But from my perspective, a good latent visual reasoning policy should teach the model to use the latent visual tokens to solve certain tasks beyond original inputs, not without original inputs. I am wondering when provided with full original inputs, whether CVPR could acheive better performance compared with previous baselines.

It would help understanding if the authors could provide results of CVPR reasoning together with the original visual inputs.

We appreciate the reviewer's perspective that latent visual reasoning should complement the original visual inputs to improve predictive performance. While CVRR uses a strict interface to establish causal reliance on latent computation, the original image remains accessible during initialization and recurrent reasoning.

Following the reviewer's suggestion, we restored the original multimodal pathway to the trained CVRR checkpoint without retraining. Unlike the causal intervention on corrupted states in Figure 3(a), this experiment evaluates clean predictions using the main generation-based benchmark protocol.

BenchmarkStrict (%)Restored (%)Δ\Delta (pp) [95% CI]
V∗V^* Overall81.282.2+1.0 [-3.1, 5.2]
MMVP Item74.775.3+0.7 [-1.7, 3.0]
MMVP Pair52.754.0+1.3 [-3.3, 6.0]
BLINK Overall55.255.7+0.5 [-0.8, 1.8]
MME-RW-Lite46.747.1+0.3 [-1.0, 1.7]

Here, Δ=Restored−Strict\Delta=\mathrm{Restored}-\mathrm{Strict}, with 95% paired bootstrap CIs (image-pair-clustered for MMVP). Restoring the multimodal pathway yields positive accuracy differences of 0.3–1.3 percentage points across all benchmarks, although the confidence intervals include zero. Differences were computed from unrounded accuracy values.

We further compared restored CVRR against existing latent visual reasoning baselines in Table 1, evaluated with their original multimodal pathways.

MethodV∗V^* OverallMMVP ItemMMVP PairBLINK OverallMME-RW-Lite
LVR81.271.044.751.851.3
Monet79.170.746.046.350.1
SkiLa77.571.747.352.844.7
Laser77.564.732.753.450.8
UniVLR82.770.042.750.251.8
HyLaR75.974.351.353.444.9
PEARL80.675.754.756.747.6
CVRR (Restored)82.275.354.055.747.1

All values represent generation-based accuracy (%). Bold and italics indicate the best and second-best results, respectively.

Restored CVRR achieves the second-highest accuracy on four of the five benchmarks, closely matching PEARL on MMVP Item, MMVP Pair, and BLINK, while outperforming it on V∗V^* Overall. Notably, CVRR achieves this performance by adapting only a single shared decoder layer with 2.88M LoRA parameters (approximately 0.035% of the backbone). These results demonstrate that CVRR can effectively combine latent recurrent reasoning with the original visual pathway while maintaining competitive predictive performance with minimal parameter adaptation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Reviews and Responses

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the importance of our research question (@eTg3), the novel problem perspective and sound methodological design (@UbSm), and the carefully chosen controls and causal evaluations (@rCWC). Following their suggestions, we conducted extensive additional experiments and strengthened the revised manuscript.

We summarize below how we addressed the reviewers' main concerns.

  • Sequential Recurrent Refinement (@UbSm): CVRR outperforms a compute-comparable non-recurrent control with more trainable parameters by 4.2 points on V∗V^* and 6.7 points on MMVP Pair Accuracy. Independently retrained step-wise ablations further demonstrate that visual re-reading at every recurrent step contributes to prediction, with all paired 95% confidence intervals excluding zero (Section 5.2, Table 3).

  • Generation-based Causal Validation (@rCWC): We repeat the same-question swap and crossed-state visual re-reading interventions under the main generation-based evaluation protocol. Same-question swapping reduces MMVP Item Accuracy from 74.7% to 25.3%, while crossed-state interventions yield +12.7-point rescue and −12.7-point corruption effects, confirming that predictions depend on image-conditioned recurrent content and persistent visual evidence (Section 6, Figures 3–4).

  • Statistical Reliability (@rCWC): We provide paired bootstrap confidence intervals and independently retrain recurrent-depth controls across three seeds. The T=4T=4 model consistently outperforms T=2T=2, with mean improvements of 3.0 points on V∗V^* and 5.3 points on MMVP Pair Accuracy, demonstrating that the benefits of recurrent refinement are not limited to a single training seed (Appendix C).

  • Baseline Fairness and Gradient Flow (@rCWC, @UbSm): We clarify the latent interfaces, training objectives, loss weights, and optimization settings of all six no-bypass baselines. Three-seed evaluations confirm consistent performance differences, while answer-CE-only gradient checks verify that answer supervision reaches latent-generation parameters (Appendix B.2).

  • Larger-Backbone Generalization and Multimodal Compatibility (@eTg3): Learned recurrence improves performance in 14 of 16 larger-backbone settings, while conventional full-multimodal SFT also degrades relative to zero-shot performance in 9 of 16 settings. Restoring the original multimodal pathway without retraining preserves competitive performance, achieving the second-highest accuracy on four of five benchmark metrics among latent visual reasoning methods evaluated with their original multimodal pathways (Section 6.4, Appendix F).

Together, these additional experiments strengthen our central conclusion that CVRR makes latent computation necessary for prediction while performing learned, prediction-relevant sequential visual refinement. We sincerely thank the reviewers for their constructive suggestions and respectfully ask the Area Chair to consider the revised manuscript and additional evidence in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.