acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence

Abstract

Latent visual reasoning (LVR) enables multimodal large language models (MLLMs) to perform intermediate computation in continuous latent tokens rather than expressing every reasoning step in words. However, unlike textual CoT, latent reasoning is not directly observable, making it difficult to supervise what latent tokens learn. In this work, we first conduct a thorough analysis of latent-token behavior and identify a latent evidence-credit gap: latent tokens respond only weakly to image perturbations that alter the correct answer. We hypothesize that this issue stems from the lack of explicit supervision during GRPO training. These findings suggest that a final-answer reward provides too little guidance on what visual evidence to preserve or how credit should be assigned across latent tokens. To bridge this gap, we propose ReaLVR, which brings visual-evidence supervision to the model’s own free-running latent trajectories. ReaLVR contrasts correct and model-generated wrong answers to determine where stronger supervision is needed, and relevant and mismatched visual evidence to specify what to preserve. Across three model families, ReaLVR consistently outperforms evaluated LVR baselines, achieving the highest five-task average of 63.7% on Qwen2.5-VL-7B. Crucially, we are the first to scale visual reasoning in latent space, showing that our framework continues to deliver robust improvements at frontier model scales up to 235B. Further analyses show more question-sensitive latent-token positions, stronger alignment with relevant visual regions, and greater fixed-context dependence on the most attended latent tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
visual reasoning, representation learning, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WSJbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies the “visual evidence–credit assignment gap” in latent visual reasoning (LVR), where models may produce correct answers without encoding or using decisive visual evidence in their latent tokens. To address this issue, the authors propose ReaLVR, which selects latent tokens based on attention differences between correct and incorrect answers and uses contrastive learning with relevant and mismatched visual evidence to supervise them, without modifying the model architecture or inference procedure. Experiments across Qwen, InternVL, and Gemma models—including a 235B-parameter model—combine benchmark comparisons, component ablations, and mechanistic analyses to evaluate whether ReaLVR improves the visual sensitivity and causal relevance of latent representations.

Critical strengths

Counterfactual image edits, latent-token replacement, and component ablations provide useful evidence that the tested LVR baselines are insufficiently sensitive to key visual changes, while ReaLVR increases the visual responsiveness of latent states and the answer’s dependence on them.

Critical weaknesses
  1. Limited and non-uniform advantage over the strongest alternative. On Qwen2.5-VL-7B, ReaLVR outperforms ILVR on MMVP, HRBench-4K, and MME-RealWorld, but underperforms it on BLINK and HRBench-8K. Its five-task average improves by only 0.8 percentage points. Thus, ReaLVR provides genuine gains, but does not demonstrate consistent superiority over the strongest alternative. Without an equal-compute comparison, the magnitude and practical significance of this advantage remain unclear.

  2. The 235B evaluation does not isolate the contribution of ReaLVR. For Qwen3-VL-235B, Table 2 compares ReaLVR only against Direct decoding and LVR-SFT, and evaluates only three benchmarks. A matched LVR-RL baseline is missing. Consequently, the reported gains cannot be attributed specifically to the proposed evidence-credit objective rather than to Stage-2 RL more generally. The experiment establishes feasibility at this scale, but not a controlled advantage of ReaLVR.

Questions for authors
  1. Was ILVR reproduced from the same Stage-1 checkpoint with matched update steps, rollout budget, image-token limit, data order, and checkpoint-selection procedure? Could the authors report paired-seed differences?

  2. What are ReaLVR’s additional GPU-hours, wall-clock time, throughput, and peak-memory requirements relative to ILVR or LVR-RL?

  3. Could the authors provide a matched LVR-RL baseline for Qwen3-VL-235B? Without it, how can the reported gain be separated from generic Stage-2 RL?

2 comments
AuthorsOpenReview ↗

Author response to Reviewer WSJb (Part 1/2)

We sincerely appreciate your careful review and your recognition of the counterfactual, latent-replacement, and ablation evidence. Thank you for identifying the comparisons needed to clarify the practical benefit and the 235B result. We address each point below.

W1/Q1. Comparison with ILVR and the size of the improvement.

ReaLVR improves the five-task average over ILVR-Stage2, the interleaved latent-reasoning baseline, under matched training conditions. The following results are reproduced from the original manuscript's Table 1 for Qwen2.5-VL-7B:

MethodMMVP (%) ↑BLINK (%) ↑HR-4K (%) ↑HR-8K (%) ↑MME-RW (%) ↑Five-task mean (%) ↑
LVR-RL64.2 ± 0.453.6 ± 0.369.6 ± 0.364.4 ± 0.350.1 ± 0.360.4 ± 0.2
ILVR-Stage269.4 ± 0.456.8 ± 0.371.0 ± 0.266.9 ± 0.350.3 ± 0.362.9 ± 0.2
ReaLVR72.0 ± 0.455.8 ± 0.371.8 ± 0.366.6 ± 0.352.2 ± 0.263.7 ± 0.2

The benefit is in average performance rather than every benchmark. The original manuscript's Table 2 also reports average gains over LVR-RL of +3.1 pp on InternVL3-8B and +2.6 pp on Gemma-3-12B.

We verified the following shared settings for the ILVR-Stage2 and ReaLVR comparison:

Training conditionShared settingMatch
Stage-1 initializationqwen2.5-vl-7b-lvr-sftIdentical
Stage-2 update count100 updatesIdentical
Rollout budget per promptG = 8, temperature 0.6Identical
Prompts per update512 (64 nodes ×\times 8 GCDs)Identical
Image-token limit2,560 tokens (2.0 MP2.0\text{ MP})Identical
Training data & orderViRL39K + Visual-CoT mixture; same seed and batch orderIdentical
Checkpoint selectionBest on 500-val splitIdentical

The within-seed differences below are ReaLVR minus ILVR-Stage2. Positive values favor ReaLVR:

Paired training seedΔ MMVP (pp)Δ BLINK (pp)Δ HR-4K (pp)Δ HR-8K (pp)Δ MME-RW (pp)Δ five-task mean (pp)
Seed 42+2.7−0.9+0.9−0.4+2.1+0.88
Seed 123+2.4−1.1+0.6−0.2+1.7+0.68
Seed 999+2.8−1.0+0.8−0.3+1.9+0.84
Mean ± SD+2.63 ± 0.21−1.00 ± 0.10+0.77 ± 0.15−0.30 ± 0.10+1.90 ± 0.20+0.80 ± 0.11

The five-task average improvement is positive in all three paired runs.

W1/Q2. Equal-compute comparison and training overhead.

At the same update count, ReaLVR costs less than ILVR-Stage2 and more than LVR-RL; under the same compute budget, it retains the highest five-task average. We profiled 100 Stage-2 updates on 64 nodes (512 AMD MI250X GCDs, global batch 512). A GCD-hour counts one ROCm-visible compute die for one hour.

Method, 100 Stage-2 updatesTotal GCD-hours ↓Additional GCD-hrs vs. LVR-RLWall-clock hours ↓Throughput (prompts/s) ↑Peak memory (GB/GCD) ↓
LVR-RL1,2340 (reference)2.41 h5.8946.4
ILVR-Stage21,512+278 (+22.5%)2.95 h4.8152.8
ReaLVR1,398+164 (+13.3%)2.73 h5.2149.7

For the equal-compute comparison, all methods receive a Stage-2 budget of B = 1,234 GCD-hours per seed. Results are from three training seeds. We select the best validation checkpoint reached within each method's budget using the same selection rule:

Method, equal Stage-2 budgetAchieved GCD-hrsUpdates completedMMVP (%) ↑BLINK (%) ↑HR-4K (%) ↑HR-8K (%) ↑MME-RW (%) ↑Five-task mean (%) ↑
LVR-RL1,23410064.2 ± 0.453.6 ± 0.369.6 ± 0.364.4 ± 0.350.1 ± 0.360.4 ± 0.2
ILVR-Stage21,2328168.3 ± 0.455.9 ± 0.370.1 ± 0.366.1 ± 0.449.5 ± 0.362.0 ± 0.2
ReaLVR1,2318871.4 ± 0.455.3 ± 0.371.2 ± 0.366.1 ± 0.351.6 ± 0.263.1 ± 0.2

The average-accuracy advantage therefore persists when the methods receive the same training-compute budget.

AuthorsOpenReview ↗

Author response to Reviewer WSJb (Part 2/2)

W2/Q3. Isolating the evidence objective at 235B.

We have completed the matched 235B LVR-RL control: it removes the evidence objective while retaining the other training conditions. The public LVR implementation did not support our distributed setup at submission, and the necessary adaptation is now complete.

The new control uses Qwen3-VL-235B-A22B on 200 nodes (1,600 GCDs). LVR-RL and ReaLVR share the Stage-1 checkpoint, training data and order, update and rollout budgets, image-token limit, and checkpoint-selection rule. The other task scores come from the original manuscript's Table 2; three-task averages are computed from the displayed task scores.

MethodSourceMMVP (%) ↑BLINK (%) ↑MME-RealWorld (%) ↑Three-task average (%) ↑
Direct decodingOriginal Table 280.073.569.874.4
LVR-SFTOriginal Table 280.974.270.475.2
LVR-RLNew matched control (λev=0\lambda_{\mathrm{ev}}=0)81.374.770.675.5
ReaLVROriginal Table 281.975.471.076.1
ReaLVR − matched LVR-RLObserved additional gain+0.6 pp+0.7 pp+0.4 pp+0.57 pp

Across these three benchmarks, outcome-only RL improves over LVR-SFT by +0.37 pp on average, and evidence supervision provides a further +0.57 pp over the matched LVR-RL control. We will include this control in Table 2 of the revised manuscript.

Thank you again for your time and valuable feedback. We hope the above content can solve your doubts. We have updated all the newly added experiments. Please let us know if there are further questions.

Reviewer_cqukOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the lack of visual grounding in latent visual reasoning (LVR). The authors identify a latent evidence-credit gap, where latent tokens fail to capture answer-relevant visual evidence. To address this issue, they propose ReaLVR, a framework that uses answer contrast to identify important latent positions and visual contrast to supervise their representations. Experiments across five benchmarks and six model backbones demonstrate consistent improvements over existing LVR methods. Additional analyses show stronger visual grounding, greater sensitivity to image changes, and increased dependence on latent tokens during answer generation.

Critical strengths
  1. ReaLVR applies contrastive visual supervision directly to autoregressively generated latent trajectories. The objective encourages latent states to align with relevant visual evidence while distinguishing them from mismatched visual prototypes. Gradients propagate through the latent generation process, reducing the discrepancy between teacher-forced supervision and free-running inference. The method requires no architectural modifications or additional inference operations.
  2. The experiments cover six backbones across three model families, with model sizes ranging from 7B to 235B parameters. On Qwen2.5-VL-7B, ReaLVR achieves an average accuracy of 63.7% across five benchmarks, outperforming LVR-RL by 3.3 percentage points and Monet-RL by 1.5 percentage points. The improvements also extend to Qwen3-VL, InternVL3, and Gemma-3, demonstrating the applicability of the method across different architectures and model scales.
  3. The ablation experiments in Table F.2 isolate the effects of answer-contrastive weighting, negative visual prototypes, on-policy latent regeneration, and gradient detachment. Removing these components reduces the average accuracy by 0.5–1.8 percentage points. The additional experiments in Table J.3 control the total supervision weight and separately evaluate position selection and visual contrast. These results provide evidence for the effectiveness of the individual components.
Critical weaknesses
  1. On Qwen2.5-VL-7B, ReaLVR achieves 63.7% average accuracy, only 0.8 percentage points above ILVR-Stage2. It also performs worse than ILVR-Stage2 on BLINK (55.8% vs. 56.8%) and HRBench-8K (66.6% vs. 66.9%). On Qwen3-VL-8B, the improvement over LVR-RL is only 0.6 percentage points, with lower accuracy on MME-RealWorld (49.3% vs. 49.7%). These results suggest that the advantages of ReaLVR are not consistent across individual benchmarks. The authors should further investigate why the proposed supervision provides limited or negative improvements on certain tasks.
  2. Section 4.3 determines latent-token supervision weights by comparing the attention distributions of correct and incorrect answers. However, attention differences do not necessarily reflect the actual contribution of each latent token to answer generation. The token-replacement experiments in Appendix I.3 provide supporting evidence, but they do not establish that attention-based credit assignment is optimal or consistently reliable. The authors should compare the proposed weighting strategy with gradient-based attribution, leave-one-out importance, or other methods that estimate the contribution of individual latent tokens.
  3. Section 4.2 constructs a single positive visual prototype by averaging the visual embeddings within the annotated region. All latent tokens are supervised using the same prototype, with only their supervision weights varying across positions. This design does not explicitly encourage different latent tokens to capture distinct objects, spatial relationships, or intermediate reasoning steps. For questions requiring multiple pieces of visual evidence, a single pooled representation may also discard important spatial information. The authors should investigate position-specific visual targets or multiple region-level prototypes to determine whether more structured supervision improves complex visual reasoning.
Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Author response to Reviewer cquk (Part 1/2)

We sincerely appreciate your careful review and your recognition of the supervision on free-running latent trajectories, the broad model coverage, and the component analyses. Below, we address your three concerns.

W1. Why are the gains limited or negative on some benchmarks?

A plausible explanation is that different tasks require different granularity of visual evidence. Shared pooling provides a visual target for generated latents, but does not explicitly preserve separate region identities, cross-view correspondences, or fine text structure. This may help explain why spatial relation, relative depth, and counting improve, while multi-view correspondence and dense OCR / complex charts are weaker. BLINK likewise evaluates these as distinct visual capabilities [1].

We ran new training experiments with three paired seeds to obtain the subtask breakdown below, extending the original manuscript's benchmark-level comparisons. The comparator is ILVR-Stage2 for Qwen2.5-VL-7B and LVR-RL for Qwen3-VL-8B. Accuracies are mean ± SD; the last column gives the paired difference, with positive values favoring ReaLVR.

Backbone / benchmarkEvaluated subsetNComparator acc. (%) ↑ReaLVR acc. (%) ↑ReaLVR − comparator (pp)
Qwen2.5-VL-7B / BLINKAll examples69756.8 ± 0.355.8 ± 0.3−1.00 ± 0.10 pp
Qwen2.5-VL-7B / BLINKSpatial Relation & Relative Depth23253.4 ± 0.455.6 ± 0.4+2.16 ± 0.38 pp
Qwen2.5-VL-7B / BLINKObject Counting & Jigsaw21858.3 ± 0.360.1 ± 0.3+1.83 ± 0.42 pp
Qwen2.5-VL-7B / BLINKMulti-view & Semantic Corresp.24758.7 ± 0.452.2 ± 0.4−6.48 ± 0.45 pp
Qwen2.5-VL-7B / HR-8KAll examples1,44066.9 ± 0.366.6 ± 0.3−0.30 ± 0.10 pp
Qwen2.5-VL-7B / HR-8KSpatial Layout & Fine Search83466.8 ± 0.368.3 ± 0.3+1.56 ± 0.28 pp
Qwen2.5-VL-7B / HR-8KDense OCR & Complex Charts60667.0 ± 0.464.2 ± 0.4−2.86 ± 0.32 pp
Qwen3-VL-8B / MME-RWAll examples1,00049.7 ± 0.349.3 ± 0.2−0.40 ± 0.14 pp
Qwen3-VL-8B / MME-RWReal-world Visual Perception58650.1 ± 0.351.5 ± 0.3+1.37 ± 0.26 pp
Qwen3-VL-8B / MME-RWDocument Parsing & Visual Math41449.1 ± 0.446.2 ± 0.3−2.90 ± 0.31 pp

W3 tests whether preserving separate regions helps within the same evidence-supervision framework. It reports overall benchmark and multi-evidence results, without a separate breakdown of the weak multi-view/correspondence and dense OCR/chart subtasks above. Those results support finer regional supervision in the tested setting; the causes of the individual subtask deficits remain an interpretation to investigate.

The original manuscript's Table F.1 also reports a +2.3 pp BLINK improvement from K=8K=8 to K=16K=16 on a fixed evaluation subset. This subset-specific result motivates task-dependent latent budgets, consistent with findings on difficulty-dependent compute allocation in language-model reasoning [2], without attributing the original Table 1 deficits to budget alone.

W2. How reliable is answer-contrastive weighting compared with other importance estimators?

Answer contrast achieves the highest mean accuracy in this comparison and trains faster than gradient attribution and LOO, though more slowly than uniform routing. We evaluated the following alternative supervision weights from the same Qwen2.5-VL-7B Stage-1 checkpoint under positive–negative visual contrast. Following Section 4.4 and Appendix J.3 of the original paper, each uses the normalized routing allocation:

qt=η/K+(1−η)γtη+(1−η)∑s=1Kγs,with fallback qt=1K if ∑sγs=0,q_t = \frac{\eta/K + (1-\eta)\gamma_t}{\eta + (1-\eta)\sum_{s=1}^K \gamma_s}, \quad \text{with fallback } q_t = \frac{1}{K} \text{ if } \sum_s \gamma_s = 0,

where γt\gamma_t is defined below and η=0.3\eta=0.3:

  1. Gradient-based attribution: γtgrad=∥∇ztlog⁡πθ(y∗∣x,z1:K)∥2\gamma_t^{\mathrm{grad}} = \Vert{}\nabla_{z_t} \log \pi_\theta(y^* \vert{} x, z_{1:K})\Vert{}_2.
  2. Leave-one-out importance: γtLOO=[log⁡πθ(y∗∣x,z1:K)−log⁡πθ(y∗∣x,z1:K(\t))]+\gamma_t^{\mathrm{LOO}} = [\log \pi_\theta(y^* \vert{} x, z_{1:K}) - \log \pi_\theta(y^* \vert{} x, z_{1:K}^{(\backslash t)})]_+, measuring local sensitivity under token replacement.
  3. Answer contrast (Ours): γt=[rt+−rt−]+\gamma_t = [r_t^+ - r_t^-]_+.
AuthorsOpenReview ↗

Author response to Reviewer cquk (Part 2/2)

Training-time position weightsMMVP (%) ↑BLINK (%) ↑HR-4K (%) ↑HR-8K (%) ↑MME-RW (%) ↑Mean (%) ↑Train time (s/step) ↓
Uniform routing (qt=1/Kq_t = 1/K)69.5 ± 0.454.6 ± 0.371.0 ± 0.365.6 ± 0.351.3 ± 0.262.4 ± 0.286.4 s
Raw correct-answer attention70.3 ± 0.354.9 ± 0.371.2 ± 0.265.8 ± 0.351.5 ± 0.262.7 ± 0.291.8 s
Gradient-based attribution70.7 ± 0.455.1 ± 0.371.4 ± 0.266.0 ± 0.351.8 ± 0.363.0 ± 0.2117.2 s
Leave-one-out importance71.3 ± 0.355.4 ± 0.271.6 ± 0.366.3 ± 0.251.9 ± 0.263.3 ± 0.2162.5 s
Answer contrast (ReaLVR)71.8 ± 0.455.7 ± 0.371.7 ± 0.266.5 ± 0.352.1 ± 0.263.6 ± 0.296.8 s

This comparison supports answer contrast as an effective balance between downstream accuracy and training cost when allocating visual supervision. This use of controlled comparisons is consistent with prior work on evaluating attention's usefulness [3].

W3. Would multiple or position-specific visual prototypes better support complex reasoning?

In response to your suggestion, multiple region prototypes improve every accuracy column in the comparison below, whereas fixed position-specific targets underperform the shared target. We tested these alternatives on Qwen2.5-VL-7B using the multi-box Visual-CoT subset (Ntrain=18,432N_{\mathrm{train}} = 18,432) under the same 100-update schedule:

  1. Shared pooled prototype: Pool the features of all annotated regions into one visual target, shared across latent positions.
  2. Multiple region prototypes: Multiple-region targets preserve a separate prototype for each annotated region and match regions to latent positions based on cosine similarity.
  3. Position-specific targets: The position-specific variant instead uses a predefined region-to-latent assignment.

All three variants use the same initialization, training subset, seeds, data order, training budget, and other training settings; only the visual-target construction differs. We evaluate all five benchmarks and a Multiple-Evidence Evaluation Subset (N=540N=540 across BLINK, HR-8K, and MME-RW), manually selected using existing annotations to identify questions requiring reasoning over 2+ spatial entities. This is a separate experiment from the original main comparison and the attribution-weight experiment above.

Visual target constructionMMVP (%) ↑BLINK (%) ↑HR-4K (%) ↑HR-8K (%) ↑MME-RW (%) ↑Five-task mean (%) ↑Multi-evidence subset (%) ↑
Shared pooled prototype71.6 ± 0.455.5 ± 0.371.5 ± 0.266.3 ± 0.351.9 ± 0.263.4 ± 0.258.2 ± 0.4
Multiple region prototypes72.1 ± 0.356.3 ± 0.371.8 ± 0.367.1 ± 0.352.2 ± 0.263.9 ± 0.262.4 ± 0.4
Position-specific visual targets70.4 ± 0.454.6 ± 0.370.9 ± 0.365.5 ± 0.451.1 ± 0.362.5 ± 0.356.5 ± 0.5

Keeping separate region targets reduces the mixing of different regions' evidence while allowing their assignments to follow the input. The shared target supplies an aggregate visual constraint without prescribing a role for each latent position. The fixed variant instead assumes that a given box order should map to a given latent order; a plausible reason for its poorer performance is that latent roles vary across inputs. Flexible entity binding also appears in object-centric learning [4].

Visual-evidence supervision of the model's own generated latent trajectory remains the core contribution. These results identify a better target construction within that framework and support finer regional supervision in this multi-evidence setting. We will include this analysis in Appendix J.4.

Thank you again for your time and valuable feedback. We hope the above content can solve your doubts. We have updated all the newly added experiments. Please let us know if there are further questions.

References

[1] 'BLINK: Multimodal Large Language Models Can See but Not Perceive', ECCV 2024.

[2] 'Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Parameters for Reasoning', ICLR 2025.

[3] 'Attention is not not Explanation', EMNLP-IJCNLP 2019.

[4] 'Object-Centric Learning with Slot Attention', NeurIPS 2020.

Reviewer_pFTPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies latent visual reasoning (LVR), where an MLLM reasons through interleaved spans of continuous latent tokens and textual answer. The authors argue that standard two-stage LVR training leaves a “latent evidence-credit gap”: In GRPO Stage, the sampled latent tokens are treated as fixed context, so the free-running latent trajectory used at inference gets no direct visual supervision. The paper proposed ReaLVR, which adds an auxiliary evidence loss to GRPO without changing the architecture or the inference procedure. Across five benchmarks and six backbones from three families, up to Qwen3-VL-235B, ReaLVR improves over LVR baselines.

Critical strengths
  1. A clear diagnosis with a targeted fix. The introduced loss term fixed a blind spot in the LVR RL training and introduced no extra training or infra effort. The proposed onpolicy latent rollouts also minimizes the inference2training gap.

  2. The method scaled LVR to frontier models and 200+B size. Scaling up is essential to show the effectiveness of model structures.

Critical weaknesses
  1. The source of the gains is unclear. The paper reports neither zero-shot results, without which, the improvement cannot be attributed to the evidence loss.
  2. The counterfactual test (App. J.1) keeps the first-turn answer in context, so LVR’s drop to 8% on edited images may reflect copying its previous answer rather than visual insensitivity. Moreover, the token axes of the H.1–H.4 maps show a textual <think> rationale and no latent or image tokens, so they do not visualize the claimed latent reasoning.
Questions for authors
  1. What are the counterfactual results under single-turn evaluation?
  2. Are the latents identical across the G rollouts?
  3. What is the training-time overhead of regeneration and multi-branch attention extraction? Did you switch to eager attention?
  4. What fraction of training examples has ROI annotations rather than the whole-image fallback?
3 comments
AuthorsOpenReview ↗

Author response to Reviewer pFTP (Part 1/3)

We sincerely appreciate the time and effort you devoted to reviewing our manuscript. Thank you for recognizing the targeted motivation, the supervision of free-running latent trajectories, and the evaluation at larger model scales. Below, we address each of your concerns and questions.

W1. Zero-shot performance and the source of the gains.

We added the pretrained model's zero-shot performance; the controlled comparison with outcome-only RL supports the contribution of evidence supervision. The new zero-shot evaluation uses the same benchmark suite. The mean accuracies in the other three rows are reproduced from the original manuscript's Table 1. The original manuscript's Table F.2 also identifies LVR-RL as the λev=0\lambda_{\mathrm{ev}}=0 control.

Qwen2.5-VL-7BSourceMMVP (%) ↑BLINK (%) ↑HR-4K (%) ↑HR-8K (%) ↑MME-RW (%) ↑Five-task mean (%) ↑
Zero-shot direct decodingNew evaluation61.450.867.261.547.957.8
LVR-SFTOriginal Table 163.653.269.063.349.559.7
LVR-RLOriginal Table 164.253.669.664.450.160.4
ReaLVROriginal Table 172.055.871.866.652.263.7

The zero-shot row places the trained models in context; the ReaLVR–LVR-RL comparison isolates the additional evidence objective.

The following selected rows from the original manuscript's Table J.3 further distinguish where visual supervision is allocated from what visual evidence it preserves, with equal total supervision weight per example:

Position weightsVisual objectiveFive-task mean (%) ↑
UniformPositive-only61.5
UniformPositive–negative contrast62.5
Answer contrastPositive–negative contrast63.6

Both contrastive visual targets and answer-dependent allocation contribute under this controlled setting.

W2(a)/Q1. Counterfactual evaluation without the previous answer.

Following your suggestion, we evaluated each original and edited image in a separate, fresh single-turn conversation. ReaLVR retains the highest edited-image accuracy and strict paired correct-flip rate across all four edit types. We use the same 2,048 pairs as the original manuscript's Appendix J.1. Direct is a no-latent control trained with the same backbone, Stage-2 data, answer reward, and update budget, allowing comparison with adaptation that does not use latent reasoning.

Each edit type contains 512 pairs. The changed/unchanged-answer counts are 442/70 (color), 417/95 (removal), 433/79 (shape), and 435/77 (spatial). Edited accuracy and strict paired correct flip use changed-answer pairs; the latter requires both original and edited views to be answered correctly. False flip measures prediction changes on unchanged-answer pairs.

New single-turn evaluation — primary outcomes. Bold denotes the best value within each edit type, including ties.

EditMethodEdited acc. (%) ↑Strict paired correct flip (%) ↑False flip (%) ↓
Color changeDirect71.954.811.4 (8/70)
Color changeLVR-RL24.28.84.3 (3/70)
Color changeReaLVR81.773.85.7 (4/70)
Object removalDirect69.353.712.6 (12/95)
Object removalLVR-RL26.69.46.3 (6/95)
Object removalReaLVR80.170.76.3 (6/95)
Shape swapDirect70.752.911.4 (9/79)
Shape swapLVR-RL24.98.37.6 (6/79)
Shape swapReaLVR79.271.15.1 (4/79)
Spatial swapDirect68.550.813.0 (10/77)
Spatial swapLVR-RL23.47.46.5 (5/77)
Spatial swapReaLVR77.969.26.5 (5/77)

These results show that ReaLVR's advantage in correctly responding to edited visual evidence also holds in the independent single-turn setting. Its false-flip rate is lower than Direct's in every edit category, although it is not always the lowest among all methods. Our original dialogue evaluation examined visual changes within a conversation; we appreciate this complementary test and will include both protocols in the revised Appendix J.1.

AuthorsOpenReview ↗

Author response to Reviewer pFTP (Part 2/3)

Additional diagnostics for the same single-turn runs. Original accuracy and prediction change use changed-answer pairs. Prediction change counts different parseable predictions and is not itself a correctness metric. Parse coverage includes both views of all pairs; parse failures count as incorrect and not as valid prediction changes.

EditMethodOriginal acc. (%) ↑Prediction change (%)Parse cov. (%) ↑
Color changeDirect75.863.399.8
Color changeLVR-RL78.512.499.8
Color changeReaLVR85.381.4100.0
Object removalDirect74.365.299.5
Object removalLVR-RL77.216.599.8
Object removalReaLVR84.478.4100.0
Shape swapDirect75.162.199.8
Shape swapLVR-RL77.814.8100.0
Shape swapReaLVR83.879.2100.0
Spatial swapDirect73.860.299.5
Spatial swapLVR-RL76.613.699.8
Spatial swapReaLVR82.877.5100.0

W2(b). The token axes in Figures H.1–H.4.

Thank you for pointing this out. Figures H.1–H.4 are supplementary case studies comparing how ReaLVR and LVR answer the same visual questions, alongside attention and saliency views of their textual readout. For example, in Case 08 ("What is touching the bed?"), ReaLVR correctly answers "teddy bear," while LVR answers "blanket." Other cases have the same answer but different token-interaction patterns. These examples illustrate the answer-level visual discrimination that motivates evidence supervision; the maps provide descriptive context for the displayed responses.

These supplementary case studies pair model outputs in the upper boxes with text-token attention and saliency maps below. Some maps include model-generated explanations marked by <think>, while others show text-token interactions leading directly to the answer. These maps illustrate response-level text interactions; analyses of continuous latent states are presented separately in Section 5.3 and Appendix I. The axes index the displayed text tokens: rows are queries and columns are keys. Attention shows query–key weights; saliency is the magnitude of attention multiplied by its gradient under the generated-answer objective (original Appendix G.3). Because these panels examine textual readout, they do not display continuous latent states. We will clarify this purpose and scope in their descriptions; the analyses of actual latent states are presented separately in Section 5.3 and Appendix I.

Q2. Are the latents identical across the G rollouts?

Yes, for a given image-question prompt, the generated latent vectors z1:Krollz_{1:K}^{\mathrm{roll}} are numerically identical across all G=8 rollouts. During Stage-2 rollouts, the model runs with dropout disabled, and the latent span z1,…,zKz_1,\dots,z_K is generated autoregressively by passing continuous hidden states directly without discrete token sampling. The sampling temperature (T=0.6) and nucleus filtering (p=1.0) are applied strictly after <|lvr_end|> to sample the discrete answer strings oio_i. Thus, stochasticity resides entirely in the post-latent answer tokens: the G completions share one latent trajectory, rather than representing independent latent paths. In parallel, the evidence branch regenerates a current-model trajectory with gradients retained (Section 4.1).

Q3. Training overhead and attention extraction.

ReaLVR adds training overhead while leaving inference-time computation unchanged. We profiled both methods for 100 Stage-2 updates on 64 nodes (512 AMD MI250X GCDs, global batch 512):

Qwen2.5-VL-7B, matched Stage-2 workloadLVR-RLReaLVRReaLVR − LVR-RL
Total training device-hours (GCD-hours) ↓1,2341,398+164 (+13.3%)
Wall-clock time (hours) ↓2.41 h2.73 h+0.32 h (+13.3%)
Throughput (prompts/s) ↑5.895.21−0.68 prompts/s
Peak memory (GB per GCD) ↓46.449.7+3.3 GB (within 64 GB cap)
Differentiable latent regeneration (s/update)—3.24—
Additional answer-readout branches (s/update)—6.81—

A GCD-hour counts one ROCm-visible compute die for one hour, following the original manuscript's Appendix A.

Both methods retain PyTorch SDPA for model attention. ReaLVR uses a separate computation to extract answer-query attention, normalized over all visible keys, and retain the K=8 latent columns; it does not switch the model to eager attention. This extraction cost is included in the reported training profile. Overall, training time and device-hours increase by 13.3%, peak memory increases by 3.3 GB, and inference computation is unchanged.

AuthorsOpenReview ↗

Author response to Reviewer pFTP (Part 3/3)

Q4. ROI annotations versus whole-image fallback.

Valid ROI masks cover 58.6% of sampled training prompts; the remaining 41.4% use whole-image supervision. A non-empty mask selects regional features; absent annotations or empty masks trigger whole-image pooling. The Stage-2 mixture contains 51,200 sampled prompts (100 updates ×\times 512 prompts), drawn 60:40 from Visual-CoT and ViRL39K:

Training sourceSampled training promptsPrompts with a valid ROI maskWhole-image fallback promptsValid-ROI share (%)
Visual-CoT30,72029,983737 (empty patch mask)97.6%
ViRL39K20,480020,480 (no native ROI, a=∅a=\emptyset)0.0%
Total51,20029,98321,21758.6%

Visual-CoT supplies bounding boxes, with 2.4% yielding empty visual-token masks after preprocessing; ViRL39K has no native region boxes. Counts include repeated sampled prompts and count each prompt once, without multiplying by its G answer rollouts. The fallback allows the same objective to use both region-annotated and box-free reasoning data, without requiring boxes for every training source.

Thank you again for your time and valuable feedback. We hope the above content can solve your doubts. We have updated all the newly added experiments. Please let us know if there are further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.