acceptodds
Under review as a conference paper at ICLR 2027

ELVIS: Evidence-Grounded Latent Visual States for Multi-Step Reasoning

Abstract

Multi-step multimodal reasoning requires sustained access to visual evidence, yet recurrent methods, constructing latent visual states sequentially, show weak sensitivity to decisive evidence and provide limited support for subsequent reasoning in our diagnostics. We introduce ELVIS, a framework that constructs evidence-grounded latent visual states in parallel and invokes new states as reasoning progresses. Learnable queries interact bidirectionally within each state while preserving causality across reasoning steps. To supervise state content, we construct AVR-112K, an Action–Visual–Readout corpus whose explicit observations guide teacher-based evidence selection. Evidence alignment and visual anchoring train latent states to capture the selected evidence and attend to the original image. An Invocation Progressive Reward encourages additional invocations when they yield higher empirical success than smaller observed invocation counts. Experiments demonstrate strong performance across visual perception and spatial reasoning tasks, including a 22.04% gain over Qwen2.5-VL-7B on Zebra Maze. At 16 latent positions and matched invocation counts, ELVIS achieves an 11.65–17.40× speedup in latent-state computation over the fastest applicable recurrent baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Reasoning, Latent Visual Reasoning, Multimodal Understanding

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_LqtbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to solve the limitations of sequential visual latent tokens, especially the weak sensitivity to decisive evidence and limited support for subsequent reasoning. This paper constructs a dataset AVR-112K and proposes a framework called ELVIS that constructs evidence grounded latent visual states in parallel and invokes new states as reasoning progresses. Diverse benchmarks, ablations, and analyses show the effectiveness of the proposed method.

Critical strengths
  1. The paper addresses important limitations of existing latent visual reasoning methods. The proposed architecture is well motivated.

  2. The experimental evaluation is comprehensive and the component ablations and latent transplantation experiments provide useful evidence for the effectiveness of the method.

Critical weaknesses

The comparison with vanilla SFT is not compute-matched, as ELVIS needs an additional computation budget, such as extra SFT stages. Notably, on Zebra Maze, vanilla SFT already improves accuracy from 15.25% to 35.59%, while the full ELVIS only further improves it to 37.29%. The paper does not provide budget-controlled comparisons with vanilla SFT. Consequently, it remains unclear whether the proposed latent mechanism provides sufficient benefits to justify its additional computational and training complexity.

Questions for authors
  1. Could the authors compare against a non-latent VLM trained with the same total computational budget, for example through additional SFT or SFT followed by GRPO? How much of the improvement can be attributed specifically to latent reasoning rather than additional training?

  2. Could the authors report end-to-end inference latency and computational cost relative to a standard VLM without latent states? Does ELVIS still provide a good accuracy–efficiency trade-off under matched inference budgets?

Minor: The AI-use statement is currently placed in Appendix A.5 rather than before the references as recommended by the template. Additionally, it would be better if you stated the data processing via Qwen2-VL (Sec. 3.3) in the AI use statement section.

2 comments
AuthorsOpenReview ↗

Part 1/2: Response to Reviewer Lqtb

W1 & Q1: Training-Budget-Matched Comparisons with Non-Latent Baselines

We thank the reviewer for raising this important concern. We first clarify that the original vanilla SFT baseline corresponds to the first-stage SFT (SFT1) of ELVIS. To examine whether ELVIS's improvements arise from additional training, we have added comparisons with non-latent baselines under matched training steps in Appendix A.5 (Tab. 11). Specifically, starting from the shared SFT1 checkpoint (3,500 steps), we continue training the vanilla model for another 1,766 steps (matching ELVIS's SFT2 stage) on AVR-112K using the standard CE objective. We further apply vanilla GRPO to both models under the same RL setup, enabling controlled comparisons at both the SFT and RL stages.

Table 11. Training-budget-matched comparisons between ELVIS and non-latent baselines.

ModelSFT(1+2) StepsRLV*HR4KJigsawMaze
Vanilla (Shared SFT1)3,500--79.0668.1348.6735.59
Vanilla (SFT1+2)3,500 + 1,766--79.5870.1246.6724.58
Vanilla (SFT1+2 + RL)3,500 + 1,766GRPO82.2069.3848.0027.97
ELVIS (SFT1+2)3,500 + 1,766--84.8271.8854.0036.44
Δ ELVIS - Vanilla (SFT)+5.24\color{green}{+5.24}+1.76\color{green}{+1.76}+7.33\color{green}{+7.33}+11.86\color{green}{+11.86}
ELVIS (SFT1+2 + RL)3,500 + 1,766GRPO83.2571.1356.6734.75
Δ ELVIS - Vanilla (GRPO)+1.05\color{green}{+1.05}+1.75\color{green}{+1.75}+8.67\color{green}{+8.67}+6.78\color{green}{+6.78}
ELVIS (SFT1+2 + RL w/ Prog.)3,500 + 1,766GRPO + Prog.85.3473.7558.0037.29

Under matched SFT training steps, ELVIS outperforms the non-latent baseline by +5.24, +1.76, +7.33, and +11.86 pp on V*, HR4K, Jigsaw, and Maze, respectively. After vanilla GRPO, ELVIS maintains improvements of +1.05, +1.75, +8.67, and +6.78 pp on the same benchmarks. These results indicate that additional training alone cannot reproduce the benefits of ELVIS's latent-state learning.

Regarding the reviewer's observation on Zebra Maze, we acknowledge that vanilla SFT1 already achieves 35.59%. Interestingly, continuing vanilla SFT for the same additional 1,766 steps reduces Maze accuracy from 35.59% to 24.58%, while Jigsaw also decreases from 48.67% to 46.67%. This may reflect overfitting to the training trajectories and the difficulty of transferring knowledge learned from intermediate images to inference without such observations. In contrast, ELVIS learns evidence-grounded latent states to bridge this gap.

Furthermore, the controlled ablations in Sec. 5.4 (Tab. 4 and Fig. 7) separately examine the contributions of latent-state construction and evidence supervision. Together with the matched-step comparisons, these results support that ELVIS's improvements arise from its latent-state learning design.

 

Q3(Minor): Relocation of the AI-Use Statement

We thank the reviewer for the reminder. We have moved the existing AI-use statement from the appendix to immediately before the references and updated it to add the use of LLM and VLM for dataset processing.

AuthorsOpenReview ↗

Part 2/2: Response to Reviewer Lqtb

Q2: End-to-End Inference Efficiency

We thank the reviewer for highlighting this distinction. We have added end-to-end inference latency and FLOPs comparisons with both tool-based and recurrent latent reasoning methods, and vanilla SFT in Appendix A.6 (Tabs. 12–13). All experiments are conducted on a single NVIDIA A800 GPU with BF16 precision, batch size 1, and a maximum generation length of 1,024 tokens.

Table 12. End-to-end inference latency and FLOPs compared with tool-based visual reasoning methods. Parentheses indicate speedups over the fastest baseline and FLOPs changes relative to the lowest-FLOP baseline.

\hline \text{Dataset} & \text{Model} & \text{Time (s)}\downarrow & \text{FLOPs (T)}\downarrow \\\\ \hline \text{V*} & \text{Pixel-Reasoner} & 6.72 & 194.88 \\\\ & \text{DeepEyes} & 7.09 & 226.22 \\\\ & \text{Thyme} & 10.59 & 155.64 \\\\ & \mathbf{ELVIS} & \mathbf{2.64}\\;(2.54\times) & \mathbf{99.19}\\;(-36.27\\%) \\\\ \hline \text{HR4K} & \text{Pixel-Reasoner} & 12.23 & 432.25 \\\\ & \text{DeepEyes} & 12.39 & 434.67 \\\\ & \text{Thyme} & 13.54 & 246.64 \\\\ & \mathbf{ELVIS} & \mathbf{4.26}\\;(2.87\times) & \mathbf{193.55}\\;(-21.53\\%) \\\\ \hline \text{Maze} & \text{Pixel-Reasoner} & 19.73 & 22.94 \\\\ & \text{DeepEyes} & 24.49 & 59.37 \\\\ & \text{Thyme} & 23.32 & 24.19 \\\\ & \mathbf{ELVIS} & \mathbf{6.34}\\;(3.11\times) & \mathbf{12.23}\\;(-46.69\\%) \\\\ \hline \end{array}$$ As shown in **Tab. 12**, ELVIS achieves **2.54×, 2.87×, and 3.11× speedups** over the fastest tool-based baselines on V*, HR4K, and Maze, respectively, with FLOPs reductions of **36.27%, 21.53%, and 46.69%**. These substantial latency gains reflect the additional overhead of tool execution in tool-based methods.   **Table 13. End-to-end inference latency and FLOPs compared with latent methods across latent-state widths K.** Parentheses indicate speedups over the fastest baseline and FLOPs changes relative to the lowest-FLOP baseline. $$\begin{array}{llcccccc} \hline \text{Dataset} & \text{Model} & \text{Time} & \text{Time} & \text{Time} & \text{FLOPs} & \text{FLOPs} & \text{FLOPs} \\\\ & & K=8 & K=16 & K=32 & K=8 & K=16 & K=32 \\\\ \hline \text{V*} & \text{LVR} & \mathbf{2.50} & 2.89 & 3.05 & \mathbf{99.14} & 100.97 & 101.22 \\\\ & \text{Monet} & 2.86 & 2.95 & 3.08 & 100.89 & 100.95 & 101.05 \\\\ & \text{ILVR} & 3.76 & 4.21 & 5.43 & 101.61 & 101.95 & 102.86 \\\\ & \mathbf{ELVIS} & 2.68\\;(0.94\times) & \mathbf{2.64}\\;(1.09\times) & \mathbf{2.65}\\;(1.15\times) & 99.20\\;(+0.06\\%) & \mathbf{99.19}\\;(-1.75\\%) & \mathbf{99.22}\\;(-1.81\\%) \\\\ \hline \text{HR4K} & \text{LVR} & 4.34 & 4.74 & 5.02 & 193.66 & 193.90 & 194.77 \\\\ & \text{Monet} & 4.45 & 4.95 & 5.49 & 194.75 & 194.89 & 195.13 \\\\ & \text{ILVR} & 4.85 & 5.69 & 6.54 & 195.13 & 195.73 & 196.34 \\\\ & \mathbf{ELVIS} & \mathbf{4.29}\\;(1.01\times) & \mathbf{4.26}\\;(1.11\times) & \mathbf{4.21}\\;(1.19\times) & \mathbf{193.45}\\;(-0.11\\%) & \mathbf{193.55}\\;(-0.18\\%) & \mathbf{193.67}\\;(-0.57\\%) \\\\ \hline \text{Maze} & \text{LVR} & 7.54 & 8.71 & 9.07 & 13.75 & 14.86 & 15.09 \\\\ & \text{Monet} & 11.40 & 11.50 & 13.73 & 16.95 & 17.03 & 18.50 \\\\ & \text{ILVR} & 6.94 & 8.90 & 11.43 & 14.12 & 17.09 & 20.46 \\\\ & \mathbf{ELVIS} & \mathbf{6.31}\\;(1.10\times) & \mathbf{6.34}\\;(1.37\times) & \mathbf{6.41}\\;(1.41\times) & \mathbf{12.21}\\;(-11.19\\%) & \mathbf{12.23}\\;(-17.68\\%) & \mathbf{12.26}\\;(-18.78\\%) \\\\ \hline \end{array}$$ Compared with recurrent latent reasoning methods (**Tab.13**), at the default $K=16$, ELVIS achieves **1.09×, 1.11×, and 1.37× speedups** on the three benchmarks, respectively. Furthermore, as $K$ increases from 8 to 32, the inference time and FLOPs of recurrent methods progressively increase, whereas **ELVIS remains nearly constant in both, yielding greater relative speedups at larger $K$**. These results confirm that the efficiency gains of ELVIS's parallel latent-state construction extend to the complete inference process.   **Table 14. End-to-end inference latency and FLOPs of ELVIS and Vanilla SFT.** | Dataset | Model | Time (s) ↓ | FLOPs (T) ↓ | | ------- | ----------- | ---------: | ----------: | | V* | Vanilla SFT | 2.57 | 99.03 | | | **ELVIS** | 2.64 | 99.19 | | HR4K | Vanilla SFT | 4.26 | 193.47 | | | **ELVIS** | 4.26 | 193.55 | | Maze | Vanilla SFT | 6.41 | 12.41 | | | **ELVIS** | 6.34 | 12.23 | Compared with vanilla SFT, ELVIS achieves **nearly identical end-to-end latency and FLOPs**, while improving accuracy by **+6.28 pp on V\* and +9.33 pp on Jigsaw** (Tab.1). Latent-state computation accounts for only **~1% of total inference time**, further demonstrating the negligible overhead of parallel latent construction.
Reviewer_rUhCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ELVIS, a method for learning evidence-grounded latent visual states for multi-step multimodal reasoning. ELVIS constructs latent visual states in parallel, using evidence alignment and visual anchoring to supervise the latent representations, and introduces a progressive reward to encourage adaptive latent-state invocations.

The paper additionally introduces AVR-112K, a dataset constructed aiming to reduce “information leakage” where relevant evidence of the intermediate image is leaked before the image and to reduce “information omission” where intermediate images are not accompanied by relevant textual descriptions.

The paper trains ELVIS on AVR-112K, comparing against GPT-4o, base Qwen2.5-VL-7B-Instruct, Qwen2.5-VL-7B-Instruct trained with vanilla SFT as well as existing tool based visual reasoning and latent visual reasoning methods. Across high resolution visual perception benchmarks (V*, HRBench, MME-RealWorldLite) and spatial reasoning benchmarks, ELVIS outperforms the base Qwen2.5-VL-7B-Instruct and vanilla SFT and is competitive with tool-based and latent visual reasoning methods. The paper also provides ablations of the components of ELVIS.

Critical strengths

S1. Well-motivated method

The paper proposes an interesting and well-motivated approach to improving latent visual reasoning. The authors first investigate the limitations of existing recurrent latent visual reasoning methods in visual sensitivity, reasoning utility and internal dynamics (Figure 1a and Appendix A.1) These experiments and quantitative results greatly strengthen the motivation of the paper.

S2. Broad evaluation and useful ablation studies

The paper evaluates ELVIS across a broad range of visual perception and spatial reasoning benchmarks, and compare against multiple categories of baselines, including textual reasoning, tool-based visual reasoning, and existing latent visual reasoning methods. In particular, the inclusion of a data matched vanilla SFT baseline trained on the same AVR-112K dataset is useful for understanding the improvements beyond training data distribution. The paper also provides useful ablation studies of the components of ELVIS, including evidence supervision, visual anchoring, latent-state formation, latent size, and the progressive reward. These experiments help demonstrate the contributions of the individual components of ELVIS.

Critical weaknesses

W1. Insufficient validation of the AVR dataset reformulation

(a) A central motivation of this paper is that existing interleaved reasoning trajectories suffer from information leakage and information omission, and that reformulating them into Action–Visual–Readout trajectories can better supervise visual reasoning. However, I find the dataset construction and validation insufficiently demonstrated.

(b) While Section 3.3 mentions using Qwen2-VL-72B to restructure the trajectories, the paper does not provide any quantiative evidence that demonstrates that the reformulation of AVR achieved its intended goal of reducing information leakage and information omission. For example, quantifying information leakage and omission before reformulation and after reformulation, and possibly even quantifying these for existing datasets.

(c) Furthermore, the paper does not have a controlled comparison between training on the original trajectories and the AVR reformulation. This will isolate the gains from training on the particular data domain of AVR dataset vs the AVR reformulation.

(d) Since AVR-112K is presented as a major contribution, I believe these missing evaluations significantly weaken the evidence supporting the proposed dataset design.

W2. Limited evidence of OOD generalization.

While ELVIS demonstrates strong performance on in-distribution benchmarks, the evidence for OOD generalization is relatively weak. In particular, the full ELVIS model achieves only 25.70% on VisuLogic, compared with 25.60% for the base Qwen2.5-VL-7B-Instruct. Furthermore, the SFT and vanilla GRPO variants achieve 24.90% and 25.90%, respectively, showing no consistent improvement on this benchmark across the training stages. Therefore, it is unclear whether the proposed method learns more generalizable visual reasoning capabilities or primarily improves performance on task distributions similar to those encountered during training.

Questions for authors

Q1. Validation of AVR dataset reformulation

Could the authors provide quantitative evidence of the prevalence of information leakage and information omission before and after AVR reformulation, perhaps through a manually annotated subset of trajectories? Additionally, have the authors considered an ablation comparing training on the original trajectories versus the AVR-reformulated trajectories under the same training setup? These results would help establish whether AVR achieves its intended goals and whether the reformulation itself contributes to the performance improvements. (Refer to W1)

Q2. OOD generalization

Could the authors explain why the improvements observed on the visual perception and spatial reasoning benchmarks do not appear to transfer to VisuLogic, where the full ELVIS model performs similarly to the base model? Have the authors evaluated ELVIS on additional OOD benchmarks with task distributions distinct from those represented in AVR-112K? Such results would help clarify whether the proposed dataset and method provide generalizable reasoning capabilities beyond the training task distributions. (Refer to W2)

3 comments
AuthorsOpenReview ↗

Part 1/3: Response to Reviewer rUhC

W1 & Q1: Validation of AVR-112K Reformulation

We thank the reviewer for highlighting this issue and apologize for not clearly describing the dataset construction process. We have expanded Appendix A.2 to detail the AVR-112K construction pipeline. We address the reviewer's concerns as follows.

(a) Dataset construction and quality verification. Briefly, our four-stage pipeline consists of (relevant system prompts provided in Figs. 18–20):

  1. Trajectory Parsing: We first employ Qwen3-32B to segment interleaved reasoning trajectories into image-anchored steps.
  2. Error Detection: For each step, Qwen2-VL-72B extracts textual visual evidence from the intermediate image to serves as explicit reference for independently identifying pre-image leakage and post-image omission. Both issues are labeled as Yes, No, or Uncertain. We found that adding explicit textual visual evidence improves detection over direct image-based judgment.
  3. AVR Reformulation: Guided by the detected errors and extracted evidence, Qwen2-VL-72B reformulates each step into an Action–Visual–Readout structure, moving prematurely disclosed observations and incorporating missing visual evidence.
  4. Quality Verification: We use Gemini 3.1 Pro for independent verification, checking both step-level visual grounding and trajectory-level reasoning consistency. Unresolved cases are filtered out, and eight researchers manually inspect a randomly sampled subset of the resulting corpus.

(b) Quantitative validation of leakage and omission. Using the same detection before and after reformulation, we calculate sample-level leakage and omission rates. Across AVR-112K (Appendix A.2, Fig. 9):

  • Leakage: 17.09% → 2.29% (−14.80 pp).
  • Omission: 31.24% → 0.001% (−31.24 pp).

We also report two subsets with particularly severe issues: Robot Planning, omission decreases from 87.59% to 0.76%, and Chart Refocus, leakage decreases from 63.68% to 0.00%.

To validate the automated results, we randomly sample 200 paired trajectories before and after reformulation for manual evaluation. Human-annotated leakage decreases from 13.5% to 0.5%, and omission from 22% to 0.5%, with 96% agreement between human and Qwen2-VL-72B judgments on the same samples.

(c) Controlled ablation of AVR reformulation. We further train ELVIS-SFT on the original and reformulated trajectories. The original post-image text and reformulated readouts are used for evidence selection, respectively. As shown in Appendix A.2 (Tab. 7), AVR reformulation improves all evaluated benchmarks, including V* (+1.57), HR4K (+2.00), BLINK Multi-view (+7.51), Zebra Maze (+4.24), and VisuLogic (+2.50). These controlled results confirm that AVR reformulation itself provides more effective supervision.

Table 7. Effect of AVR Reformulation on ELVIS-SFT Performance.

ModelV*HR4KHR8KMME-LiteBLINK JigsawBLINK Multi-v.Zebra MazeVisuLogic
ELVIS-SFT (Original)83.2569.8864.1351.1752.6750.3832.2022.40
ELVIS-SFT (Reformulated)84.8271.8866.1352.4254.0057.8936.4424.90
Δ vs. Original+1.57\color{green}{+1.57}+2.00\color{green}{+2.00}+2.00\color{green}{+2.00}+1.25\color{green}{+1.25}+1.33\color{green}{+1.33}+7.51\color{green}{+7.51}+4.24\color{green}{+4.24}+2.50\color{green}{+2.50}
AuthorsOpenReview ↗

Part 2/3: Response to Reviewer rUhC

(d) Training existing LVR methods on AVR-112K. To further examine the contribution of the corpus, we additionally retrained LVR and ILVR on AVR-112K on AVR-112k. As neither SCOLAR nor DeepLatent has released its code, we reproduced DeepLatent from its paper as a parallel baseline. All methods are evaluated at the SFT stage to exclude RL effects (Appendix A.4, Tab. 10).

Table 10. SFT-stage performance of latent visual reasoning methods trained on AVR-112K.

ModelV*HRBench4KHRBench8KMME-LiteBLINK JigsawBLINK Multi-v.Zebra MazeVisuLogic
LVR-SFT80.1068.1361.8851.5952.6736.0922.0322.60
LVR-SFT on AVR79.5870.0061.8850.3453.3342.8627.9723.40
Δ vs. Original SFT (LVR)−0.52\color{red}{-0.52}+1.87\color{green}{+1.87}0−1.25\color{red}{-1.25}+0.66\color{green}{+0.66}+6.77\color{green}{+6.77}+5.94\color{green}{+5.94}+0.80\color{green}{+0.80}
ILVR-SFT72.7758.1353.0039.5541.3345.8621.1921.40
ILVR-SFT on AVR78.0164.7556.2547.7945.3348.1221.1923.80
Δ vs. Original SFT (ILVR)+5.24\color{green}{+5.24}+6.62\color{green}{+6.62}+3.25\color{green}{+3.25}+8.24\color{green}{+8.24}+4.00\color{green}{+4.00}+2.26\color{green}{+2.26}0+2.40\color{green}{+2.40}
DeepLatent-SFT on AVR75.9261.8852.2541.0642.0038.3519.4920.70
ELVIS-SFT84.8271.8866.1352.4254.0057.8936.4424.90
Δ vs. Best AVR Baseline+5.24\color{green}{+5.24}+1.88\color{green}{+1.88}+4.25\color{green}{+4.25}+2.08\color{green}{+2.08}+0.67\color{green}{+0.67}+9.77\color{green}{+9.77}+8.47\color{green}{+8.47}+1.10\color{green}{+1.10}

Retraining improves several baseline results, particularly ILVR, whose accuracy increases by +5.24 pp on V* and +6.62 pp on HR4K relative to its original SFT results. This confirms that AVR-112K benefits existing methods as well. Nevertheless, ELVIS still outperforms the strongest AVR-trained baseline by +5.24 pp on V*, +1.88 pp on HR4K, and +8.47 pp on Zebra Maze, supporting the complementary contributions of AVR reformulation and our latent-state learning design.

AuthorsOpenReview ↗

Part 3/3: Response to Reviewer rUhC

W2 & Q2: OOD Generalization

We believe the limited improvement on VisuLogic may stem from its particularly challenging reasoning requirements. Unlike visual perception tasks that primarily rely on grounding information in the input image, or visual planning tasks such as Maze that reason over an existing scene, VisuLogic often requires constructing and manipulating abstract visual states, as well as inductively inferring the underlying transformation rules. This involves high-level "visual imagination", such as reasoning from 2D representations to imagined 3D states (Fig. 17). Although AVR-112K includes viewpoint changes and visual transformations, and ELVIS demonstrates strong performance on related tasks such as BLINK Multi-view, the combination of visual imagination and logical reasoning in VisuLogic presents a more challenging OOD setting.

Following the reviewer's suggestion, we have expanded Sec. 5.2 (OOD Generalization, Tab. 2) with three additional OOD benchmarks to further assess whether ELVIS generalizes to such tasks: LogicVista, which also evaluates visual logical and inductive reasoning; MathVista, which focuses on visual mathematical reasoning; and MMMU-Science, which requires scientific understanding. ELVIS consistently outperforms the base model on all three benchmarks, achieving improvements of +1.12 pp on LogicVista, +2.80 pp on MathVista, and +2.59 pp on MMMU-Science.

Table 2. Accuracy (%) on OOD benchmarks.

ModelVisuLogicLogicVistaMathVistaMMMU-Sci
Non-Latent Baselines
Qwen2.5-VL-7B25.6042.0673.8740.33
+ vanilla SFT23.9036.9163.8939.50
Latent Visual Reasoning Models
LVR25.3013.2055.5625.47
Monet24.207.6167.4133.84
COVT25.6039.1575.0040.80
ILVR21.401.3415.005.19
ELVIS (SFT)24.9041.3974.8138.09
ELVIS (Full)25.7043.1876.6742.92
Δ vs. Qwen2.5-VL-7B+0.10\color{green}{+0.10}+1.12\color{green}{+1.12}+2.80\color{green}{+2.80}+2.59\color{green}{+2.59}

Notably, most existing latent reasoning methods exhibit substantial performance degradation on these OOD benchmarks, potentially reflecting their primary focus on visual search tasks. These results suggest that the benefits of ELVIS extend to broader logical, mathematical, and scientific reasoning tasks.

Reviewer_yj3KOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a latent visual reasoning framework called ELVIS. In this framework a shared bank of K learnable queries are inserted after a <latent_start> token. These token attends bidirectionally within the block and causally to the preceding trajectory.

This work constructs a corpus of action-visual-trajectory called AVR-112K, by rewriting the trajectories of existing becnmarks in a way such that the observations appears in post-image readout. They use this corpus for supervision in ELVIS.

The training is done in three stages: (I) A teacher is explicitely finetune on trajectories that include intermediate images. (II) A student replaces each intermediate image with the latent block. The student loss contains three components (a) the CE loss, (b) an evidence alignment loss and (c) a KL loss on the attention to the original image, and (iii) they run GRPO with an invocation Progressive Reward that rewards rollouts whose invocation of the visual reasoning steps ( the latent reasoning) has higher success rate. They reported results on V*, HRBench4K/8K, MME-RealWorld-Lite, BLINK Jigsaw and Multi-view, a held-out Zebra Maze split, and VisuLogic.

Critical strengths
  1. The works seems to be nicely motivated, specially with the probing study that has been done on the existing methods.
  2. The Action-Visual-Readout looks like an interesting approach to reduce the information leakage in the reasoning trajectories.
  3. The design of the supervision loss ( Evidence alignment and visual anchoring ) is sensible and a well argued choice. Addition to the GRPO rewards ( the progressive reward) is well motivated and intuitively explained.
  4. The ablations and the diagonistic study was well thought out.
Critical weaknesses
  1. In Table 1, the reported relative improvements is shown with respect to the base model. However, vanilla SFT on same training corpus does better than the base model. So it would better if the table is labeled acoordingly.

  2. The LVR models are trained on their own data. so I feel Table 1 conflates the gains form the methods with the gains from the AVR-112K data. Retraining at least one or two baselines (e.g., LVR and ILVR) on AVR-112K with the same backbone and budget would make the central claim much more convincing.

  3. In the RL setup, only 16 rollouts per prompts are used. Since the rollouts can be distributed across multiple invocation counts, some success-rate estimates may be based on very few samples. It would be better if the RL data is described in more details

  4. The paper itself discusses several prior works in parallel latent construction in SCOLAR and DeepLatent, and evidence selection or representation alignment in related latent visual reasoning methods. The current ablations comparing recurrent, parallel-causal, and parallel-bidirectional construction are helpful. However, direct comparisons with relevant prior parallel latent reasoning approaches would provide stronger evidence for the architectural contribution.

  5. The anchoring loss would train the student to match its attention over the Image to that of the teacher. That would be helpful for perception task, but that wouldn't help the model in "visual imagination". So, for example tasks in visulogic, would not depend on perceiving what on the image but on how to reason with that. The latent-to-image attention maps in Figs. 4 and 14 show where the model looks. They do not show that it represents a transformed state or how the model reasons with intermediate images that contain information absent from the Image ( like mental transformation or rotations). This can explain the not improvement in visulogic. How does it perform on OOD datasets that are perception heavy.

  6. For latency, the comparison covers only latent-state computation. It would provide a much better insight into the latency of this work if end-to-end wall clock time is reported.

  7. The AVR-112K corpus is rewritten by Qwen2-VL-72B. However no human verification of a subset of the dataset brings into question the quality control of the dataset. A small study on human agreement of the rewritten corpus would address this issue.

Questions for authors
  1. In Table 2, for V* GRPO -> +Prog, before and after accuracy is reported as Before: 73.91% and After: 86.96%. However in Table 1 the overall numbers for V* for SFT+ GRPO and SFT + GRPO + prog are different. Explaining this would help in reading the Tables.

  2. Can you report results for LVR and ILVR retrained on AVR-112K with the same backbone and budget ?

  3. How much of the performance improvement can be attributed specifically to latent-state construction and evidence alignment, rather than the AVR dataset and additional supervised training?

  4. How does ELVIS compare against existing parallel latent reasoning methods, particularly SCOLAR and DeepLatent, under comparable training and inference conditions?

  5. How sensitive is the Invocation Progressive Reward to the number of sampled rollouts and the distribution of invocation counts?

  6. What is the end-to-end inference latency of ELVIS, and how does its accuracy–compute tradeoff compare with recurrent and non-latent baselines?

  7. How reliable is readout-to-visual attention as an evidence-selection mechanism? Have alternative methods of identifying relevant visual regions been considered?

5 comments
AuthorsOpenReview ↗

Part 1/5: Response to Reviewer yj3K

W1: Clarification of Relative Improvements

We thank the reviewer for pointing this out. We have clarified in Tab.1 that the reported improvements are relative to the base model (Qwen2.5-VL-7B). We additionally report the improvements over vanilla SFT below.

Table A. Improvements against Qwen2.5-VL-7B and Vanilla SFT.

ComparisonV*HR4KHR8KMME-LiteJigsawMulti-v.Maze
Δ vs. Base+8.90+5.87+4.13+9.75-4.67+7.52+22.04
Δ vs. Vanilla SFT+6.28+5.62+3.88+7.71+9.33+8.27+1.70

 

Q1: Clarification of Invocation Changes in Tab.3

We thank the reviewer for pointing out this ambiguity. The accuracy in Tab. 3 are evaluated only on the subset whose invocation counts change between the two training stages, whereas Tab. 1 reports accuracy on the entire testing data. We have revised the caption of Tab. 3 to clarify this.

 

W2 & Q2, W4 & Q4: Comparisons with Recurrent and Parallel LVR Methods on AVR-112K

We thank for these insightful suggestions. To isolate the data contribution, we have retrained LVR and ILVR on AVR-112K. As neither SCOLAR nor DeepLatent has released its code, we reproduced DeepLatent on AVR-112K from its paper details as a parallel latent baseline. All methods are evaluated at the SFT stage to exclude RL effects (note that results in Tab.1 include RL). Implementation details are provided in Appendix A.4 (Tab. 10).

Table 10 (part). SFT-stage performance of latent visual reasoning methods trained on AVR-112K.

ModelV*HR4KHR8KMME-LiteJigsawMulti-v.MazeVisuLogic
LVR-SFT80.1068.1361.8851.5952.6736.0922.0322.60
LVR-SFT on AVR79.5870.0061.8850.3453.3342.8627.9723.40
Δ vs. Original SFT (LVR)−0.52\color{red}{-0.52}+1.87\color{green}{+1.87}0−1.25\color{red}{-1.25}+0.66\color{green}{+0.66}+6.77\color{green}{+6.77}+5.94\color{green}{+5.94}+0.80\color{green}{+0.80}
ILVR-SFT72.7758.1353.0039.5541.3345.8621.1921.40
ILVR-SFT on AVR78.0164.7556.2547.7945.3348.1221.1923.80
Δ vs. Original SFT (ILVR)+5.24\color{green}{+5.24}+6.62\color{green}{+6.62}+3.25\color{green}{+3.25}+8.24\color{green}{+8.24}+4.00\color{green}{+4.00}+2.26\color{green}{+2.26}0+2.40\color{green}{+2.40}
DeepLatent-SFT on AVR75.9261.8852.2541.0642.0038.3519.4920.70
ELVIS-SFT84.8271.8866.1352.4254.0057.8936.4424.90
Δ vs. Best AVR Baseline+5.24\color{green}{+5.24}+1.88\color{green}{+1.88}+4.25\color{green}{+4.25}+2.08\color{green}{+2.08}+0.67\color{green}{+0.67}+9.77\color{green}{+9.77}+8.47\color{green}{+8.47}+1.10\color{green}{+1.10}

Retraining on AVR-112K improves several results, particularly ILVR, confirming the contribution of our training corpus. Nevertheless, ELVIS outperforms the strongest baselines +5.24 on V*, +1.88 on HR4K, +0.67 on Jigsaw, and +8.47 on Maze. Compared with DeepLatent, ELVIS achieves substantial gains of +8.90 on V*, +10.00 on HR4K, and +16.95 on Maze, while introducing only 0.78M additional parameters (learnable queries), compared with 386M for DeepLatent. These results support the effectiveness of ELVIS over representative LVR methods beyond the benefits of AVR-112K.

AuthorsOpenReview ↗

Part 2/5: Response to Reviewer yj3K

Q3: Disentangling the Contributions of Training Data, Training Budget, and Latent-State Learning

To disentangle the contributions of training data, additional training, and latent-state design, we conduct three complementary comparisons: (1) retraining existing LVR methods on AVR-112K to control for data differences, (2) matching the training budget with non-latent baselines to isolate additional training, and (3) ablating latent-state construction and evidence supervision to examine their individual contributions.

First, controlling for training data, as discussed in above W2 & Q2, W4 & Q4, ELVIS consistently outperforms these baselines, indicating that its advantages beyond training corpus.

Second, controlling for additional training, we further train the vanilla SFT model for the same training steps as SFT2 and apply the same GRPO setup (Appendix A.5, Tab. 11). Under the matched settings, ELVIS achieves improvements of +5.24, +1.76, +7.33, and +11.86 pp at the SFT stage, and maintains gains of +1.05, +1.75, +8.67, and +6.78 pp after GRPO on V*, HR4K, Jigsaw, and Maze.

Table 11. Training-budget-matched comparisons between ELVIS and non-latent baselines.

ModelSFT(1+2) StepsRLV*HR4KJigsawMaze
Vanilla (Shared SFT1)3,500--79.0668.1348.6735.59
Vanilla (SFT1+2)3,500 + 1,766--79.5870.1246.6724.58
Vanilla (SFT1+2 + RL)3,500 + 1,766GRPO82.2069.3848.0027.97
ELVIS (SFT1+2)3,500 + 1,766--84.8271.8854.0036.44
Δ ELVIS - Vanilla (SFT)+5.24\color{green}{+5.24}+1.76\color{green}{+1.76}+7.33\color{green}{+7.33}+11.86\color{green}{+11.86}
ELVIS (SFT1+2 + RL)3,500 + 1,766GRPO83.2571.1356.6734.75
Δ ELVIS - Vanilla (GRPO)+1.05\color{green}{+1.05}+1.75\color{green}{+1.75}+8.67\color{green}{+8.67}+6.78\color{green}{+6.78}
ELVIS (SFT1+2 + RL w/ Prog.)3,500 + 1,766GRPO + Prog.85.3473.7558.0037.29

Finally, isolating the contributions of latent-state construction and evidence supervision, Referring to ablations in Sec. 5.4 (Tab. 4 and Fig. 7), with Tab. 4 reorganized below for clarity. Fig. 7 shows that parallel-bidirectional latent construction outperforms recurrent and parallel-causal alternatives. Under the same architecture, adding evidence alignment (MSE) and visual anchoring (KL) to CE-only training further improves +2.62, +1.75, +3.33, and +1.69 pp across four benchmarks. These results demonstrate the individual contributions of our latent architecture and evidence supervision.

Table 4. Training component ablation of ELVIS.

\hline \text{Variant} & \mathbf{CE} & \mathbf{MSE} & \mathbf{KL} & \mathbf{RL} & \mathbf{V*} & \mathbf{HR4K} & \mathbf{Jigsaw} & \mathbf{Maze} \\\\ \hline \text{SFT1 (Vanilla SFT)} & \checkmark & - & - & - & 79.06 & 68.13 & 48.67 & 35.59 \\\\ \hline \text{SFT2 Ablation} & & & & & & & & \\\\ \hline \text{No Auxiliary Sup. (Base Init.)} & \checkmark & - & - & - & 79.58 & 68.13 & 52.00 & 27.12 \\\\ \text{No Auxiliary Sup. (SFT1 Init.)} & \checkmark & - & - & - & 82.20 & 70.13 & 50.67 & 34.75 \\\\ \text{Pooling Supervision} & \checkmark & \checkmark & - & - & 82.20 & 69.75 & 47.33 & 34.75 \\\\ \text{Evidence Supervision} & \checkmark & \checkmark & - & - & 83.77 & 70.75 & 52.00 & 35.59 \\\\ \text{Evidence + Anchor} & \checkmark & \checkmark & \checkmark & - & 84.82 & 71.88 & 54.00 & 36.44 \\\\ \hline \text{RL Ablation} & & & & & & & & \\\\ \hline \text{GRPO} & \checkmark & \checkmark & \checkmark & \text{GRPO} & 83.25 & 71.13 & 56.67 & 34.75 \\\\ \text{GRPO + Prog.} & \checkmark & \checkmark & \checkmark & \text{GRPO + Prog.} & \mathbf{85.34} & \mathbf{73.75} & \mathbf{58.00} & \mathbf{37.29} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Part 3/5: Response to Reviewer yj3K

W5: Visual Imagination and OOD Generalization

We appreciate this insightful concern. First, we clarify that visual anchoring (KL) and evidence alignment (MSE) serve different purposes. Visual anchoring maintains grounding in the original image, whereas evidence alignment supervises the content of latent states from intermediate images, which contain transformed visual states absent from the original input. For example, AVR-112K includes multi-view object counting and embodied reasoning tasks involving viewpoint changes and scene transformations. Through evidence-grounded MSE alignment, the student learns to approximate these transferred visual representations.

We believe Fig. 4 already provides qualitative evidence of imagination-like visual reasoning. In the Maze example, latent-state attention maps trace evolving paths, resembling human visual imagination during planning. And the reasoning text generated after latent-state invocation can also provide indirect evidence of whether the model can reason about transformed visual states. We additionally provide a BLINK Multi-view example below, illustrating reasoning about viewpoint changes.

Question: 
The images are frames from a video. Is the camera
moving clockwise (left) or counter-clockwise (right)? (A) left (B) right

ELVIS:
The foreground mug is in front of the two bottles.
Starting from the first image, I will test a rightward camera
movement and compare the resulting view with the second image.
<latent_state>
The resulting image shows that the nearer mug shifts left
relative to the bottles, roughly matching the second image.

Next, I will test a leftward change in viewpoint.
<latent state>
The mug shifts further to the right relative to the bottles,
contradicting the second image.

Therefore, the camera moves right (counter-clockwise). The final answer is \boxed{B}.

VisuLogic often requires high-level constructing and manipulating abstract visual states, as well as abstract logical rules, such as reasoning from 2D representations to imagined 3D states (Fig. 17). So it is more challenging than our in-distribution tasks. To further assess whether ELVIS generalizes to such reasoning tasks, we have expanded Sec. 5.2 (OOD Generalization, Tab. 2) with three additional OOD benchmarks: LogicVista, which also evaluates visual logical and inductive reasoning; MathVista, which focuses on visual mathematical reasoning; and MMMU-Science, which requires scientific understanding. ELVIS consistently outperforms the base model on all three benchmarks, achieving improvements of +1.12 pp on LogicVista, +2.80 pp on MathVista, and +2.59 pp on MMMU-Science.

Table 2. Accuracy (%) on OOD benchmarks.

ModelVisuLogicLogicVistaMathVistaMMMU-Sci
Non-Latent Baselines
Qwen2.5-VL-7B25.6042.0673.8740.33
+ vanilla SFT23.9036.9163.8939.50
Latent Visual Reasoning Models
LVR25.3013.2055.5625.47
Monet24.207.6167.4133.84
COVT25.6039.1575.0040.80
ILVR21.401.3415.005.19
ELVIS (SFT)24.9041.3974.8138.09
ELVIS (Full)25.7043.1876.6742.92
Δ vs. Qwen2.5-VL-7B+0.10\color{green}{+0.10}+1.12\color{green}{+1.12}+2.80\color{green}{+2.80}+2.59\color{green}{+2.59}

Notably, most existing latent reasoning methods exhibit substantial performance degradation on these OOD benchmarks, potentially reflecting their primary focus on visual search tasks. These results suggest that the benefits of ELVIS extend to broader logical, mathematical, and scientific reasoning tasks.

Reviewer_4WaeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study explores how to improve the use of visual evidence in multi-step reasoning. The authors first construct a dataset, named AVR-112K, where each step of the reasoning process is organized into three components: action, intermediate image, and observation. Furthermore, authors propose the ELVIS framework, which employs a teacher model to select top-K visual evidence and train a student model to construct latent visual states in parallel. Reinforcement learning is then used to improve the adaptive retrieval of visual evidence. Experimental results demonstrate the effectiveness of the proposed method and the each proposed component.

Critical strengths

1.This study finds the limitation in existing implicit visual reasoning methods: their latent states may fail to effectively capture the visual evidence required for multi-step reasoning. 2.The authors construct the AVR-112K dataset which is used to guide the selection of visual features, providing supervision for the latent states. 3.The authors also propose a parallel latent state construction method, which reduces the time required to compute latent states, compared to previous sequential generation approaches.

Critical weaknesses

1.The acceleration mentioned in the experimental part is the Latent-state computation latency, instead of the complete end-to-end question and answer process. The overall reasoning time is still unclear. 2.The study does not explicitly state whether teacher model is necessary. It's easy to directly use the proposed dataset to train the student model, where the teacher model is not necessary. So the authors need to explain why the teacher model is needed.

Questions for authors

1.How does the complete end-to-end inference time of ELVIS compare to other existing methods? 2.If training is performed directly on AVR-112K, without both teacher-based evidence alignment and visual ranking, how does the student model perform? 3.Could the authors report results from multiple runs or provide further analysis of the improvements on Zebra Maze?

2 comments
AuthorsOpenReview ↗

Part 1/2: Response to Reviewer 4Wae

W1 & Q1: End-to-End Inference Efficiency

We thank the reviewer for highlighting this distinction. We have added end-to-end inference latency and FLOPs comparisons with both tool-based and recurrent latent reasoning methods, and vanilla SFT in Appendix A.6 (Tabs. 12–13). All experiments are conducted on a single NVIDIA A800 GPU with BF16 precision, batch size 1, and a maximum generation length of 1,024 tokens.

Table 12. End-to-end inference latency and FLOPs compared with tool-based visual reasoning methods. Parentheses indicate speedups over the fastest baseline and FLOPs changes relative to the lowest-FLOP baseline.

\hline \text{Dataset} & \text{Model} & \text{Time (s)}\downarrow & \text{FLOPs (T)}\downarrow \\\\ \hline \text{V*} & \text{Pixel-Reasoner} & 6.72 & 194.88 \\\\ & \text{DeepEyes} & 7.09 & 226.22 \\\\ & \text{Thyme} & 10.59 & 155.64 \\\\ & \mathbf{ELVIS} & \mathbf{2.64}\\;(2.54\times) & \mathbf{99.19}\\;(-36.27\\%) \\\\ \hline \text{HR4K} & \text{Pixel-Reasoner} & 12.23 & 432.25 \\\\ & \text{DeepEyes} & 12.39 & 434.67 \\\\ & \text{Thyme} & 13.54 & 246.64 \\\\ & \mathbf{ELVIS} & \mathbf{4.26}\\;(2.87\times) & \mathbf{193.55}\\;(-21.53\\%) \\\\ \hline \text{Maze} & \text{Pixel-Reasoner} & 19.73 & 22.94 \\\\ & \text{DeepEyes} & 24.49 & 59.37 \\\\ & \text{Thyme} & 23.32 & 24.19 \\\\ & \mathbf{ELVIS} & \mathbf{6.34}\\;(3.11\times) & \mathbf{12.23}\\;(-46.69\\%) \\\\ \hline \end{array}$$ As shown in **Tab. 12**, ELVIS achieves **2.54×, 2.87×, and 3.11× speedups** over the fastest tool-based baselines on V*, HR4K, and Maze, respectively, with FLOPs reductions of **36.27%, 21.53%, and 46.69%**. These substantial latency gains reflect the additional overhead of tool execution in tool-based methods. &nbsp; **Table 13. End-to-end inference latency and FLOPs compared with latent methods across latent-state widths K.** Parentheses indicate speedups over the fastest baseline and FLOPs changes relative to the lowest-FLOP baseline. $$\begin{array}{llcccccc} \hline \text{Dataset} & \text{Model} & \text{Time} & \text{Time} & \text{Time} & \text{FLOPs} & \text{FLOPs} & \text{FLOPs} \\\\ & & K=8 & K=16 & K=32 & K=8 & K=16 & K=32 \\\\ \hline \text{V*} & \text{LVR} & \mathbf{2.50} & 2.89 & 3.05 & \mathbf{99.14} & 100.97 & 101.22 \\\\ & \text{Monet} & 2.86 & 2.95 & 3.08 & 100.89 & 100.95 & 101.05 \\\\ & \text{ILVR} & 3.76 & 4.21 & 5.43 & 101.61 & 101.95 & 102.86 \\\\ & \mathbf{ELVIS} & 2.68\\;(0.94\times) & \mathbf{2.64}\\;(1.09\times) & \mathbf{2.65}\\;(1.15\times) & 99.20\\;(+0.06\\%) & \mathbf{99.19}\\;(-1.75\\%) & \mathbf{99.22}\\;(-1.81\\%) \\\\ \hline \text{HR4K} & \text{LVR} & 4.34 & 4.74 & 5.02 & 193.66 & 193.90 & 194.77 \\\\ & \text{Monet} & 4.45 & 4.95 & 5.49 & 194.75 & 194.89 & 195.13 \\\\ & \text{ILVR} & 4.85 & 5.69 & 6.54 & 195.13 & 195.73 & 196.34 \\\\ & \mathbf{ELVIS} & \mathbf{4.29}\\;(1.01\times) & \mathbf{4.26}\\;(1.11\times) & \mathbf{4.21}\\;(1.19\times) & \mathbf{193.45}\\;(-0.11\\%) & \mathbf{193.55}\\;(-0.18\\%) & \mathbf{193.67}\\;(-0.57\\%) \\\\ \hline \text{Maze} & \text{LVR} & 7.54 & 8.71 & 9.07 & 13.75 & 14.86 & 15.09 \\\\ & \text{Monet} & 11.40 & 11.50 & 13.73 & 16.95 & 17.03 & 18.50 \\\\ & \text{ILVR} & 6.94 & 8.90 & 11.43 & 14.12 & 17.09 & 20.46 \\\\ & \mathbf{ELVIS} & \mathbf{6.31}\\;(1.10\times) & \mathbf{6.34}\\;(1.37\times) & \mathbf{6.41}\\;(1.41\times) & \mathbf{12.21}\\;(-11.19\\%) & \mathbf{12.23}\\;(-17.68\\%) & \mathbf{12.26}\\;(-18.78\\%) \\\\ \hline \end{array}$$ Compared with recurrent latent reasoning methods (**Tab. 13**), at the default $K=16$, ELVIS achieves **1.09×, 1.11×, and 1.37× speedups** on the three benchmarks, respectively. Furthermore, as $K$ increases from 8 to 32, the inference time and FLOPs of recurrent methods progressively increase, whereas **ELVIS remains nearly constant in both, yielding greater relative speedups at larger $K$**. These results confirm that the efficiency gains of ELVIS's parallel latent-state construction extend to the complete inference process. &nbsp; **Table 14. End-to-end inference latency and FLOPs of ELVIS and Vanilla SFT.** | Dataset | Model | Time (s) ↓ | FLOPs (T) ↓ | | ------- | ----------- | ---------: | ----------: | | V* | Vanilla SFT | 2.57 | 99.03 | | | **ELVIS** | 2.64 | 99.19 | | HR4K | Vanilla SFT | 4.26 | 193.47 | | | **ELVIS** | 4.26 | 193.55 | | Maze | Vanilla SFT | 6.41 | 12.41 | | | **ELVIS** | 6.34 | 12.23 | Compared with vanilla SFT, ELVIS achieves **nearly identical end-to-end latency and FLOPs**, while improving accuracy by **+6.28 pp on V\* and +9.33 pp on Jigsaw** (Tab.1). Latent-state computation accounts for only **~1% of total inference time**, further demonstrating the negligible overhead of parallel latent construction.
AuthorsOpenReview ↗

Part 2/2: Response to Reviewer 4Wae

W2 & Q2: Necessity of the Teacher Model

We thank the reviewer for this important question. We added a SFT2 ablation initialized directly from the base model in Sec. 5.4 (Tab. 4), trained on AVR-112K using only CE loss, without teacher-based supervision.

Table 4. Training component ablation of ELVIS.

\hline \text{Variant} & \mathbf{CE} & \mathbf{MSE} & \mathbf{KL} & \mathbf{RL} & \mathbf{V*} & \mathbf{HR4K} & \mathbf{Jigsaw} & \mathbf{Maze} \\\\ \hline \text{SFT1 (Vanilla SFT)} & \checkmark & - & - & - & 79.06 & 68.13 & 48.67 & 35.59 \\\\ \hline \text{SFT2 Ablation} & & & & & & & & \\\\ \hline \text{No Auxiliary Sup. (Base Init.)} & \checkmark & - & - & - & 79.58 & 68.13 & 52.00 & 27.12 \\\\ \text{No Auxiliary Sup. (SFT1 Init.)} & \checkmark & - & - & - & 82.20 & 70.13 & 50.67 & 34.75 \\\\ \text{Pooling Supervision} & \checkmark & \checkmark & - & - & 82.20 & 69.75 & 47.33 & 34.75 \\\\ \text{Evidence Supervision} & \checkmark & \checkmark & - & - & 83.77 & 70.75 & 52.00 & 35.59 \\\\ \text{Evidence + Anchor} & \checkmark & \checkmark & \checkmark & - & 84.82 & 71.88 & 54.00 & 36.44 \\\\ \hline \text{RL Ablation} & & & & & & & & \\\\ \hline \text{GRPO} & \checkmark & \checkmark & \checkmark & \text{GRPO} & 83.25 & 71.13 & 56.67 & 34.75 \\\\ \text{GRPO + Prog.} & \checkmark & \checkmark & \checkmark & \text{GRPO + Prog.} & \mathbf{85.34} & \mathbf{73.75} & \mathbf{58.00} & \mathbf{37.29} \\\\ \hline \end{array}$$ Compared with SFT1 initialization, direct base-model initialization yields lower accuracy on most benchmarks, particularly Maze (**27.12% vs. 34.75%**). With our full evidence supervision, performance further improves by **2.62, 1.75, 3.33, and 1.69 pp** on V*, HR4K, Jigsaw, and Maze, respectively. The teacher serves two purposes. **(1) SFT1 adapts the base model to interleaved visual reasoning.** Since the base model is not specifically trained to reason with intermediate images, SFT1 enables it to better utilize them. To verify this, we track the ground-truth readout NLL with and without intermediate images during SFT1 (**Sec. 5.4 Teacher visual dependency, Fig. 6**). The increasing NLL gap indicates that the teacher progressively learns to rely more on intermediate visual evidence, laying the foundation for SFT2. **(2) The teacher provides readout-conditioned visual representations as supervision targets for SFT2 latent-state learning.** As shown in **Tab. 4**, our evidence-based supervision consistently outperforms pooling-based supervision by **1.57, 1.00, 4.67, and 0.84 pp** across all benchmarks. These results indicate the teacher plays an important role in learning effective, evidence-grounded latent states. &nbsp; ## Q3: Robustness and Further Analysis on Zebra Maze We further evaluated Zebra Maze over **five seeds (42, 27, 123, 168, 2027)**, using fixed model checkpoints and identical settings (temperature = 0.7, top-p = 0.9). **Table A. Five-seed Zebra Maze robustness** | Model | Seed = 42 | Seed = 27 | Seed = 123 | Seed = 168 | Seed = 2027 | Mean | Std | | ------------- | --------- | --------- | ---------- | ---------- | ----------- | --------------------- | -------- | | Qwen2.5-VL-7B | 15.25 | 16.95 | 20.34 | 15.25 | 14.41 | 16.44 | 2.37 | | ELVIS | **37.29** | **38.14** | **37.29** | **38.98** | **37.29** | **37.80 (+21.36 pp)** | **0.76** | Across five runs, **ELVIS achieves 37.80 ± 0.76% accuracy**, versus **16.44 ± 2.37%** for Qwen2.5-VL-7B (**+21.36 pp**), indicating its robustness to inference randomness. We have observed that vanilla SFT already substantially improves Maze accuracy (**15.25% → 35.59%, Tab. 1**), suggesting AVR-112K's contribution. We further isolate latent-state learning through **training-budget-matched non-latent comparisons** on AVR-112K (**Appendix A.5, Tab. 11**). The Zebra Maze results are summarized below: **Table B. Training-budget-matched comparisons between ELVIS and baselines.** | Training Stage | Baseline | ELVIS | Δ | | ------------------------------------- | -------: | --------: | ---------: | | SFT1 + SFT2 | 24.58 | **36.44** | **+11.86** | | SFT1 + SFT2 + GRPO | 27.97 | **34.75** | **+6.78** | | SFT1 + SFT2 + GRPO (+Prog. for ELVIS) | 27.97 | **37.29** | **+9.32** | Under matched SFT training steps, ELVIS outperforms the baseline by **11.86 pp**. With the same GRPO setup, ELVIS maintains a **6.78 pp advantage**, which further increases to **9.32 pp** with our progressive reward. These results of Maze support the contribution of evidence-grounded latent-state learning.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Main Revisions

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive suggestions. In response, we have substantially revised the manuscript and conducted additional experiments. The main revisions are summarized below, with changes highlighted in blue in the revised manuscript.

  1. AVR-112K construction and validation (Reviewers rUhC, yj3K) (Sec. 3.3, Appendix A.2). We detail the complete four-stage dataset construction pipeline, include quality verification procedures, quantify information leakage and omission before and after reformulation, and compare ELVIS trained on original versus reformulated trajectories to verify the effectiveness of AVR-112k.
  2. Comparisons with recurrent and parallel latent methods on AVR-112k (Reviewers yj3K, rUhC) (Appendix A.4, Tab. 10). We retrain LVR and ILVR and reproduce DeepLatent on AVR-112K, disentangling the contributions of the training corpus and our methodological design.
  3. Training-budget-matched comparisons with baselines (Reviewers yj3K, Lqtb, 4Wae) (Appendix A.5, Tab. 11). We further train non-latent baselines with matched training steps at both SFT and RL stages, demonstrating that additional training alone cannot account for ELVIS's improvements.
  4. Analysis of teacher model necessity (Reviewer 4Wae) (Sec. 5.4, Fig. 6, Tab. 4). We add a direct SFT2 training ablation without SFT1 initialization and analyze the teacher's visual dependency throughout SFT1, clarifying the role of the teacher model.
  5. Expanded OOD evaluation (Reviewers yj3K, rUhC) (Sec. 5.2, Tab. 2). We extend the OOD evaluation to LogicVista, MathVista, and MMMU-Science, further validating ELVIS's generalization across diverse reasoning tasks.
  6. End-to-end inference efficiency (Reviewers 4Wae, yj3K, Lqtb) (Appendix A.6, Tabs. 12–14). We supplement latent-state latency measurements with end-to-end inference latency and FLOPs comparisons against recurrent latent reasoning, tool-based methods, and Vanilla SFT baseline, demonstrating ELVIS's efficiency.

We hope these additional experiments and clarifications address the reviewers' concerns and further strengthen the empirical evidence supporting ELVIS. We will publicly release our code and AVR-112k to facilitate reproducibility and future research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.