acceptodds
Withdrawn from ICLR 2027

Probing Internal World Models for Interleaved Visual-Textual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Humans often solve spatial problems by mentally simulating visual transformations. In contrast, conventional vision-language models (VLMs) reason primarily through language. We investigate whether VLMs can solve spatial problems by reasoning with both text and generated visual states. To this end, we introduce WM-VLM, which augments a pretrained VLM with a lightweight world model branch for generating intermediate visual states. Our two-stage training first teaches the model to generate the next visual state and then to use that state for reasoning. We programmatically construct spatial reasoning tasks with verifiable intermediate visual states. These tasks allow us to evaluate how well the model generates visual states and how much it relies on them to answer the question. On 2D and 3D mental rotation tasks, WM-VLM consistently outperforms the supervised fine-tuned backbone, with gains of up to 39.25 percentage points. Ablations suggest that these gains depend on the generated visual states, as removing or corrupting them sharply reduces performance. Together, these results suggest that internal world models offer a promising path toward VLMs that reason in both language and visual space.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Interleaved visual-textual reasoning, internal world model, vision-language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e9FaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether generating intermediate visual states can improve spatial reasoning in vision language models (VLMs). It introduces WM-VLM, which augments a pretrained VLM with a visual-generation branch. Training first teaches the branch to predict visual-state embeddings using flow matching, then jointly optimizes reasoning with generated embeddings. The authors construct programmatically verifiable 2D and 3D rotation tasks and evaluate an additional navigation task. WM-VLM outperforms fine-tuned backbone on rotation tasks. Retrieval analyses, latent perturbations, and ablations examine the relationship between visual-state prediction and answer accuracy, as well as the effects of training objectives and architectural choices.

Critical strengths

The gains over the fine-tuned backbone are supported by ablations. On Tetris-2D, accuracy improves from 48.25% to 87.50% in distribution and from 46.20% to 75.00% out of distribution. Improvements also extend to the 3D splits. Experiments on token count, loss balance, branch placement, and training stages provide useful evidence about when this approach works.

Critical weaknesses
  1. The comparison with image-based reasoning is not convincing. The paper generates images using an MLP trained with a pixel reconstruction loss. This differs from the diffusion or flow-matching generators commonly used for image generation.
  2. The latent ablations leave an important question unanswered. Replacing latents with zeros or random vectors hurts accuracy, but these inputs are very different from what the model normally receives. The model might need normal-looking latent inputs without needing them to represent the correct visual state.
  3. Minor typo: On page 9, “Whhich one is the better reasoning medium?” should be “Which one is the better reasoning medium?”
Questions for authors
  1. Would the advantage over image-based reasoning remain if the image variant used a diffusion generator instead of MLP-based pixel reconstruction?
  2. Could you replace the intermediate latents with latents from another image, such as a different rotation of the same object? My concern is that the model may only need inputs similar to those seen during training, rather than the correct visual content. This experiment would help distinguish the two explanations.
Reviewer_v4XGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether vision-language models can improve spatial reasoning by predicting intermediate visual states on their own. The authors propose WM-VLM, which adds a lightweight generation branch to a pretrained VLM. The model is trained in two stages: first to generate visual states, then to use them for reasoning. On the Tetris-2D and Tetris-3D rotation tasks, WM-VLM improves accuracy by 16.30–39.25 percentage points across five test splits compared with a supervised fine-tuned baseline using the same backbone.

Critical strengths

(1) The paper goes beyond final-answer accuracy to study whether the model uses generated intermediate visual states for reasoning. The authors build rotation tasks with verifiable intermediate states and final answers, with test splits covering seen shapes, unseen shapes, and out-of-distribution cases.

(2) Large performance gains across multiple test splits using the same backbone. In Table 1, WM-VLM achieves 87.50% and 75.00% accuracy on the Tetris-2D ID and OOD splits, compared with 48.25% and 46.20% for the SFT baseline. It also improves accuracy by 16.30–21.75 percentage points across the three 3D splits.

Critical weaknesses

(1) Limited evidence that the model uses correct visual-state information. Zeroing or randomizing visual tokens shows reliance on a latent channel, but does not prove correct spatial simulation. On Tetris-3D-OOD, retrieval accuracy is only 13.40%, yet answer accuracy when retrieval fails remains 69.52%. Retrieval failure does not necessarily mean incorrect states, but these results leave their role in spatial reasoning unclear.

(2) The comparisons do not clearly isolate the source of the gains. Matching SFT training steps does not control for added capacity, visual-state supervision, or optimization differences. A comparison that changes only the state-prediction objective while keeping other factors fixed is missing. Also, ThinkMorph’s 0% on 3D tasks reflects failure to generate images and answers, not direct evidence against pixel-based reasoning.

(3) The paper has problems with figure quality and layout. Figure 3 has excessive empty space above and below it. The figures are also poorly designed. For example, Figure 2 has text that is too small despite substantial unused space, and its overall design is not visually appealing.

Questions for authors

(1) Add tests that replace generated states with incorrect rotations of the same shape, apply counterfactual action interventions, and use ground-truth intermediate states in WM-VLM itself. These would help distinguish reliance on latent variables from reliance on correct spatial states.

(2) Add controls using a capacity-matched auxiliary branch and direct latent regression with the same architecture. Report how baseline failures were investigated, how hyperparameters were chosen, and the actual compute budgets. Otherwise, it remains unclear whether the gains come from world-state supervision, the generation method, or added capacity and training.

Reviewer_mMMDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method that adds a generation branch to a VLM, allowing it to predict the visual latents resulting from actions. It first trains the generation branch to predict the correct visual states, then jointly trains both branches to use these generated visual latents for answer prediction. Experiments on 2D/3D Tetris tasks show good performance improvements.

Critical strengths
  • Interleaved reasoning is a promising direction.

  • The method shows good improvements on the evaluated tasks.

Critical weaknesses
  • Limited novelty compared to prior work. Interleaved reasoning has been extensively explored, and the core idea of combining textual reasoning with generated intermediate visual states is closely related to MVoT [1], which is also cited in the paper. The authors should better clarify how their method differs from MVoT.

  • The evaluation is limited to relatively simple, synthetic tasks.

  • The generalization ability remains unclear. Although the paper includes OOD evaluations, these are still largely within the same task settings.

[1] Imagine while Reasoning in Space: Multimodal Visualization-of-Thought

Questions for authors
  • It would be interesting to see whether the spatial reasoning ability learned from 2D tasks can transfer to 3D tasks. For example, could pretraining on Tetris-2D improve performance or training efficiency on Tetris-3D?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.