- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether generating intermediate visual states can improve spatial reasoning in vision language models (VLMs). It introduces WM-VLM, which augments a pretrained VLM with a visual-generation branch. Training first teaches the branch to predict visual-state embeddings using flow matching, then jointly optimizes reasoning with generated embeddings. The authors construct programmatically verifiable 2D and 3D rotation tasks and evaluate an additional navigation task. WM-VLM outperforms fine-tuned backbone on rotation tasks. Retrieval analyses, latent perturbations, and ablations examine the relationship between visual-state prediction and answer accuracy, as well as the effects of training objectives and architectural choices.
Critical strengths
The gains over the fine-tuned backbone are supported by ablations. On Tetris-2D, accuracy improves from 48.25% to 87.50% in distribution and from 46.20% to 75.00% out of distribution. Improvements also extend to the 3D splits. Experiments on token count, loss balance, branch placement, and training stages provide useful evidence about when this approach works.
Critical weaknesses
- The comparison with image-based reasoning is not convincing. The paper generates images using an MLP trained with a pixel reconstruction loss. This differs from the diffusion or flow-matching generators commonly used for image generation.
- The latent ablations leave an important question unanswered. Replacing latents with zeros or random vectors hurts accuracy, but these inputs are very different from what the model normally receives. The model might need normal-looking latent inputs without needing them to represent the correct visual state.
- Minor typo: On page 9, “Whhich one is the better reasoning medium?” should be “Which one is the better reasoning medium?”
Questions for authors
- Would the advantage over image-based reasoning remain if the image variant used a diffusion generator instead of MLP-based pixel reconstruction?
- Could you replace the intermediate latents with latents from another image, such as a different rotation of the same object? My concern is that the model may only need inputs similar to those seen during training, rather than the correct visual content. This experiment would help distinguish the two explanations.