- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work teaches a unified multimodal model to "think in images" by emitting intermediate visual states during reasoning. The authors assemble Sperry-43K — 42,682 interleaved text-image traces across six tasks spanning mental simulation/imagination and 3D spatial intelligence — together with the 2,249-sample SperryBench, and pair them with a full post-training recipe: two-stage SFT that trains the understanding and generation branches in sequence, followed by GSPPO, an RL stage in which the generation branch is optimized on prompts sampled from the model's own text policy to close the train–inference mismatch. Built on Bagel, the 14B model moves from 19.1% to 51.9% on SperryBench, surpassing far larger open models such as Qwen3.5-397B and Gemma-4-31B, as well as a latent-reasoning baseline trained on identical data.
Critical strengths
- A complete and well-motivated training recipe. Simulator-rendered data makes intermediate states correct by construction; sequential two-branch SFT resolves the mismatched optimization horizons of text and image modeling; GSPPO directly targets the prompt-distribution gap faced by the generation branch. The pipeline is a useful reference for post-training UMMs for reasoning.
- Well-chosen task scope. Mental simulation and spatial intelligence are exactly where text-only chains fall short, and carrying state updates through generated intermediate images is a sensible and promising paradigm.
Critical weaknesses
- The headline evaluation is confined to the authors' own benchmark, whose six task families mirror the training distribution; there is little evidence of generalization to independent benchmarks, and 51.9% absolute accuracy leaves the method's real competitiveness in open settings unclear.
- Gaps in comparison and supervision. The latent-reasoning baseline comparison does not establish matched capacity or training budget, and the RL reward verifies only the generated images programmatically — the textual reasoning receives no process-level supervision.
Questions for authors
- Are there generalization results on established spatial-reasoning or geometry benchmarks outside SperryBench?
- How much of the improvement comes from GSPPO versus DiffusionNFT with dataset prompts?
- What are the per-image generation latency and the overall inference cost?
6 comments
Rebuttal for reviewer zMUU (2/2)
3. Contribution of GSPPO vs. DiffusionNFT (Question 2). We agree that this split should be made explicit. The two RL variants share the same pipeline, data and reward and differ only in where the generator's prompts come from.
| SperryBench (%) | Maze | BlockCount | CubeView | LineTrack | Counting | Jigsaw | Overall |
|---|---|---|---|---|---|---|---|
| SFT | 41.8 | 26.8 | 51.8 | 42.3 | 62.1 | 36.2 | 43.5 |
| + RL with dataset prompts (DiffusionNFT) | 59.7 | 27.9 | 54.6 | 55.9 | 64.3 | 41.0 | 50.6 (+7.1) |
| + RL with self-written prompts (GSPPO) | 70.0 | 31.4 | 54.0 | 56.7 | 59.2 | 40.2 | 51.9 (+1.3) |
Of the 8.4 points that RL adds, 7.1 come from generation-branch RL and 1.3 from self-prompting. GSPPO trains the generator on prompts written by the model itself, which are exactly the prompts it receives at inference, and its extra gain is concentrated on Maze and BlockCount, the two tasks whose answer is read off the generated image. Self-prompting therefore adds a further gain on top of an already strong image-RL stage. We will state this split explicitly in Sec. 5.3.
4. Per-image latency and inference cost (Question 3). Measured over our SperryBench evaluation runs:
| Sperry on one GB200 GPU | Time |
|---|---|
| One intermediate image (50 denoising steps) | 6.8 s |
| Image generation per problem (3.4 images on average) | about 23 s |
| Full problem (text and images) | 35 s |
| Per problem | Text written | Time | SperryBench (%) |
|---|---|---|---|
| Sperry, one H100 | a few hundred tokens and 3.4 images | 46.0 s | 51.9 |
| Qwen3.5-9B with vLLM | about 24k thinking tokens | about 3 min | 36.1 |
Image generation takes about two thirds of Sperry's time, but Qwen3.5-9B has to spell out every intermediate state in words. A few generated images cost less time than the long textual chains of text-only thinking, so reasoning through images makes Sperry faster as well as more accurate. We will add the per-image latency to Sec. 5.3.
Thank you for the additional experiments. The new results help address my concern regarding cross-benchmark generalization. However, two important concerns remain:
-
Matching the 7B understanding branch does not establish system-level capacity matching, as Sperry additionally employs a 7B image generator. Could the authors clarify the total parameter counts and actual training computational budgets of both systems? If they are not matched, the claim of superiority over latent visual reasoning should be appropriately qualified.
-
Does GSPPO optimize only the image generation branch, or does it also update the text reasoning branch? If the latter receives no direct reward or process-level supervision, how does the method ensure the correctness of the textual reasoning that guides image generation?
While I appreciate the additional results, these concerns remain insufficiently addressed. I therefore maintain my current rating pending further clarification.
Rebuttal for reviewer zMUU (1/2)
Thank you for the detailed review and for recognizing both the recipe and the task scope. We answer your points in order.
1. Generalization to independent benchmarks (Weakness 1, Question 1). We agree that generalization has to be tested beyond SperryBench, since its six task families mirror the training data. Our experiments test it on independent benchmarks that probe the spatial-reasoning and visual-geometry abilities you ask about, and test retention on general benchmarks.
Sperry scores higher than the much larger Qwen3-VL-235B on four of the five BabyVision subtypes, and the VisWorld-Eval mazes were generated and rendered independently of ours. In addition, we also evaluated SpatialViz-Bench (ICLR 2026), whose mental-rotation tasks use transformations that none of our training tasks contains.
Sperry therefore generalizes beyond SperryBench to independent spatial-reasoning benchmarks, including rotation tasks outside its training data, while largely retaining Bagel's general abilities. We will highlight these results in the revised paper.
2. Matched capacity and budget in the Monet comparison (Weakness 2). We agree that the comparison with the latent-reasoning baseline Monet is meaningful only at matched capacity and training budget, and our comparison is matched on both.
Sperry's second 7B branch is the image generator, the very component whose effect the comparison measures. Under these matched conditions:
Even without RL, Sperry is 7.7 points ahead of Monet with RL, and the margin is largest on the tasks that require simulating state changes (Maze, BlockCount and LineTrack). With matched capacity and budget, reasoning with explicit generated images therefore clearly beats reasoning with latent visual tokens. We will state both points explicitly in Sec. 5.2.