acceptodds
Under review as a conference paper at ICLR 2027

Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs

Abstract

Multimodal Large Language Models (MLLMs) excel at visual understanding via text reasoning but fall short in scenarios requiring visual imagination. Existing approaches rely on predefined external toolkits or in-line image generation, whereas humans flexibly interleave visual and textual imagination within a unified mental space. Motivated by this, and by the fact that MLLMs already encode visual and textual information in the same feature space, we argue that visual tokens can be seamlessly inserted into text-token reasoning, so that visual imagination can be carried entirely by latent features. To this end, we propose Sketch-in-Latents (SkiLa), a unified multimodal reasoning paradigm that extends the auto-regressive capability of MLLMs to natively generate continuous visual embeddings, termed latent sketch tokens, as visual thoughts. SkiLa is trained in two stages: SFT explicitly grounds latent sketch tokens via a latent visual semantics reconstruction mechanism, while RL with masked GRPO on our curated SkiLa-RL-4K further implicitly refine them. SkiLa can dynamically alternates between a textual thinking mode that produces text tokens and a visual sketching mode that produces latent sketch tokens. Extensive experiments show that SkiLa achieves superior performance on vision-centric tasks and generalizes well to diverse multimodal benchmarks. Code, model and datasets will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal, latent reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XWQoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SkiLa, a framework for training interleaved latent visual reasoning with segments of continuous "latent sketch tokens". Training follows the SFT -> masked GRPO, which only supervises the discrete tokens.

Critical strengths

I don't particularly feel there are critical strengths. The idea of latent visual reasoning is already explored in previous works like LVR, Monet, ILVR etc. This paper is quite similar to prior works without notable novelty.

Critical weaknesses
  1. I think the novelty is quite constrained compared to previous LVR works. One contribution that the paper claims is the multi-segment text and visual reasoning, which is also present in prior work (ILVR). The masked GRPO is also not new. Therefore, I am not clear about the actual novelty of the paper.
  2. The main comparisons in Table 1 may not be fair: the previous works (LVR, Monet, CoVT) primarily adopt Qwen2.5-VL, whereas Skila uses Qwen3VL.
  3. Some ablations are missing: the effect on the number of text / visual segments, the latency/throughput comparisons (as efficiency is a main advantage proposed by the paper), the design choices over the supervision of visual tokens, etc.
Questions for authors

Please see weaknesses.

3 comments

1. Contribution and Relation to Previous Latent Reasoning (W1)

The specific differences between SkiLa and prior methods are as follows.

1) Compared with LVR: different reasoning patterns and supervision targets. LVR's intermediate reasoning during SFT is supervised as latent-only visual reconstruction. SkiLa instead uses interleaved hybrid autoregressive reasoning, alternating discrete textual thoughts with continuous latent hidden states. Moreover, LVR reconstructs crop/ROI regions, whereas SkiLa supervises intermediate thinking images, such as maze paths and board states, extending region reconstruction to multi-step visual-state modeling.

2) Compared with Monet: direct joint supervision versus multi-stage distillation. Monet uses three-stage SFT (warm-up, observation-guided alignment, and latent distillation), followed by explicit latent optimization with VLPO. SkiLa uses external-encoder semantic reconstruction jointly with textual SFT, without a separate distillation stage, then implicitly constrains latent hidden states through final-answer rewards with masked GRPO.

3) Compared with ILVR: SFT + RL versus two-stage SFT. ILVR first uses a momentum teacher for visual-feature alignment, then performs latent relaxation under textual supervision; both stages are SFT. SkiLa first establishes semantic grounding through external-encoder supervision, then performs RL refinement using final-answer rewards.

In summary, SkiLa combines multi-step text–vision interleaved reasoning, explicit semantic supervision from intermediate thinking images, and implicit latent optimization through final-answer rewards, making visual thoughts both semantically grounded and useful for solving tasks.

2. Backbone Comparability (W2)

We add reproductions of LVR and Monet on Qwen3-VL-8B to directly compare different methods on the same backbone.

1) LVR with matched data and backbone. LVR and SkiLa use the same training data and Qwen3-VL-8B backbone. SkiLa already outperforms LVR on all four benchmarks after SFT, and leads by 2.7/3.6/4.0/4.3 percentage points after RL. Its supervision and reasoning design thus remain effective when data and backbone are controlled.

Method (SFT / SFT + RL)V*HR4KHR8KMMStar
LVR84.3 / 85.377.9 / 77.971.9 / 73.565.3 / 67.0
SkiLa85.3 / 88.078.0 / 81.572.3 / 77.566.9 / 71.3

2) Monet with a matched backbone. Monet uses three-stage SFT with a different supervision construction from SkiLa; this reproduction therefore does not use SkiLa's training data. We reproduce Monet on the same Qwen3-VL-8B backbone. SkiLa leads by 1.1/3.0/2.2/3.2 percentage points across the four benchmarks, showing that its advantage is not solely due to a newer backbone.

Method (Qwen3-VL-8B)V*HR4KHR8KMMStar
Backbone83.876.571.364.7
Monet86.978.575.368.1
SkiLa-RL88.081.577.571.3

3. Segments, Efficiency, and Visual Supervision (W3)

1) Visual-segment distribution. We count the visual reasoning turns in responses on HR8K and MMStar. On HR8K, 94.7% of examples use one turn; on MMStar, 88.33% use one or two turns, while other examples use zero or more turns. These distributions demonstrate variable visual reasoning depth across queries.

HRBench8K (turn)0124
Distribution3.4%94.7%0.9%1.0%
MMStar (turn)0123458
Distribution2.40%83.33%5.00%2.07%4.07%1.73%1.40%

2) End-to-end inference efficiency. We compare end-to-end inference time over the complete evaluation sets on the same A100-80GB GPU. SkiLa reduces runtime by 40.3% and 45.7% on HR8K and MMStar, respectively. By directly feeding back continuous hidden states for visual reasoning rather than verbalizing intermediate visual thoughts step by step, SkiLa achieves practical efficiency gains.

End-to-End Inference Time (HH:MM:SS)HR8KMMStar
Qwen3-VL-8B-Instruct00:42:0601:20:51
SkiLa-8B00:25:0900:43:56

3) Visual supervision design. The main text separately evaluates the sketch encoder, target-token count, and reconstruction weight. SigLIP2, 27 target tokens, and a reconstruction weight of 0.1 perform best among the tested settings, supporting the combination of semantic features, a moderate reconstruction length, and a small reconstruction weight.

Supervision choiceResultsSetting
QwenViT / CLIP / SigLIP2V*: 83.8 / 84.3 / 85.3; HR4K: 77.0 / 77.5 / 78.0Main text Table 4
1 / 9 / 27 / 81 target tokensV*: 83.2 / 84.3 / 85.3 / 84.8Main text Table 5
0.1 / 0.2 / 0.5 / 1.0 reconstruction weightV*: 85.3 / 84.3 / 83.8 / 82.7Main text Table 6

Dear Reviewer XWQo,

Thank you again for your time and valuable feedback. As the discussion deadline approaches, we would like to follow up on our responses. If any concerns remain, or if we have misunderstood any aspect of your comments, please let us know. We would be glad to provide further clarification.

Sincerely,

The Authors

Dear Reviewer XWQo,

As the discussion period draws to a close, we would welcome any further feedback on our rebuttal and would be glad to address any remaining concerns. Thank you again for your time.

Sincerely,

The Authors

Reviewer_HyZTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SkiLa, a framework that enables VLMs to interleave textual reasoning with latent visual reasoning tokens, represented as sketch tokens, within a single autoregressive generation process. During training, an external sketch module converts intermediate sketch images into visual targets to supervise the sketch token generation. This enables the model to combine language and visual reasoning in a unified trajectory. Experiments across multiple benchmarks show that SkiLa performs well against conventional text-only, visual-only, and recent latent reasoning methods.

Critical strengths
  • The topic is interesting. Whether VLMs are capable of performing multimodal reasoning seamlessly without external tools is an interesting topic for research. This paper further provide some signals for understanding what the latent tokens express, which is often absent in previous latent multimodal reasoning papers.
  • The experiments are comprehensive. The authors provide 6 tables covering ablations and hyperparameter selection in the main paper, as well as case studies.
Critical weaknesses
  • One weakness is the comparison in Table 1. SkiLa is compared with three multimodal latent reasoning methods. But all three methods were proposed over one year ago. I am wondering whether authors could include more recent baselines. Also, given they are proposed one year ago, they are using weaker base models, like Qwen2.5-VL, and the authors just use their numbers in their paper. Given the capability difference in base models, whether it is fair to directly compare is also questionable.
  • If computational resource allows, one comparison with same data and base model, just different method design, would provide a more direct understanding of SkiLa's effectiveness.
Questions for authors
  • One thing I am curious about SkiLa is the reliability of the visual latents. One good signal is that SkiLa provides the textual description of the latent tokens, providing a probing into the latents. I am wondering that, whether adding certain level of random noise to the latent tokens, will affect the performance a lot. My question is that whether the subsequent <sketch_end> token or textual description will calibrate the latent bias.
  • Currently the number of the sketch tokens is fixed by SFT performance. Do the authors think that the number of sketch tokens could be trained to be adaptive after the RL stage? My intuition is that different level visual thoughts may rely on different level of latent cot tokens. One simple task could even be solved with one or two sketch token. In Table 5, using only one sketch token only sacrifices 2 points performance on V*.
  • The 4k dataset would be valuable to the research community if it is of high quality. The authors could introduce that more to show its quality.
3 comments

1. Recent Baselines and Controlled Comparison (W1, W2)

We add reproductions of LVR and Monet on Qwen3-VL-8B to compare different methods on the same backbone.

1) LVR with matched data and backbone. LVR and SkiLa use the same training data and Qwen3-VL-8B backbone. SkiLa already outperforms LVR on all four benchmarks after SFT, and leads by 2.7/3.6/4.0/4.3 percentage points after RL. Its supervision and reasoning design thus remain effective when data and backbone are controlled.

Method (SFT / SFT + RL)V*HR4KHR8KMMStar
LVR84.3 / 85.377.9 / 77.971.9 / 73.565.3 / 67.0
SkiLa85.3 / 88.078.0 / 81.572.3 / 77.566.9 / 71.3

2) Monet with a matched backbone. Monet uses three-stage SFT with a different supervision construction from SkiLa; this reproduction therefore does not use SkiLa's training data. We reproduce Monet on the same Qwen3-VL-8B backbone. SkiLa leads by 1.1/3.0/2.2/3.2 percentage points across the four benchmarks, showing that its advantage is not solely due to a newer backbone.

Method (Qwen3-VL-8B)V*HR4KHR8KMMStar
Backbone83.876.571.364.7
Monet86.978.575.368.1
SkiLa-RL88.081.577.571.3

2. Reliability under Noise (Q1)

We add masking and noise experiments to evaluate the model when its latent states are perturbed.

1) Masking intervention. Randomly masking a fraction of the sketch tokens at inference causes accuracy to decrease monotonically. Thus, even with subsequent textual reasoning retained, masking latent information still affects the answers.

Masked Sketch TokensHR4KHR8KMMStar
0% (clean)81.577.571.3
33%80.175.669.4
66%79.372.567.8
100%78.572.066.7

2) Noise intervention. We add Gaussian noise ε ∼ N(0, σ²) to the latent tokens. Accuracy decreases as σ increases, further showing that subsequent answers depend on the latent content.

Noise Level (σ)HR4KHR8KMMStar
0 (clean)81.577.571.3
0.0578.273.167.4
0.174.970.662.5

3. Adaptive Sketch-Token Counts after RL (Q2)

After RL, the actual sketch length can vary rather than being fixed to the SFT reconstruction length. We report the distributions of latent-token counts and visual reasoning turns on HR8K below.

1) Latent-token counts. Generated lengths span multiple values on HR8K, and 3.4% of examples generate no latent tokens. This shows that the model does not always use the same visual-token budget.

Latent Tokens09273654
HR8K3.4%2.3%8.5%1.5%84.3%

2) Visual reasoning turns. Most HR8K examples use one sketch turn, while others use zero or multiple turns, showing that visual reasoning depth is also variable.

HRBench8K (turn)0124
Distribution3.4%94.7%0.9%1.0%

4. Quality and Composition of SkiLa-RL-4K (Q3)

Appendix A.2 describes the curation and composition of SkiLa-RL-4K, which addresses both prompt quality and informative RL learning signals.

1) Quality filtering. We use Qwen3-VL-235B as an automatic reviewer to filter out corrupted or low-resolution images, ambiguous or under-specified questions, and image–text inconsistencies, retaining prompts suitable for answer verification.

2) Difficulty filtering. We generate eight responses per prompt with the backbone and retain only prompts with both correct and incorrect responses. During RL, groups with zero reward variance are further discarded dynamically, focusing updates on examples that provide a relative learning signal.

3) Composition. The 4,130 prompts comprise Visual Perception (47.4%; fine-grained recognition and chart understanding), Visual Reasoning (42.6%; image-grounded mathematics and geometry), and Capability Enhancement (10.0%; captioning, OCR, and chart description). This combination covers visual perception, reasoning, and general instruction following rather than visual search alone.

We will release SkiLa-RL-4K as a reusable resource for research on visual latent reasoning and RL training.

Dear Reviewer HyZT,

Thank you again for your time and valuable feedback. As the discussion deadline approaches, we would like to follow up on our responses. If any concerns remain, or if we have misunderstood any aspect of your comments, please let us know. We would be glad to provide further clarification.

Sincerely,

The Authors

Dear Reviewer HyZT,

As the discussion period draws to a close, we would welcome any further feedback on our rebuttal and would be glad to address any remaining concerns. Thank you again for your time.

Sincerely,

The Authors

Reviewer_so7cOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Sketch-in-Latents, where An MLLM interleaves textual thoughts with continuous “latent sketch tokens” that are generated by feeding hidden states back as inputs. The model is trained to generate latent tokens of imagined views as reasoning trace. On Qwen3-VL-8B, SkiLa improves over the backbone and prior latent and tool-based methods on V*, HRBench, MME-RealWorld and general VQA benchmarks.

Critical strengths
  1. A simple, general recipe for multi-step interleaved latent reasoning. It needs no tools at inference and no pixel generation.
  2. Consistent gains over the Qwen3-VL-8B backbone on eight benchmarks, including general VQA benchmarks such as MMStar (+6.6).
Critical weaknesses
  1. Most of the gain also comes from RL (Table 3), and the RL data (Thyme, VisualProbe, DeepEyesV2) is in-domain for V* and HRBench. There is no text-only GRPO baseline on the same data, so the gains cannot be attributed to the latent sketches.

  2. There is no evidence that the latents are actually used. There is no inference-time intervention, such as removing or replacing sketch tokens. The similarity analysis is computed with a projector trained jointly under the MSE loss, which also risks collapse. The ablations are SFT-only and run on V* alone, where 1 point is about 2 questions.

Questions for authors
  1. How do Monet and text-only GRPO perform on the same backbone and data?
  2. How does accuracy change if sketch tokens are removed or replaced at inference?
  3. Is the sketch projector jointly trained, and how is target collapse prevented? Does the QwenViT variant use the native merger output as its target?
3 comments

1. Source of Gains and Matched RL Baselines (W1, Q1)

We add a text-only GRPO baseline with the same backbone and RL data, together with a reproduction of Monet on Qwen3-VL-8B, to examine the effects of RL training and backbone differences.

1) Comparison with text-only RL. The text-only baseline uses the same SkiLa-RL-4K data and RL settings as SkiLa-RL, but its intermediate reasoning consists only of textual CoT. SkiLa-RL outperforms it by 5.3/4.0/5.2/4.0 percentage points on V/HR4K/HR8K/MMStar, showing that SkiLa’s gains cannot be attributed to RL alone.

Method (Qwen3-VL-8B)V*HR4KHR8KMMStar
Backbone83.876.571.364.7
Text-only SFT + GRPO82.777.572.367.3
SkiLa-RL88.081.577.571.3

2) Comparison with Monet. Monet uses three-stage SFT with a different supervision construction from SkiLa; this reproduction therefore does not use SkiLa's training data. We reproduce Monet on the same Qwen3-VL-8B backbone. SkiLa leads by 1.1/3.0/2.2/3.2 percentage points across the four benchmarks, showing that its advantage is not solely due to a newer backbone.

Method (Qwen3-VL-8B)V*HR4KHR8KMMStar
Backbone83.876.571.364.7
Monet86.978.575.368.1
SkiLa-RL88.081.577.571.3

2. Inference-Time Intervention on Sketch Tokens (W2, Q2)

Beyond the attention and conditioned-captioning analyses in Figure 5 and Appendix B, we conduct masking and noise interventions on the final RL model to directly test the contribution of sketch tokens to the final answer.

1) Masking intervention. We randomly mask a fraction of the sketch tokens at inference. As shown below, accuracy decreases monotonically as the masked fraction increases, indicating that sketch tokens contribute to the answers.

Masked Sketch TokensHR4KHR8KMMStar
0% (clean)81.577.571.3
33%80.175.669.4
66%79.372.567.8
100%78.572.066.7

2) Noise intervention. We add Gaussian noise ε ∼ N(0, σ²) to the latent tokens. Accuracy decreases as σ increases, further showing that subsequent answers depend on the latent content.

Noise Level (σ)HR4KHR8KMMStar
0 (clean)81.577.571.3
0.0578.273.167.4
0.174.970.662.5

3. Sketch Projector and Reconstruction Targets (W2, Q3)

The sketch projector is jointly trained during SFT; the QwenViT variant directly uses native merger outputs as its visual targets.

1) Joint training. As described in Sec. 3.2, frozen SigLIP2 features are mapped into the LLM embedding space by a single-layer sketch projector, then pooled into 27 target vectors. The projector and LLM are jointly optimized through MSE. The sketch module is not used during RL or inference; sketch tokens are generated by the LLM itself.

2) Optimization constraints. The frozen encoder supplies pretrained visual-semantic features, while subsequent text is conditioned on the generated latents. The joint NTP objective therefore provides task-driven supervision for the latents, encouraging them to support both visual reconstruction and subsequent reasoning rather than merely minimizing MSE.

3) QwenViT targets. The QwenViT variant directly uses embeddings from the native vision encoder after the merger, without an additional sketch projector.

Two critical concerns are addressed and I've increased my rating.

We sincerely appreciate your response and are glad that our rebuttal has addressed your key concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.