acceptodds
Under review as a conference paper at ICLR 2027

Structured Latent Reasoning: Refactoring Reasoning with Reusable Primitives for Multimodal LLMs

Abstract

Chain-of-Thought reasoning has significantly advanced problem solving in multimodal large language models (MLLMs), yet autoregressive generation of lengthy reasoning traces incurs substantial token-level redundancy and computational overhead. Existing approaches compress reasoning into continuous latent states or discrete latent representations, but typically represent heterogeneous reasoning content within a shared latent space, making it difficult to efficiently encode both reusable reasoning patterns and instance-specific information. In this paper, we present **SLR**, a **S**tructured **L**atent **R**easoning framework that separates *reusable reasoning operations* from their *instance-specific executions*. SLR segments reasoning traces into unit spans, decomposes each span into an abstract operation and its concrete execution, and clusters recurring operations to construct a compact discrete *Reasoning Codebook*. During training, operation descriptions are replaced by discrete codebook tokens and interleaved with explicit execution text, allowing recurring procedural patterns to be represented compactly while preserving problem-specific computations. We evaluate SLR across diverse multimodal reasoning benchmarks with three MLLM configurations. Compared with standard SFT, SLR reduces mean output length by 10-35%, while matching or exceeding standard SFT accuracy in all settings. These results demonstrate that explicitly separating reusable reasoning structure from instance-specific execution provides an effective approach to reducing reasoning redundancy while maintaining multimodal reasoning performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Latent Reasoning, Multimodal Reasoning, Chain-of-Thought, Reasoning Abstraction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sKgSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a structured latent tokenization framework SLR that explicitly disentangles reusable reasoning patterns from instance-specific execution and represents it with latent operation tokens. SLR allows compact reusable operations to be combined with instance-specific computation, enabling interleaved latent and explicit reasoning.

Critical strengths
  • The proposed SLR successfully separates reusable procedural patterns from instance-specific computations, unlike prior methods that compress heterogeneous information into a shared space.
  • SLR allows precise control over compression granularity and codebook size using intuitive hyperparameters.
  • Extensive evaluation across diverse multimodal reasoning benchmarks exhibits that SLR reduces mean output token length by 10%-35% while matching or exceeding standard SFT accuracy.
Critical weaknesses
  • The quality of the discrete reasoning codebook relies heavily on pipeline components like automated span segmentation, decomposition prompts, and greedy leader clustering, which may introduce parsing errors or propagate noise into codebook tokens.
  • Ablation studies show that randomizing operation-token assignments or removing semantic grounding leads to a sharp drop in token utility and downstream accuracy, indicating that the model's performance is sensitive to the semantic integrity of the induced primitives.
  • Experiments are limited to Qwen model families, which restrict the evaluation of the method's generalization
Questions for authors
  • The paradigm of SLR is similar to classical vector quantization and neuro-symbolic approaches. Can you clarify the core breakthrough of SLR over existing discrete latent reasoning methods and classical codebook compression, beyond applying a span-segmentation and clustering recipe to multimodal CoT?
  • Can the learned reasoning codebooks be transferred zero-shot to other model architectures? And what is the performance of SLR on models out of the Qwen families, which is related to the generalization of the method?
  • Can SLR be applied to open-ended or non-algorithmic tasks where rigid procedural primitives cannot not be cleanly segmented?
  • What is the total computational cost of corpus-wide text segmentation, decomposition, and semantic clustering prior to training? Furthermore, does the lookup and decoding of codebook tokens introduce measurable inference latency overhead in practice?
3 comments

[W1] Quality of codebook construction

We agree that automated pipelines over large-scale data can introduce errors. To mitigate such noise in the final codebook, our construction pipeline includes quality-control and refinement mechanisms at both the decomposition and clustering stages. We briefly mentioned them in Sec. 4.3 and added additional details in Appendix A.

More broadly, SLR is designed as a modular framework. Its segmentation, decomposition, and clustering components can each be modified or improved as needed without changing the overall framework. Our results show the framework is effective with the current practices, despite the possibility of residual noise. This suggests that its effectiveness does not require a perfectly error-free construction process. More advanced segmentation or clustering methods can also be incorporated directly.


[W2] Semantic consistency and ablations on operation tokens

We thank the reviewer for raising this point, but we consider this ablation as supporting our approach rather than revealing a weakness. They serve as sanity checks to verify that the induced operation tokens carry meaningful, learnable semantics, rather than providing benefits merely through the introduction of additional tokens or a structured output format. Specifically, randomizing token assignments and removing semantic grounding are controlled interventions that disrupt different aspects of the intended operation-to-token correspondence. The resulting degradation is therefore expected and confirms the importance of semantically meaningful primitives.

More importantly, randomizing token assignments leads to a substantial reduction in operation-token utilization. When the same token is assigned to unrelated operations under different contexts across various training instances, there’s no longer a consistent function for the model to learn. Consequently, the model largely stops using these tokens and falls back on its standard reasoning. This behavior is informative because rather than blindly adopting arbitrary tokens, the model learns to avoid tokens that provide no consistent operation-specific information. This, in turn, demonstrates that when operation tokens are assigned meaningful and consistent semantics in the training data, the model can learn to associate them with their corresponding reasoning functions and leverage them to improve reasoning.

Crucially, semantic consistency is not an external condition on which SLR happens to depend; it is an explicit objective of our codebook induction procedure. We construct each operation token to represent a consistent, reusable reasoning function, and train the model to associate that token with the appropriate execution context. Semantic initialization provides an informative prior for the token embeddings, while the model learns to use these discrete representations during training. Deliberately breaking the operation-to-token correspondence would undermine the learnability of these discrete representations. Thus, these ablations demonstrate that SLR's gains arise from learning and utilizing semantically consistent reasoning primitives, rather than from the presence of additional tokens alone.


[W3] / [Q2] Generalization beyond the Qwen family

To directly evaluate generalization beyond the Qwen family, we conduct additional experiments on InternVL3.5-2B, an architecture from a different model family. We apply both SLR and standard SFT using the same training data, hyperparameters and evaluation settings.

MethodMATH-VisionMMStar
AccTokensAccTokens
Base model22.70178857.27591
SFT28.95226760.67512
SLR30.26202861.07405

SLR achieves higher accuracy than standard SFT while generating shorter responses. These results are consistent with our findings on Qwen models, demonstrating that the benefits of SLR are not specific to the Qwen family. Together with our main experiments, SLR is now validated on four backbones across two model scales (2B and 8B) and two model families (Qwen and InternVL), providing further evidence of its generalizability.

[W3] / [Q2] Generalization beyond the Qwen family

SLR is model-agnostic by construction. Its codebook is induced from reasoning traces by extracting and consolidating recurring abstract operations, independently of the target model's architecture, weights, or tokenizer. The resulting operation definitions and interleaved training data can therefore be reused across different model architectures. Applying SLR to a new model requires only adding the corresponding operation tokens to its vocabulary and training on the interleaved data using standard SFT. The operation-token embeddings are learned separately for each target model during training. In our InternVL3.5 experiments, we reuse the same codebook and data pipeline without modification, demonstrating that the induced reasoning primitives transfer across model families without requiring model-specific codebook construction.


[Q1] Clarifying the core breakthrough of SLR

Our core contribution is not simply applying span segmentation and clustering to multimodal CoT. Rather, SLR introduces a different formulation of discrete reasoning by changing what is represented by discrete tokens: instead of quantizing entire reasoning spans, it discretizes only the reusable operations underlying those spans, while keeping their instance-specific executions explicit. Consequently, no instance-specific information passes through the codebook, which allows a compact set of reusable operation tokens to represent recurring reasoning patterns.

In addition, SLR is lightweight and fully compatible with standard SFT pipelines. It introduces no additional encoder or decoder modules, no architectural changes, and has no extra reconstruction or alignment objectives. The model is trained directly with the standard next-token prediction objective over an expanded vocabulary, which makes SLR straightforward to integrate into existing training pipelines.

In summary, SLR's central contribution is a structured formulation of discrete latent reasoning that explicitly separates reusable procedural knowledge from instance-specific information. By discretizing only the former and retaining the latter in the reasoning sequence, SLR provides a compact and interpretable vocabulary of reasoning primitives that the model can learn to invoke during generation, without requiring additional architectural components or training objectives.


[Q3] Applicability beyond rigid procedural reasoning tasks

We thank the reviewer for this question. We clarify that SLR does not rely on rigid procedural boundaries for segmentation. Instead, it discovers recurring patterns across reason traces, and such patterns are not limited to algorithmic procedures. For example, non-algorithmic visual reasoning can also involve recurring operations, such as ANALYZE_SPATIAL_LAYOUT and LOCATE_TARGET_OBJECT. Moreover, these primitives are optional to the model: the model uses them when a reusable pattern applies, and otherwise continues reasoning in natural language. Thus less structured reasoning stays fully expressible.

Our evaluation already provides evidence that SLR is applicable beyond rigid procedural tasks:

  • MMVP evaluates basic visual capabilities, including orientation, position, and viewpoint recognition. SLR improves accuracy by 2-3% over standard SFT while generating fewer reasoning tokens.
  • MMStar covers six capability dimensions, including coarse and fine-grained perception. SLR achieves slightly higher accuracy than standard SFT on both subsets.

These results suggest that SLR is not restricted to rigid algorithmic reasoning. It can also capture recurring patterns in less procedural forms of reasoning, while retaining natural language as a fallback for steps that do not correspond to reusable primitives.


[Q4] Codebook construction cost and inference latency

Total cost prior to training. SLR introduces a one-time offline step for codebook construction. The construction pipeline makes approximately 2 LLM calls per training trace on average to decompose reasoning traces and construct the codebook. Approximately 3 short operation strings per training trace are called for embeddings using Gemini Embedding on average. All calls are independent and can be executed in parallel. Importantly, the resulting codebook and processed training data can be reused across different models and multiple training runs. The construction costs are incurred only once for each dataset, rather than being repeated for each model or training run.

[Q4] Codebook construction cost and inference latency

Latency overhead. We measure inference latency for Qwen3-VL-2B-Instruct trained with standard SFT and with SLR. Both models were evaluated on a single NVIDIA L40S GPU, using the same Hugging Face Transformers inference setup, with batch size 1, greedy decoding, and a maximum of 4,096 new tokens. We report the average tokens per sample, mean per-sample latency, and decoding throughput:

MethodMATH-VisionMMStar
TokensLatency (s)Tokens/sTokensLatency (s)Tokens/s
Standard SFT240594.925.338715.125.6
SLR197877.525.534713.825.1

SLR reduces latency by 18.3% on MATH-Vision and 8.6% on MMStar, corresponding to output token reduction of 17.8% and 10.3%, respectively. Decoding throughput remains nearly identical for both models (approximately 25 tokens/s). This is expected because SLR retains the same model architecture and autoregressive decoding procedure as standard SFT, with only a small number of additional codebook tokens added to the vocabulary (less than 0.1% of the backbone model vocabulary). The inference follows the normal autoregressive next-token prediction paradigm. Therefore, the latency reductions primarily result from shorter generated sequences.

Reviewer_45hcOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Long cot helps MLLMs reason, but the traces are full of repeated procedural text (“let’s solve step by step"). Prior compression work pushes reasoning into continuous latents or a shared discrete codebook, and those approaches squeeze reusable reasoning patterns and instance-specific content into the same space. SLR separates the two. An LLM (Gemini) segments each training trace into spans and splits each span into an abstract operation and its concrete execution. Frequent operations are clustered into a small vocabulary of named special tokens and the model is SFT’d on interleaved sequences where each operation is a single token and the execution stays in plain text. The main novelty is this operation/execution disentanglement, done as pure data refactoring with no architecture or loss changes. On Qwen3-VL-2B/8B across benchmarks, SLR cuts output length by roughly 10–35% relative to standard SFT and matches or slightly beats its accuracy.

Critical strengths
  • The idea is simple and well motivated. Separating “how to reason” from “what it’s applied to” is a cleaner answer to the shared-codebook capacity problem than just making the codebook bigger.
  • It needs no architecture changes, auxiliary heads, or custom losses, so it plugs straight into any SFT pipeline.
  • The token ablations are well designed. Random-uniform, frequency-matched permutation, a single unified marker, and random init each isolate a different factor. On MATH-Vision, semantic assignment clearly matters (44.4 for SLR vs. ~37 for the randomized variants).
Critical weaknesses
  • An ablation experiment wort investigating is that most of the compression may come from the LLM rewriting the traces, not from the codebook. Executions are Gemini-rewritten “concise text,” and the Unified variant (one generic marker, no operation identities) already drops MATH-Vision length from 1920 to 1331, while SLR reaches 1245. Most of the length reduction therefore appears without distinct operation tokens. The missing control is SFT on the decomposed traces with the operations kept as natural-language text.
Questions for authors
1 comment

We thank the reviewer for the positive assessment and for suggesting this controlled experiment on the source of compression. To separate the contribution of the codebook from teacher rewriting, we train Qwen3-VL-2B-Instruct on the same decomposed traces under two additional settings

  • Execution text only: all operation tokens are removed and only the instance-specific execution text is kept. This is an equally concise natural-language baseline.
  • With operation descriptions: natural-language operation descriptions are included for training, followed by the same execution text.

The accuracy and average tokens per sample are shown below:

MethodMATH-Vision
AccTokens
SLR25.001978
Execution text only22.931932
With operation descriptions23.462071

On MATH-Vision, SLR outperforms both ablations with a comparable output length. We agree that the length reduction mainly comes from the concise execution text. Both the Unified variant and Execution text only shortens outputs at the expense of reasoning quality. While SLR keeps outputs at a similarly short length and achieves higher accuracy. These results suggest that the codebook tokens can provide explicit operation-level signals that guide the subsequent execution text. Writing the operations out in natural language doesn’t give the same benefit. It recovers part of the accuracy and produces the longer outputs, because the model must generate multiple tokens to express each operation rather than invoking it with a single one.

Reviewer_7sSyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SLR, which separates reasoning traces into reusable operations and instance-specific execution text. Operations are clustered and replaced with special tokens for SFT. Experiments on four multimodal benchmarks show shorter outputs with comparable or slightly improved accuracy over standard SFT.

Critical strengths
  1. The distinction between reusable operations and instance-specific content is sensible. Keeping calculations and visual observations explicit may help avoid information loss during compression.

  2. Results are consistent across the two Qwen3-VL sizes: output length decreases on all four benchmarks without an observed accuracy drop.

Critical weaknesses
  1. The contribution of the codebook is not isolated from that of teacher-generated data rewriting. A baseline trained on equally concise, teacher-rewritten natural-language traces is missing.

  2. The interpretation of the global-permutation ablation is questionable. A fixed permutation preserves a learnable operation-to-token mapping; degradation may instead reflect a mismatch with semantic initialization.

  3. The efficiency evaluation reports output tokens rather than measured latency or total cost. And several training and codebook details are missing.

Questions for authors
  1. How does SLR compare with the same decomposed training data using full operation descriptions, or execution text alone?
  2. Were the Figure 2 baselines retrained with the same data and budget?
  3. How were checkpoints and hyperparameters selected for Figure 3?
2 comments

[W1] / [Q1] Isolating the contribution of the codebook from teacher rewriting

We thank the reviewer for pointing this out. To isolate the contribution of the codebook from teacher rewriting, we train Qwen3-VL-2B-Instruct on the same decomposed traces under two additional settings:

  • Execution text only: all operation tokens are removed and only the instance-specific execution text is kept. This is an equally concise natural-language baseline.
  • With operation descriptions: natural-language operation descriptions are included for training, followed by the same execution text.

The accuracy and average tokens per sample are shown below:

MethodMATH-Vision
AccTokens
SLR25.001978
Execution text only22.931932
With operation descriptions23.462071

On MATH-Vision, SLR outperforms both ablations with a comparable output length. Removing operation tokens reduces accuracy, suggesting that they provide explicit operation-level signals that guide the subsequent execution text. Writing the operation descriptions out produces longer outputs, as the model must generate multiple subword tokens to express each operation rather than invoking it with a single token. These results show the codebook provides benefits beyond teacher rewriting.


[W2] Interpretation of the global-permutation ablation

We thank the reviewer for this point and agree that "globally permutes" was ambiguous. To clarify, we do not apply a fixed permutation of the operation-to-token mapping, which would preserve a consistent, learnable mapping. Instead, we randomly shuffle the operation tokens across all operation positions in the training data. This keeps only the exact token frequency distribution and the structural positions where operations appear, while decoupling each token from the actual operation performed at that position. No consistent mapping can be learned from the shuffled data.

The reviewer’s alternative explanation, a mismatch with semantic initialization, is tested by our random-init ablation (Tab. 2). It keeps the consistent operation-to-token mapping but removes the semantic initialization. The two ablations exhibit different token utilization behaviors:

  • Random init: Despite removing semantic initialization, the model learns to use the operation tokens extensively, emitting 2.51 operation tokens per response on MATH-Vision and using them in 95.7% of MMStar responses.
  • Random (shuffle): Breaking the operation-to-token correspondence substantially reduces token utilization, to 0.22 tokens per response on MATH-Vision and 71.5% of MMStar responses. The model also tends to fall back to the most frequent token (Fig. 6) rather than learning meaningful operation-specific token usage. This is the expected behavior when token identity carries no predictable information about the operation.

We have revised Sec. 5.3.2 to clarify the procedure more precisely.


[W3] Measured latency and training/codebook details

Inference efficiency (measured latency). Following the reviewer’s suggestion, we measure inference latency for Qwen3-VL-2B-Instruct trained with standard SFT and with SLR. Both models were evaluated on a single NVIDIA L40S GPU, using the same Hugging Face Transformers inference setup, with batch size 1, greedy decoding, and a maximum of 4,096 new tokens. We report the average tokens per sample, mean per-sample latency, and decoding throughput:

MethodMATH-VisionMMStar
TokensLatency (s)Tokens/sTokensLatency (s)Tokens/s
Standard SFT240594.925.338715.125.6
SLR197877.525.534713.825.1

SLR reduces latency by 18.3% on MATH-Vision and 8.6% on MMStar, corresponding to output token reduction of 17.8% and 10.3%, respectively. Decoding throughput remains nearly identical for both models (approximately 25 tokens/s). This is expected because SLR retains the same model architecture and autoregressive decoding procedure as standard SFT, with only a small number of additional codebook tokens (less than 0.1% of the backbone model vocabulary). Therefore, the latency reductions primarily result from shorter generated sequences.

Training details. Sec. 5.1 lists the backbones, training datasets, the number of training steps, and codebook token initialization. Standard SFT and SLR use identical training configurations, with the only difference being the target sequences, where SLR has operation tokens interleaved with execution texts. We use the AdamW optimizer with a learning rate of 1×10−51 \times 10^{-5} and a cosine learning rate schedule. Training is run with a global batch size of 128 across 8 NVIDIA H100 GPUs. We conduct full-parameter language model training, while keeping the vision encoder and multimodal projector frozen.

[W3] Measured latency and training/codebook details

Codebook details. In the main results, we use Fmin=400F_{min} = 400 and τsim=0.84τ_{sim} = 0.84 to construct the codebook. This gives a codebook of 93 operation tokens, and each training sample contains 3.4 operation tokens on average. Appendix Tab. 3 lists the most frequent operation tokens used on each benchmark.


[Q2] Baseline evaluation protocol

We did not retrain the baselines. Instead, following the common practice of evaluating against existing methods [1, 2], we use their publicly available checkpoints. All methods in Fig. 2, including SLR, are built on the same Qwen2.5-VL-7B-Instruct backbone, providing a consistent model foundation for comparison.

The training data and procedures differ across methods because they are intrinsic to their respective design. For example, LVR requires auxiliary images for each training sample, while Monet relies on dedicated interleaved text-image reasoning traces that our data does not include. Their training data is also built around their own training objectives. Retraining them on our data, or our method on theirs, would therefore not be a faithful reproduction of either. For a controlled comparison, we compare SLR against the SFT baseline, using the same backbone, training data, and training budget (Tab. 1). This comparison is the primary evidence for its effectiveness.

[1] Wang, Qixun, et al. "Monet: Reasoning in latent visual space beyond image and language." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[2] Cui, Jin, et al. "Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning." arXiv preprint arXiv:2605.07106 (2026).


[Q3] Checkpoint and hyperparameter selection

All reported results use a fixed training budget: every model, including standard SFT, is evaluated at its final 3,300-step checkpoint (circles in Fig. 3). The triangles indicate the peak accuracy reached during training and are included for illustrative purposes only. For the hyperparameter analysis, Fig. 3 sweeps Fmin∈F_{min} \in {50, 100, 200, 300, 400, 500} and τsim∈τ_{sim} \in {0.80, 0.84, 0.88}. All other codebook construction and training settings are held fixed. The results show that our conclusions are robust to these choices: all configurations produce shorter reasoning traces than standard SFT, and most achieve comparable or higher accuracy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.