acceptodds
Under review as a conference paper at ICLR 2027

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains

Abstract

Explicit chain-of-thought (CoT) reasoning substantially improves the reasoning ability of large language models (LLMs), but incurs high inference cost due to lengthy autoregressive traces. Existing latent reasoning methods offer a promis- ing alternative, yet they often treat reasoning as uniformly compressible, causing precision-critical intermediate steps to be overly compressed and thereby degrading reasoning accuracy. In this work, we propose Selective Latent Thinking (SLT), a reasoning paradigm that selectively compresses redundant reasoning spans into latent representations while preserving precision-critical spans as explicit CoT within the same reasoning trajectory. Our primary instantiation uses a lightweight decoder to anticipate a short upcoming reasoning span and applies confidence- based gating to determine the longest span that can be reliably compressed; the resulting selective compression policy is further optimized with trajectory-level reinforcement learning to balance answer correctness against reasoning cost. Ex- periments across four mathematical reasoning benchmarks show that which spans are compressed is critical: with the same compressor, compressing arbitrary spans reduces the GSM-Hard accuracy of Qwen3-4B from 54.7% to 17.3%, whereas keeping numerical tokens explicit retains 51.8%. SLT reduces explicit reasoning length by 58.4% with only a 2.8-point accuracy drop compared to explicit CoT on Llama-3.2-1B. Notably, the principle extends beyond this design: instantiated within existing latent reasoning models via a learned compression gate, it improves accuracy at equal reasoning length (+2.6 points over Latent-SFT) and at shorter length (+5.6 points over Latent-GRPO with 13% fewer tokens).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_oiEpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Selective Latent Thinking (SLT), a reasoning paradigm that adaptively interleaves implicit latent representations with explicit CoT within a single reasoning trajectory, addressing the high inference cost of explicit chain-of-thought reasoning and the problem that existing latent reasoning methods adopt a "homogeneous compressibility assumption," causing precision-critical spans to be over-compressed. To this end, the authors introduce a lightweight feature decoder that anticipates short-horizon future reasoning trajectories, and a confidence gate that selects the longest continuous prefix that can be reliably compressed; this prefix is encoded by a cross-attention compressor into a fixed-length latent block, while the remaining precision-critical spans are preserved as explicit CoT. The framework is trained in three stages (latent compression learning, reliability-aware feature decoding, and trajectory-level GRPO optimization), with a dual-ascent constraint on the latent share to prevent the model from retreating to fully explicit reasoning. The discussion of the deficiencies of the homogeneous compressibility assumption is generally clear, and the design of the compression and reliability mechanisms is reasonably well motivated. The main contribution lies in shifting the "what to compress" decision from a uniform assumption to span-wise decisions based on per-span reliability, trading off answer correctness against explicit token cost via trajectory-level reinforcement learning, and transferring this principle to continuous-thought models (Latent-SLT). The authors conduct systematic experiments and ablations on mathematical reasoning benchmarks such as GSM8k-AngNL: explicit reasoning length is reduced by 58.4% with only a 2.8-point accuracy drop on Llama-3.2-1B, and SLT surpasses RoT by 17.33% at a comparable compression ratio on Qwen3-4B; ablations show that random compression drops GSM-Hard accuracy from 54.76% to 17.3%, confirming the decisive role of which spans are compressed. Finally, comparisons against Latent-SFT, Latent-GRPO, and other methods, together with general-ability retention results (MMLU, ARC, HellaSwag), are used to argue for SLT's superiority and generality.

Overall, this paper could be a significant algorithmic contribution, but several theoretical and experimental details require clarification: there is an objective mismatch between the reliability losses and the "longest continuous reliable prefix" rule at inference time (with continuity never explicitly optimized), the rationale for the hyperparameter ρ = 0.73 is not provided, and comparisons against more established non-uniform/adaptive compression baselines are missing. Given clarifications on these points in an author response, I would be willing to increase the score.

Critical strengths
  • The paper designs a dynamic CoT compression method that compresses only those spans amenable to predictable segment-level compression, thereby preserving reasoning accuracy as much as possible. This further confirms that the homogeneous compressibility assumption is fundamentally flawed.

  • The framework is technically complete, with each component supported by ablations, and the empirical results are compelling. This demonstrates that the method is not only conceptually interesting but also practically effective and controllable — the accuracy-length trade-off can be freely adjusted at inference time, which is precisely what determines whether such a method can be adopted in practice.

  • Control experiments (e.g., the acc-only control in Table 8, where the latent share collapses to 16.9% and the length doubles) also show that the authors understand and have addressed the failure modes of their own objective function, which increases my confidence in the experimental conclusions.

Critical weaknesses
  • At inference time, the compression decision follows the rule "take the longest continuous reliable prefix satisfying c_j ≥ τ," but both L_eosch and L_gate on the training side are per-position objectives. No term optimizes the length of the continuous reliable prefix or the temporal continuity of reliability. Consequently, the model may well learn a "jagged" reliability pattern in which position j is reliable, j+1 is unreliable, and j+2 is reliable again — and the prefix rule would discard all reliable spans after j along with it, unintentionally suppressing the compression rate. The paper's response is to explicitly defer long-horizon consequences to Stage 3 trajectory-level RL ("the first two stages establish local mechanisms... but they do not directly optimize the long-horizon consequences"), but the RL reward is merely an ex-post correction (and it only penalizes explicit tokens without constraining prefix structure), not a structural guarantee at the level of the training objective. This is critical, because it undermines the paper's central selling point — that "selective compression decisions are learned through a reliability-aware mechanism"; the disconnect between the decision mechanism and the training signal means that the observed gains may stem in part from RL's global redistribution rather than from reliability estimation itself. In addition, key design details such as the span selection rule in Stage 1 of the main framework and the specific value of the latent block length N are left unspecified, further complicating mechanistic attribution.

  • The experimental baselines consist mainly of pure explicit CoT (SFT-CoT, SFT-w/o CoT), token-level reduction (TokenSkip), and uniform/fully implicit methods (CoLaR, RoT, Coconut, Latent-SFT/GRPO). The problem is this: the paper's core claim is that "selectivity" is what brings the gains, yet apart from two heuristic ablations (Compress-Random, Compress-SkipNum) and one learned-gate control (Ours-RL-gateω), no existing non-uniform/adaptive compression method is included as a baseline. This leaves one alternative explanation unexcluded: the gains may come from "any form of non-uniform compression" (e.g., entropy-based, semantic-boundary-based, or sentence/step-level adaptive compression) rather than from the specific "reliability-aware gating" mechanism proposed in this paper. Meanwhile, although the related work section discusses hybrid ideas such as Coconut, Medusa, and multi-token prediction, no hybrid explicit-implicit reasoning method is directly compared against in the experiments. This point is decisive for my overall assessment.

Questions for authors
  • Inference uses the "longest continuous reliable prefix satisfying c_j ≥ τ" rule: please clarify: (a) how the trailing reliable spans abandoned after prefix truncation are handled at inference time (are they re-considered for compression at subsequent steps)? (b) Have you observed the actual impact of jagged reliability distributions on the compression rate? (c) Have you considered structured objectives on prefix length or continuity?

  • Rationale for key design parameters: (a) How was ρ = 0.73 determined? (b) What is the specific value of the latent block length N? (c) Is the span segmentation rule for constructing hybrid sequences in Stage 1 based on random sampling or some other criterion?

  • Stage 3 RL globally reshapes the compression distribution. Please provide an end-to-end breakdown of compression rate and accuracy using only the Stage 2 gate (without RL), to disentangle how much of the gain comes from "reliability estimation" itself versus trajectory-level RL redistribution.

  • Stage 3 only penalizes explicit tokens (β = 0.01) and does not penalize the number of latent blocks. Under acc-only rewards or extreme thresholds, compression collapse and doubled length have been observed. Please describe under what conditions the method may degenerate into being "longer than SFT-CoT," and whether there are diagnostic metrics to help practitioners avoid this situation.

Reviewer_QNrLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose SLT. It is a hybrid reasoning framework that selectively replaces predictable spans of explicit chain-of-thought with latent representations while preserving precision-critical spans explicitly. SLT combines a feature decoder, a confidence-based gate, a latent compressor, and trajectory-level GRPO optimization. Experiments on some benchmarks using Llama-3.2-1B and Qwen3-4B show substantial reductions in explicit reasoning length with moderate accuracy loss. Finally, extensions to Latent-SFT and Latent-GRPO suggest that selective compression is useful beyond the proposed architecture.

Critical strengths

The paper identifies a clear and important limitation of existing latent reasoning methods of treating all reasoning spans as equally compressible. The proposed selective-compression mechanism is intuitive, technically coherent, and supported by informative ablations. Results are generally consistent across several mathematical benchmarks and model sizes, and the extension to existing latent reasoning models strengthens the claim that selectivity is a general principle rather than an artifact of one compressor.

Critical weaknesses

Several baseline results are taken from prior work or use different experimental settings, while the paper reports no multi-seed variance or statistical uncertainty. Explicit-token length is an imperfect efficiency proxy, and the wall-clock evaluation uses a single GPU without reporting throughput, memory, batch-size effects, or the complete overhead of the feature decoder and compressor. Finally, the main RL reward appears to penalize length only for correct trajectories, the authors should clarify the effect of this design and the protocol used to select thresholds and operating points.

Questions for authors
  1. Some matched baseline reruns and multi-seed uncertainty estimates could further improve the papers strength, please refer to the Critical Weaknesses above.
  2. At inference time, does the compressor operate on sampled or argmax tokens produced by the feature decoder, or directly on hidden states?
  3. How frequently do the decoder's predicted spans disagree with the primary LLM, and what is the accuracy impact of using predicted spans instead of teacher spans?
  4. Eq 9 seems to assign zero reward to incorrect trajectories and apply the length penalty only when the answer is correct. Is this intentional, and how does an unconditional length penalty affect the learned policy and accuracy-efficiency Pareto frontier? Please also clarify how the final threshold is selected without test-set tuning.
Reviewer_6JkqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes selective latent thinking. It argues that reasoning chains shouldn’t be compressed uniformly. Predictable parts can be squeezed into latent vectors, while hard parts like new calculations should stay as explicit text. Its method uses a small draft head to decide which spans to compress, and is trained with RL to trade accuracy against length. The core insight holds up: which spans you compress matters more than how much. The method and results are not so convincing - It is no better than simply deleting unimportant tokens, its accuracy cost grows on stronger models, and it was only tested on small models and grade-school math.

Critical strengths
  1. Core idea makes sense. The experiments show fairly clearly that which spans you compress matters a lot. Compressing arbitrary spans hurts badly, keeping the hard parts explicit recovers most of the accuracy, and a learned gate beats random selection. This idea is also not tied to a specific architecture. Adding the gate to an existing latent reasoning model gives similar accuracy.
Critical weaknesses
  1. This paper overlooks a set of closely related prior works that tackle exactly the same problem. Most of them are not cited, and the two that are cited (CODI, LightThinker) are not compared against. The most important group already mixes latent and explicit reasoning inside one trajectory, which is the paper’s central idea:
  • System-1.5 Reasoning (NeurIPS 2025) starts from almost the same motivation: existing latent methods treat all steps uniformly. It also skips trivial steps and keeps critical ones.
  • SwiReasoning (ICLR 2026) and DyLaR (AAAI 2026) switch between latent and explicit modes based on the model’s confidence or entropy. This is essentially the paper’s gating idea, done without extra training.
  • HRPO (NeurIPS 2025) learns a gate between token embeddings and hidden states with RL.
  • Token Assorted (ICML 2025) interleaves latent and text tokens in one chain.

Given these, “homogeneous compressibility” is not an unaddressed gap, and the paper’s novelty needs to be argued against them.

  1. The comparisons don’t show the method is actually better. The large gains over prior latent methods mostly happen because those baselines barely work in this setting; some are no better than skipping reasoning entirely. Against simpler options the advantage disappears. Just deleting unimportant tokens does about as well, and the draft head could be used for standard speculative decoding, which speeds things up without losing any accuracy, but this is never compared. The explicit-CoT baseline also isn’t RL-trained, while their method is, so the reported accuracy gap is likely understated.

  2. The scope is narrow. All experiments are done on small models and grade-school math with short reasoning chains.

  3. The results are not scalable; it even gets worse as models get stronger. The accuracy cost grows from a few points on the 1B model to much larger drops on 4B and 7B. This makes it hard to believe the approach scales to the long-reasoning settings where compression would actually matter.

Reference

  • System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts. Xiaoqiang Wang, Suyuchen Wang, Yun Zhu, Bang Liu. NeurIPS 2025.
  • SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs. Dachuan Shi, Abedelkadir Asi, Keying Li, Xiangchi Yuan, Leyan Pan, Wenke Lee, Wen Xiao. ICLR 2026.
  • Beyond Tokens: Dynamic Latent Reasoning via Semantic Residual Refinement. Fangrui Lv, Lei Wang, Ruixin Hong, Yong Du, Xiangyu Wu, Tingting Gao, Guorui Zhou, Changshui Zhang. AAAI 2026.
  • Hybrid Latent Reasoning via Reinforcement Learning. Zhenrui Yue, Bowen Jin, Huimin Zeng, Honglei Zhuang, Zhen Qin, Jinsung Yoon, Lanyu Shang, Jiawei Han, Dong Wang. NeurIPS 2025.
  • Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning. Dijia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, Qinqing Zheng. ICML 2025.
Questions for authors
  1. Is the latent vector actually needed? Keep the same gate decisions, but replace the compressed vector with a zero or mean embedding, or simply drop the accepted span. If accuracy barely changes, the gains come from skipping predictable tokens (much like TokenSkip) rather than from latent reasoning.

  2. Comparison with speculative decoding. The drafter plus gate is essentially a speculative-decoding draft head with an acceptance predictor. If the same drafter were used for lossless speculative decoding on the SFT-CoT model, it would keep full accuracy. How does that compare to SLT in wall-clock time? What does SLT gain beyond a shorter KV cache? Please give latency and memory numbers with the batch size stated.

  3. Fair explicit baselines. SLT is RL-trained, but the explicit-CoT baseline is not. Please add explicit CoT trained with the same GRPO budget, using the same correctness-gated length penalty. Please also run the “keep numbers explicit” heuristic end to end at matched length, and include CODI and SwiReasoning on the same backbone.

  4. Sampling temperature. All results use greedy decoding, but explicit-CoT models are often run with temperature sampling, both to reach less common reasoning paths and for pass@k or majority voting. How does SLT behave at, say, T = 0.6 / 1.0? I’d expect high temperature to make the drafter agree with the sampled tokens less often, so the gate accepts fewer spans and the length savings shrink. It’s also possible that compressing spans reduces trajectory diversity and hurts pass@k. Please report accuracy, explicit length and gate acceptance rate across temperatures, plus pass@k and majority-vote results. Please also clarify whether the drafter’s tokens are sampled or taken greedily at inference, and whether temperature is applied to the drafter, the gate, or only the main model.

Reviewer_Eb9LOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective Latent Thinking (SLT), a framework that shortens reasoning traces by selectively compressing spans into latent representations while retaining explicit generation for potentially precision-critical steps. A lightweight decoder anticipates upcoming tokens, and a confidence gate determines which spans to compress. The framework is trained through supervised learning followed by reinforcement learning to balance answer correctness and reasoning cost.

Critical strengths
  1. The paper addresses a practically relevant problem: improving reasoning efficiency while limiting accuracy degradation. The design is well motivated, and content-dependent compression offers greater flexibility than applying a uniform compression policy throughout the reasoning trajectory.
  2. The experiments provide evidence that the choice of compression locations matters. In particular, Table 7 shows that a learned gate achieves higher accuracy and shorter reasoning trajectories than a content-independent random policy on the same student model.
Critical weaknesses
  1. The gate supervision primarily measures agreement between the feature decoder and the backbone’s token predictions. This supports identifying predictable spans more directly than identifying precision-critical spans: agreement does not necessarily imply that compression will preserve the information needed for downstream correctness. Although RL may improve the policy, further analysis is needed to validate the proposed interpretation of the gate.
  2. Average accuracy obscures substantial losses on individual benchmarks. In Table 1, Ours-RL underperforms SFT-CoT by 7.02 percentage points on GSM for Llama-3.2-1B and by 15.13 percentage points on GSM-Hard for Qwen3-4B. These results suggest that accuracy is better preserved on simpler tasks, while challenging numerical reasoning remains sensitive to compression.
  3. The comparison in Table 4 does not control for compression budget. On GSM-Hard with Qwen3-4B, Compress-Random retains approximately 30% of the original reasoning length, whereas Compress-SkipNum retains approximately 80%. Consequently, the accuracy gap cannot be attributed solely to preserving numerical tokens, since the latter also retains substantially more context. Although Table 7 provides complementary evidence for learned selectivity, a matched-budget comparison is needed to isolate the benefit of preserving numerical spans.
Questions for authors
  1. Under Equation (9), with α = 1 and β = 0.01, a correct trajectory with more than 100 counted explicit CoT tokens would receive a negative reward, whereas an incorrect trajectory would receive zero. Is this behavior intended? Does the implementation apply clipping, normalization, or another mechanism to address this issue?
  2. How is reasoning length calculated across methods? Does it include only explicit tokens, or both explicit tokens and latent positions? How is the computational cost of the auxiliary decoder accounted for?
  3. Can the authors provide direct evidence that the gate identifies compression-sensitive spans rather than merely predictable ones—for example, by measuring the downstream accuracy impact of compressing accepted versus rejected spans under matched compression budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.