acceptodds
Under review as a conference paper at ICLR 2027

SuperThoughts: Reasoning Tokens in Superposition

Abstract

Long Chain-of-Thought (CoT) reasoning improves LLM problem-solving but is computationally expensive due to sequential token generation. While recent works explore reasoning in continuous latent spaces to bypass discrete token generation, they often struggle with training stability and fail to scale to complex, long-horizon tasks due to lack of supervision signal. We propose SuperThoughts, which compresses pairs of consecutive CoT tokens into single latent representations and decodes two tokens per step via a lightweight Multi-Token Prediction (MTP) module. This preserves discrete token supervision at training time while increasing throughput at inference time. We post-train Qwen2.5-Math-1.5B-Instruct, Qwen2.5-Math-7B-Instruct, Qwen2.5-Math-14B-Instruct, and evaluate on MATH500, AMC, OlympiadBench, and GPQA-Diamond. With a confidence-based adaptive mechanism that falls back to standard decoding when uncertain, SuperThoughts achieves 20-35% CoT length reduction while maintaining accuracy within 1-2 points of the baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Superposition, Reasoning, Adaptive Compute

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZvorOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SuperThoughts makes the generation of a chain of thought faster by producing two text tokens in a single pass of the language model. The main model predicts the first token, and a smaller prediction module predicts the second one. Then, the learned compressor combines these two tokens into a single input for the next forward pass of the main model. The paper examines this method with Qwen models with 1.5B, 7B, and 14B parameters on the MATH500, OlympiadBench, and AMC23 datasets. The take-home message is mainly the empirical performance, which results in higher accuracy with a reduction in reasoning length compared to the baseline HAMburger at the 1.5B scale. The authors show that generation time is approximately 21 to 28% less compared to the chain-of-thought method.

Critical strengths
  1. The method has been tested on a wide and good range of reasoning tasks with strong empirical performance For instance, the 1.5B projection model scores 73.0% on MATH500 compared to 72.4% for the standard CoT, and it reduces the correct-chain length from 494.1 to 314.8 positions (Table 1).

    The paper also includes relevant comparisons in runtime measurements. Figure 5 shows that using fewer decodings results in less generation time, taking the extra modules and the fallback overhead in the implementation into consideration.

  2. The paper is well-written and clear to follow. The motivation and experiments are clearly presented and understandable enough to let readers easily digest this work.

Critical weaknesses
  1. The paper does not provide a clear enough advantage when compared to related methods. HAMburger, for instance, combines several tokens into a single input representation, employs a smaller decoder to produce multiple tokens during each pass of the main model, and modifies the generation process by applying a confidence threshold. The paper does not identify which of the changes is responsible for the improvement. I would expect a more thorough ablation study to test this, for example, by examining the first distillation stage. This is a novelty concern, and to me, the benefit seems to stem from an improved training and decoding procedure within an existing architectural approach.

    The discussion of supervised latent reasoning is also incomplete. SIM-CoT and MUX introduce the intermediate supervision and improves latent reasoning benchmarks. Neither approach is discussed or evaluated, even though intermediate supervision is central to the paper's motivation (Introduction, lines 38–48). Appendix A states that a CODI reproduction attains an accuracy of approximately 0.1%, which is a clear failure; thus, a good understanding of the reasoning behind it is necessary. The paper includes no information regarding the configuration or the diagnostic results of that experiment (lines 778–783). These distillation methods are reducing the latency comparatively more without reducing accuracy too much.

  2. The timing results need clearer experimental settings and more baselines to compare.

    The wall-clock experiment is practically useful, but it doesn't state the hardware specifications or the serving batch size (Section 4.2, lines 454–465). Moreover, it fails to give the accuracy figure together with each timed configuration.

    In addition, I think this paper requires a better comparison with existing related works. TokenSkip reduces text length using a different method and should be included as an efficiency baseline (Appendix A, lines 761–766). A speculative-decoding comparison geared to the measured workload would be useful in assessing the practical value of the savings reported too. The combination of these omissions means that it is difficult to assess the actual speed gain.

Questions for authors
  1. What is SuperThoughts' performance when compared to SIM-CoT and MUX? Please carry out a comparison of accuracy and computational requirements using the same base model, training and test datasets, and include a calibrated reasoning-step budget for each of the methods.

  2. Why was the accuracy of CODI only about 0.1%? Could you please provide details of its training setup and the number of internal reasoning steps it used?

  3. How does SuperThoughts perform against other methods designed to speed up generation? Please compare it with speculative decoding methods and TokenSkip, if applicable.

  4. How reliable are the small accuracy differences? How were the confidence thresholds determined? Also include uncertainty estimates concerning the accuracy differences.

Reviewer_6SbqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SuperThoughts, which combines pairs of CoT tokens into a single input embedding and predicts two tokens per backbone step using a lightweight MTP module. A confidence-based fallback switches to single-token decoding when needed. Experiments show reduced decoding steps and wall-clock time while largely preserving reasoning accuracy.

Critical strengths
  1. The approach reduces backbone computation while retaining token-level supervision, which makes it a practical way to compress reasoning. The adaptive fallback provides a useful accuracy–efficiency tradeoff, avoiding the large accuracy drops seen with uniform two-token decoding.

  2. The paper demonstrates actual wall-clock improvements rather than relying only on shorter generation lengths. The results across 1.5B, 7B, and 14B models, together with a CoT baseline trained on the same data, provide meaningful evidence of practical efficiency gains.

Critical weaknesses
  1. The novelty over HAMburger is somewhat limited: both combine token compression with a lightweight decoder to generate multiple tokens per backbone step.
  2. The attribution of the benefits is unclear. e.g., the adaptive and non-adaptive models use different MTP loss weights (0.02 versus 1.0), so the accuracy recovery cannot be attributed to fallback alone. A detailed comparison with the closest baseline with ablations of the proposed difference is also missing.
Questions for authors
  1. Is my understanding correct? The proposed method reduced the flexibility of the action space (e.g., fixing the token merging to be only on 2 tokens with a fallback to 1 token instead of being adaptive length, the applications move from general text generation to CoT compression etc.). A simpler design could still be valuable, but the paper does not clearly establish which changes lead to the improved tradeoff in Figure 4, or whether that advantage holds beyond the 1.5B setting.
  2. Can you provide a detailed study on what accoutns for the improvement over HAMburger in Figure 4. E.g., the fixed two-token grouping, hidden-state distillation, or confidence-based fallback?
  3. How's performance comparison with the MTP loss weight held fixed for adaptive v.s. non-adaptive inference?
Reviewer_2BCEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SuperThoughts, which fuses each pair of CoT tokens into one input embedding and decodes two tokens per step with a lightweight MTP module, thus preserving token-level supervision. The compressor is first aligned via hidden-state distillation and then trained jointly with the model. At inference, the model falls back to single-token decoding when MTP confidence is low. On Qwen2.5 models (1.5B–14B) across four reasoning benchmarks, it cuts decoding steps by ~20–35% with 1–2 points of accuracy loss.

Critical strengths
  1. A practical approach to token compression. The method reduces main-model decoding steps while preserving supervision for every discrete token.
  2. Experiments show real efficiency gains. Evaluation across three model sizes includes a same-data HAMburger comparison and wall-clock measurements, with reported generation-time reductions of roughly 21–28%.
Critical weaknesses
  1. The need for the full training pipeline is unclear. The method combines complex procedures: compressor distillation, MTP warm-up, joint training, and adaptive inference, but does not isolate the contribution of each stage. Moreover, the adaptive and non-adaptive models use different MTP loss weights (0.02 vs. 1.0), so their comparison does not isolate the effect of fallback. Controlled ablations are needed to show which components are necessary for the reported gains.
  2. Advantage over standard MTP is claimed but not shown: The paper argues that SuperThoughts reduces FLOPs, while standard MTP only helps latency through speculative decoding (App. A). Yet it reports no throughput or FLOP measurements and no comparison with self-speculative decoding using the same MTP module. Since speculative decoding is lossless and SuperThoughts loses 1–3 points of accuracy, it is unclear when SuperThoughts is the better choice.
Questions for authors
  1. Can you report measured FLOPs, latency and results across batch sizes to support the claimed advantage in compute-bound settings?
  2. Are all training stages necessary? Ablations removing compressor distillation or MTP warm-up would help justify the added complexity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.