- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SuperThoughts makes the generation of a chain of thought faster by producing two text tokens in a single pass of the language model. The main model predicts the first token, and a smaller prediction module predicts the second one. Then, the learned compressor combines these two tokens into a single input for the next forward pass of the main model. The paper examines this method with Qwen models with 1.5B, 7B, and 14B parameters on the MATH500, OlympiadBench, and AMC23 datasets. The take-home message is mainly the empirical performance, which results in higher accuracy with a reduction in reasoning length compared to the baseline HAMburger at the 1.5B scale. The authors show that generation time is approximately 21 to 28% less compared to the chain-of-thought method.
Critical strengths
-
The method has been tested on a wide and good range of reasoning tasks with strong empirical performance For instance, the 1.5B projection model scores 73.0% on MATH500 compared to 72.4% for the standard CoT, and it reduces the correct-chain length from 494.1 to 314.8 positions (Table 1).
The paper also includes relevant comparisons in runtime measurements. Figure 5 shows that using fewer decodings results in less generation time, taking the extra modules and the fallback overhead in the implementation into consideration.
-
The paper is well-written and clear to follow. The motivation and experiments are clearly presented and understandable enough to let readers easily digest this work.
Critical weaknesses
-
The paper does not provide a clear enough advantage when compared to related methods. HAMburger, for instance, combines several tokens into a single input representation, employs a smaller decoder to produce multiple tokens during each pass of the main model, and modifies the generation process by applying a confidence threshold. The paper does not identify which of the changes is responsible for the improvement. I would expect a more thorough ablation study to test this, for example, by examining the first distillation stage. This is a novelty concern, and to me, the benefit seems to stem from an improved training and decoding procedure within an existing architectural approach.
The discussion of supervised latent reasoning is also incomplete. SIM-CoT and MUX introduce the intermediate supervision and improves latent reasoning benchmarks. Neither approach is discussed or evaluated, even though intermediate supervision is central to the paper's motivation (Introduction, lines 38–48). Appendix A states that a CODI reproduction attains an accuracy of approximately 0.1%, which is a clear failure; thus, a good understanding of the reasoning behind it is necessary. The paper includes no information regarding the configuration or the diagnostic results of that experiment (lines 778–783). These distillation methods are reducing the latency comparatively more without reducing accuracy too much.
-
The timing results need clearer experimental settings and more baselines to compare.
The wall-clock experiment is practically useful, but it doesn't state the hardware specifications or the serving batch size (Section 4.2, lines 454–465). Moreover, it fails to give the accuracy figure together with each timed configuration.
In addition, I think this paper requires a better comparison with existing related works. TokenSkip reduces text length using a different method and should be included as an efficiency baseline (Appendix A, lines 761–766). A speculative-decoding comparison geared to the measured workload would be useful in assessing the practical value of the savings reported too. The combination of these omissions means that it is difficult to assess the actual speed gain.
Questions for authors
-
What is SuperThoughts' performance when compared to SIM-CoT and MUX? Please carry out a comparison of accuracy and computational requirements using the same base model, training and test datasets, and include a calibrated reasoning-step budget for each of the methods.
-
Why was the accuracy of CODI only about 0.1%? Could you please provide details of its training setup and the number of internal reasoning steps it used?
-
How does SuperThoughts perform against other methods designed to speed up generation? Please compare it with speculative decoding methods and TokenSkip, if applicable.
-
How reliable are the small accuracy differences? How were the confidence thresholds determined? Also include uncertainty estimates concerning the accuracy differences.