- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MemoSight to improve the information fidelity and reasoning efficiency of online chain-of-thought (CoT) compression. To address the issues in existing methods—namely, allocating a fixed number of memory tokens to reasoning steps of different lengths, placing memory tokens too far from the original content, and using a short supervision range during training—the authors propose three components: Adaptive Memory Allocation, which allocates memory capacity according to the length of each reasoning step; Uniform Position Layout, which evenly places memory tokens within the span of the compressed content; and Lookahead Multi-Token Prediction, which leverages distant-token prediction to enhance long-range supervision.
In addition, the lookahead tokens obtained during training can serve as draft predictions at inference time for lossless speculative decoding. The authors conduct experiments on four reasoning benchmarks and three model families. Results show that, compared with LightThinker, MemoSight achieves higher reasoning accuracy at comparable KV-cache usage and further improves generation speed through speculative decoding, thereby improving the overall trade-off among accuracy, memory footprint, and inference speed.
Critical strengths
-
The method design is clear and practical. MemoSight improves CoT compression from three aspects—memory capacity, positional layout, and training supervision—without modifying the base model architecture.
-
It balances accuracy and efficiency. Experiments show that the method outperforms LightThinker under similar KV-cache usage, and uses lookahead tokens to achieve lossless speculative decoding, further improving generation speed.
Critical weaknesses
-
The experiments are insufficient to support the paper’s central claim about “long CoT compression.” The main experiments focus on GSM8K, MMLU, BBH, and GPQA, and many of these samples do not require genuinely ultra-long reasoning chains, making it difficult to adequately test long-term information compression and error accumulation. Although the paper adds MATH500 and AIME26, it reports only accuracy and does not simultaneously provide key metrics such as reasoning length, KV-cache usage, and end-to-end latency. Therefore, the current results cannot yet demonstrate that MemoSight can maintain a favorable accuracy–memory–speed trade-off in genuinely ultra-long, information-dense reasoning scenarios.
-
The speed comparison is not sufficiently fair, as the benefits of compression and speculative decoding are conflated. MemoSight’s speedup mainly comes from the additionally introduced lookahead multi-token prediction and speculative decoding, whereas core baselines such as LightThinker still use standard autoregressive decoding. The paper does not equip the baselines with a similar speculative decoding mechanism, so it is impossible to determine whether the acceleration stems from MemoSight’s CoT compression design or mainly from general speculative decoding techniques. The authors should compare end-to-end latency under a unified decoding framework and separately report the independent benefits brought by the compression mechanism and speculative decoding.
Questions for authors
see the above Weaknesses
3 comments
Response to Reviewer hyZf - part 2/2
W2. Fairness of the speed comparison; separating compression from speculative decoding
We agree that the two benefits should be reported separately. MemoSight's compression design (AMA and UPL) and its MTP training improve accuracy, and under autoregressive decoding it runs at a speed similar to LightThinker's; the speedup over LightThinker comes from speculative decoding (SD). Table 1 lists MemoSight's speed as "AR → SD" (autoregressive → speculative decoding) for this reason, but the text did not make the attribution explicit enough, and §4.2 now states it explicitly.
(a) Separate contributions under one decoding framework (Qwen3-8B, averages over the four benchmarks; from Table 1 and G3).
| Method | Decoding | Acc↑ | Speed↑ (tok/s) | Extra params for drafting |
|---|---|---|---|---|
| Vanilla (uncompressed) | AR | 81.68 | 9.57 | – |
| LightThinker | AR | 77.48 | 10.92 | – |
| MemoSight | AR | 79.25 | 11.40 | – |
| LightThinker + Medusa (new) | SD | 77.48 | 8.98 | 33.6M |
| MemoSight | SD (built-in) | 79.25 | 17.03 | 4K |
- Compression alone (AR decoding for all). Against Vanilla, MemoSight's compression lowers the average peak KV by 60% (1,046 vs. 2,603 tokens) and raises the speed by 19% (11.40 vs. 9.57 tok/s). Against LightThinker under the same AR decoding, the speed is similar on all three model families (23.79 vs. 23.91, 11.40 vs. 10.92 and 21.88 vs. 21.98 tok/s), and the benefit of our design (AMA, UPL and MTP training) is accuracy (+3.90, +1.77 and +2.13 points) at a comparable peak KV.
- SD alone. SD raises MemoSight's speed by a further 24.4%, 49.4% and 30.2% on the three model families. Drafts are verified against the autoregressive prediction, so the output and the accuracy are unchanged.
- End-to-end latency. Speed in tok/s is not latency, because the compressed models generate longer outputs. The time columns in W1 include this cost: on MATH500 and AIME26, MemoSight needs 28% and 45% less time per problem than Vanilla and 24% and 51% less than LightThinker, and under the 24K limit its total time is 31% and 48% lower than Vanilla's.
(b) The baseline equipped with a comparable SD mechanism. We added Medusa-1 heads [5] to LightThinker (Appendix C.6): the backbone and the shared LM head are frozen, and two future-prediction heads (33.6M parameters) are trained for 5 epochs on the same training data; drafts are verified with the official Medusa tree (mc_sim_7b_63) restricted to its first two heads, which leaves 39 nodes. Medusa commits a similar number of tokens per step (2.18 vs. 2.10 for MemoSight), but in our implementation its speed is 0.82× that of AR LightThinker, because each step verifies a tree of up to 39 nodes to commit at most 3 tokens. MemoSight verifies a single chain of d=2 drafts and obtains 1.49×. A more optimized Medusa implementation may narrow this difference, but it would still need a separately trained head.
(c) Why SD is still part of the method and not a general technique added on top. Foresight tokens are introduced to supervise the memory tokens during training (+1.85 average accuracy, Table 3). Because they already predict future tokens, the same tokens serve as drafts at inference, with 4K extra parameters and no extra training. A baseline without foresight tokens can obtain SD only by training a separate draft module, as in (b).
Regarding presentation, we now spell out the abbreviations AR and SD where they first appear, and we would welcome pointers to any passages that were hard to follow.
Thank you again for your review. We hope the new experiments address both concerns, and we would be grateful if you would take them into account in your final assessment. We are happy to answer any further questions.
[5] Cai et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024.
Kind Request for Feedback
Dear Reviewer hyZf,
We hope all is well with you. Since the discussion period is ending soon, we wanted to send a brief follow-up to our response.
We now report reasoning length, peak KV and latency on MATH500 and AIME26, together with a new evaluation under a 24K-token generation limit (W1). We have also separated the contribution of compression from that of speculative decoding, and equipped LightThinker with a comparable speculative decoding mechanism (W2). These results are included in the revised manuscript.
We would appreciate knowing if this resolves your concerns, and we are happy to clarify further.
Thank you again for your time and review.
Response to Reviewer hyZf - part 1/2
Dear Reviewer hyZf,
Thank you for the careful review and for the accurate summary of our method. Both of your concerns are about whether the experiments support the claims, and we have run new experiments for each. Full tables are in the General Response (G1 and G3); here we give the results that answer your points directly.
W1. Evidence for long CoT compression; reasoning length, KV-cache usage and latency on MATH500 and AIME26
We agree that accuracy alone does not show the full picture. We now report reasoning length, peak KV and latency on MATH500 and AIME26, and we added an evaluation with a 24K-token generation limit.
(a) The missing metrics for Table 2 (Qwen3-8B, 10,240-token limit; accuracy as in Table 2).
Peak KV is the peak number of tokens held in the KV cache, averaged over samples. Time per problem is the average end-to-end generation time. MemoSight decodes with its built-in speculative decoding (SD). Peak KV and Speed are now in Table 2, and the full table is in Appendix C.5 (Table 8).
MemoSight keeps 41% and 33% of Vanilla's peak KV and needs 28% and 45% less time per problem. This holds although the compressed models generate longer outputs than Vanilla (3,881 vs. 3,119 and 9,587 vs. 7,925 tokens), a cost that the time column includes. Against LightThinker, accuracy is 7.33 and 13.34 points higher at a slightly higher peak KV.
(b) Ultra-long reasoning: a 24K-token limit on AIME26 and HMMT Feb 2025. In the submitted version, the generation limit is 10,240 tokens, and Vanilla's average output on AIME26 (7,925 tokens) is already close to it. We therefore raised the limit to 24,576 tokens and added HMMT Feb 2025 (30 problems each), using the same checkpoints without retraining.
Time is the sum of per-request generation times; the three methods run under the same hardware and concurrency setting. These results are in Appendix C.5 (Table 9).