acceptodds
Under review as a conference paper at ICLR 2027

MemoSight: Improving the Fidelity of Chain-of-Thought Compression with Memory and Foresight Tokens

Abstract

Long chain-of-thought (CoT) reasoning enables LLMs to solve increasingly challenging tasks, but incurs substantial inference costs as the KV cache grows with the reasoning length. Online CoT compression reduces this footprint by replacing completed reasoning steps with compact memory tokens. However, existing methods compress every step into a *fixed, length-agnostic* set of memory tokens that are *positioned far from the content they summarize* and trained with *short-horizon next-token supervision*, limiting the fidelity of the compressed representation and degrading reasoning accuracy. We propose **MemoSight**, a lightweight framework that improves the fidelity of compressed reasoning while accelerating inference, through three designs targeting *distinct sources* of information loss. 1) **Adaptive Memory Allocation** allocates memory tokens in proportion to the length of each reasoning step, providing longer steps with greater capacity to preserve their information. 2) **Uniform Positional Layout** assigns memory-token position IDs uniformly within the span of the step they summarize, shortening their positional distance to the content and encouraging them to cover different parts of the step. 3) **Foresight Multi-Token Prediction** introduces foresight tokens that predict reasoning tokens farther ahead rather than only the next one, providing longer-horizon supervision that encourages richer memory representations. Across four reasoning benchmarks and three model families, MemoSight improves average reasoning accuracy by 2.6 points over LightThinker at comparable peak KV-cache usage. Moreover, the same foresight tokens used for training supervision are also used to generate draft predictions for speculative decoding, accelerating inference by 24.4%-49.4% across model families. Together, these results establish a better accuracy-efficiency trade-off for CoT compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
CoT Compression, Reasoning Efficiency, Multi-Token Prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hyZfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemoSight to improve the information fidelity and reasoning efficiency of online chain-of-thought (CoT) compression. To address the issues in existing methods—namely, allocating a fixed number of memory tokens to reasoning steps of different lengths, placing memory tokens too far from the original content, and using a short supervision range during training—the authors propose three components: Adaptive Memory Allocation, which allocates memory capacity according to the length of each reasoning step; Uniform Position Layout, which evenly places memory tokens within the span of the compressed content; and Lookahead Multi-Token Prediction, which leverages distant-token prediction to enhance long-range supervision.

In addition, the lookahead tokens obtained during training can serve as draft predictions at inference time for lossless speculative decoding. The authors conduct experiments on four reasoning benchmarks and three model families. Results show that, compared with LightThinker, MemoSight achieves higher reasoning accuracy at comparable KV-cache usage and further improves generation speed through speculative decoding, thereby improving the overall trade-off among accuracy, memory footprint, and inference speed.

Critical strengths
  1. The method design is clear and practical. MemoSight improves CoT compression from three aspects—memory capacity, positional layout, and training supervision—without modifying the base model architecture.

  2. It balances accuracy and efficiency. Experiments show that the method outperforms LightThinker under similar KV-cache usage, and uses lookahead tokens to achieve lossless speculative decoding, further improving generation speed.

Critical weaknesses
  1. The experiments are insufficient to support the paper’s central claim about “long CoT compression.” The main experiments focus on GSM8K, MMLU, BBH, and GPQA, and many of these samples do not require genuinely ultra-long reasoning chains, making it difficult to adequately test long-term information compression and error accumulation. Although the paper adds MATH500 and AIME26, it reports only accuracy and does not simultaneously provide key metrics such as reasoning length, KV-cache usage, and end-to-end latency. Therefore, the current results cannot yet demonstrate that MemoSight can maintain a favorable accuracy–memory–speed trade-off in genuinely ultra-long, information-dense reasoning scenarios.

  2. The speed comparison is not sufficiently fair, as the benefits of compression and speculative decoding are conflated. MemoSight’s speedup mainly comes from the additionally introduced lookahead multi-token prediction and speculative decoding, whereas core baselines such as LightThinker still use standard autoregressive decoding. The paper does not equip the baselines with a similar speculative decoding mechanism, so it is impossible to determine whether the acceleration stems from MemoSight’s CoT compression design or mainly from general speculative decoding techniques. The authors should compare end-to-end latency under a unified decoding framework and separately report the independent benefits brought by the compression mechanism and speculative decoding.

Questions for authors

see the above Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer hyZf - part 1/2

Dear Reviewer hyZf,

Thank you for the careful review and for the accurate summary of our method. Both of your concerns are about whether the experiments support the claims, and we have run new experiments for each. Full tables are in the General Response (G1 and G3); here we give the results that answer your points directly.

W1. Evidence for long CoT compression; reasoning length, KV-cache usage and latency on MATH500 and AIME26

We agree that accuracy alone does not show the full picture. We now report reasoning length, peak KV and latency on MATH500 and AIME26, and we added an evaluation with a 24K-token generation limit.

(a) The missing metrics for Table 2 (Qwen3-8B, 10,240-token limit; accuracy as in Table 2).

BenchmarkMethodAcc↑Output tokens (avg)Peak KV↓Speed↑ (tok/s)Time per problem↓ (s)
MATH500Vanilla74.893,1193,4339.83317
MATH500LightThinker64.473,7471,21012.52299
MATH500MemoSight (d=2, SD)71.803,8811,40317.12227
AIME26Vanilla46.677,9258,2969.99793
AIME26LightThinker13.339,4512,46810.61891
AIME26MemoSight (d=2, SD)26.679,5872,70721.83439

Peak KV is the peak number of tokens held in the KV cache, averaged over samples. Time per problem is the average end-to-end generation time. MemoSight decodes with its built-in speculative decoding (SD). Peak KV and Speed are now in Table 2, and the full table is in Appendix C.5 (Table 8).

MemoSight keeps 41% and 33% of Vanilla's peak KV and needs 28% and 45% less time per problem. This holds although the compressed models generate longer outputs than Vanilla (3,881 vs. 3,119 and 9,587 vs. 7,925 tokens), a cost that the time column includes. Against LightThinker, accuracy is 7.33 and 13.34 points higher at a slightly higher peak KV.

(b) Ultra-long reasoning: a 24K-token limit on AIME26 and HMMT Feb 2025. In the submitted version, the generation limit is 10,240 tokens, and Vanilla's average output on AIME26 (7,925 tokens) is already close to it. We therefore raised the limit to 24,576 tokens and added HMMT Feb 2025 (30 problems each), using the same checkpoints without retraining.

BenchmarkMethodSolvedOutput tokens (avg)>16KPeak KV↓Speed↑Time↓ (h)
AIME26Vanilla15/3010,1974/3010,5676.4913.08
AIME26LightThinker4/3014,90314/304,4347.9715.58
AIME26MemoSight (d=2, SD)9/3013,5348/303,93512.499.03
HMMTVanilla9/3011,8765/3012,2297.9412.46
HMMTLightThinker3/3015,08213/303,97610.0012.57
HMMTMemoSight (d=2, SD)6/3013,46711/303,77017.186.53

Time is the sum of per-request generation times; the three methods run under the same hardware and concurrency setting. These results are in Appendix C.5 (Table 9).

  • Error accumulation. On these chains (13.5K tokens on average, 27% and 37% of them above 16K), MemoSight solves 9 vs. 4 and 6 vs. 3 problems compared with LightThinker, with shorter outputs and a lower average peak KV. Its advantage in fidelity is therefore kept as the chains grow.
  • Accuracy–memory–speed trade-off. Compared with Vanilla, the average peak KV is 62.8% and 69.2% lower and the total time is 31% and 48% lower.
  • What we do not claim. Accuracy is still below Vanilla (9 vs. 15 and 6 vs. 9 problems). Lossy compression has a real cost on such long chains, and with 30 problems per benchmark we report counts and do not claim statistical significance. §4.3 now reports Vanilla's results next to ours, and Appendix C.5 states this gap, so that the claims about long CoT match this evidence.
AuthorsOpenReview ↗

Response to Reviewer hyZf - part 2/2

W2. Fairness of the speed comparison; separating compression from speculative decoding

We agree that the two benefits should be reported separately. MemoSight's compression design (AMA and UPL) and its MTP training improve accuracy, and under autoregressive decoding it runs at a speed similar to LightThinker's; the speedup over LightThinker comes from speculative decoding (SD). Table 1 lists MemoSight's speed as "AR → SD" (autoregressive → speculative decoding) for this reason, but the text did not make the attribution explicit enough, and §4.2 now states it explicitly.

(a) Separate contributions under one decoding framework (Qwen3-8B, averages over the four benchmarks; from Table 1 and G3).

MethodDecodingAcc↑Speed↑ (tok/s)Extra params for drafting
Vanilla (uncompressed)AR81.689.57–
LightThinkerAR77.4810.92–
MemoSightAR79.2511.40–
LightThinker + Medusa (new)SD77.488.9833.6M
MemoSightSD (built-in)79.2517.034K
  • Compression alone (AR decoding for all). Against Vanilla, MemoSight's compression lowers the average peak KV by 60% (1,046 vs. 2,603 tokens) and raises the speed by 19% (11.40 vs. 9.57 tok/s). Against LightThinker under the same AR decoding, the speed is similar on all three model families (23.79 vs. 23.91, 11.40 vs. 10.92 and 21.88 vs. 21.98 tok/s), and the benefit of our design (AMA, UPL and MTP training) is accuracy (+3.90, +1.77 and +2.13 points) at a comparable peak KV.
  • SD alone. SD raises MemoSight's speed by a further 24.4%, 49.4% and 30.2% on the three model families. Drafts are verified against the autoregressive prediction, so the output and the accuracy are unchanged.
  • End-to-end latency. Speed in tok/s is not latency, because the compressed models generate longer outputs. The time columns in W1 include this cost: on MATH500 and AIME26, MemoSight needs 28% and 45% less time per problem than Vanilla and 24% and 51% less than LightThinker, and under the 24K limit its total time is 31% and 48% lower than Vanilla's.

(b) The baseline equipped with a comparable SD mechanism. We added Medusa-1 heads [5] to LightThinker (Appendix C.6): the backbone and the shared LM head are frozen, and two future-prediction heads (33.6M parameters) are trained for 5 epochs on the same training data; drafts are verified with the official Medusa tree (mc_sim_7b_63) restricted to its first two heads, which leaves 39 nodes. Medusa commits a similar number of tokens per step (2.18 vs. 2.10 for MemoSight), but in our implementation its speed is 0.82× that of AR LightThinker, because each step verifies a tree of up to 39 nodes to commit at most 3 tokens. MemoSight verifies a single chain of d=2 drafts and obtains 1.49×. A more optimized Medusa implementation may narrow this difference, but it would still need a separately trained head.

(c) Why SD is still part of the method and not a general technique added on top. Foresight tokens are introduced to supervise the memory tokens during training (+1.85 average accuracy, Table 3). Because they already predict future tokens, the same tokens serve as drafts at inference, with 4K extra parameters and no extra training. A baseline without foresight tokens can obtain SD only by training a separate draft module, as in (b).

Regarding presentation, we now spell out the abbreviations AR and SD where they first appear, and we would welcome pointers to any passages that were hard to follow.

Thank you again for your review. We hope the new experiments address both concerns, and we would be grateful if you would take them into account in your final assessment. We are happy to answer any further questions.

[5] Cai et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024.

AuthorsOpenReview ↗

Kind Request for Feedback

Dear Reviewer hyZf,

We hope all is well with you. Since the discussion period is ending soon, we wanted to send a brief follow-up to our response.

We now report reasoning length, peak KV and latency on MATH500 and AIME26, together with a new evaluation under a 24K-token generation limit (W1). We have also separated the contribution of compression from that of speculative decoding, and equipped LightThinker with a comparable speculative decoding mechanism (W2). These results are included in the revised manuscript.

We would appreciate knowing if this resolves your concerns, and we are happy to clarify further.

Thank you again for your time and review.

Reviewer_H4W6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MemoSight on top of LightThinker-style CoT compression. It allocates memory tokens proportional to step length, spreads their position IDs inside the step, and adds foresight tokens trained to predict tokens further ahead, which are then reused as drafts for SD. It improves accuracy over LightThinker by a few points at similar peak KV and gets 24–49% speedup from SD.

Critical strengths
  • The three bottlenecks are well motivated, each fix is simple, and the ablation shows each one helps.
  • Reusing foresight tokens for lossless SD without a separate drafter is a nice addition.
  • The attention visualization for UPL is a clear sanity check.
Critical weaknesses
  • There is still a large gap to Vanilla on hard tasks like AIME26, and on easier benchmarks H2O is only a few points behind. Stronger reasoning-aware KV compression baselines are missing.
  • Most benchmarks are short reasoning, GSM8K/MMLU/BBH. Training is truncated at 4K tokens and peak KV is only around 1K tokens, which is not the long CoT setting where compression really matters.
  • The speed numbers look like single-request, unoptimized decoding. The SD speedup may not hold in vLLM or SGLang, and there is no comparison to Medusa or EAGLE.
Questions for authors
  • Results with 16K+ CoT?
  • Throughput in vLLM/SGLang with batch > 1?
  • Comparison with recent reasoning KV eviction methods at matched peak KV?
4 comments
AuthorsOpenReview ↗

Response to Reviewer H4W6 - part 1/2

Dear Reviewer H4W6,

Thank you for the positive assessment and for the questions about long reasoning and deployment, which led to most of our new experiments. We answer each point below. Full tables are in the General Response (G1–G3); here we give the main results and address the points specific to your review.

W1 / Q3. Gap to Vanilla on hard tasks, H2O on easier benchmarks, and reasoning-aware KV eviction baselines

(a) We added R-KV [2] and LazyEviction [3], including a comparison at a similar peak KV (G2; Appendix C.4). Both are applied to the same fine-tuned Vanilla checkpoint (Qwen3-8B).

SettingR-KV AccLazyEviction AccMemoSight (d=2) AccPeak KV (eviction vs. MemoSight)
Avg. of 4 benchmarks, 1,024 budget76.8673.8079.251,011 / 1,013 vs. 1,046
GPQA, 1,024 budget47.9835.3552.531,037 / 1,035 vs. 1,788
GPQA, 1,800 budget48.9944.9552.531,775 / 1,775 vs. 1,788

At a similar average peak KV, MemoSight is more accurate than both eviction methods, by 2.39 and 5.45 points on average, and by 3.54 and 7.58 points on GPQA after their budget is raised to bring the peaks close (1,775 vs. 1,788). With its built-in SD it is also faster (16.01 vs. 10.45 and 8.54 tok/s on GPQA).

Setup: R-KV keeps a retention budget of 896 tokens with a 128-token buffer (observation window 8), and LazyEviction a budget of 992 with an eviction window of 32, i.e., a steady-state target of 1,024 KV tokens, the budget used for H2O and SepLLM in Table 1. For the 1,800 setting, R-KV's retention budget is raised to 1,672 with the buffer and window unchanged, and LazyEviction's budget is raised to 1,672 and its window to 128. These parameters were fixed in advance and not tuned on test accuracy.

(b) H2O is close on the easier benchmarks because little needs to be evicted there. On Qwen3-8B, Vanilla's own peak is 1,264 tokens on GSM8K and 1,708 on BBH, so a 1,024-token budget keeps 81% and 60% of the full cache. In this regime H2O stays within 0.4 and 4.7 points of Vanilla, and we agree that eviction is a strong and simple choice (R-KV is even slightly above MemoSight on GSM8K). Two differences remain:

  • MemoSight reaches a comparable accuracy (93.18 and 86.46, against 93.32 and 86.80 for H2O) with a lower peak (685 and 838 tokens, against 1,024), and it is faster than Vanilla, whereas H2O is slower than Vanilla in Table 1 (8.18 vs. 9.57 tok/s on average).
  • The difference is larger on GPQA (Vanilla peak 5,532): H2O drops to 33.06 and SepLLM to 30.81, while MemoSight stays at 52.53, within 2.02 points of Vanilla. The stronger eviction methods narrow this difference considerably but remain below MemoSight at a similar peak, as shown in (a).

(c) We agree that a clear gap to Vanilla remains on AIME26, and the revision reports it explicitly. In problem counts (30 problems), Vanilla solves 14, LightThinker 4, and MemoSight 8 (d=2) and 10 (d=4) in Table 2. With the generation limit raised to 24K tokens (G1), the counts are 15, 4 and 9. MemoSight therefore recovers a good part of what LightThinker loses, at 33%–37% of Vanilla's peak KV, but lossy compression still has a real cost on the hardest problems. We do not claim otherwise: §4.3 now reports Vanilla's results next to ours under both limits, and Appendix C.5 discusses the gap as the current cost of lossy compression on such long chains.

[2] Cai et al. R-KV: Redundancy-aware KV Cache Compression for Reasoning Models. NeurIPS 2025.

[3] Zhang et al. LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning. ACL 2026.

AuthorsOpenReview ↗

Response to Reviewer H4W6 - part 2/2

W2 / Q1. Short reasoning, 4K training length, ~1K peak KV; results with 16K+ CoT

We ran a new evaluation with a 24K-token generation limit, in which a large share of the outputs exceed 16K tokens (G1; Appendix C.5).

  • The ~1K peak is the size after compression, not the reasoning length. Uncompressed, the reasoning takes 5.5K tokens on GPQA (Table 1), and 3.4K on MATH500 and 8.3K on AIME26 (Vanilla's peak KV, now added to Table 2).
  • 16K+ CoT. With the limit raised to 24,576 tokens on AIME26 and HMMT Feb 2025, MemoSight generates 13.5K tokens on average, and 8/30 and 11/30 of its outputs exceed 16K. Against LightThinker it solves 9 vs. 4 and 6 vs. 3 problems with a lower average peak KV. Against Vanilla, its average peak KV is 62.8% and 69.2% lower (3,935 vs. 10,567 and 3,770 vs. 12,229) and its total generation time is 9.03 h vs. 13.08 h and 6.53 h vs. 12.46 h. Its accuracy is still below Vanilla's (9 vs. 15 and 6 vs. 9 problems), as discussed in W1 (c).
  • 4K training length. These are the checkpoints of the paper, trained with a 4,096-token length and not retrained. The learned compression therefore still gives the gains above on chains more than three times longer than those seen in training. Training on longer sequences may reduce the remaining gap to Vanilla, which we leave to future work.

W3 / Q2. Single-request decoding, vLLM/SGLang, batch > 1, and Medusa / EAGLE

(a) Batch > 1: the speedup is retained under batched inference (G3; Appendix C.7). At batch sizes 4, 16 and 64 (GSM8K, Qwen3-8B, a single GPU), MemoSight with speculative decoding (SD) improves throughput over Vanilla by 1.48×, 1.62× and 1.43×, with a 48%–52% lower peak KV. The gain from SD alone (over MemoSight with autoregressive decoding) is 1.39×, 1.31× and 1.23×: it shrinks as the batch grows and is still clear at batch 64. Since MemoSight's outputs are 10%–13% longer, its total time is 22%–32% lower than Vanilla's.

(b) vLLM/SGLang: we have not integrated MemoSight into these engines, and we limit our claim accordingly. Our batched implementation uses static batching in PyTorch. MemoSight needs two operations that, to our knowledge, these engines do not expose out of the box: evicting the KV states of a completed step separately for each request, and assigning non-consecutive position IDs to the memory tokens. Memento [9] reports a vLLM extension for a related block-level KV masking, which suggests that such an integration is feasible, but it is engineering work that we could not complete during the rebuttal. Our claim is therefore that the speedup is not limited to single-request decoding, and we do not claim a specific speedup under continuous batching. You are also right that our decoding is not heavily optimized: all methods share the same implementation and backend (Appendix B.5), so the speeds are meant as relative comparisons.

(c) Medusa: we equipped LightThinker with Medusa-1 heads [5] (G3; Appendix C.6). Two heads with 33.6M parameters, trained for 5 epochs on the same data, commit 2.18 tokens per step (MemoSight: 2.10). In our implementation the speed is 8.98 tok/s, against 10.92 for autoregressive LightThinker, because each step verifies a tree of up to 39 nodes to commit at most 3 tokens. MemoSight verifies a single chain of d=2 drafts and goes from 11.40 to 17.03 tok/s (1.49×) with 4K extra parameters. A more optimized Medusa implementation may narrow this difference.

(d) EAGLE: we did not run this comparison. We chose Medusa because it is the closest to our mechanism: both draft several future tokens in the same forward pass of the backbone, whereas EAGLE [11] drafts autoregressively with a separately trained draft layer. We do not claim that foresight-token SD is faster than a dedicated drafter such as EAGLE; our claim is that the tokens that supervise the memory during training also give lossless drafts with no extra module and no extra training. Such a drafter is orthogonal to compression and could be added on top of MemoSight, which we discuss in Appendix C.9.

On presentation, we now define AR and SD at first use, and if specific passages were hard to follow, we would be glad to revise them.

Thank you again for the constructive review. We hope the new results address your concerns, and we are happy to continue the discussion.

[5] Cai et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024.

[9] Kontonis et al. Memento: Teaching LLMs to Manage Their Own Context. COLM 2026.

[11] Li et al. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024.

Thanks for the authors' reply, most of my concerns are solved. I will maintain my score. :->

Dear Reviewer H4W6,

Thank you for your reply and for your constructive review, which has helped us improve the paper.

Reviewer_pBTLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MemoSight, a framework for online chain-of-thought compression that aims to improve reasoning accuracy while reducing KV-cache usage and accelerating inference. It combines three components: AMA, ULP and MTP. Experiments across three models and four main reasoning benchmarks report an average accuracy improvement of 2.6 percentage points over LightThinker, alongside a 24.4%–49.4% throughput improvement over MemoSight’s autoregressive decoding.

Critical strengths

1.The structure of this paper is clear. Easy for readers to follow. 2.The evlutation is solid. including component ablations, compression-ratio sweeps, comparisons with conventional MTP methods, and runtime analysis. 3.Practical relevance and empirical gains. The paper addresses the important accuracy–memory trade-off in long-chain reasoning.

Critical weaknesses

1.Over claim for AMA. The abstract says existing methods have fixed, length-agonstic. This is true for LightThinker, but not true for all methods. 2.Limited novelty. UPL appears to follow the same core positional-assignment strategy as Position IDs Matter. Moreover, MTP and speculative decoding are already well established. 3.Lack of statistical uncertainty analysis. This paper gives the relatively modest accuracy gains in some settings, their robustness remains unclear.

Questions for authors

1.Could you clarify what technical contributions go beyond adapting and combining these existing techniques (MTP and UPL) for CoT compression? 2.Why only LightThinker? Could you report more baselines ? 3.Could you provide results from multiple independent runs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer pBTL - part 1/2

Dear Reviewer pBTL,

Thank you for the careful review, and for recognizing the clear structure, the solid evaluation and the practical relevance of the paper. We answer each of your points below. Several points are answered in full in the General Response (G2, G4, G5); here we give the main result for each.

W1. Over-claim for AMA

We agree, and we have narrowed the claim. The sentence in the abstract describes latent step-level methods such as LightThinker, which compress each step into a fixed number of memory-token hidden states. It should not have been written about "existing methods" in general. At least two families fall outside it:

  • Methods that rewrite completed reasoning into shorter text (e.g., PENCIL [7], InftyThink [8] and Memento [9], cited in §2) produce summaries whose length varies with the content.
  • In input-side context compression, ratio-based allocation is already used. For example, Activation Beacon [10] inserts one beacon token per fixed-size unit of the context, so the number of compressed tokens grows with the input length.

What AMA contributes is therefore not length-proportional allocation as a general idea. It brings this idea to online latent CoT compression, where steps are generated during decoding and differ widely in length, and shows that it matters there: replacing AMA with a fixed budget of 9 memory tokens lowers the average accuracy from 66.84 to 63.07 (Table 3), and MemoSight gives a better accuracy–memory trade-off than every fixed-budget LightThinker setting in Figure 5. We state this relation to Activation Beacon in Appendix E.1.

In the revision, "existing methods compress every step into a fixed, length-agnostic set of memory tokens" in the abstract becomes "existing latent compression methods such as LightThinker compress every step into a fixed, length-agnostic set of memory tokens", and §1 is revised accordingly.

W2 / Q1. Novelty and technical contributions beyond combining MTP and UPL

We do not claim MTP, speculative decoding (SD) or the layout principle of EPL [6] ("Position IDs Matter") as new. Our contributions are the diagnosis of three sources of information loss in online CoT compression, and a new role for MTP. The full answer is in G5; in short:

  • UPL and EPL. UPL builds on EPL's principle of placing the position IDs of special tokens inside the span they compress, and we now credit EPL explicitly in §3.2 and discuss the difference in Appendix E.1. The difference is in how it is used: EPL places a fixed number of tokens over an input context that is given in advance and compressed once, whereas UPL is applied online to every newly generated step, and with AMA the number of memory tokens grows with the step length.
  • MTP as a training signal for compressed memory. Foresight tokens are trained under the compression mask, so they see earlier steps only through the memory tokens. Predicting farther ahead therefore encourages the memory tokens to keep more of what later reasoning needs (+1.85 average accuracy, Table 3). To our knowledge, MTP has not been used for this purpose before.
  • The form of MTP matters under compression. Our foresight tokens follow the lightweight special-token form of MTP [12] (cited in §2), which we do not claim as new. What we show is that this choice matters in our setting: conventional MTP with added Transformer blocks (103M parameters) reaches 62.59 (MTP Last) and 64.45 (MTP Middle), both below the model without any MTP (64.99), whereas foresight tokens add 4K parameters and reach 66.84. Both forms gradient-train the memory tokens; we hypothesize that the large modules absorb part of the signal in their own weights. This comparison is now in the main text (Table 3 and the paragraph "Supervision: Foresight tokens vs. conventional MTP" in §5).
  • Lossless SD comes for free. The same foresight tokens serve as drafts at inference, with no draft model or head and no extra training. As a reference, Medusa heads [5] on LightThinker need 33.6M separately trained parameters and did not speed up its decoding in our experiments (G3).

[5] Cai et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024.

[6] Zhao et al. Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models. Findings of EMNLP 2025.

[7] Yang et al. PENCIL: Long Thoughts with Short Memory. ICML 2025.

[8] Yan et al. InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models. ICLR 2026.

[9] Kontonis et al. Memento: Teaching LLMs to Manage Their Own Context. COLM 2026.

[10] Zhang et al. Long Context Compression with Activation Beacon. ICLR 2025.

[12] Gerontopoulos et al. Multi-Token Prediction Needs Registers. NeurIPS 2025.

AuthorsOpenReview ↗

Response to Reviewer pBTL - part 2/2

Q2. Why only LightThinker? More baselines

Table 1 compares with LightThinker and two training-free KV pruning methods, alongside three uncompressed references, and we have now added three recent baselines. Trainable online CoT compression into latent memory tokens is a recent direction with few published methods so far. LightThinker is its representative method and the one MemoSight builds on, so we used it as the trained compression baseline, together with the training-free KV pruning methods H2O and SepLLM. Following your suggestion, we added a recent method in this direction, HybridThinker [1], and two recent reasoning-aware KV eviction methods, R-KV [2] and LazyEviction [3], under the setting of Table 1. The full table and setup are in G2 and Appendix C.4; the averages over the four benchmarks on Qwen3-8B are:

MethodAvg. Acc↑Avg. Speed↑ (tok/s)Avg. Peak KV↓
Vanilla + R-KV (new)76.8611.611011
Vanilla + LazyEviction (new)73.809.591013
LightThinker77.4810.92910
HybridThinker (new)79.4710.051177
MemoSight (d=2)79.2517.031046
MemoSight (d=4)79.7016.171057
  • MemoSight is more accurate than both eviction methods at a similar average peak KV (+2.39 over R-KV and +5.45 over LazyEviction at d=2). On GPQA, where we also raise their budget so that the average peaks are similar (1,775 vs. 1,788 tokens), MemoSight remains 3.54 and 7.58 points higher.
  • MemoSight is on par with HybridThinker in accuracy (79.47 lies between our d=2 and d=4 results), with a 10%–11% lower peak KV and, with its built-in SD, about 1.6–1.7× its speed. We do not claim to outperform it in accuracy. HybridThinker gains from keeping recent steps uncompressed and MemoSight from making the compressed memory more faithful, so the two are complementary.

W3 / Q3. Statistical uncertainty and multiple independent runs

The gain is reproduced by independent training runs, and it is statistically significant on GSM8K and BBH. Details are in G4 and Appendix C.8.

  • Independent runs (Qwen3-8B). Three MemoSight runs average 79.36 ± 0.10 (79.25, 79.41, 79.44) and two LightThinker runs give 77.48 and 77.12. Every MemoSight run is above both LightThinker runs on GSM8K, MMLU and BBH, and the smallest difference in average accuracy (1.77 points) is far above the run-to-run variation.
  • Paired bootstrap over test questions (10,000 resamples). The 95% confidence interval of the gain excludes zero on GSM8K (+2.27, [+0.76, +3.72]) and BBH (+3.22, [+1.41, +5.05]). It includes zero on MMLU (+1.27, [−0.88, +3.41]) and GPQA (+2.52, [−2.02, +6.57]).
  • Consistency across model families (Table 1). Over the 12 model–benchmark pairs, MemoSight is above LightThinker on 10, equal on 1 and below on 1, and its average gain is 3.90, 1.77 and 2.13 points on Qwen2.5-7B, Qwen3-8B and Llama3.1-8B. The tie and the loss are both on GPQA.

We agree with you that the gains are modest in some settings, and the revision says so (Appendix C.8): the improvement is clear on GSM8K and BBH, smaller but reproduced by every run on MMLU, and mixed on GPQA, which has only 198 questions (+3.54 on Qwen2.5-7B, 0.00 on Qwen3-8B and −1.51 on Llama3.1-8B in Table 1), so we do not claim a consistent improvement there.

We hope these answers and the new results address your concerns, and we would be grateful if you would take them into account in your final assessment. We are happy to answer any further questions.

[1] Liu et al. HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps. EMNLP 2026.

[2] Cai et al. R-KV: Redundancy-aware KV Cache Compression for Reasoning Models. NeurIPS 2025.

[3] Zhang et al. LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning. ACL 2026.

AuthorsOpenReview ↗

Kind Request for Feedback

Dear Reviewer pBTL,

Hope you are doing well. As the discussion period nears its close, we are writing to follow up on our response.

We have narrowed the claim about existing methods in the abstract (W1) and clarified what we do and do not claim as new (W2/Q1). We have also added three recent baselines, HybridThinker, R-KV and LazyEviction (Q2), and independent training runs with a paired bootstrap test (W3/Q3). These changes are included in the revised manuscript.

Please let us know whether this addresses your concerns, and we remain happy to clarify anything further.

Thank you again for your time and careful review.

Comments on the paper

8 comments
AuthorsOpenReview ↗

General Response to All Reviewers - part 1/8

Dear Reviewers and Area Chair,

We sincerely thank all of you for your time and constructive feedback. We are encouraged that the reviewers commonly recognized the strengths of our work, including the well-motivated diagnosis of three distinct sources of information loss and the simple, practical design that requires no architectural change (pBTL Strength 3; H4W6 Strength 1; hyZf Strength 1), the thorough ablations and analyses (pBTL Strength 2; H4W6 Strength 1, 3), and the reuse of foresight tokens as drafts for lossless speculative decoding (SD) without a separate drafter (H4W6 Strength 2; hyZf Strength 2). We are equally grateful for the weaknesses raised, which have made the paper better.

Below we address the main concerns (G1–G5); the remaining points are answered in the individual responses. All new results are included in the revised PDF, with changes marked in blue.

  • G1. Evaluation under genuinely long CoT (H4W6, hyZf). We add peak KV and speed to Table 2 and run a new evaluation with a 24K-token limit on AIME26 and HMMT. MemoSight solves 9 vs. 4 and 6 vs. 3 problems compared with LightThinker, with a 63%–69% lower peak KV than Vanilla. A gap to Vanilla in accuracy remains (9 vs. 15 and 6 vs. 9).
  • G2. Stronger baselines (pBTL, H4W6). We add HybridThinker, R-KV and LazyEviction. MemoSight is more accurate than both eviction methods at a similar peak KV (+2.39 and +5.45 on average), and on par with HybridThinker in accuracy with a 10%–11% lower peak KV and, with its built-in SD, about 1.6–1.7× its speed.
  • G3. Separating compression from speculative decoding, and deployment efficiency (hyZf, H4W6). The accuracy gain is obtained under autoregressive decoding, and the speedup comes from SD. MemoSight's foresight tokens (4K parameters) commit as many tokens per step as 33.6M-parameter Medusa heads on LightThinker (2.10 vs. 2.18) and give a 1.49× speedup, against 0.82× for Medusa in our implementation, and MemoSight keeps a 1.43–1.62× throughput gain over Vanilla at batch sizes 4–64.
  • G4. Statistical robustness (raised by pBTL). Three independent MemoSight runs average 79.36 ± 0.10, against 77.48 and 77.12 for two LightThinker runs. A paired bootstrap shows a significant gain on GSM8K and BBH, and a positive but not significant gain on MMLU and GPQA.
  • G5. Novelty and technical contributions (raised by pBTL). We do not claim multi-token prediction (MTP), SD or the layout principle of "Position IDs Matter" as new. Our contributions are the diagnosis of three sources of information loss, the use of MTP as a training signal for compressed memory (conventional MTP lowers accuracy in this setting), and lossless SD obtained from the same tokens at no extra cost.
AuthorsOpenReview ↗

General Response to All Reviewers - part 2/8

G1. Evaluation under genuinely long CoT (Reviewer H4W6 W2/Q1; Reviewer hyZf W1)

We thank Reviewers H4W6 and hyZf for this point. We agree that long chains are where compression matters most, and that accuracy should be reported together with reasoning length, memory and speed. We did two things: we completed Table 2 with the missing metrics, and we ran a new evaluation with the generation limit raised to 24K tokens.

(a) We add peak KV and speed to the MATH500 and AIME26 results of Table 2 (Qwen3-8B, 10,240-token generation limit).

MethodMATH500 Acc↑Peak KV↓Speed↑AIME26 Acc↑Peak KV↓Speed↑
Vanilla (uncompressed)74.893,4339.8346.678,2969.99
LightThinker64.471,21012.5213.332,46810.61
MemoSight (d=2, SD)71.801,40317.1226.672,70721.83

Accuracy is as reported in Table 2. Peak KV is the peak number of tokens held in the KV cache, averaged over samples, and Speed is generation speed in tok/s. MemoSight decodes with its built-in speculative decoding (SD). Vanilla keeps every token, so its peak KV is its actual reasoning length (prompt plus generated tokens).

The reasoning here is already long: 3.4K tokens on MATH500 and 8.3K on AIME26. The peak of about 1K tokens noted by Reviewer H4W6 is what remains after compression. MemoSight keeps 41% and 33% of Vanilla's peak KV and decodes 1.74× and 2.19× faster, and it improves over LightThinker by 7.33 and 13.34 points at a slightly higher peak, since AMA gives longer steps more memory tokens.

(b) We raised the generation limit to 24K tokens on AIME26 and HMMT Feb 2025. On AIME26, Vanilla's reasoning is already close to the 10,240-token limit, so we raised the limit to 24,576 tokens and added HMMT Feb 2025 as a second hard math benchmark (30 problems each). We use the same Qwen3-8B checkpoints, trained with a 4,096-token training length and not retrained, with the same decoding settings; MemoSight uses d=2 with its built-in SD.

BenchmarkMethodAcc↑Peak KV↓Output tokens (avg)>16KSpeed↑Time↓ (h)
AIME26Vanilla50.00 (15/30)10,56710,1974/306.4913.08
AIME26LightThinker13.33 (4/30)4,43414,90314/307.9715.58
AIME26MemoSight (d=2, SD)30.00 (9/30)3,93513,5348/3012.499.03
HMMTVanilla30.00 (9/30)12,22911,8765/307.9412.46
HMMTLightThinker10.00 (3/30)3,97615,08213/3010.0012.57
HMMTMemoSight (d=2, SD)20.00 (6/30)3,77013,46711/3017.186.53

These results are in Appendix C.5 (Table 9). Peak KV and Speed are defined as in (a). >16K counts the samples whose output exceeds 16K tokens. Time is the sum of per-request generation times. The three methods are run under the same hardware and concurrency setting.

The reasoning in this setting is long. MemoSight generates 13.5K tokens on average on both benchmarks, and 27% and 37% of its outputs exceed 16K tokens.

MemoSight's gain over LightThinker holds on these longer chains. Accuracy is 16.67 and 10.00 points higher (9 vs. 4 and 6 vs. 3 problems solved), with outputs that are 9.2% and 10.7% shorter and an average peak KV that is 11.3% and 5.2% lower.

Memory stays far below Vanilla. MemoSight's average peak KV is 62.8% lower than Vanilla's on AIME26 and 69.2% lower on HMMT, although its outputs are longer, and it has the highest speed and the lowest total time on both benchmarks.

A gap to Vanilla remains, and we state it in the revision (§4.3 and Appendix C.5). Accuracy is still below Vanilla by 20.00 and 10.00 points, which is the current cost of lossy compression on such long chains. With 30 problems per benchmark, one problem is 3.33 points, so we report these results as problem counts and do not claim statistical significance.

AuthorsOpenReview ↗

General Response to All Reviewers - part 3/8

G2. Stronger baselines (Reviewer pBTL Q2; Reviewer H4W6 W1/Q3)

We thank Reviewers pBTL and H4W6 for asking for more baselines. We added three recent ones under the setting of Table 1, and for the two KV eviction methods we also ran a comparison at a similar peak KV, following Reviewer H4W6's request for a matched-peak comparison.

(a) We compare with HybridThinker, R-KV and LazyEviction under the setting of the main table (Qwen3-8B).

  • HybridThinker [1] is a trainable soft-compression method. It additionally keeps the most recent thought steps uncompressed, so it is not a pure soft-compression method in the strict sense, but we include it as the closest recent work. We retrain it with the same initialization, data and hyperparameters as LightThinker and MemoSight.
  • R-KV [2] and LazyEviction [3] are recent reasoning-aware KV eviction methods. They are training-free and applied to the same fine-tuned Vanilla checkpoint, with a KV budget of 1,024 tokens, the budget used for H2O and SepLLM in Table 1.
  • We also considered Breadcrumbs Reasoning [4], which likewise compresses reasoning into learned tokens. Its contribution lies in the training framework (joint distillation and reinforcement learning), so it is not directly comparable under our supervised fine-tuning setting, and we discuss it in the additional related work (Appendix E.1).

All methods use the evaluation setting of Table 1 (greedy decoding, repetition penalty 1.1, at most 10,240 generated tokens).

MethodGSM8K Acc↑Speed↑Peak↓MMLU Acc↑Speed↑Peak↓GPQA Acc↑Speed↑Peak↓BBH Acc↑Speed↑Peak↓Avg. Acc↑Speed↑Peak↓
Vanilla (uncompressed)93.7110.38126486.959.67190954.557.75553291.5210.48170881.689.572603
+ R-KV (new)93.7811.2599483.2511.19100947.9811.49103782.4212.52100576.8611.611011
+ LazyEviction (new)92.659.7299983.159.57101135.359.47103584.049.61100873.809.591013
LightThinker91.4311.0463383.5411.1876352.5310.23144382.4211.2180177.4810.92910
HybridThinker (new)91.8910.2691085.5910.01103552.539.46176687.8810.4699879.4710.051177
MemoSight (d=2)93.1817.2368584.8115.3287452.5316.01178886.4619.5783879.2517.031046
MemoSight (d=4)92.9517.7269884.6214.9187453.5415.51182387.6816.5383279.7016.171057

As in Table 1, Speed is in tok/s and Peak is the average peak number of KV-cache tokens. MemoSight decodes with its built-in speculative decoding, which is part of the method and does not change the output. The Vanilla, LightThinker and MemoSight (d=2) rows are those of Table 1. This table and the one in (b) are in Appendix C.4 (Tables 6 and 7).

MemoSight is more accurate than both eviction methods at a similar average peak KV. Its average accuracy is 79.25 (d=2) and 79.70 (d=4), against 76.86 for R-KV and 73.80 for LazyEviction, at an average peak of about 1.05K tokens against 1.01K, and with its built-in SD it is also faster. The eviction methods stay close on GSM8K and MMLU (R-KV is slightly higher on GSM8K); the difference comes from BBH and, above all, GPQA.

MemoSight is on par with HybridThinker in accuracy, with less memory and higher speed. HybridThinker averages 79.47, between MemoSight at d=2 (79.25) and at d=4 (79.70), while MemoSight's average peak KV is 10%–11% lower and its speed with built-in SD is about 1.6–1.7× higher. HybridThinker's gain comes from keeping recent steps uncompressed and MemoSight's from making the compressed memory more faithful, so the two are complementary and can be combined.

[1] Liu et al. HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps. EMNLP 2026.

[2] Cai et al. R-KV: Redundancy-aware KV Cache Compression for Reasoning Models. NeurIPS 2025.

[3] Zhang et al. LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning. ACL 2026.

[4] Monea et al. Breadcrumbs Reasoning: Memory-Efficient Reasoning with Compression Beacons. arXiv:2510.13797, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.