acceptodds
Under review as a conference paper at ICLR 2027

KV-STREAMS FOR EFFICIENT COMPACTION IN AGEN- TIC REINFORCEMENT LEARNING

Abstract

Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5× wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, compaction, post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SD7mOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes reusing the KV cache instead of re-prefilling the context after compaction to accelerate rollouts in RL training. To maintain consistency between training and inference, they introduce custom attention masks during training. Experimental results show their method substantially reduces RL training time with little accuracy loss.

Critical strengths
  1. Although the proposed KV cache reuse approach is direct and simple, the custom attention mask maintains the training-inference consistency and demonstrates its effect in RL training.
Critical weaknesses
  1. Although accuracy is evaluated to a limited extent, long-term training stability and convergence remain unverified.
Questions for authors
  1. What speedup does your method achieve when applied to agentic LLM inference?
2 comments

We appreciate the reviewer’s time and feedback. We have uploaded a new version of the paper that includes additional experiments in response to the reviews, improved text and improved visuals.

Q1. Are long-term training stability and convergence verified?

We train for 500 steps on TextWorld and 200 on ALFWorld and SWE-bench, by which point the baselines converge on the training set. KV-Streams does not always converge within this budget, but we use an equal step budget for fairness. Figure 4 truncates some runs for visual clarity, and we have added the untruncated curves to Appendix C. Regardless, KV-Streams consistently matches baseline accuracy. We thank the reviewer for this insightful question and are happy to clarify further.

Q2. What speedup does the method achieve for agentic LLM inference?

While our main focus is training performance, we agree that a dedicated study of inference is important, so we added an ablation with Qwen3.5-4B and Qwen3.5-9B (Appendix E). Using sliding-window compaction, we measured the time to sample 1024 episodes of 32k, 64k and 128k tokens. The gap between KV-Streams and re-prefill grows consistently with model size and sequence length. Across all lengths, KV-Streams cuts inference time by 1.7–2.3× (1.7–1.9× for 4B, 2.1–2.3× for 9B). We thank the reviewer for this suggestion, which has helped improve the paper.

Dear reviewer,

The deadline for the discussion period is very soon. We appreciate the time taken to review this paper. Please let us know if we have addressed your concerns and if it has increased your confidence or perception of our work, or if there are any remaining concerns.

Reviewer_ftgmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses repeated prefilling in context-compacted agentic RL. KV-streams evicts selected KV entries while preserving the remaining cache at inference, and reproduces the eviction history with a training-time attention mask. It is evaluated with Summary, Markovian Thinker, Markovian Pick, and Sliding Window on TextWorld, ALFWorld, and software-engineering tasks. Additional warm-start and synthetic retrieval experiments investigate whether RL alone can teach retained KVs to carry information from evicted text.

Critical strengths
  1. The paper separates the choice of what to retain from the efficient execution of compaction. Section 2 makes the cost of repeated prefilling explicit, particularly at high retention ratios. Coordinating inference-time eviction with training-time masking is a useful systems contribution.
  2. The software-engineering experiments indicate that KV-streams can approach full-context task performance with less training time. This provides useful evidence for budget-constrained long-horizon agents and connects the efficiency improvement to practical task quality.
  3. Section 6 studies warm-start training and retrieval after eviction, with additional prompt and entity transfer experiments. Appendix G documents eviction timing, answer-contamination checks, and alternative training objectives, making the mechanism experiments more inspectable.
Critical weaknesses
  1. Checking that the counting text contains no candidate answer does not exclude indirect target-dependent coding through numbers, whitespace, length, or formatting. A stronger control would force the identical counting-token sequence for every target, then compare retained KVs against KVs recomputed without the assignment and KVs swapped across targets while keeping the visible text fixed. The current results are consistent with KV memory but do not fully isolate it.
  2. Asynchronous cache consistency needs validation. Section 4 allows in-flight weight updates without flushing cached KVs, so a rollout can contain states produced by different policy versions. Recomputing the trajectory with current weights and the eviction mask reproduces attention visibility, but not necessarily those historical cache values. The paper should explain how rollout log-probabilities, importance ratios, and cache versions are handled, and quantify this approximation through frozen-weight rollouts and an asynchronous-lag sweep, reporting both quality and throughput.
  3. Universal compatibility and long-horizon scaling remain under-tested. The evidence centers on two 4B models and bounded trajectories. Preserving KVs changes the conditional state relative to re-prefilling, so implementation compatibility does not imply quality preservation for arbitrary compaction rules. Summary-cache construction, positional handling beyond the original range, sparse-mask implementation, and trainer memory versus total trajectory length need clarification. Bounded inference-cache memory should be distinguished from bounded training memory.
Questions for authors

See the Weakness above

5 comments

We appreciate the reviewer’s time and feedback. We have uploaded a new version of the paper that includes additional experiments in response to the reviews, improved text and improved visuals.

Q1. Could the model encode the target indirectly through the counting text?

We thank the reviewer for this point and agree that the model may encode this information indirectly. We anticipated this, and all experiments already force-decodes the exact sequence (1, …, k), which removes this effect. We have made this explicit in Section 6.2.

Q2. How are stale cache values, importance ratios and asynchronous lag handled?

Keeping stale KVs is central to KV-Streams’ success. Staleness has been studied extensively in prior work and shown to be compute-optimal without hurting performance (PipelineRL [1], ScaleRL [2]). In-flight weight updates are also standard in frontier asynchronous post-training pipelines. Because KV-Streams keeps only part of each trace on GPU, flushing the cache would force a re-prefill and substantially reduce the efficiency gains. We ran an experiment comparing the trainer–inference mismatch between re-prefilling and keeping the KVs (Appendix F). We observe little to no difference, and the mismatch is similar to or lower than what PipelineRL [1] reports for full re-prefill.

For the text-based games, we use prime-rl’s CISPO importance-sampling correction with a KL coefficient of 0.001 (Appendix B.1). The maximum off-policy lag is fixed per task, at three steps for TextWorld and ALFWorld and two steps for SWE-bench (Table 1). Within each task, every algorithm uses the same lag, so all comparisons are made under identical staleness. We do not sweep the lag because our focus is efficient KV-cache handling rather than tolerance to sample staleness.

Q3. Does the method hold for longer horizons, and how is trainer memory bounded?

Our experiments are fairly long-horizon, with SWE tasks using a 64k context window. As stated in the paper, we preserve positional encodings after the cache is evicted. On the trainer side, KV-Streams is more memory-efficient than full attention because we use FlexAttention, which supports memory-efficient sparse attention masks.

We agree that bounded inference memory differs from bounded trainer memory. In practice, though, trainer memory is fully controllable because samples are complete by the time they reach the trainer. The main memory challenge is at inference, where sample lengths are unknown and many requests run concurrently, while the trainer can always process a fixed token budget to maximize throughput and memory use. We also scale the sequence budget with task complexity, using 16k tokens for ALFWorld, 32k for TextWorld and 64k for SWE-bench (Appendix B.1).

To show that the method generalizes across context length and model size, we ran an inference ablation with Qwen3.5-4B and Qwen3.5-9B (Appendix E). Using sliding-window compaction, we measured the time to sample 1024 episodes of 32k, 64k and 128k tokens. The gap between KV-Streams and re-prefill grows consistently with model size and sequence length. KV-Streams cuts inference time by 1.7–2.3× (1.7–1.9× for 4B, 2.1–2.3× for 9B).

We hope this clarifies the reviewer’s concerns. We have updated the manuscript with the missing details, and we thank the reviewer for helping us improve our work. Given the new experiments and clarifications, we hope the reviewer will consider raising their score.

[1] PipelineRL. https://arxiv.org/abs/2509.19128

[2] ScaleRL. https://arxiv.org/abs/2510.13786

Dear reviewer,

The deadline for the discussion period is very soon. We appreciate the time taken to review this paper. Please let us know if we have addressed your concerns and if it has increased your confidence or perception of our work, or if there are any remaining concerns.

I thank the authors for the rebuttal. Yet, I still find the performance claim overstated. On SWE-bench, with Markovian Thinker policy held fixed, accuracy drops from 53.4% with Re-prefill to about 49% with KV-streams. I also could not find the promised explanations of CISPO and FlexAttention or the clarification of the scope of training in the revised manuscript. Given these unresolved concerns, I maintain my original score of 2.

Reviewer_9BuoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KV-streams, a method for efficient context compaction in agentic RL. Instead of recomputing the retained context after each compaction, it reuses existing KV-cache entries and applies matching attention masks during training. Experiments on TextWorld, ALFWorld, and SWE-bench Verified demonstrate substantial training speedups with comparable task performance.

Critical strengths
  1. Efficient context compaction is an important problem for long-horizon agentic RL. The proposed method is simple and practically useful.

  2. The evaluation covers several compaction strategies and realistic workloads, including SWE-bench Verified. The end-to-end training improvements are meaningful.

Critical weaknesses
  1. Novelty compared to prior work. MEMENTO [1] already explores preserving KV states after context eviction and using the retained states to carry information forward. KV-streams extends this idea to multiple compaction strategies and agentic RL training. However, the paper could better clarify what the key technical differences are beyond this extension.

  2. Limited evaluation of KV-cache consistency. Section 4 mentions retaining KV states across policy updates without flushing the cache. Since these KV states were computed using older model parameters, it's unclear how the resulting mismatch affects training stability and model quality. Also, TextWorld and SWE experiments only use one training seed, limiting confidence in the reported performance differences.

[1] MEMENTO: Teaching LLMs to Manage Their Own Context, COLM 2026.

Questions for authors
  1. Can the authors clarify the key technical differences from MEMENTO, especially regarding KV-cache reuse and training-time masking?

  2. Can the authors provide an ablation on retaining KV states across policy updates? How much does policy staleness affect model performance?

4 comments

We appreciate the reviewer’s time and feedback. We have uploaded a new version of the paper that includes additional experiments in response to the reviews, improved text and improved visuals.

Q1. What are the key technical differences from Memento?

We agree that Memento is closely related, and we acknowledge it throughout the paper. Memento is a strong contribution, and our work pushes its ideas further with four key scientific and engineering innovations.

Agentic and multi-turn compaction. Memento evicts within a single generation and does not need prefix caching. KV-Streams enables KV eviction while preserving the prefix cache, which is crucial in multi-turn settings. This makes it conceptually more complex, since it must handle special-token boundaries, coordinate the inference engine and the trainer, and efficiently communicate eviction events between them.

Compaction methodology. Memento relies on special compaction tokens that the model must learn, whereas our models compact without any training overhead. Memento also focuses on SFT, with a single RL experiment, while we focus on RL post-training across agentic tasks. Memento reports performance degradation, which matches our findings for summary compaction. We also show that other strategies, such as sliding window, become scalable with KV-Streams without degrading performance. Without KV-Streams, these strategies add too much overhead because the context must be re-prefilled.

Pseudo-recurrence. We show that streamed KVs can act as a pseudo-recurrent state. By retaining KVs that depend on evicted context, the model carries forward information that is no longer in the context window. Section 6.2 shows that RL alone can induce this behaviour, without the expensive SFT that every Memento experiment relies on.

Engineering. We also remove major engineering bottlenecks. We build infrastructure to scale KV-Streams in both SGLang and vLLM. We identify and quantify the real overhead of re-prefill compaction in both the trainer and the inference engine, and implement a solution. Finally, we use components of PipelineRL (in-flight weight updates) to train at scale, and explain why this matters.

Q2. How does retaining KV states across policy updates affect performance?

Keeping stale KVs is central to KV-Streams’ success. We understand the reviewer’s concern, but staleness has been studied extensively in prior work and shown to be compute-optimal without hurting performance (PipelineRL [1], ScaleRL [2]). In-flight weight updates are also standard in frontier asynchronous post-training pipelines (Magistral [3], GLM-5 [4]).

Because KV-Streams keeps only part of each trace on GPU, re-prefilling the full trace can introduce trainer–inference mismatch and reduces the efficiency gains. To validate keeping the cache, we compared the trainer–inference mismatch over several training steps with the KV cache kept versus re-prefilled (Appendix F). We observe little to no difference, and the mismatch is similar to or lower than what PipelineRL [1] reports for full re-prefill.

Q3. Do single training seeds limit confidence in the results?

We agree that one seed is not ideal. ALFWorld uses two seeds, and we have now run three seeds for the final SWE-bench evaluation, which further supports our claim of no performance degradation.

We lack the resources to run more. The TextWorld baseline alone ran for 6.25 days on a full H100 node, so we prioritized more baselines and experiments over additional seeds.

We are also careful not to overclaim. We do not claim to improve on the baselines, only that we observe no performance degradation, since better performance cannot be claimed confidently without more seeds.

Finally, seeds matter less for our main result. The effect in Figure 2 is (nearly) deterministic because it concerns re-prefill cost, not the performance of a randomised training or sampling algorithm.

[1] PipelineRL. https://arxiv.org/abs/2509.19128

[2] ScaleRL. https://arxiv.org/abs/2510.13786

[3] Rastogi, Abhinav, et al. “Magistral.” arXiv preprint arXiv:2506.10910 (2025).

[4] Zeng, Aohan, et al. “GLM-5: From Vibe Coding to Agentic Engineering.” arXiv preprint arXiv:2602.15763 (2026).

Dear reviewer,

The deadline for the discussion period is very soon. We appreciate the time taken to review this paper. Please let us know if we have addressed your concerns and if it has increased your confidence or perception of our work, or if there are any remaining concerns.

Thanks the Authors for the response. I do have a few follow-up questions popped out of mind. Can KV-Stream compatible with the hybrid models, such as Desne-Attention&Mamba Hybrid or Dense-Attention&Sliding Window Hybrid?

We thank the reviewer for bringing this up, it does indeed work with hybrid architectures.

We only apply KV-streams to the full attention layers. In fact we have verified this and have implementations for this for Qwen 3.5 series of models which is a hybrid full attention & delta net variant. In fact, we have added experiments to Appendix E measuring the efficiency gains using static token budgets (32k, 64k, 128k) for Qwen 3.5 4b, 9b and intend to add 27b for the camera ready.

Specifically with these hybrid architectures, across all lengths, KV-Streams cuts inference time by 1.7–2.3× (1.7–1.9× for 4B, 2.1–2.3× for 9B). Further, we have updated the text to specifically reflect your point on hybrid architectures (lines 1147-1149).

We thank the reviewer for bringing this up and hope this increases your confidence or perceived score in our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.