- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes reusing the KV cache instead of re-prefilling the context after compaction to accelerate rollouts in RL training. To maintain consistency between training and inference, they introduce custom attention masks during training. Experimental results show their method substantially reduces RL training time with little accuracy loss.
Critical strengths
- Although the proposed KV cache reuse approach is direct and simple, the custom attention mask maintains the training-inference consistency and demonstrates its effect in RL training.
Critical weaknesses
- Although accuracy is evaluated to a limited extent, long-term training stability and convergence remain unverified.
Questions for authors
- What speedup does your method achieve when applied to agentic LLM inference?
2 comments
Reminder
Dear reviewer,
The deadline for the discussion period is very soon. We appreciate the time taken to review this paper. Please let us know if we have addressed your concerns and if it has increased your confidence or perception of our work, or if there are any remaining concerns.
Rebuttal
We appreciate the reviewer’s time and feedback. We have uploaded a new version of the paper that includes additional experiments in response to the reviews, improved text and improved visuals.
Q1. Are long-term training stability and convergence verified?
We train for 500 steps on TextWorld and 200 on ALFWorld and SWE-bench, by which point the baselines converge on the training set. KV-Streams does not always converge within this budget, but we use an equal step budget for fairness. Figure 4 truncates some runs for visual clarity, and we have added the untruncated curves to Appendix C. Regardless, KV-Streams consistently matches baseline accuracy. We thank the reviewer for this insightful question and are happy to clarify further.
Q2. What speedup does the method achieve for agentic LLM inference?
While our main focus is training performance, we agree that a dedicated study of inference is important, so we added an ablation with Qwen3.5-4B and Qwen3.5-9B (Appendix E). Using sliding-window compaction, we measured the time to sample 1024 episodes of 32k, 64k and 128k tokens. The gap between KV-Streams and re-prefill grows consistently with model size and sequence length. Across all lengths, KV-Streams cuts inference time by 1.7–2.3× (1.7–1.9× for 4B, 2.1–2.3× for 9B). We thank the reviewer for this suggestion, which has helped improve the paper.