acceptodds
Under review as a conference paper at ICLR 2027

Reversible Memory Management for LLM Agents with Support for Token-Level Supervision

Abstract

Summarizing agent histories can discard details needed later, while recalling historical text repeats prefill computation. We introduce RETURN, a framework for reversible context management that keeps active context compact while retaining historical KV states. It restores selected turns with updated positions, without re-prefilling their text. An ephemeral probe selects history from turn digests, then rolls back its tokens and execution state to keep memory management outside the task trace. This enables supervision from a full-context teacher without management labels. Our backend offloads inactive KV to CPU, while path-aware parallel replay reconstructs recorded history selections and position mappings and propagates gradients through reused states. Across memory and agentic benchmarks, RETURN generally delivers stronger and more consistent performance than text-recall baselines without additional training. It also improves accuracy beyond the native context window. Despite probing overhead with short outputs, reduced GPU KV memory enables higher concurrency and improves serving throughput. On-policy distillation with parallel replay improves task performance, demonstrating compatibility with token-level supervision on formal assistant tokens.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Memory Management, KV Cache Reuse, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uGGtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a comprehensive architecture integrating KV-level reversible context management, isolated memory selection, and training replay. The method decouples memory management from task execution and further implements RoPE positional adaptation, CPU offloading, parallel training replay, and gradient propagation. Its effectiveness is validated via extensive experiments on memory benchmark, agent benchmark, system throughput evaluation, ablation studies, and OPD training verification.

Critical strengths
  • The motivation is clear, and the problem is practical. The paper addresses two limitations in existing long-horizon agent memory management: summary compression tends to discard critical information; memory retrieval requires repeated prefill. The proposed reversible KV management resolve both limitations.
  • The method is well-designed. ReTurn integrates KV restoration, RoPE position adaptation, CPU offloading and Ephemeral Probe. It also decouples memory management operations from official task trajectories. This avoids extra management tokens interfering with agent execution and reduces the complexity of training adaptation.
  • ReTurn proposed Path-aware Parallel Replay reconstructs dynamic contexts at different timesteps and enables gradient propagation over reused KV states. It works not only for inference but also for OPD training.
  • The experiments are relatively comprehensive.
Critical weaknesses
  • KV reuse lacks sufficient analysis of its effective boundary: historical KV is computed under the original context, but when restored, the context and positions may have changed, and adjusting RoPE alone can't fully fix this.
  • Training experiments are insufficient: results only show ReTurn makes OPD training feasible with performance gains, but lack direct comparison to full-context OPD and systematic analysis of Parallel Replay's training overhead.
Questions for authors
  • ReTurn reuses historical KV states by directly remapping RoPE positions. While this effectively reduces computational overhead, does significant performance degradation occur when the reused historical KV differs substantially from the current context (there is weak correlation between the restored KV)? Could you provide a case study comparing actual outputs in such scenarios?
  • What is the additional training cost of Path-aware Parallel Replay? Compared with standard full-context training, what are its training throughput, peak memory usage and total computational overhead? Besides, the proposed method introduces complexity, and it remains unclear whether it can be stably applied to training at scale for long-horizon tasks.
  • ReTurn lacks discussion and ablations on its triggering mechanism. Agent tasks vary in their reliance on historical context, and probing at every turn may introduce extra overhead, especially for simpler tasks. A more adaptive triggering strategy could improve its practical usage.
  • Adding more discussions on related methods for context compression and KV reuse is beneficial. Summarizing their pros and cons can help better evaluate the novelty and effectiveness of this work.
2 comments

Q3. Triggering mechanism

ReTurn lacks discussion and ablations on its triggering mechanism. Agent tasks vary in their reliance on historical context, and probing at every turn may introduce extra overhead, especially for simpler tasks.

The probe is already conditional rather than unconditional. It runs only when folding is enabled and something has actually been folded, so on short trajectories, and on the early turns of every trajectory, no probe fires at all. In sample 135 above, 110 probes run across 114 assistant turns because that trajectory folds early and often, but a task whose history never exceeds the active window never pays for a probe.

Beyond this condition the probe fires on every turn, and we think you have identified one of the more promising directions for this line of work. The interesting question is not only how to save the cost of an unnecessary probe, but when restoration is needed at all. Trajectories differ a great deal in this respect. Some, like the one above, repeatedly return to evidence found a hundred turns earlier, while others never look back and would be served just as well with no restoration. A method that could recognize which regime it is in, and switch probing on and off accordingly, would be more useful than our fixed policy, and would also tell us something about how agents actually use their history.

We are taking this up as the direction of our next piece of work. Rather than hand-tuning a rule, we want the policy to be learned: a model that decides at each step whether a probe is needed, trained together with the probe itself so that its selections also become more accurate. We will say this in the paper, both as a limitation of the present design and as what we intend to build on it.

Q4. Discussion of related context compression and KV reuse methods

Adding more discussions on related methods for context compression and KV reuse is beneficial. Summarizing their pros and cons can help better evaluate the novelty and effectiveness of this work.

We will expand Section 5 of the paper with a structured comparison.

FamilyExamplesStrengthLimitationRelation to RETURN
Summary compressionReSum, MEM1, MemAgent, Context-FoldingBounded active context, no raw history storedMust anticipate future needs, discarded detail is unrecoverableFigure 2 shows the cost of compressing before the question is known
Text recallHiAgent, MemexHistory stays recoverableRecalled text is re-prefilled on every recallSame recall ability without re-encoding, 2.37x faster on our serving workload
Sparse attentionQUEST, InfLLMStored history unchanged, fine-grainedRule-based scores, selection repeated per decoding step or per blockSemantic turn-level selection once per turn, so selected KV stays resident and inactive KV can stay on CPU
KV reuse across segmentsCacheBlend, KVLinkReuses precomputed statesIndependent segments lack cross-segment dependencies, needing recomputation or trained linkingAgent turns are encoded under the active prefix, so only position must be adapted

Why RETURN does not need the repair that CacheBlend and KVLink require. CacheBlend and KVLink are both KV reuse methods, and both report that naively concatenating reused KV degrades quality, so each adds a mechanism to recover it. CacheBlend selectively recomputes the KV of a subset of tokens, and KVLink trains link tokens and adjusts positions to restore the missing interaction. The degradation they repair comes from how their caches are built. In the RAG setting each context paragraph is prefilled separately, so the KV of one paragraph is computed without ever attending to the others, and the cross-paragraph dependencies are simply absent from the cache. RETURN starts from a different position. Every turn of an agent trajectory is encoded inside one continuous context, where it attends to the earlier turns that were active when it was produced, so the dependencies between turns are already written into the stored KV. Restoration brings these states back in chronological order and only has to adapt their positions, which is why Figure 3 shows restored history reaching the full-context level without any recomputation or trained linking.


We thank the reviewer for these questions. Several identified reporting gaps rather than defects in the method, and addressing them has made the training and boundary analysis more complete.

Thank you for the detailed rebuttal. It addresses most of my concerns, and I will maintain my positive rating.

Reviewer_j2ZbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ReTurn manages agent history by deactivating and restoring cached KV states at turn granularity. Keys are retained before rotary position encoding and assigned updated positions when restored. A temporary probe selects history from short digests, then its execution state is rolled back so memory management does not enter the formal task trace. CPU offloading supports serving, and a parallel replay procedure enables token-level training along recorded memory paths. The evaluation covers memory retrieval, browsing, coding, serving efficiency, and on-policy distillation.

Critical strengths
  • The separation of memory management from task generation has a clear practical consequence. It allows a full-context teacher to supervise ordinary assistant tokens without supplying memory-operation labels. The replay procedure addresses the changing positions of reused keys, and the distillation experiment shows that this training interface works in practice.

  • The evaluation connects task quality to actual system behavior. Results cover several model sizes and both retrieval and interactive workloads, with particularly competitive quality for the larger models. Removing history restoration, randomizing selection, re-prefilling selected text, and disabling CPU offloading test meaningful parts of the design. The serving comparison also accounts for the GPU used by the summarizer. Together, these provide a positive reason to accept the work beyond the conceptual appeal of reversible memory.

Critical weaknesses
  • All evaluated backbones are hybrid models whose recurrent layers continue to process the complete history. The paper acknowledges this, but it leaves an important ambiguity in the main empirical argument. In the oracle restoration experiment, the recurrent state is shared across selections and already contains information from the entire history. Recovering good answers after restoring a small fraction of KV therefore does not isolate how much information the restored KV contributes. The same issue limits generalization to models without this persistent history channel. A control that separates recurrent-state retention from KV restoration, or an evaluation on a full-attention backbone, would clarify whether the proposed reuse mechanism preserves quality independently of hybrid memory. Until then, the strongest supported claim is about reversible attention management for the evaluated hybrid models.
Questions for authors

See weaknesses.

Reviewer_dJjLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a reversible context-management framework for long-horizon LLM agents called RETURN. It retains each turn’s pre-RoPE KV, folds turns out of active attention, and restores selected turns with recomputed RoPE positions without re-prefill instead of summarizing history or re-prefilling recalled text. On hybrid Qwen3.5/3.6/3.8 models across 6 benchmarks, RETURN stays close to full-context accuracy, surpasses it beyond the native window on RULER, and improves BrowseComp via OPD.

Critical strengths
  1. Ephemeral probe rollback cleanly separates memory management from task generation, enabling training-free use and full-context teacher supervision without management labels.

  2. Evaluation spans 6 benchmarks and 3 models, staying close to full-context accuracy with ~74–82% active-context savings and ~2× faster workload completion, conducted useful selection and serving ablations.

Critical weaknesses
  1. Missing comparisons with closely related KV-selection methods such as InfLLM[1] / QUEST[2] or other SOTA methods, and insufficiently matched retrieval budgets limit attribution of the gains. The claim of being generally stronger than text recall is not consistently supported, e.g., Memex outperforms RETURN on BrowseComp-4B, 33.5 vs 26.0, and no variance is reported.

  2. Serving gains are demonstrated under selected long-trajectory workloads against a GPU-resident baseline capped at 256 concurrency. The gains appear to come mainly from CPU offloading, without offloading, RETURN is slower than the baseline (6650 s vs 5938 s in Tables 1 and 4). And reported throughput also includes probe tokens.

  3. Parallel replay lacks numerical validation against sequential execution and training-efficiency measurements. The single OPD experiment has no enough ablation, e.g., the same OPD applied to a full-context student, so the improvement can't be attributed to RETURN.

[1] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory. In Advances in Neural Information Processing Systems, volume 37, 2024.

[2] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: Query-aware sparsity for efficient long-context LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47901–47911. PMLR, 2024.

Questions for authors
  1. How does KV restoration compare with text re-prefill in both accuracy and latency under identical history selections and token budgets, and with InfLLM/Quest where applicable?

  2. Do the conclusions hold for a full-attention backbone, and how much do retained recurrent states contribute?

  3. Against a full-context baseline with CPU offloading, what are the task-token throughput, tail latency, and CPU memory costs across request rates?

  4. Does parallel replay match sequential logits and parameter gradients, and what training speed x and memory benefits does it provide? How does the OPD gain compare with applying the same recipe to a full-context student?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.