acceptodds
Withdrawn from ICLR 2027

Rewind: Restoring Evicted Prefix States for Hybrid LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-context LLM agents repeatedly revisit shared histories during retries and branch exploration, making efficient prefix reuse essential for responsive serving. However, hybrid models cannot reconstruct an earlier recurrent state from the latest one, so cache eviction forces requests to recompute long prefixes even when they add only a short suffix. Optimizing checkpoint placement alone cannot recover the state once it has been discarded. Our key insight is that preserving complete boundary checkpoints can decouple prefix reuse from the lifetime of native cache entries. We present Rewind, a serving system that restores evicted prefixes for hybrid LLM agents. First, Rewind persists recurrent state, convolution history, and attention keys and values at a common token boundary, coordinating their installation across workers before execution resumes. Second, it uses cost-based checkpoint admission and retention to prioritize expected saved computation under storage pressure. Finally, its restore gate compares complete installation cost with the additional prefill avoided beyond the native cache hit. On a 16-session Qwen3.5-27B-FP8 replay using two RTX 4090 GPUs, Rewind’s additional checkpoint store reduces prefill by 91.7%, replay duration by 30.5%, and return-round p95 latency by 26.5% relative to native caching. Under forced eviction at matched storage capacity, selective Rewind performs 41% less prefill than Snapshot-LRU, which saves every boundary and evicts by recency.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Hybrid LLM Serving, Prefix Reuse, Cache Eviction, Recurrent State Checkpointing, Agent Workloads

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GpRXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Rewind is a vLLM extension that using a separate store of complete hybrid-model prefix checkpoints. It saves recurrent states, convolution histories and attention KV at a common token boundary, then reinstalls them across workers when a request revisits an evicted prefix. Capture and retention policies decide which checkpoints to keep, and a restore gate compares installation cost with avoided prefill. On constructed agent replays, the additional store reduces 27B replay duration by 30% over native caching, while at matched storage capacity, the full selective configuration reduces prefill by 41% and replay duration by 8.7% over the default snapshot-LRU.

Critical strengths
  • Reusing an evicted hybrid prefix is a practical serving problem. Rewind correctly captures the state components needed at a common bondary and restores them before execution resumes.
  • It shows improvement than snapshot-LRU when the capacity matches. It shows that the selective configuration preserves more reusable computation under storage pressure, beyond the benefit of adding a checkpoint store alone.
Critical weaknesses
  • Limited novelty beyond hybrid checkpointing and cache persistence. My main concern is what Rewind adds beyond storing complete checkpoints outside the native cache. Marconi already considers recurrent checkpoints, reuse likelihood, and compute savings per byte; the spacing and admission models in Section 3.3 follow established results. Coordinating capture and installation is necessary engineering, but the paper does not identify a new mechanism or demonstrate an advantage that establishes a sufficient research contribution. There is no end-to-end comparison with the published Marconi system: the Marconi-style baseline adapts its admission policy inside Rewind's own persistence layer and installer.

  • Limited evidence for end-to-end value and serving scalability. The 30.5% headline reduction comes with extra checkpoint storage. As for matched capacity, the 41% prefill reduction becomes only an 8.7% replay-time reduction, however, with considerable serving stack complexity increases; Figure 5 does not show clear speedup from the length-aware gate over always-restore. The deep-context experiments run on a single node with at most two RTX 4090 GPUs and three active requests. They do not establish how checkpoint storage, eviction, and coordinated restoration behave with more workers, higher concurrency, or multiple nodes, in a real industrial serving system. Appendix E also reports memory exhaustion in a deep-context run at concurrency six. The current results demonstrate a small-scale prototype, but do not show that its additional policy and coordination machinery provides a worthwhile benefit at serving scale.

Questions for authors
  • What concrete mechanism is new beyond complete-state persistence and existing hybrid-cache management? How faithfully does the Marconi-style baseline reproduce the published admission and eviction policies, and what prevents an end-to-end comparison with the original system?
  • Are there token-bearing agent traces on which the policy was evaluated without selecting return-heavy sessions or enforcing token-identical returns? What recoverable reuse and end-to-end benefit were observed on those traces?
  • How would checkpoint ownership, eviction, and installation coordination work across multiple nodes or a larger worker pool? What bounds restoration-buffer memory and prevents concurrent restores from exhausting resources? If possible, please report any larger-scale or higher-concurrency evaluation in a more realistic setup.
Reviewer_orrROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Rewind, a system that restores evicted prefix states for hybrid LLMs by offloading complete checkpoints to CPU memory and disk. It introduces cost-aware checkpoint admission, eviction, and restoration policies to reduce redundant prefill computation. Experiments on hybrid LLMs show up to 91.7% less prefill than native caching and 41% less than Snapshot-LRU under matched storage budgets.

Critical strengths
  1. Important and timely problem. Hybrid models have recently gained significant attention (e.g., Qwen3.5 and Kimi K3), making efficient prefix caching for these models an increasingly important research problem.
Critical weaknesses

Weaknesses:

  1. Limited novelty. The main contribution appears to be extending existing offloading systems to hybrid models, which seems more like an engineering effort than a fundamental research contribution. It is unclear whether adapting these systems to hybrid models introduces significant new technical challenges that justify a dedicated system design.
  2. Limited evaluation scope. Although the evaluation uses real-world user traces from TraceLab, the authors selectively evaluate traces exhibiting particularly frequent return behaviors, which may favor the proposed approach. It remains unclear how representative these workloads are and whether the proposed system can deliver meaningful benefits in more general real-world production settings.
Questions for authors

What are the fundamental technical challenges in adapting existing offloading systems to hybrid models? Could the authors clarify which challenges cannot be addressed by straightforward extensions of existing approaches?

Reviewer_4Aq8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focused on prefix cache (state) issues of hybrid LLM agents, identifing the significance of compelete boundry states. To improve the performance, they proposed Rewind, a serving system that restores evicted prefixes for hybrid LLM agents. Rewind couples complete-state persistence with coordinated installation to recover evicted prefixes, and uses capture, retention, and restore decisions to control its storage and transfer costs, shceduling those boundry checkpoints efficiently. According to the experinmental results presented in this paper, Rewind could largely reduce the ratio of prefill, and outperform existing works over representitive models and benchmarks.

Critical strengths
  1. This paper identified the importance of complete boundry states, some quantitative numbers were provided to illustrate the issue.

  2. Rewind couples complete-state persistence with coordinated installation to recover evicted prefixes, and uses capture, retention, and restore decisions to control its storage and transfer costs, shceduling those boundry checkpoints efficiently.

  3. The overview figure is drawn clearly, easy to understand how the whole system works.

  4. Experinment on representitive models and benchmarks performs well and the detailed setting is provided.

Critical weaknesses
  1. A brief introduction of the recurrent layer in hybrid LLMs could make the presentation better.

  2. It seems that only one benchmark is used for evaluation and the limitation section points out those replays are constructed. Evaluating on a wider range of benchmarks would be more convincing.

  3. The result of Ring and Kimi-Linear presented in Appendix not as good as Qwen 3.5. In particular, Rewind underperforms compared to Snapshot-LRU on Kimi-Linear without calibrating adimission. After calibrating, the 4.85M prefill is still bigger. Also, the results for Marconi on these two models are missing.

  4. In the experiment you mentioned that "the 4B settings are 0.73 for native caching and approximately 0.77 for Rewind", so why is the memory setting defferent?

Once these concerns are addressed, I would be willing to increase my rating.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.