- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a KV residency scheme for long-context agentic RL rollouts. It keeps the full KV history in host memory and maintains bounded GPU pools, fetching selected entries before attention reads them. A full-length index-key cache remains on the GPU, allowing the sparse indexer to select from the complete history. Pool allocation and initialization use recent prefill access patterns, while LRU and periodic maintenance adapt the cached entries during decoding. The paper also explores early indexing for prefetching. It reports reduced KV traffic and about 6 GiB less GPU allocation at 262K context, with parity against each sparse branch's resident reference in the tested runs. The modeled 1.24× decode speedup concerns early-indexer variants and does not apply to the selected d=0
Critical strengths
-
The profiling provides useful evidence about sparse-attention access patterns. Demand varies across layers and sequences, and entries can be reused after long gaps. These observations help explain why uniform allocation and recency-based permanent eviction can be poor choices.
-
The cache policies follow naturally from the measurements. Ablations examine allocation, initialization, replacement and periodic maintenance, while the memory measurements and parity checks support the proposed residency mechanism.
Critical weaknesses
-
The full-length index-key cache remains in GPU memory. Its footprint grows with context length and concurrency, leaving a scaling constraint after the KV pools have been reduced. It would help to quantify when this cache becomes the next capacity bottleneck.
-
The practical benefit of early indexing remains unclear. The selected d=0 configuration has no cross-layer prefetch window, while the other branches do not clear the reported quality gate. The paper should distinguish the demonstrated benefits of residency management from the benefits projected for the prefetch schedule.
-
The connection to agentic RL is less developed than the motivation suggests. Although the formulation distinguishes logical admission from GPU residency, the pool policies do not appear to use tool-wait duration or active transitions when deciding what to retain. A comparison with offloading the pool during a wait and restoring it on resume would clarify when keeping a bounded GPU pool is useful.
Questions for authors
-
How does the full system compare with a plain LRU offload baseline and a compatible system such as ECHO, under the same memory budget and workload? A HiCache comparison would also be useful where its configuration and role are comparable.
-
Could early indexing be used to prefetch candidates, followed by the target layer's native selection and fetching any remaining misses? How much transfer latency could this hide after accounting for the additional selection work?
-
Under what tool-wait durations and resume patterns is retaining a GPU pool preferable to offloading it entirely? Can the existing traces quantify the effect on resume latency and admitted concurrency?
5 comments
Part 2
Table 3: RULER variable tracking (VT)
| RU_VT_8k | RU_VT_32k | RU_VT_64k | RU_VT_128k | RU_VT_256k | Avg on all | |
|---|---|---|---|---|---|---|
| d=0 | 82.00 | 86.60 | 87.20 | 85.40 | 54.80 | 79.12 |
| d=1 | 89.80 (+7.80) | 94.20 (+7.60) | 90.40 (+3.20) | 84.60 (−0.80) | 56.60 (+1.80) | 82.43 (+3.31) |
| d=2 | 86.60 (+4.60) | 97.20 (+10.60) | 95.40 (+8.20) | 77.20 (−8.20) | 48.60 (−6.20) | 79.33 (+0.21) |
| d=3 | 78.20 (−3.80) | 81.00 (−5.60) | 89.40 (+2.20) | 62.00 (−23.40) | 64.00 (+9.20) | 81.46 (+2.34) |
On average, the overall accuracy of d=1 is 3.31% higher than d=0, d=3 comes next (+2.34%), and d is about the same as d=0.
In addition, we also conducted an accuracy test on the τ-bench (tool-agent-user benchmark) to ensure the reliability, as we are focusing on the Agentic RL.
τ-bench : 165 questions per branch × 4 rollouts per question. Budget for each trajectory: 131072/rollout, 128-round cycle, 7200s per trajectory, 4096 tokens/tern.
| Mean reward | Pass¹ (%) | Pass⁴ (%) | compared with d=0 [95% CI] | |
|---|---|---|---|---|
| d=0 | 6.97 | 6.06 | 19.39 | - |
| d=1 | 11.97 | 18.18 | 30.91 | +5.00 [+1.82, +8.18] |
| d=2 | 5.61 | 6.06 | 14.55 | −1.36 [−3.94, +1.21] |
| d=3 | 11.06 | 10.30 | 28.48 | +4.09 [+1.06, +7.12] |
Under agentic-tool-user serving, d=1 and d=3 are significantly higher than d=0 under same token budget. And we find d=2 is lower than d=0 mainly due to length truncation, because the generation for d = 2 is a bit lengthy.
Then we report the efficieny results on a single H800.
Capacity and throughput : Residency budget is set to 64K tokens of KV per group. At approximately 128K and 256K, offloading increases the maximum batch size from 4 to 16 and from 1 to 16, respectively. The numbers are {maximum batch size (Throughputs tok/s/GPU)}
| Context | Lead 0, no offload | Lead 0, offload | Lead 1/2/3, no offload | Lead 1/2/3, offload |
|---|---|---|---|---|
| ≈64K (control) | 16 (132.1) | — | 16 (137.4) | — |
| ≈128K | 4 (71.3) | 16 (124.9) | 4 (81.3) | 16 (134.0) |
| ≈256K | 1 (29.0) | 16 (123.0) | 1 (35.2) | 16 (131.9) |
TPOT(ms) with context length and batch size
| Scenario | Lead 0, no offload | Lead 0, offload | Lead 1/2/3, no offload | Lead 1/2/3, offload |
|---|---|---|---|---|
| 128K, B1 | 33.90 | 40.9 | 27.98 | 28.7 |
| 128K, B8 | — | 78.3 | — | 69.1 |
| 128K, B16 | — | 128.1 | — | 119.4 |
| 256K, B1 | 34.4 | 41.4 | 28.4 | 29.1 |
| 256K, B16 | — | 130.1 | — | 121.3 |
As batch size increases, offload overhead increases, reducing the benefit of offloading for lead 1/2/3. But the overhead is limited by the and kernel implementation and can be alleviated with TP=4. Our offload operators are not yet optimized to their theoretical limit.
Operator shares of the offload latency (%) are:
| Configuration | plan | write_current | attention | copies | slots | attn_launch | book | plan_wait | scatter | gather | attn_wait | reuse+flush |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 128K, B1 | 40.2 | 11.6 | 9.7 | 8.5 | 6.9 | 6.4 | 4.7 | 4.7 | 3.4 | 1.2 | 2.1 | 0.4 |
| 256K, B1 | 43.3 | 14.8 | 7.4 | 6.5 | 5.6 | 4.9 | 2.5 | 9.2 | 3.0 | 0.8 | 1.6 | 0.3 |
| 64K, B16 | 37.0 | 4.0 | 5.9 | 19.9 | 3.9 | 3.7 | 3.9 | 0.2 | 3.7 | 15.2 | 1.6 | 1.0 |
The overhead is concentrated in CPU-side planning: plan accounts for 40% of total latency at 128K and 43% at 256K. Offload does not add main-path attention latency: attention wait is constant at 0.19 ms/token (0.5% of offload TPOT). We referenced the offload paths in HiSparse and the vLLM KV connector. Our operator overhead still includes process-side planning and copy-submission costs ( and ), so it has not reached the theoretical optimum.
Operator overhead does not vary much with context length. Each round replaces an average of 64–256 tokens per group cross different context. and increase with batch size because their work is per request and layer, while PCIe use remains far below its bandwidth limit.
The efficiency results separate into two contributions. Residency management provides the capacity gain: batch rises from 4 to 16 at 128K and from 1 to 16 at 256K, measured on a single H800. Also, offloading introduce CPU kernel overhead, but this can be minimized by optimized prefetch kernel.Early indexing provides a separate benefit as it decouples indexing and prefetch from critial attention path: it reduces the offload tax from +7.0 ms/token (lead 0) to +0.72 ms/token (lead 1/2/3).
** Part 3**
Question 1: Competitive baselines
Plain LRU: we run a same-stack plain-LRU arm under the same memory budget and workload. At 16K single request, TPOT is policy-insensitive across LRU, uniform, and demand-based variants (42.6–43.6 ms). The difference is not the replacement policy itself but the admission-time demand-based allocation and periodic reallocation, which reduce replacements by up to 41% compared to uniform allocation. Also in our offload kernel latency profling, the PCIe transfer latency is low, so the H2D volume impact is minor.
HiCache is an SGLang-native prefix/radix cache with a whole-block interface. It does not expose a per-step sparse-row fetch path. Its role (prefix reuse for prefill) differs from ours (per-step sparse-row residency for decode).
Our contribution is a residency and scheduling scheme on top of the serving engine, not a replacement for it. Our method is cheap to train and easy to deploy. We chose Qwen3.5 specifically because it is a hybrid attention architecture, which matches the structure of most current open-source models. Importantly, we propose a full range of solutions with training-inference consistency, which is different from most inference optimizations.
Question 2: Two-stage selection
The idea of prefetching indexer candidates and then letting the target layer re-select is a reasonable direction, but it has two costs in our setting.
First, if the target layer's attention still selects from the full KV history, the long-context Top-k compute remains significant (about ~40% in 1M context attention) and is coupled to the main attention path, where it cannot be hidden. Second, the missing-KV fetch is exposed on the critical path. Even when the fetched volume is small, the dominant cost is CPU kernel launch and copy overhead, which exceeds the PCIe transfer latency itself.
A compromise is a coarse-to-fine two-stage selection: the indexer selects a coarse set (e.g., top 16K), those KV entries are prefetched to HBM, and the target attention then precisely selects top 2048 from the 16K. This reduces the Top-k compute from n×n to 16K×n and eliminates the CPU fetch. However, the Top-k compute remains on the attention path, and the accuracy guarantee shifts to whether the indexer's coarse selection contains the precise Top-k. We think this is a promising direction and will explore it in future work to find a better balance between accuacy and efficiency.
Weakness 3 and Question 3: Agentic RL connection and tool-wait retention
The pool policies are deliberately wait-agnostic: they do not predict tool-wait duration or use active transitions to decide retention. The scheduling is handled by vLLM's engine layer, our policy affects only two things per sequence: how much HBM its pool occupies, and when missing entries are fetched. The residency guarantee is logical reachability, where every selected entry stays reachable through the complete host backing, regardless of whether the sequence is actively generating or parked in a tool wait. The pool's role is to keep the working set warm so that resume is near-instant.
To directly test whether retaining a bounded pool is preferable to offloading it entirely during a wait, we compared three configurations at 128K, B=1, lead=1:
| Metric | Full offload (pool=2049) | Coarse residency (64K budget) | No offload |
|---|---|---|---|
| TPOT (ms/token) | 53.58 | 28.7 | 27.98 |
| TTFT (s) | 52.7 | 19.6 | 15.7 |
| Pool hit rate | 0.664 | 0.996 | — |
| H2D volume | 1.68 GB (4120 transfers) | ~2 MB per decode | 0 |
Full offload increases TPOT by 86% and TTFT by 2.7× over coarse residency. A parked sequence that retains its bounded pool resumes at near-resident cost. A fully offloaded sequence pays scattered transfers on every historical access after resume. The design choice is: during a tool wait, the sequence's pool stays in HBM (bounded by the residency budget, a small fraction of the full KV). On resume, the pool is already warm and the first decode step proceeds at near-resident latency. Offloading the pool during the wait would free a small amount of HBM but require a large re-fetch on resume — the worse trade-off.
In addition, we chose partial residensy is due to the consideration of cost-effectiveness. Short sequences stay fully resident with KV holding on HBM, as offloading would add transfer cost without freeing meaningful capacity. Long sequences are offloaded: the freed HBM admits additional sequences. For example, offloading one 256K sequence releases enough capacity to admit six 32k sequences at full residency. This is a joint trade-off among HBM capacity, PCIe transfer volume, and the number of resident sequences.
** Part 1 **
We thank the reviewer for the detailed comments and the constructive questions. Below we address each concern.
Weakness 1: Index-key cache scaling
The full-length index-key cache is resident in HBM, but its footprint is much smaller than the attention KV cache. Our trained indexer uses FP8 keys (1 byte per element) and stores only K, not V. At 256K context on Qwen3.5-4B, a single request's index-K cache is 0.54 GB, while its full attention K+V cache (FP16) is 8.59 GB, as the indexer is approximately 16× smaller than the KV it indexes.
We retain the complete and precise indexer-K on HBM for the sake of model accuracy. The selection and calculation of the Top-k for attention remain unchanged. Compared to inference serving, RL training is more crucial in this regard, as it can lead to errors in the trajectory scoring of this batch, and even cause a regression in the model's capabilities.
We considered further compressing the index-key cache by merging 8 index-K entries into one block, also reducing the Top-k compute from n to n/4. However, experiments show this reduces the attention mass coverage from ~80-90% to ~60-70% and mild degrades downstream benchmark accuracy. RL training has stricter accuracy requirements than inference serving, so this compression is not worth the accuracy cost. The KV cache saving from bounded residency provides a much larger benefit than further index-K compression.
Weakness 2: Early indexing and offload benefit
We apologize for some typos in the paper that caused misunderstanding. We integrate the model into vLLM (torch 2.10.0+cu129, vLLM 0.19.0) and test it on H800 to better demonstrate our contribution.
We would like to clarify the reported number '15/16' for d=1 in Sec 5.1. The native-quality evaluation contains 15 RULER task-by-length comparisons (each with 100 examples), and one GovReport score computed over 200 examples, each sub-task is counted separately. The '15/16' means d=1 has only one sub-task (niah_multikey_1 262K) shows lower score (96 v.s. 97), while the other 15 sub-tasks is higher or equal to d=0 baseline, such as niah_multikey_1 64k (d=1:97 v.s. d=0:93) and nial_multikey_1 131k (d=1:92 v.s. d=0:80). So '15/16' alone do not show that either branch was significantly less accurate.
To make the accuracy results easier to present, we report the full per-task scores both for testbd in paper and additional benchmarks.
Benchmarks:
Table 1: LongBench (LB) + LongCodeBench (LCB)
Table 2: RULER needle retrieval (MQ + MV)