acceptodds
Under review as a conference paper at ICLR 2027

Coarse KV Residency: Decoupling KV Reachability from GPU Memory in Long-Context RL Rollouts

Abstract

Agentic RL rollouts alternate generation with tool waits, during which each sequence's complete KV state must remain in GPU memory. In a replay of a production turn structure, 85.6% of resident KV belongs to sequences idle in such waits. However, this footprint cannot be reduced by eviction or approximation: rollouts are the training data of RL post-training, and perturbing the sparse top-k selection changes the policy being optimized. We propose coarse KV residency, which decouples logical KV reachability from physical residency: the host stores every sequence's complete KV state, while the GPU maintains a bounded pool per sequence, layer and KV group. Pool capacities are sized at admission from measured sparse-attention demand and maintained by warm-start initialization, LRU retention and periodic reallocation. Residency never changes the model's own top-k selection, and missing selected entries are fetched before attention reads them. On Qwen3.5-4B, pooled decoding is bit-exact against full residency and the selected configuration passes all sixteen registered noninferiority endpoints, while allocation remains within 2.5-13% of an oracle, admission seeding reduces K/V traffic by 25.7%, and periodic reallocation reduces replacements by 40.3% on long rollouts. The freed residency yields up to 88% higher aggregate decode throughput through additional concurrent rollouts.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV Cache Management, GPU–CPU Offloading and Prefetching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_14jMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a KV residency scheme for long-context agentic RL rollouts. It keeps the full KV history in host memory and maintains bounded GPU pools, fetching selected entries before attention reads them. A full-length index-key cache remains on the GPU, allowing the sparse indexer to select from the complete history. Pool allocation and initialization use recent prefill access patterns, while LRU and periodic maintenance adapt the cached entries during decoding. The paper also explores early indexing for prefetching. It reports reduced KV traffic and about 6 GiB less GPU allocation at 262K context, with parity against each sparse branch's resident reference in the tested runs. The modeled 1.24× decode speedup concerns early-indexer variants and does not apply to the selected d=0

Critical strengths
  1. The profiling provides useful evidence about sparse-attention access patterns. Demand varies across layers and sequences, and entries can be reused after long gaps. These observations help explain why uniform allocation and recency-based permanent eviction can be poor choices.

  2. The cache policies follow naturally from the measurements. Ablations examine allocation, initialization, replacement and periodic maintenance, while the memory measurements and parity checks support the proposed residency mechanism.

Critical weaknesses
  1. The full-length index-key cache remains in GPU memory. Its footprint grows with context length and concurrency, leaving a scaling constraint after the KV pools have been reduced. It would help to quantify when this cache becomes the next capacity bottleneck.

  2. The practical benefit of early indexing remains unclear. The selected d=0 configuration has no cross-layer prefetch window, while the other branches do not clear the reported quality gate. The paper should distinguish the demonstrated benefits of residency management from the benefits projected for the prefetch schedule.

  3. The connection to agentic RL is less developed than the motivation suggests. Although the formulation distinguishes logical admission from GPU residency, the pool policies do not appear to use tool-wait duration or active transitions when deciding what to retain. A comparison with offloading the pool during a wait and restoring it on resume would clarify when keeping a bounded GPU pool is useful.

Questions for authors
  1. How does the full system compare with a plain LRU offload baseline and a compatible system such as ECHO, under the same memory budget and workload? A HiCache comparison would also be useful where its configuration and role are comparable.

  2. Could early indexing be used to prefetch candidates, followed by the target layer's native selection and fetching any remaining misses? How much transfer latency could this hide after accounting for the additional selection work?

  3. Under what tool-wait durations and resume patterns is retaining a GPU pool preferable to offloading it entirely? Can the existing traces quantify the effect on resume latency and admitted concurrency?

5 comments

** Part 1 **

We thank the reviewer for the detailed comments and the constructive questions. Below we address each concern.

Weakness 1: Index-key cache scaling

The full-length index-key cache is resident in HBM, but its footprint is much smaller than the attention KV cache. Our trained indexer uses FP8 keys (1 byte per element) and stores only K, not V. At 256K context on Qwen3.5-4B, a single request's index-K cache is 0.54 GB, while its full attention K+V cache (FP16) is 8.59 GB, as the indexer is approximately 16× smaller than the KV it indexes.

We retain the complete and precise indexer-K on HBM for the sake of model accuracy. The selection and calculation of the Top-k for attention remain unchanged. Compared to inference serving, RL training is more crucial in this regard, as it can lead to errors in the trajectory scoring of this batch, and even cause a regression in the model's capabilities.

We considered further compressing the index-key cache by merging 8 index-K entries into one block, also reducing the Top-k compute from n to n/4. However, experiments show this reduces the attention mass coverage from ~80-90% to ~60-70% and mild degrades downstream benchmark accuracy. RL training has stricter accuracy requirements than inference serving, so this compression is not worth the accuracy cost. The KV cache saving from bounded residency provides a much larger benefit than further index-K compression.

Weakness 2: Early indexing and offload benefit

We apologize for some typos in the paper that caused misunderstanding. We integrate the model into vLLM (torch 2.10.0+cu129, vLLM 0.19.0) and test it on H800 to better demonstrate our contribution.

We would like to clarify the reported number '15/16' for d=1 in Sec 5.1. The native-quality evaluation contains 15 RULER task-by-length comparisons (each with 100 examples), and one GovReport score computed over 200 examples, each sub-task is counted separately. The '15/16' means d=1 has only one sub-task (niah_multikey_1 262K) shows lower score (96 v.s. 97), while the other 15 sub-tasks is higher or equal to d=0 baseline, such as niah_multikey_1 64k (d=1:97 v.s. d=0:93) and nial_multikey_1 131k (d=1:92 v.s. d=0:80). So '15/16' alone do not show that either branch was significantly less accurate.

To make the accuracy results easier to present, we report the full per-task scores both for testbd in paper and additional benchmarks.

Benchmarks:

  • longbench (LB): hotpotqa / musique (multi-hop QA, token F1), passage_retrieval_en (retrieval accuracy).
  • longcodebench (LCB): long-code multiple choice, scored by first A-D match.
  • RULER (RU): niah_multiquery (MQ) and niah_multivalue (MV) needle retrieval, variable tracking (VT), each with 8k, 32k, 64k, 128k and 256k context.

Table 1: LongBench (LB) + LongCodeBench (LCB)

LB_hotpotqaLB_musiqueLB_retr_enLCB_4kLCB_8kLCB_16kLCB_32kLCB_64kLCB_128kLCB_256k
d=064.9239.76100.0080.0066.6790.7085.0585.8290.8583.77
d=163.98 (−0.94)38.16 (−1.59)100.00 (+0.00)80.00 (+0.00)66.67 (+0.00)90.70 (+0.00)84.58 (−0.47)85.07 (−0.75)90.52 (−0.33)83.25 (−0.52)
d=264.90 (−0.02)39.98 (+0.22)100.00 (+0.00)80.00 (+0.00)66.67 (+0.00)89.53 (−1.16)84.11 (−0.93)84.33 (−1.49)89.87 (−0.98)82.20 (−1.57)
d=364.72 (−0.20)40.18 (+0.43)100.00 (+0.00)80.00 (+0.00)66.67 (+0.00)90.70 (−0.00)85.05 (+0.00)85.07 (−0.75)90.20 (−0.65)84.29 (+0.52)

Table 2: RULER needle retrieval (MQ + MV)

RU_MQ_8kRU_MQ_32kRU_MQ_64kRU_MQ_128kRU_MQ_256kRU_MV_8kRU_MV_32kRU_MV_64kRU_MV_128kRU_MV_256k
d=098.7589.2595.2596.5098.7593.7560.0056.2554.7551.25
d=1100.00 (+1.25)97.00 (+7.75)98.00 (+2.75)96.00 (−0.50)99.25 (+0.50)96.00 (+2.25)77.25 (+17.25)74.75 (+18.50)60.25 (+5.50)63.75 (+12.50)
d=2100.00 (+1.25)89.75 (+0.50)95.50 (+0.25)95.25 (−1.25)97.25 (−1.50)92.00 (−1.75)61.25 (+1.25)56.50 (+0.25)57.00 (+2.25)52.25 (+1.00)
d=399.75 (+1.00)97.25 (+8.00)97.00 (+1.75)98.00 (+1.50)98.50 (−0.25)96.75 (+3.00)80.75 (+20.75)79.00 (+22.75)65.00 (+10.25)63.00 (+11.75)

Part 2

Table 3: RULER variable tracking (VT)

RU_VT_8kRU_VT_32kRU_VT_64kRU_VT_128kRU_VT_256kAvg on all
d=082.0086.6087.2085.4054.8079.12
d=189.80 (+7.80)94.20 (+7.60)90.40 (+3.20)84.60 (−0.80)56.60 (+1.80)82.43 (+3.31)
d=286.60 (+4.60)97.20 (+10.60)95.40 (+8.20)77.20 (−8.20)48.60 (−6.20)79.33 (+0.21)
d=378.20 (−3.80)81.00 (−5.60)89.40 (+2.20)62.00 (−23.40)64.00 (+9.20)81.46 (+2.34)

On average, the overall accuracy of d=1 is 3.31% higher than d=0, d=3 comes next (+2.34%), and d is about the same as d=0.

In addition, we also conducted an accuracy test on the τ-bench (tool-agent-user benchmark) to ensure the reliability, as we are focusing on the Agentic RL.

τ-bench : 165 questions per branch × 4 rollouts per question. Budget for each trajectory: 131072/rollout, 128-round cycle, 7200s per trajectory, 4096 tokens/tern.

Mean rewardPass¹ (%)Pass⁴ (%)compared with d=0 [95% CI]
d=06.976.0619.39-
d=111.9718.1830.91+5.00 [+1.82, +8.18]
d=25.616.0614.55−1.36 [−3.94, +1.21]
d=311.0610.3028.48+4.09 [+1.06, +7.12]

Under agentic-tool-user serving, d=1 and d=3 are significantly higher than d=0 under same token budget. And we find d=2 is lower than d=0 mainly due to length truncation, because the generation for d = 2 is a bit lengthy.

Then we report the efficieny results on a single H800.

Capacity and throughput : Residency budget is set to 64K tokens of KV per group. At approximately 128K and 256K, offloading increases the maximum batch size from 4 to 16 and from 1 to 16, respectively. The numbers are {maximum batch size (Throughputs tok/s/GPU)}

ContextLead 0, no offloadLead 0, offloadLead 1/2/3, no offloadLead 1/2/3, offload
≈64K (control)16 (132.1)—16 (137.4)—
≈128K4 (71.3)16 (124.9)4 (81.3)16 (134.0)
≈256K1 (29.0)16 (123.0)1 (35.2)16 (131.9)

TPOT(ms) with context length and batch size

ScenarioLead 0, no offloadLead 0, offloadLead 1/2/3, no offloadLead 1/2/3, offload
128K, B133.9040.927.9828.7
128K, B8—78.3—69.1
128K, B16—128.1—119.4
256K, B134.441.428.429.1
256K, B16—130.1—121.3

As batch size increases, offload overhead increases, reducing the benefit of offloading for lead 1/2/3. But the overhead is limited by the plan_plan and copies_copies kernel implementation and can be alleviated with TP=4. Our offload operators are not yet optimized to their theoretical limit.

Operator shares of the offload latency (%) are:

Configurationplanwrite_currentattentioncopiesslotsattn_launchbookplan_waitscattergatherattn_waitreuse+flush
128K, B140.211.69.78.56.96.44.74.73.41.22.10.4
256K, B143.314.87.46.55.64.92.59.23.00.81.60.3
64K, B1637.04.05.919.93.93.73.90.23.715.21.61.0

The overhead is concentrated in CPU-side planning: plan accounts for 40% of total latency at 128K and 43% at 256K. Offload does not add main-path attention latency: attention wait is constant at 0.19 ms/token (0.5% of offload TPOT). We referenced the offload paths in HiSparse and the vLLM KV connector. Our operator overhead still includes process-side planning and copy-submission costs ( plan_plan and copies_copies ), so it has not reached the theoretical optimum.

Operator overhead does not vary much with context length. Each round replaces an average of 64–256 tokens per group cross different context. plan_plan and copies_copies increase with batch size because their work is per request and layer, while PCIe use remains far below its bandwidth limit.

The efficiency results separate into two contributions. Residency management provides the capacity gain: batch rises from 4 to 16 at 128K and from 1 to 16 at 256K, measured on a single H800. Also, offloading introduce CPU kernel overhead, but this can be minimized by optimized prefetch kernel.Early indexing provides a separate benefit as it decouples indexing and prefetch from critial attention path: it reduces the offload tax from +7.0 ms/token (lead 0) to +0.72 ms/token (lead 1/2/3).

** Part 3**

Question 1: Competitive baselines

Plain LRU: we run a same-stack plain-LRU arm under the same memory budget and workload. At 16K single request, TPOT is policy-insensitive across LRU, uniform, and demand-based variants (42.6–43.6 ms). The difference is not the replacement policy itself but the admission-time demand-based allocation and periodic reallocation, which reduce replacements by up to 41% compared to uniform allocation. Also in our offload kernel latency profling, the PCIe transfer latency is low, so the H2D volume impact is minor.

HiCache is an SGLang-native prefix/radix cache with a whole-block interface. It does not expose a per-step sparse-row fetch path. Its role (prefix reuse for prefill) differs from ours (per-step sparse-row residency for decode).

Our contribution is a residency and scheduling scheme on top of the serving engine, not a replacement for it. Our method is cheap to train and easy to deploy. We chose Qwen3.5 specifically because it is a hybrid attention architecture, which matches the structure of most current open-source models. Importantly, we propose a full range of solutions with training-inference consistency, which is different from most inference optimizations.

Question 2: Two-stage selection

The idea of prefetching indexer candidates and then letting the target layer re-select is a reasonable direction, but it has two costs in our setting.

First, if the target layer's attention still selects from the full KV history, the long-context Top-k compute remains significant (about ~40% in 1M context attention) and is coupled to the main attention path, where it cannot be hidden. Second, the missing-KV fetch is exposed on the critical path. Even when the fetched volume is small, the dominant cost is CPU kernel launch and copy overhead, which exceeds the PCIe transfer latency itself.

A compromise is a coarse-to-fine two-stage selection: the indexer selects a coarse set (e.g., top 16K), those KV entries are prefetched to HBM, and the target attention then precisely selects top 2048 from the 16K. This reduces the Top-k compute from n×n to 16K×n and eliminates the CPU fetch. However, the Top-k compute remains on the attention path, and the accuracy guarantee shifts to whether the indexer's coarse selection contains the precise Top-k. We think this is a promising direction and will explore it in future work to find a better balance between accuacy and efficiency.

Weakness 3 and Question 3: Agentic RL connection and tool-wait retention

The pool policies are deliberately wait-agnostic: they do not predict tool-wait duration or use active transitions to decide retention. The scheduling is handled by vLLM's engine layer, our policy affects only two things per sequence: how much HBM its pool occupies, and when missing entries are fetched. The residency guarantee is logical reachability, where every selected entry stays reachable through the complete host backing, regardless of whether the sequence is actively generating or parked in a tool wait. The pool's role is to keep the working set warm so that resume is near-instant.

To directly test whether retaining a bounded pool is preferable to offloading it entirely during a wait, we compared three configurations at 128K, B=1, lead=1:

MetricFull offload (pool=2049)Coarse residency (64K budget)No offload
TPOT (ms/token)53.5828.727.98
TTFT (s)52.719.615.7
Pool hit rate0.6640.996—
H2D volume1.68 GB (4120 transfers)~2 MB per decode0

Full offload increases TPOT by 86% and TTFT by 2.7× over coarse residency. A parked sequence that retains its bounded pool resumes at near-resident cost. A fully offloaded sequence pays scattered transfers on every historical access after resume. The design choice is: during a tool wait, the sequence's pool stays in HBM (bounded by the residency budget, a small fraction of the full KV). On resume, the pool is already warm and the first decode step proceeds at near-resident latency. Offloading the pool during the wait would free a small amount of HBM but require a large re-fetch on resume — the worse trade-off.

In addition, we chose partial residensy is due to the consideration of cost-effectiveness. Short sequences stay fully resident with KV holding on HBM, as offloading would add transfer cost without freeing meaningful capacity. Long sequences are offloaded: the freed HBM admits additional sequences. For example, offloading one 256K sequence releases enough capacity to admit six 32k sequences at full residency. This is a joint trade-off among HBM capacity, PCIe transfer volume, and the number of resident sequences.

Reviewer_3J9NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the GPU KV memory bottleneck in long-context agentic RL rollouts, where many sequences remain idle during tool waits. It proposes coarse KV residency, which stores complete KV states on CPU while maintaining smaller GPU pools through demand-based capacity allocation, warm-start initialization, LRU retention, and periodic reallocation. The paper also explores early-indexer prefetching to reduce KV transfer overhead, although the final selected configuration does not use early lookahead. Experiments on sparse-attention traces and a sparse-trained Qwen3.5-4B model show that the proposed method reduces KV transfers while preserving model outputs, and suggest potential throughput improvements through increased concurrency.

Critical strengths
  1. The paper identifies a relevant GPU memory bottleneck in long-context agentic RL rollouts, particularly the KV memory occupied by idle sequences during tool waits.

  2. The proposed KV residency mechanism is straightforward and well-motivated, using observed attention patterns to guide memory allocation and cache management while preserving the model's original attention selection.

  3. The paper provides useful characterizations of KV access patterns and ablation studies to examine the effectiveness of its cache management components.

Critical weaknesses
  1. The paper aims to improve rollout concurrency and throughput by reducing GPU KV residency, but lacks an end-to-end evaluation demonstrating these benefits in realistic agentic RL workloads. The reported throughput gains mainly rely on separate batch-scaling experiments rather than the complete proposed system, while the actual offloading implementation introduces substantial latency and CPU-side contention.

  2. Several key techniques, including KV offloading, LRU-based cache management, and early-indexer prefetching, have been explored in prior work. The more distinctive contributions appear to be demand-based capacity allocation and periodic reallocation, but their evaluations mainly rely on offline replay and comparisons with simple cache policies, without sufficient comparisons against recent competitive systems or evidence of their end-to-end benefits.

  3. Although the paper emphasizes the memory occupied by idle sequences during tool waits, the proposed method does not explicitly leverage these characteristics and appears largely applicable to general long-context sparse-attention inference. Moreover, the main experiments rely on general long-context workloads and a separately sparse-trained Qwen3.5-4B model, providing limited evidence of effectiveness in realistic agentic RL settings.

  4. Despite receiving substantial attention in both the method and experiments, the final selected configuration is (d=0), which provides no early lookahead. The potential acceleration of other configurations is mainly supported by a modeled schedule, while these configurations do not satisfy all downstream quality requirements. This leaves a considerable gap between the proposed system design and its demonstrated benefits.

Questions for authors
  1. Can the authors directly demonstrate that the reduced GPU KV residency enables higher actual rollout concurrency and throughput under the same hardware budget, rather than inferring these benefits from separate batch-scaling experiments, especially given the measured offloading overhead and CPU contention?

  2. What are the key innovations over existing KV offloading systems, and can their practical benefits be demonstrated against competitive baselines? In particular, can the effectiveness of periodic reallocation be validated in real long-running decoding rather than primarily through offline replay?

  3. How does the proposed method specifically address the characteristics of agentic RL, particularly idle sequences during tool waits? Can its effectiveness be demonstrated on realistic agentic RL workloads and existing native sparse-attention models?

  4. Given that the final selected configuration uses (d=0) without early lookahead, what actual contribution does early-indexer prefetching make to the validated system, and what measured acceleration supports it beyond the modeled estimates?

6 comments

Part 1

We thank the reviewer for the detailed comment and recognition. We have integrated the model into vLLM (torch 2.10.0+cu129, vLLM 0.19.0) and tested it on H800. Below we split the gains into parts to make them easier to understand.

Weakness 1 and 4: Complete-system benefit and the role of early indexing

To make the accuracy results easier to inspect, we report the full per-task scores both for testbd in paper and additional benchmarks.

Benchmarks:

  • longbench (LB): hotpotqa / musique (multi-hop QA, token F1), passage_retrieval_en (retrieval accuracy).
  • longcodebench (LCB): long-code multiple choice, scored by first A-D match.
  • RULER (RU): niah_multiquery (MQ) and niah_multivalue (MV) needle retrieval, variable tracking (VT), each with 8k, 32k, 64k, 128k and 256k context.

Table 1: LongBench (LB) + LongCodeBench (LCB)

LB_hotpotqaLB_musiqueLB_retr_enLCB_4kLCB_8kLCB_16kLCB_32kLCB_64kLCB_128kLCB_256k
d=064.9239.76100.0080.0066.6790.7085.0585.8290.8583.77
d=163.98 (−0.94)38.16 (−1.59)100.00 (+0.00)80.00 (+0.00)66.67 (+0.00)90.70 (+0.00)84.58 (−0.47)85.07 (−0.75)90.52 (−0.33)83.25 (−0.52)
d=264.90 (−0.02)39.98 (+0.22)100.00 (+0.00)80.00 (+0.00)66.67 (+0.00)89.53 (−1.16)84.11 (−0.93)84.33 (−1.49)89.87 (−0.98)82.20 (−1.57)
d=364.72 (−0.20)40.18 (+0.43)100.00 (+0.00)80.00 (+0.00)66.67 (+0.00)90.70 (−0.00)85.05 (+0.00)85.07 (−0.75)90.20 (−0.65)84.29 (+0.52)

Table 2: RULER needle retrieval (MQ + MV)

RU_MQ_8kRU_MQ_32kRU_MQ_64kRU_MQ_128kRU_MQ_256kRU_MV_8kRU_MV_32kRU_MV_64kRU_MV_128kRU_MV_256k
d=098.7589.2595.2596.5098.7593.7560.0056.2554.7551.25
d=1100.00 (+1.25)97.00 (+7.75)98.00 (+2.75)96.00 (−0.50)99.25 (+0.50)96.00 (+2.25)77.25 (+17.25)74.75 (+18.50)60.25 (+5.50)63.75 (+12.50)
d=2100.00 (+1.25)89.75 (+0.50)95.50 (+0.25)95.25 (−1.25)97.25 (−1.50)92.00 (−1.75)61.25 (+1.25)56.50 (+0.25)57.00 (+2.25)52.25 (+1.00)
d=399.75 (+1.00)97.25 (+8.00)97.00 (+1.75)98.00 (+1.50)98.50 (−0.25)96.75 (+3.00)80.75 (+20.75)79.00 (+22.75)65.00 (+10.25)63.00 (+11.75)

Table 3: RULER variable tracking (VT)

RU_VT_8kRU_VT_32kRU_VT_64kRU_VT_128kRU_VT_256kAvg on all
d=082.0086.6087.2085.4054.8079.12
d=189.80 (+7.80)94.20 (+7.60)90.40 (+3.20)84.60 (−0.80)56.60 (+1.80)82.43 (+3.31)
d=286.60 (+4.60)97.20 (+10.60)95.40 (+8.20)77.20 (−8.20)48.60 (−6.20)79.33 (+0.21)
d=378.20 (−3.80)81.00 (−5.60)89.40 (+2.20)62.00 (−23.40)64.00 (+9.20)81.46 (+2.34)

On average, the overall accuracy of d=1 is 3.31% higher than d=0, d=3 comes next (+2.34%), and d=2 is about the same as d=0.

The offload policy itself does not enter these numbers: it does not change the top-k selection or the attention computation.

Capacity, throughput and TPOT.

The same HBM now holds more sessions. With a 64K KV residency budget per group on single H800, the maximum batch size and the aggregate throughput (tokens/s) are:

ContextLead 0, no offloadLead 0, offloadLead 1, no offloadLead 1, offload
≈64K (control)16 (132.1)—16 (137.4)—
≈128K4 (71.3)16 (124.9)4 (81.3)16 (134.0)
≈256K1 (29.0)16 (123.0)1 (35.2)16 (131.9)

Each cell is maximum batch size with aggregate throughput (tok/s/GPU) in parentheses. At 128K the batch rises from 4 to 16 and throughput from 71.3 (lead 0, no offload) to 134.0 (lead 1, offload); at 256K the batch rises from 1 to 16 and throughput from 29.0 to 131.9.

TPOT (ms) Per-token latency at B1:

ScenarioLead 0, no offloadLead 0, offloadLead 1, no offloadLead 1, offload
128K, B133.9040.927.9828.7
128K, B8—78.3—69.1
128K, B16—128.1—119.4
256K, B134.441.428.429.1
256K, B16—130.1—121.3

At the same batch B1, the d=1 offload system is faster per token than the d=0 full-residency reference: 28.7 vs. 33.90 ms at 128K and 29.1 vs. 34.4 ms at 256K. At B16 only the offload configuration can serve the batch. Per-token latency rises with batch in all arms.

** Part2 **

What early indexing contributes. With d=0 the index-Q is projected from the target layer's own entry residual, so selection, fetch and wait all sit in front of attention and cannot overlap the main path. With d≥1 the index-Q projection is finished one to three half-layers earlier, so scoring and fetch fit inside the layer's own QKV window. At 128K B1 lead-1(no offload) shows up as 5.92 ms/token saved under full residency compared to lead-0(no offload), and lead-1(offload) shows 12.2ms improvement under offload. Early indexing decouples most of the offload cost from the main path attention, so the configuration that carries the prefetch window is also the one that keeps near-resident latency. The remaining overhead is mostly dominated by CPU-side planning:

Configurationplanwrite_currentattentioncopiesslotsattn_launchbookplan_waitscattergatherattn_waitreuse+flush
128K, B140.211.69.78.56.96.44.74.73.41.22.10.4
256K, B143.314.87.46.55.64.92.59.23.00.81.60.3
64K, B1637.04.05.919.93.93.73.90.23.715.21.61.0

Operator shares of the offload latency (%). Our operator overhead still includes process-side planning and copy-submission costs ( plan_plan and copies_copies ), so it has not reached the theoretical optimum (e.g., compared with HiSparse).

Our test is integrated into vllm 0.19.0. The serving path is vLLM's standard scheduler: admission, queueing and batching are the engine's decisions. Coarse residency changes only where a sequence's KV lives and when a missing selected entry is fetched. A tool wait changes when a sequence is parked, not these per-round terms. What the wait changes is how much KV sits in HBM while the sequence is parked, and that is what the capacity table measures.

Our results show that under the same HBM capacity, Coarse Residency achieves a higher batch size with more throughput. This enables more admitted sessions to complete over the same wall-clock.

Weakness 2: Novelty and comparison to existing systems

Our goal is to optimize the overall training wall-clock, including TPOT, HBM footprint, PCIe traffic, and admitted concurrency. We therefore do not claim a new offloading, LRU, or look-ahead indexing paradigm. Our contribution is identifying the resource constraints and access structure of RL, and combining index scheduling, selective prefetching, non-uniform physical residency and cold recovery into a coordinated design based on the rollout cost of the entire batch.

RL rollouts differ from ordinary inference serving in their resource demands. Agentic rollouts alternate generation with tool waits, producing the batch hole. Our algorithm and system design are built on the sparse attention demands and characteristics profiled from real model inference (DeepSeek-V3.2-Exp). Early indexing and prefetch move the auxiliary work off the main critical path, and the HBM saved by bounded residency is converted into more effective concurrency and lower recovery cost. The result is a shorter wall-clock for the entire rollout batch on fixed hardware with no accuracy loss.

The method is cheap to train and easy to deploy. Training only changes the source of the index-Q during the indexer warm up. The inference serving flow is fully compatible with the current vLLM engine. Although the final evaluation is on Qwen3.5-4B, we also ran horizon ablations from d=0 to d=8 on Qwen3.6-35B-A3B. We chose Qwen3.5 specifically because it is a hybrid attention architecture, which matches the structure of most current open-source models. Because there are fewer full layers, the end-to-end benefit is actually much less than that of the full-attention model (e.g., Qwen3).

Importantly, we propose a full range of solutions with training-inference consistency, which is different from most inference optimizations.

** Part 3**

Weakness 3: Agentic characteristics and model scope

Agentic benchamrk accuarcy. In addition, we conducted an accuracy test on the τ-bench (tool-agent-user benchmark) to ensure the reliability, as we are focusing on the Agentic RL.

τ-bench Scale: 165 questions per branch × 4 rollouts per question. Budget for each trajectory: 131072/rollout, 128-round cycle, 7200s per trajectory, 4096 tokens/tern.

Mean rewardPass¹ (%)Pass⁴ (%)compared with d=0 [95% CI]
lead=06.976.0619.39-
lead=111.9718.1830.91+5.00 [+1.82, +8.18]
lead=25.616.0614.55−1.36 [−3.94, +1.21]
lead=311.0610.3028.48+4.09 [+1.06, +7.12]

Under agentic-tool-user serving, d=1 and d=3 are significantly higher than d=0 under same token budget. And we find d=2 is lower than d=0 mainly due to length truncation, because the generation for d = 2 is a bit lengthy.

Retaining a bounded pool vs. full offload. To directly compare the cost of keeping a bounded GPU pool against fully offloading a sequence, we run lead=1 at 128K context, B=1, 512 outputs, with the residency pool reduced to 2049 (top-2048 + current K).

MetricFull offload (pool=2049)Coarse residency (64K budget)No offload
TPOT (ms/token)53.5828.727.98
TTFT (s)52.719.615.7
Pool hit rate0.6640.996—
H2D volume1.68 GB (4120 transfers)~2MB each decode0

Full offload increases TPOT by 86% over coarse residency (53.58 vs 28.7 ms) and TTFT by 2.68× (52.7 vs 19.6 s). Actually, we have already discussed the infeasibility of full offloading in our characterizing in Section 3 . As shown in Figure 3, the overlap of the Top-k index for adjacent decoding steps is low and varying cross layers, and the efficiency of PCIe for transmitting scattered bytes is extremely far from achieving the ideal bandwidth.

Reviewer_KhZVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes coarse KV residency for long-context RL rollouts. Complete KV states remain in host memory, while bounded GPU pools are managed through demand-based allocation, warm-start seeding, LRU retention, and periodic reallocation. The system preserves each sparse model’s top-k selection by fetching missing entries before attention. Evaluation includes sparse-attention profiling, Qwen3.5-4B correctness and quality tests, and capacity measurements.

Critical strengths

The residency design has a clear correctness boundary. Offloading changes where KV entries reside without restricting the model’s candidate set. Across five branches and two context lengths, pooled execution matches the corresponding engine references on the tested log-probabilities and top-k sets. Releasing external GPU KV also reduces net allocation by 6.00 GiB at 262K context.

The profiling supports demand-aware memory management. Across 540 profiled pools, admission-time working-set sizes vary by 9.2×. Demand-based allocation reduces replacements by up to approximately 41%, while warm-start seeding reduces measured K/V traffic by 25.7%. These results support allocating memory according to heterogeneous demand rather than uniformly.

Critical weaknesses

The quality-preserving operating point does not realize the proposed prefetch acceleration. Only d=0 passes all 16 downstream noninferiority endpoints; d=1 passes 15, and d=2/3 pass 12 each. However, d=0 has no lead window. The reported 1.24× decoding acceleration instead comes from a counterfactual decoupled schedule for branches with positive lead. Thus, a single configuration demonstrating both the required quality and that acceleration remains unestablished.

The net throughput benefit of the complete offload system is not directly demonstrated. The headline 88% gain comes from increasing a feasible batch from 4 to 8 at 255K context, rather than comparing full residency against the proposed offload system. It measures the potential value of additional concurrency without establishing the gain after transfer, synchronization, and host-memory contention. Production telemetry and synthetic occupancy replay motivate the workload but do not close this experimental gap.

Questions for authors

What are the directly measured maximum concurrency and aggregate throughput of the complete d=0 offload system versus full residency under the same GPU-memory budget?

How do concurrent KV transfers and mixed-length sessions affect throughput and exposed stalls?

4 comments

Part 1

We thank the reviewer for the detailed comment and recognition. We apologize for some typos in the paper that caused misunderstanding. We integrate the model and system into vLLM (torch 2.10.0+cu129, vLLM 0.19.0) and test on H800. Below we split the gains into parts to make them easier to understand.

W1 Accuracy

We would like to clarify the reported number '15/16' for d=1 in Sec 5.1. The native-quality evaluation contains 15 RULER task-by-length comparisons (each with 100 examples), and one GovReport score computed over 200 examples, each sub-task is counted separately. The '15/16' means d=1 has only one sub-task (niah_multikey_1 262K) that shows lower score (96 v.s. 97), while the other 15 sub-tasks is higher or equal to d=0 baseline, such as niah_multikey_1 64k (d=1:97 v.s. d=0:93) and nial_multikey_1 131k (d=1:92 v.s. d=0:80). So '15/16' alone do not show that either branch was significantly less accurate.

To make the accuracy results easier to present, we report the full per-task scores both for testbed in paper and additional benchmarks.

Benchmarks:

  • longbench (LB): hotpotqa / musique (multi-hop QA, token F1), passage_retrieval_en (retrieval accuracy).

  • longcodebench (LCB): long-code multiple choice, scored by first A-D match.

  • RULER (RU): niah_multiquery (MQ) and niah_multivalue (MV) needle retrieval, variable tracking (VT), each with 8k, 32k, 64k, 128k and 256k context.

Table 1: LongBench (LB) + LongCodeBench (LCB)

LB_hotpotqaLB_musiqueLB_retr_enLCB_4kLCB_8kLCB_16kLCB_32kLCB_64kLCB_128kLCB_256k
d=064.9239.76100.0080.0066.6790.7085.0585.8290.8583.77
d=163.98 (−0.94)38.16 (−1.59)100.00 (+0.00)80.00 (+0.00)66.67 (+0.00)90.70 (+0.00)84.58 (−0.47)85.07 (−0.75)90.52 (−0.33)83.25 (−0.52)
d=264.90 (−0.02)39.98 (+0.22)100.00 (+0.00)80.00 (+0.00)66.67 (+0.00)89.53 (−1.16)84.11 (−0.93)84.33 (−1.49)89.87 (−0.98)82.20 (−1.57)
d=364.72 (−0.20)40.18 (+0.43)100.00 (+0.00)80.00 (+0.00)66.67 (+0.00)90.70 (−0.00)85.05 (+0.00)85.07 (−0.75)90.20 (−0.65)84.29 (+0.52)

Table 2: RULER needle retrieval (MQ + MV)

RU_MQ_8kRU_MQ_32kRU_MQ_64kRU_MQ_128kRU_MQ_256kRU_MV_8kRU_MV_32kRU_MV_64kRU_MV_128kRU_MV_256k
d=098.7589.2595.2596.5098.7593.7560.0056.2554.7551.25
d=1100.00 (+1.25)97.00 (+7.75)98.00 (+2.75)96.00 (−0.50)99.25 (+0.50)96.00 (+2.25)77.25 (+17.25)74.75 (+18.50)60.25 (+5.50)63.75 (+12.50)
d=2100.00 (+1.25)89.75 (+0.50)95.50 (+0.25)95.25 (−1.25)97.25 (−1.50)92.00 (−1.75)61.25 (+1.25)56.50 (+0.25)57.00 (+2.25)52.25 (+1.00)
d=399.75 (+1.00)97.25 (+8.00)97.00 (+1.75)98.00 (+1.50)98.50 (−0.25)96.75 (+3.00)80.75 (+20.75)79.00 (+22.75)65.00 (+10.25)63.00 (+11.75)

Table 3: RULER variable tracking (VT)

RU_VT_8kRU_VT_32kRU_VT_64kRU_VT_128kRU_VT_256kAvg on all
d=082.0086.6087.2085.4054.8079.12
d=189.80 (+7.80)94.20 (+7.60)90.40 (+3.20)84.60 (−0.80)56.60 (+1.80)82.43 (+3.31)
d=286.60 (+4.60)97.20 (+10.60)95.40 (+8.20)77.20 (−8.20)48.60 (−6.20)79.33 (+0.21)
d=378.20 (−3.80)81.00 (−5.60)89.40 (+2.20)62.00 (−23.40)64.00 (+9.20)81.46 (+2.34)

On average, the overall accuracy of d=1 is 3.31% higher than d=0, d=3 comes next (+2.34%), and d=2 is about the same as d=0.

In addition, we also conducted an accuracy test on the τ-bench (tool-agent-user benchmark) to ensure the reliability, as we are focusing on the Agentic RL.

τ-bench : 165 questions per branch × 4 rollouts per question. Budget for each trajectory: 131072/rollout, 128-round cycle, 7200s per trajectory, 4096 tokens/turn.

Mean rewardPass¹ (%)Pass⁴ (%)compared with d=0 [95% CI]
d=06.976.0619.39-
d=111.9718.1830.91+5.00 [+1.82, +8.18]
d=25.616.0614.55−1.36 [−3.94, +1.21]
d=311.0610.3028.48+4.09 [+1.06, +7.12]

Under agentic-tool-user serving, d=1 and d=3 are significantly higher than d=0 under same token budget. And we find d=2 is lower than d=0 mainly due to length truncation, because the generation for d = 2 is a bit lengthy.

Finaly, we need to state that our offloading strategy only determines which KVs will be stored in the HBM based on indexer topk, and does not change the selection of the top-k and the attention. Therefore, offloading does not affect the model's accuracy.

Part 2

Weakness 2 Efficiency on single H800

Capacity and throughput. Residency budget is set to 64K tokens of KV per group. At approximately 128K and 256K, offloading increases the maximum batch size from 4 to 16 and from 1 to 16, respectively. The numbers are {maximum batch size (Throughputs tok/s/GPU)}

ContextLead 0, no offloadLead 0, offloadLead 1/2/3, no offloadLead 1/2/3, offload
≈64K (control)16 (132.1)—16 (137.4)—
≈128K4 (71.3)16 (124.9)4 (81.3)16 (134.0)
≈256K1 (29.0)16 (123.0)1 (35.2)16 (131.9)

TPOT(ms) with context length and batch size

ScenarioLead 0, no offloadLead 0, offloadLead 1/2/3, no offloadLead 1/2/3, offload
128K, B133.9040.927.9828.7
128K, B8—78.3—69.1
128K, B16—128.1—119.4
256K, B134.441.428.429.1
256K, B16—130.1—121.3

As batch size increases, offload overhead increases, reducing the benefit of offloading for lead 1/2/3. But the overhead is limited by the plan_plan and copies_copies kernel implementation and can be alleviated with TP=4. Our offload operators are not yet optimized to their theoretical limit.

Operator shares of the offload latency (%) are:

Configurationplanwrite_currentattentioncopiesslotsattn_launchbookplan_waitscattergatherattn_waitreuse+flush
128K, B140.211.69.78.56.96.44.74.73.41.22.10.4
256K, B143.314.87.46.55.64.92.59.23.00.81.60.3
64K, B1637.04.05.919.93.93.73.90.23.715.21.61.0

The overhead is concentrated in CPU-side planning: plan accounts for 40% of total latency at 128K and 43% at 256K. Offload does not add main-path attention latency: attention wait is constant at 0.19 ms/token (0.5% of offload TPOT). We referenced the offload paths in HiSparse and the vLLM KV connector. Our operator overhead still includes process-side planning and copy-submission costs ( _plan and _copies ), so it has not reached the theoretical optimum.

Operator overhead does not vary much with context length. Each round replaces an average of 64–256 tokens per group cross different context. _plan and _copies opration increase with batch size because their work is per request and layer, while PCIe use remains far below its bandwidth limit.

We chose Qwen3.5 specifically because it is a hybrid attention architecture, which matches the structure of most current open-source models. Because there are fewer full layers, the end-to-end benefit is actually much less than that of the full-attention model (e.g., Qwen3).

Weakness 3 Mixed-length session:

Mixed-length sessions and concurrent transfers. Mixed-length scheduling is handled by vLLM's engine layer (admission, queueing, and batching). Our residency policy is orthogonal to it. The policy affects only two things per sequence: how much HBM its pool occupies, and when missing entries are fetched.

In a mixed-length batch, short sequences stay fully resident with KV holding on HBM, as offloading would add transfer cost without freeing meaningful capacity. Long sequences are offloaded: the freed HBM admits additional sequences. For example, offloading one 256K sequence releases enough capacity to admit six 32k sequences at full residency. This is a joint trade-off among HBM capacity, PCIe transfer volume, and the number of resident sequences.

When several sequences miss in the same step, each pool batches its missing entries into a single coalesced DMA. The number of transfers per step therefore equals the number of pools with misses, not the total number of missed entries. This keeps the per-step volume small and well within PCIe bandwidth. Attention only waits for the pools it reads in the current layer, so the stall is bounded by one layer's fetch time.

Thank you for reviewing our paper and for your thoughtful feedback. With just 1 day left in the discussion period, please review our author response and let us know if any specific concerns remain. This will give us time to address them before the discussion closes. If our response has addressed your concerns, we will be grateful if you can raise the scores accordingly.

Dear Reviewer KhZV,

As the discussion period concludes in less than 12 hours, we would like to follow up on our rebuttal. We invite you to review our response and share any new thoughts or questions. If our rebuttal has addressed your concerns, we hope you might consider updating your score.

Best regards, The authors of Paper 61004

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.