acceptodds
Under review as a conference paper at ICLR 2027

KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

Abstract

Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards by mitigating long-tail generation. However, sustaining higher concurrency compresses the same context-processing work into fewer decode steps, increasing the per-step cost of KV-cache processing. We characterize this KV concentration and show that KV-cache quantization provides substantially greater acceleration than weight-only or weight-activation quantization under the resulting workload. We further find that INT4 KV quantization can degrade learning despite importance correction. To address this, we propose KV4-RL, which combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. This bounded protection reduces sampler mismatch while retaining efficient execution over the compressed cache. In Qwen3-8B-Base GRPO training, KV4-RL achieves rollout and full-iteration speedup over BF16 across training, with rollout speedup reaching as responses lengthen. Under BF16 evaluation, final-policy mean accuracy across six math benchmarks remains within 0.5 percentage points of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
reinforcement learning with verifiable rewards, asynchronous rollout, KV cache quantization, efficient LLM training, reasoning models, mixed-precision inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mC3gOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

KV4-RL identifies a bottleneck created by asynchronous RL rollout: because generation stops at a completion quota and unfinished prefixes carry across updates, the same KV work is packed into fewer decode steps ("KV concentration"), making KV cache traffic dominate decode cost. Asynchronous rollout showed larger active batch and resident KV volume than synchronous, so KV quantization may beat weight/activation quantization under this setting. This work characterizes KV concentration as a structural effect of asynchronous scheduling; the first study of sub-8-bit KV quantization for RLVR rollout; diagnosing uniform-INT4 learning instability and proposing selective INT8 protection; and validating the method at two model scales.

Critical strengths

S1. The paper identifies a valid and timely systems problem. Asynchronous and partial rollouts with prefix carry shift decoding costs toward KV cache traffic, making KV quantization a practically motivated target. Connecting KV concentration to the structure of asynchronous rollout provides a novel framing grounded in the challenges of real RLVR systems.

S2. The evaluation demonstrates end-to-end gains beyond improvements in sampler KL. Across full-parameter GRPO training on Qwen base models and multiple math benchmarks, KV4-RL closely matches BF16 accuracy while substantially accelerating decoding and cumulative rollout. The ablations show that selective protection reduces sampler KL while preserving most of the throughput of uniform low-bit quantization, supporting its role as a targeted correction rather than a regularizer. The controlled diagnosis of learning degradation under uniform quantization, despite TIS, followed by a focused intervention that recovers performance, makes the empirical case convincing and practically relevant.

Critical weaknesses

W1. The technical contribution seems weak because the quantization method looks like a straight-forward migration of existing mixed-precision and Hadamard-rotation-based KV quantization to RL rollout problem, not a significant algorithmic breakthrough. Higher bit for attention sink and recent tokens, and Hadamard rotation for keys are common techniques for KV quantization.

W2. Evaluations are performed on Qwen3-4B/8B. Can this algorithm work for larger models? Can this algorithm scale up, especially under different setting of DP, PP, TP on larger clusters? How does this algorithm work along with existing KV management systems, when CPU KV offloading and KV propagation across nodes is taken into account? The authors are recommended to discuss these in detail to improve the practicality of this algorithm.

Questions for authors
  1. Please validate the key novelty of your quantization method. Is it merely a migration from existing methods (higher bit for important part, Hadamard rotation) to a new problem.

  2. Please analyze or validate if your method applies to larger models beyond 4B/8B.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mC3g (Part 1/3)

We thank the reviewer for the thoughtful feedback and for recognizing KV concentration as a novel workload-level perspective and the practical significance of our results. We have expanded our experiments and analyses to address the concerns regarding technical novelty, generalization, and scalability, as detailed below.


W1 & Q1. Clarifying the Technical Contributions and Novelty

We appreciate the reviewer's concern regarding the novelty of the quantization method. We agree that individual techniques such as Hadamard rotation and higher-precision protection of attention sinks and recent tokens have been explored in prior KV quantization research. Our work builds on these techniques to address the distinct execution and learning challenges of KV quantization for asynchronous RLVR, with contributions spanning workload characterization, learning-fidelity analysis, and efficient quantization design.

Our analysis of KV concentration identifies a workload characteristic that motivates a different optimization target from existing quantized-rollout approaches, which have primarily focused on weights and activations. By sustaining high generation concurrency, asynchronous rollout increasingly amortizes model-weight traffic while concentrating KV processing into fewer, more heavily populated decode steps. Our workload comparisons show that KV quantization provides substantially greater acceleration than the evaluated weight-only and weight–activation alternatives. This provides a workload-driven rationale for targeting KV cache quantization in asynchronous RLVR.

However, directly applying INT4 KV quantization does not necessarily preserve RLVR learning performance. Although it substantially accelerates rollout, our unprotected KV4 baseline exhibits learning degradation despite importance correction (TIS). This highlights a learning-fidelity requirement that conventional inference-oriented quantization evaluations do not fully capture. Meanwhile, improving fidelity uniformly across the cache, for example through smaller quantization groups or more elaborate reconstruction and error-compensation operations, can introduce additional overhead throughout the concentrated KV workload, potentially compromising the acceleration benefits.

KV4-RL addresses these competing requirements through bounded INT8 protection of attention-sink and recent positions, guided by sampler-fidelity and execution-efficiency measurements. Our ablations show that the selected allocation reduces sampler KL by 7.0× relative to unprotected KV4 while retaining approximately 94% of its decode throughput. The resulting method recovers learning performance close to BF16, with additional multi-model and three-seed experiments providing further evidence of robustness (General Response G1).

Therefore, our findings connect the execution characteristics of asynchronous rollout with its learning-specific fidelity requirements and demonstrate how these considerations motivate an efficient KV quantization design. We have revised the contribution statement in the Introduction to better emphasize the learning-fidelity challenges of aggressive KV quantization in RLVR and the efficiency–fidelity considerations underlying KV4-RL's design. We hope this clarification helps convey the technical contributions and motivation of our work beyond the individual quantization components.

AuthorsOpenReview ↗

Response to Reviewer mC3g (Part 2/3)

W2 & Q2. Scalability to Larger Models and Distributed Systems

Larger models and architectures. As detailed in General Response G1, we extend our evaluation to Ministral-3-14B (dense, a different model family) and Qwen3-30B-A3B (MoE), with three independent training seeds per configuration. KV4-RL achieves 1.82× and 1.77× late-training rollout throughput, respectively, while preserving final mean accuracy within 0.8 percentage points of BF16. These results extend our findings beyond the original Qwen3-4B/8B dense models.

Applicability to distributed rollout systems. KV4-RL operates on the local KV representation and attention execution of each GPU, without changing how requests, attention heads, or model layers are partitioned across devices. Its compression mechanism therefore naturally extends to DP, TP, and PP, without introducing additional cross-device synchronization for quantization itself. Importantly, the two mechanisms that accelerate single-node rollouts, namely reduced KV memory traffic and increased effective cache capacity, remain applicable under distributed execution.

To demonstrate this locality of acceleration, we replay the same 1,024 late-training responses across DP/TP configurations on four H200 GPUs, keeping the generated response lengths identical:

ModelParallel LayoutBF16 (s)KV4-RL (s)
Qwen3-8BDP4 × TP1361.2197.0 (1.83×)
DP2 × TP2379.2208.7 (1.82×)
DP1 × TP4375.8217.0 (1.73×)
Qwen3-30B-A3BDP2 × TP2405.9287.9 (1.41×)
DP1 × TP4357.2257.5 (1.39×)

Rollout t90t_{90} in seconds, with speedup relative to BF16 under the same parallel layout.

KV4-RL maintains 1.73–1.83× speedup on Qwen3-8B and 1.39–1.41× on Qwen3-30B-A3B across the evaluated configurations. BF16 and KV4-RL execute the same number of decode steps within each layout, indicating that these gains arise from more efficient local KV processing rather than increased resident concurrency. These results support the locality of KV4-RL's acceleration: its benefits are largely preserved across different parallel configurations because KV compression operates independently on each GPU's local KV cache. The slightly smaller gain at TP4 for 8B is consistent with increased communication accounting for a larger fraction of execution time, thereby diluting the relative benefit of reduced KV memory traffic.

Extension to larger multi-node deployments. Large-scale rollout systems commonly distribute independent model replicas across nodes using DP. When models exceed single-node capacity, a natural alternative is to combine intra-node TP with inter-node PP. In either configuration, KV caches remain associated with the GPUs responsible for their corresponding requests, attention heads, and layers. Since KV4-RL accelerates the processing of these local KV caches, the same KV-traffic and capacity benefits can be realized independently within each replica or pipeline stage, without introducing additional communication or synchronization for quantization. We therefore expect its acceleration mechanism to extend naturally to multi-node deployments, although the overall speedup will also reflect the communication and scheduling costs of the underlying distributed execution.

Compatibility with KV cache management. A key advantage of KV4-RL is that it preserves the existing KV cache management structure while introducing only a bounded INT8 side cache. The full context remains available in INT4 through the original cache-management mechanism, while the protected sink and recent positions are maintained separately in INT8. Since our attention kernel processes these two regions through independent computation paths, the protected cache can be managed independently of the main cache.

This separation makes KV4-RL straightforward to adapt to different cache-management schemes. For paged KV caches, the main cache can retain its existing paging structure in INT4, while the bounded INT8 side cache can use a small flat buffer or a separate page pool. Similarly, prefix-sharing systems such as radix-based KV caches can retain their existing prefix-management structure for the main INT4 cache, with the protected INT8 entries managed separately for each active sequence as needed. Thus, KV4-RL does not require replacing the underlying cache allocation, paging, or prefix-sharing mechanisms.

AuthorsOpenReview ↗

Response to Reviewer mC3g (Part 3/3)

The same principle extends to CPU KV offloading and cross-node KV propagation. The main KV cache can be offloaded or transferred directly in its compressed INT4 representation using the existing management mechanism, while the bounded INT8 side cache can be handled through a separate transfer or offloading path. This preserves the benefits of KV compression during cache movement, potentially reducing both storage requirements and transfer volume. Since the side cache has a fixed capacity per sequence, its additional management and transfer costs remain bounded independently of context length.

Overall, the simplicity and locality of KV4-RL's design make it straightforward to extend across different parallelization strategies and KV cache management systems. By preserving existing cache-management mechanisms and introducing only a bounded, independently managed INT8 side cache, KV4-RL can retain its local acceleration benefits across diverse deployment settings with minimal additional integration overhead. We believe this flexibility is an important practical advantage for scaling KV4-RL to larger RLVR systems.

Scope and revision. While our available computational resources did not permit direct evaluation of production-scale models or multi-node deployments, we have sought to address these concerns through expanded model evaluations, DP/TP comparisons, and an analysis of compatibility with distributed execution and KV management systems. Together, these results and analyses provide encouraging evidence for KV4-RL's applicability to larger-scale RLVR systems. We have incorporated the DP/TP measurements into Appendix E.8 (Table A10) and expanded the discussion of distributed deployment and KV management in the same section, with additional scaling considerations discussed in Section 6.


We hope these additional experiments and clarifications address your concerns regarding the technical contributions and practical scalability of KV4-RL. If they sufficiently resolve your concerns, we would greatly appreciate your reconsideration of the rating. We would also welcome any remaining questions or concerns that we could address further. Thank you again for your thoughtful and constructive feedback.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer mC3g,

As the discussion period closes on November 18 (AoE), we wanted to kindly follow up on our response regarding the technical contributions and scalability of KV4-RL, including the additional results on larger models and distributed settings.

We would be glad to know whether our responses have addressed your concerns. If so, we would greatly appreciate it if you could consider reflecting this in your final assessment. We are also happy to clarify anything further before the deadline.

Best regards,

The Authors

Reviewer_3r9MOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes KV-cache overhead in RLVR and proposes KV4-RL, a mixed INT4/INT8 KV-cache quantization method that retains higher precision for initial and recent tokens. Experiments on Qwen3-4B and Qwen3-8B demonstrate promising acceleration while maintaining final model accuracy close to the BF16-rollout baseline. The paper provides detailed analysis, implementation descriptions, and code to support reproduction.

Critical strengths
  1. The method achieves a useful balance between rollout speed and final model accuracy. On Qwen3-8B, KV4-RL improves final average benchmark accuracy from 47.8% with uniform INT4 to 52.8%, comparable to the BF16 baseline’s 52.4%.
  2. The authors demonstrate actual training acceleration. The custom attention implementation achieves 1.80× rollout and 1.27× full-iteration speedup on Qwen3-8B. Detailed implementation descriptions and released code support reproducibility.
Critical weaknesses
  1. Validation is limited to an in-house execution stack and a narrow model selection. The method is not evaluated in an established RL framework such as slime or verl with a mature inference engine. Although the shared stack supports a meaningful internal comparison, it remains unclear how much the reported benefit depends on the authors’ kernel implementations and inference-engine efficiency. Experiments also cover only two similar dense models, Qwen3-4B and Qwen3-8B, leaving generalization to other model families and MoE architectures unverified.
  2. The source of accuracy recovery is not fully isolated. For initial and recent tokens, KV4-RL increases both KV storage precision (INT4 to INT8) and PV computation precision (INT8 to BF16). Consequently, the improvement cannot be attributed solely to higher-precision KV storage. An ablation separating these changes is needed to clarify their respective contributions.
Questions for authors
  1. Is there a matched comparison against an established RL/inference stack that would help assess whether the reported speedups extend beyond the in-house implementation?
  2. Are results available for another model family or a MoE model to support the generality of the method?
  3. Can the authors separately vary KV storage precision and PV computation precision to identify their contributions to accuracy recovery?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 3r9M (Part 1/2)

We thank the reviewer for the thoughtful feedback and for recognizing KV4-RL's favorable speed–accuracy trade-off and practical training acceleration. We have conducted additional experiments to address the concerns regarding framework dependence, model generalization, and the contributions of individual precision choices.


W1 & Q1. Validation Against an Established RL/Inference Framework

To verify that KV4-RL's acceleration is not specific to our in-house framework, we implement KV4-RL using verl with vLLM as its rollout engine and perform matched-workload comparisons against our in-house framework.

Our in-house framework incorporates the key optimizations of modern inference engines, including paged attention, continuous batching, chunked prefill, CUDA graphs, and attention kernels from FlashAttention-3 and FlashInfer. Its learner uses FSDP2 following verl's training backend. For independent validation, we implement KV4-RL in verl 0.9.0 + vLLM 0.28.0. Importantly, BF16 rollout uses vLLM's native attention backend, while KV8, KV4, and KV4-RL use our quantized attention kernels registered through vLLM's plugin interface. All configurations retain vLLM's scheduler and CUDA-graph execution. This allows us to directly compare KV4-RL against an optimized native BF16 baseline within an established inference framework.

We compare both stacks on 4×H200 GPUs, replaying identical response lengths from BF16 training runs of Qwen3-8B and Qwen3-30B-A3B at different training stages. Rollout time is measured by t90t_{90}, the time until requests accounting for 90% of generated tokens have completed.

Baseline efficiency. The following table compares BF16 rollout and learner update times (seconds).

Component / Framework8B Start8B Middle8B End30B Start30B End
Rollout, verl + vLLM38.3286.7372.250.1396.2
Rollout, ours37.4281.1362.751.4390.3
Update, verl36.8143.1197.143.2226.4
Update, ours29.7137.7183.332.7201.6

Our BF16 rollout performance stays within 3% of vLLM using its native BF16 attention backend across all evaluated workloads, demonstrating that our in-house rollout engine achieves comparable inference efficiency. Our learner update times are also 4–24% shorter than those of verl. Regarding the training backend, both implementations use FSDP2 for Qwen3-8B. For Qwen3-30B-A3B, however, our implementation uses FSDP2, whereas verl uses its default Megatron training backend for MoE models. Thus, the 8B learner comparison uses the same training backend, while the 30B comparison involves different training backends.

Reproducing KV4-RL acceleration in verl + vLLM. We further measure rollout speedup relative to the BF16 baseline of each framework.

KV Cache / Framework8B Start8B Middle8B End30B Start30B End
KV8, verl + vLLM1.13×1.47×1.55×1.06×1.37×
KV8, ours1.09×1.43×1.47×1.05×1.29×
KV4, verl + vLLM1.21×1.85×2.00×1.08×1.57×
KV4, ours1.16×1.80×1.89×1.05×1.45×
KV4-RL, verl + vLLM1.13×1.77×1.93×1.03×1.52×
KV4-RL, ours1.09×1.72×1.83×1.02×1.40×

KV4-RL achieves 1.93× and 1.52× late-training rollout speedup over vLLM's native BF16 baseline on 8B and 30B-A3B, respectively, compared with 1.83× and 1.40× in our stack. All quantized formats achieve comparable or greater acceleration in vLLM, with gains increasing as generation contexts grow.

Together, these results establish that (1) our in-house BF16 baseline achieves efficiency comparable to vLLM's native implementation, and (2) KV4-RL delivers substantial acceleration over vLLM's optimized native BF16 baseline under its own scheduling and execution infrastructure. This directly addresses the concern that our reported speedups may depend on inefficiencies specific to our custom execution stack. The full comparison is included in Appendix B.4 (Table A1) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 3r9M (Part 2/2)

W1 & Q2. Generalization Across Model Families and MoE Architectures

As detailed in General Response G1, we extend our evaluation to Qwen3-30B-A3B (MoE) and Ministral-3-14B (a different model family), with three independent training seeds per configuration.

Across all four models, KV4-RL achieves 1.77–1.99× late-training rollout throughput and 1.27–1.42× cumulative overall training speedup, while preserving final mean accuracy within 0.8 percentage points of BF16. On the two newly evaluated models:

  • Qwen3-30B-A3B: 1.77× rollout throughput, 1.27× overall training speedup, and 60.8 ± 0.6% mean accuracy (BF16: 61.5 ± 0.5%).
  • Ministral-3-14B: 1.82× rollout throughput, 1.30× overall training speedup, and 50.5 ± 0.8% mean accuracy (BF16: 50.4 ± 1.6%).

These results demonstrate that KV4-RL's efficiency–accuracy benefits extend beyond the original dense Qwen3 models. The expanded evaluation is incorporated into Section 5 of the revised manuscript.


W2 & Q3. Disentangling KV Storage and PV Computation Precision

We address this concern through a 2×2 factorial ablation of protected KV storage (INT4/INT8) and PV computation (INT8/BF16), presented in General Response G2.

The results show that selective INT8 KV storage is the primary driver of sampler-fidelity recovery, reducing shared-prefix KL by approximately an order of magnitude regardless of PV precision. In contrast, BF16 PV alone provides no comparable improvement. We further find that BF16 and INT8 PV yield nearly identical sampler KL and decode performance, despite the lower arithmetic error of BF16 PV. Additional numerical-error analysis helps explain this observation by showing that PV arithmetic error is substantially smaller than KV storage quantization error.

The full factorial ablation and numerical-error analysis are presented in General Response G2 and Appendix E.3 (Fig. A6 and Table A7) of the revised manuscript.


We hope that the additional validation against verl + vLLM, together with the expanded model evaluations and precision ablations, addresses your concerns and strengthens confidence in KV4-RL's practical effectiveness. If these results resolve your concerns, we would sincerely appreciate your reconsideration of our rating. We would also be happy to clarify any remaining questions or concerns. Thank you again for your thoughtful feedback.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 3r9M,

As the discussion period closes on November 18 (AoE), we wanted to kindly follow up on our response. Following your suggestions, we have added validation in verl + vLLM, results on additional model families including MoE, and a factorial ablation of KV storage and PV precision.

If our responses have addressed your concerns, we would sincerely appreciate it if you could consider updating your score. We are also happy to clarify anything further before the deadline.

Best regards,

The Authors

Reviewer_cxqBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper characterizes how asynchronous RL rollouts concentrate KV-cache processing into fewer, heavier decoding steps. It proposes INT4 KV quantization with selective INT8 protection for attention sinks and recent tokens. Experiments on Qwen3-4B/8B report near-BF16 final accuracy, with 1.80× cumulative rollout and 1.27× full-iteration speedups at 8B.

Critical strengths

The workload analysis connects asynchronous scheduling to a concrete systems bottleneck, supported by theory, execution traces, and matched-workload timing.

The evaluation separates execution speed from changes in generated response length. Common BF16 evaluation and protection ablations provide convincing evidence that selective protection recovers the observed INT4 learning loss.

Critical weaknesses

Each configuration has only one training run; the error bars reflect evaluation sampling. Consequently, the claimed accuracy preservation within 0.5 points is not established across training randomness.

Evidence is limited to two sizes of one model family, one hardware platform, and one partial-rollout recipe. This limits confidence in how broadly the proposed precision allocation and speedups transfer.

Questions for authors

Does the accuracy preservation and recovery over unprotected INT4 hold across independent training seeds?

Protection changes both cache precision and probability–value multiplication precision. How much of the recovery comes from each change?

5 comments
AuthorsOpenReview ↗

Response to Reviewer cxqB (Part 1/3)

We thank the reviewer for the thoughtful feedback and for recognizing our workload analysis, theoretical characterization of KV concentration, and matched-workload evaluation. We have substantially expanded our experiments to address the concerns regarding training robustness and generalization across models, hardware platforms, and rollout recipes, as detailed below.


W1 & Q1. Robustness Across Training Seeds

To assess the robustness of KV4-RL beyond the original single-run evaluation, we repeat all evaluated training configurations with three independent seeds, as detailed in General Response G1. These experiments cover four models, including the newly evaluated Qwen3-30B-A3B (MoE) and Ministral-3-14B, and report the mean and standard deviation of final performance across six mathematical reasoning benchmarks.

Across all four models, KV4-RL preserves final mean accuracy within 0.8 percentage points of BF16, with consistently small run-to-run variability. In contrast, unprotected KV4 exhibits substantial accuracy degradation and seed-dependent instability on several models. For example, on Qwen3-8B, the six-benchmark mean is 54.3 ± 0.2% for BF16, 49.9 ± 3.3% for KV4, and 53.9 ± 0.5% for KV4-RL. The larger variance of KV4 reflects divergent learning outcomes across seeds: some runs achieve competitive performance, while others experience substantial degradation.

These results provide additional evidence that selective INT8 protection not only recovers the mean learning performance of BF16 rollout but also improves the consistency of training outcomes relative to unprotected INT4 KV quantization. The complete three-seed results, including learning trajectories and detailed evaluation metrics, are presented in General Response G1 and Section 5 of the revised manuscript (Fig. 5 and Table 1).

W2. Generalization Across Models and Hardware Platforms

Model scales, families, and architectures. As detailed in General Response G1, we extend our evaluation beyond the original Qwen3-4B/8B models to Qwen3-30B-A3B (MoE) and Ministral-3-14B (a different model family). Across all four models, KV4-RL achieves 1.77–1.99× late-training rollout throughput and 1.27–1.42× cumulative overall training speedup, while preserving BF16-level learning performance. These results demonstrate that the efficiency–accuracy benefits of KV4-RL extend across the evaluated dense and MoE architectures and beyond the original model family.

Hardware characteristics and expected acceleration. KV compression can accelerate decoding through two complementary mechanisms, whose relative importance depends on both hardware characteristics and workload intensity. First, reducing KV cache precision decreases the amount of KV data read at each decoding step. The resulting acceleration depends on how much of the decoding time is spent processing KV data relative to other operations, such as model computation, scheduling, and kernel overheads. On GPUs with higher memory bandwidth, KV reads become faster, potentially reducing the fraction of execution time that can be accelerated through KV compression. Second, reducing KV cache memory consumption allows more concurrent requests to remain resident when memory capacity constrains the rollout batch size. This benefit is particularly relevant on GPUs with limited memory capacity or under workloads with high concurrency and long contexts. Importantly, both mechanisms become more significant as the KV workload increases: longer contexts increase KV traffic per decoding step, while higher concurrency increases the aggregate resident KV footprint.

Measurements across A100, H200, and B200. To examine these effects, we evaluate KV4-RL on three GPU generations with different HBM bandwidth and capacity: A100-SXM4-80GB (2.0 TB/s bandwidth, 80 GB VRAM), H200 (4.8 TB/s, 141 GB), and B200 (8.0 TB/s, 180 GB). We replay identical workloads recorded from late-stage BF16 training on DAPO-Math prompts. Each request generates exactly its recorded number of tokens, ensuring that every GPU and KV configuration processes the same generation workload. We use TP=1 for Qwen3-8B and TP=2 for Qwen3-30B-A3B, with (G=8) responses per prompt and a 16K response cap.

We report rollout t90t_{90}, the time until requests accounting for 90% of generated tokens have completed, along with speedups relative to BF16 on the corresponding hardware.

KV CacheA100 8BH200 8BB200 8BA100 30BH200 30BB200 30B
BF16931.0359.3261.01118.3406.3297.3
KV8591.1 (1.57×)243.9 (1.47×)176.2 (1.48×)725.2 (1.54×)316.4 (1.28×)234.5 (1.27×)
KV4450.6 (2.07×)188.4 (1.91×)146.0 (1.79×)599.8 (1.86×)281.7 (1.44×)220.2 (1.35×)
KV4-RL461.5 (2.02×)195.8 (1.84×)153.0 (1.71×)612.2 (1.83×)291.2 (1.40×)230.1 (1.29×)
AuthorsOpenReview ↗

Response to Reviewer cxqB (Part 2/3)

KV4-RL accelerates rollout across all three GPU generations, with the observed differences consistent with the two mechanisms described above. On A100, both reduced KV traffic and increased cache capacity contribute to acceleration. Under the evaluated 8B workload, BF16 cannot keep all requests resident and processes an average of 46.9 active requests per decode step, compared with 89.0 for KV4-RL. KV compression therefore improves effective concurrency in addition to reducing memory traffic. A100's lower memory bandwidth also makes KV processing relatively expensive, further increasing the benefit of compression.

On H200 and B200, the default BF16 workloads fit within the available memory, so acceleration primarily comes from reducing KV-processing costs rather than increasing resident concurrency. The smaller relative speedup on B200 is consistent with its substantially higher bandwidth: compared with H200, B200 provides 1.67× greater memory bandwidth, but completes the BF16 rollout only 1.37–1.38× faster. This suggests that execution components other than KV memory traffic account for a larger fraction of rollout time on B200, leaving a smaller fraction to be accelerated by KV compression.

Scaling to heavier workloads. The two mechanisms also suggest that KV4-RL's relative benefit should increase when generation becomes more KV-intensive, even on GPUs with high memory bandwidth and capacity. To examine this, we construct heavier workloads by doubling the number of responses per prompt ((G=16)), doubling recorded response lengths up to a 32K cap, or applying both changes. These experiments evaluate rollout execution only, without additional RL training.

Workload / KV CacheA100 8BH200 8BB200 8BA100 30BH200 30BB200 30B
G=16, 16K cap
BF161834.6669.0441.92191.2646.3452.2
KV4-RL777.4 (2.36×)328.2 (2.04×)242.8 (1.82×)867.4 (2.53×)407.8 (1.58×)320.2 (1.41×)
G=8, 32K cap
BF163489.41231.6837.33841.41152.3843.1
KV4-RL1281.8 (2.72×)560.3 (2.20×)432.6 (1.94×)1512.8 (2.54×)709.5 (1.62×)552.2 (1.53×)
G=16, 32K cap
BF166824.22563.71745.97495.02438.71751.8
KV4-RL2622.5 (2.60×)974.4 (2.63×)738.3 (2.36×)3035.1 (2.47×)1108.4 (2.20×)852.2 (2.06×)

The results support this expectation. KV4-RL's speedup increases under heavier workloads across all three GPU generations. On B200, for example, speedup increases from 1.71× to 2.36× for Qwen3-8B and from 1.29× to 2.06× for Qwen3-30B-A3B when both workload dimensions are doubled.

The results also help distinguish the two sources of acceleration. For Qwen3-30B-A3B on B200, increasing response lengths to the 32K cap raises the speedup from 1.29× to 1.53×, even though the BF16 workload still fits in GPU memory. This is consistent with a larger fraction of decoding time being spent processing KV data as contexts grow. When both concurrency and response lengths increase, the workload exceeds BF16 KV cache capacity even on B200's 180 GB memory, introducing an additional capacity-related benefit.

Together, these experiments show that KV4-RL's acceleration depends on the KV workload relative to the available hardware resources, rather than being specific to the H200 platform. While GPUs with higher bandwidth and memory capacity can exhibit smaller speedups under a fixed workload, increasing context lengths and concurrency can make KV compression increasingly beneficial even on newer hardware.

Revision. We have added the complete cross-hardware comparisons, heavier-workload experiments, and per-step concurrency measurements to Appendix E.7 (Table A9) of the revised manuscript.


W2. Generalization Across Partial-Rollout Recipes

How partial-rollout recipes differ. Partial-rollout methods differ primarily in how they bound response collection and control the staleness of unfinished generations. Token-budget recipes, such as Kimi k1.5 [1], limit the number of tokens generated per response within an iteration and continue longer responses in subsequent iterations. Quota-based recipes, such as APRIL [2], maintain an oversubscribed request pool and stop collection once a prescribed completion quota is reached, carrying unfinished responses forward. Fully asynchronous systems, such as AReaL [3] and ROLL Flash [4], decouple generation from policy optimization, allowing rollout to continue during learner updates while controlling policy staleness. Our implementation follows the quota-based approach: collection stops when (T) prompt groups are completed, and an age limit bounds the staleness of carried groups.

AuthorsOpenReview ↗

Response to Reviewer cxqB (Part 3/3)

Why KV concentration extends across recipes. Despite their different collection and staleness rules, these methods share a common objective: maintaining generation concurrency rather than waiting for a small number of long-tail responses to finish. The mechanism underlying KV concentration therefore does not depend on a particular stopping rule. For a fixed decode workload, sustaining higher concurrency distributes the required KV processing over fewer decoding steps, increasing the amount of KV data processed per step. This is the structural effect characterized by Proposition 1 in Section 3.1.

Different partial-rollout recipes affect the degree of KV concentration primarily through how effectively they sustain concurrency. As illustrated in Fig. 1(a), our quota-based collection still allows concurrency to decline within each collection before the completion quota is reached. Fully asynchronous systems can replenish requests more continuously, including during policy updates, and may therefore approach the continuous-refill regime of Proposition 1 more closely. Although the actual degree of concentration depends on admission control, staleness constraints, and resource allocation, the underlying relationship between sustained concurrency and per-step KV processing remains applicable across these scheduling mechanisms.

Applicability of KV4-RL. KV4-RL operates at the level of KV cache storage and attention computation, independently of how requests are collected or scheduled. Its selective INT8 protection is applied to each sequence's attention-sink and recent positions, while the remaining history uses compressed INT4 KV storage. Consequently, the method can be applied to these different rollout recipes without changing their collection or scheduling rules.

We therefore expect KV4-RL's execution benefits to extend to other partial-rollout recipes that sustain high concurrency and incur substantial KV-processing workloads. The magnitude of acceleration will depend on the resulting workload characteristics, while learning fidelity may additionally depend on the recipe's policy-staleness behavior. Our direct training validation remains limited to the quota-based recipe, and we distinguish this evaluated setting from the broader applicability of the method in Section 6 of the revised manuscript.


Q2. Disentangling KV Storage and PV Computation Precision

We address this concern through a 2×2 factorial ablation of KV storage precision (INT4/INT8) and probability–value (PV) computation precision (INT8/BF16) for the protected attention-sink and recent positions, as detailed in General Response G2.

The results show that selective INT8 KV storage is the primary driver of sampler-fidelity recovery, reducing shared-prefix KL by approximately an order of magnitude regardless of PV precision. In contrast, switching PV computation to BF16 while retaining INT4 storage provides no comparable improvement. We further find that BF16 and INT8 PV yield nearly identical sampler KL and decode performance for the bounded protected region, despite the lower arithmetic error of BF16 PV. Additional numerical-error analysis helps explain this observation: the arithmetic error is substantially smaller than the error introduced by KV storage quantization, limiting the effect of PV precision on the resulting attention output.

The full factorial ablation and numerical-error analysis are presented in General Response G2 and Appendix E.3 (Fig. A6 and Table A7) of the revised manuscript.

References

[1] Team, Kimi, et al. "Kimi k1. 5: Scaling reinforcement learning with llms." arXiv preprint arXiv:2501.12599 (2025).

[2] Zhou, Yuzhen, et al. "April: Active partial rollouts in reinforcement learning to tame long-tail generation." arXiv preprint arXiv:2509.18521 (2025).

[3] Fu, Wei, et al. "Areal: A large-scale asynchronous reinforcement learning system for language reasoning." Advances in Neural Information Processing Systems 38 (2026): 36256-36282.

[4] Lu, Han, et al. "Part II: ROLL Flash--Accelerating RLVR and Agentic Training with Asynchrony." arXiv preprint arXiv:2510.11345 (2025).


We hope these additional experiments and analyses address your concerns regarding training robustness and generalization across models, hardware platforms, and rollout recipes. If these results sufficiently resolve your concerns, we would be grateful if you could reconsider your rating accordingly. If any questions remain or further clarification would be helpful, please let us know so that we can address them. We sincerely appreciate your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.