- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
KV4-RL identifies a bottleneck created by asynchronous RL rollout: because generation stops at a completion quota and unfinished prefixes carry across updates, the same KV work is packed into fewer decode steps ("KV concentration"), making KV cache traffic dominate decode cost. Asynchronous rollout showed larger active batch and resident KV volume than synchronous, so KV quantization may beat weight/activation quantization under this setting. This work characterizes KV concentration as a structural effect of asynchronous scheduling; the first study of sub-8-bit KV quantization for RLVR rollout; diagnosing uniform-INT4 learning instability and proposing selective INT8 protection; and validating the method at two model scales.
Critical strengths
S1. The paper identifies a valid and timely systems problem. Asynchronous and partial rollouts with prefix carry shift decoding costs toward KV cache traffic, making KV quantization a practically motivated target. Connecting KV concentration to the structure of asynchronous rollout provides a novel framing grounded in the challenges of real RLVR systems.
S2. The evaluation demonstrates end-to-end gains beyond improvements in sampler KL. Across full-parameter GRPO training on Qwen base models and multiple math benchmarks, KV4-RL closely matches BF16 accuracy while substantially accelerating decoding and cumulative rollout. The ablations show that selective protection reduces sampler KL while preserving most of the throughput of uniform low-bit quantization, supporting its role as a targeted correction rather than a regularizer. The controlled diagnosis of learning degradation under uniform quantization, despite TIS, followed by a focused intervention that recovers performance, makes the empirical case convincing and practically relevant.
Critical weaknesses
W1. The technical contribution seems weak because the quantization method looks like a straight-forward migration of existing mixed-precision and Hadamard-rotation-based KV quantization to RL rollout problem, not a significant algorithmic breakthrough. Higher bit for attention sink and recent tokens, and Hadamard rotation for keys are common techniques for KV quantization.
W2. Evaluations are performed on Qwen3-4B/8B. Can this algorithm work for larger models? Can this algorithm scale up, especially under different setting of DP, PP, TP on larger clusters? How does this algorithm work along with existing KV management systems, when CPU KV offloading and KV propagation across nodes is taken into account? The authors are recommended to discuss these in detail to improve the practicality of this algorithm.
Questions for authors
-
Please validate the key novelty of your quantization method. Is it merely a migration from existing methods (higher bit for important part, Hadamard rotation) to a new problem.
-
Please analyze or validate if your method applies to larger models beyond 4B/8B.
4 comments
Response to Reviewer mC3g (Part 2/3)
W2 & Q2. Scalability to Larger Models and Distributed Systems
Larger models and architectures. As detailed in General Response G1, we extend our evaluation to Ministral-3-14B (dense, a different model family) and Qwen3-30B-A3B (MoE), with three independent training seeds per configuration. KV4-RL achieves 1.82× and 1.77× late-training rollout throughput, respectively, while preserving final mean accuracy within 0.8 percentage points of BF16. These results extend our findings beyond the original Qwen3-4B/8B dense models.
Applicability to distributed rollout systems. KV4-RL operates on the local KV representation and attention execution of each GPU, without changing how requests, attention heads, or model layers are partitioned across devices. Its compression mechanism therefore naturally extends to DP, TP, and PP, without introducing additional cross-device synchronization for quantization itself. Importantly, the two mechanisms that accelerate single-node rollouts, namely reduced KV memory traffic and increased effective cache capacity, remain applicable under distributed execution.
To demonstrate this locality of acceleration, we replay the same 1,024 late-training responses across DP/TP configurations on four H200 GPUs, keeping the generated response lengths identical:
| Model | Parallel Layout | BF16 (s) | KV4-RL (s) |
|---|---|---|---|
| Qwen3-8B | DP4 × TP1 | 361.2 | 197.0 (1.83×) |
| DP2 × TP2 | 379.2 | 208.7 (1.82×) | |
| DP1 × TP4 | 375.8 | 217.0 (1.73×) | |
| Qwen3-30B-A3B | DP2 × TP2 | 405.9 | 287.9 (1.41×) |
| DP1 × TP4 | 357.2 | 257.5 (1.39×) |
Rollout in seconds, with speedup relative to BF16 under the same parallel layout.
KV4-RL maintains 1.73–1.83× speedup on Qwen3-8B and 1.39–1.41× on Qwen3-30B-A3B across the evaluated configurations. BF16 and KV4-RL execute the same number of decode steps within each layout, indicating that these gains arise from more efficient local KV processing rather than increased resident concurrency. These results support the locality of KV4-RL's acceleration: its benefits are largely preserved across different parallel configurations because KV compression operates independently on each GPU's local KV cache. The slightly smaller gain at TP4 for 8B is consistent with increased communication accounting for a larger fraction of execution time, thereby diluting the relative benefit of reduced KV memory traffic.
Extension to larger multi-node deployments. Large-scale rollout systems commonly distribute independent model replicas across nodes using DP. When models exceed single-node capacity, a natural alternative is to combine intra-node TP with inter-node PP. In either configuration, KV caches remain associated with the GPUs responsible for their corresponding requests, attention heads, and layers. Since KV4-RL accelerates the processing of these local KV caches, the same KV-traffic and capacity benefits can be realized independently within each replica or pipeline stage, without introducing additional communication or synchronization for quantization. We therefore expect its acceleration mechanism to extend naturally to multi-node deployments, although the overall speedup will also reflect the communication and scheduling costs of the underlying distributed execution.
Compatibility with KV cache management. A key advantage of KV4-RL is that it preserves the existing KV cache management structure while introducing only a bounded INT8 side cache. The full context remains available in INT4 through the original cache-management mechanism, while the protected sink and recent positions are maintained separately in INT8. Since our attention kernel processes these two regions through independent computation paths, the protected cache can be managed independently of the main cache.
This separation makes KV4-RL straightforward to adapt to different cache-management schemes. For paged KV caches, the main cache can retain its existing paging structure in INT4, while the bounded INT8 side cache can use a small flat buffer or a separate page pool. Similarly, prefix-sharing systems such as radix-based KV caches can retain their existing prefix-management structure for the main INT4 cache, with the protected INT8 entries managed separately for each active sequence as needed. Thus, KV4-RL does not require replacing the underlying cache allocation, paging, or prefix-sharing mechanisms.
Response to Reviewer mC3g (Part 3/3)
The same principle extends to CPU KV offloading and cross-node KV propagation. The main KV cache can be offloaded or transferred directly in its compressed INT4 representation using the existing management mechanism, while the bounded INT8 side cache can be handled through a separate transfer or offloading path. This preserves the benefits of KV compression during cache movement, potentially reducing both storage requirements and transfer volume. Since the side cache has a fixed capacity per sequence, its additional management and transfer costs remain bounded independently of context length.
Overall, the simplicity and locality of KV4-RL's design make it straightforward to extend across different parallelization strategies and KV cache management systems. By preserving existing cache-management mechanisms and introducing only a bounded, independently managed INT8 side cache, KV4-RL can retain its local acceleration benefits across diverse deployment settings with minimal additional integration overhead. We believe this flexibility is an important practical advantage for scaling KV4-RL to larger RLVR systems.
Scope and revision. While our available computational resources did not permit direct evaluation of production-scale models or multi-node deployments, we have sought to address these concerns through expanded model evaluations, DP/TP comparisons, and an analysis of compatibility with distributed execution and KV management systems. Together, these results and analyses provide encouraging evidence for KV4-RL's applicability to larger-scale RLVR systems. We have incorporated the DP/TP measurements into Appendix E.8 (Table A10) and expanded the discussion of distributed deployment and KV management in the same section, with additional scaling considerations discussed in Section 6.
We hope these additional experiments and clarifications address your concerns regarding the technical contributions and practical scalability of KV4-RL. If they sufficiently resolve your concerns, we would greatly appreciate your reconsideration of the rating. We would also welcome any remaining questions or concerns that we could address further. Thank you again for your thoughtful and constructive feedback.
Gentle Reminder
Dear Reviewer mC3g,
As the discussion period closes on November 18 (AoE), we wanted to kindly follow up on our response regarding the technical contributions and scalability of KV4-RL, including the additional results on larger models and distributed settings.
We would be glad to know whether our responses have addressed your concerns. If so, we would greatly appreciate it if you could consider reflecting this in your final assessment. We are also happy to clarify anything further before the deadline.
Best regards,
The Authors
Response to Reviewer mC3g (Part 1/3)
We thank the reviewer for the thoughtful feedback and for recognizing KV concentration as a novel workload-level perspective and the practical significance of our results. We have expanded our experiments and analyses to address the concerns regarding technical novelty, generalization, and scalability, as detailed below.
W1 & Q1. Clarifying the Technical Contributions and Novelty
We appreciate the reviewer's concern regarding the novelty of the quantization method. We agree that individual techniques such as Hadamard rotation and higher-precision protection of attention sinks and recent tokens have been explored in prior KV quantization research. Our work builds on these techniques to address the distinct execution and learning challenges of KV quantization for asynchronous RLVR, with contributions spanning workload characterization, learning-fidelity analysis, and efficient quantization design.
Our analysis of KV concentration identifies a workload characteristic that motivates a different optimization target from existing quantized-rollout approaches, which have primarily focused on weights and activations. By sustaining high generation concurrency, asynchronous rollout increasingly amortizes model-weight traffic while concentrating KV processing into fewer, more heavily populated decode steps. Our workload comparisons show that KV quantization provides substantially greater acceleration than the evaluated weight-only and weight–activation alternatives. This provides a workload-driven rationale for targeting KV cache quantization in asynchronous RLVR.
However, directly applying INT4 KV quantization does not necessarily preserve RLVR learning performance. Although it substantially accelerates rollout, our unprotected KV4 baseline exhibits learning degradation despite importance correction (TIS). This highlights a learning-fidelity requirement that conventional inference-oriented quantization evaluations do not fully capture. Meanwhile, improving fidelity uniformly across the cache, for example through smaller quantization groups or more elaborate reconstruction and error-compensation operations, can introduce additional overhead throughout the concentrated KV workload, potentially compromising the acceleration benefits.
KV4-RL addresses these competing requirements through bounded INT8 protection of attention-sink and recent positions, guided by sampler-fidelity and execution-efficiency measurements. Our ablations show that the selected allocation reduces sampler KL by 7.0× relative to unprotected KV4 while retaining approximately 94% of its decode throughput. The resulting method recovers learning performance close to BF16, with additional multi-model and three-seed experiments providing further evidence of robustness (General Response G1).
Therefore, our findings connect the execution characteristics of asynchronous rollout with its learning-specific fidelity requirements and demonstrate how these considerations motivate an efficient KV quantization design. We have revised the contribution statement in the Introduction to better emphasize the learning-fidelity challenges of aggressive KV quantization in RLVR and the efficiency–fidelity considerations underlying KV4-RL's design. We hope this clarification helps convey the technical contributions and motivation of our work beyond the individual quantization components.