acceptodds
Withdrawn from ICLR 2027

SR: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fast rollout generation for long-context reinforcement learning with verifiable rewards (RLVR) requires reducing KV-cache traffic without changing the response groups used for policy optimization. In our profiled RLVR workloads, attention accounts for 72–76% of autoregressive decoding kernel time. Sparse attention reduces these reads, but sampling directly from a sparse policy changes the response groups used for learning. For group-relative optimization, we show that even exact response-wise reweighting does not generally recover the expected dense-policy update because the advantage of each response depends on the rewards of other group members. We propose Self-Sparse Speculative Rollouts (\modelname). It uses sparse attention to propose tokens, then generates responses through full-attention verification and exact speculative sampling. Drafting and verification share the same policy parameter, eliminating the need to train or update a separate drafter. The resulting response groups follow the dense-policy distribution. Under \modelname, we characterize speedup conditions and the costs of drafting and verification. We found that dense verification still requires the complete historical KV cache. A KV-aware scheduler therefore limits KV-cache repeated offloading and restoration while maintaining rollout parallelism. % OR thrashing In a matched ablation with an aggressive sparse drafting budget, \modelname maintains stable training while direct sparse sampling with training-side correction becomes unstable. On long-context mathematical reasoning, \modelname achieves the end-to-end training throughput of autoregressive rollouts on Qwen3 model, with comparable task performance. Our code is available at https://anonymous.4open.science/r/S3R-7666

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM reinforcement learning, efficient post-training, speculative decoding, sparse attention, KV-cache management

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZLpjOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Self-Sparse Speculative Rollouts (S³R) to accelerate long-context reinforcement learning with verifiable rewards. The current policy serves as both drafter and verifier: dynamic sparse attention produces draft tokens, while full-attention verification and exact speculative sampling preserve the dense policy’s response distribution. The paper further argues that response-wise importance weighting does not generally recover the dense-policy GRPO update because group-relative advantages depend on other responses in the group. A KV-aware scheduler supports execution under memory pressure. Experiments on Qwen3-1.7B and Qwen3-8B report approximately 1.35–1.39× end-to-end training throughput over autoregressive rollouts, with comparable task performance.

Critical strengths
  1. The paper addresses an important practical bottleneck and reports end-to-end benefits. Accelerating RL rollout generation is valuable, and the evaluation includes full training-step throughput rather than only isolated decoding kernels.

  2. The group-level distribution analysis is useful. The paper explains why correcting individual responses need not recover the expected dense-policy update when advantages depend on group composition. This provides a principled motivation for preserving the response-group distribution before reward and advantage computation.

Critical weaknesses

1. The central efficiency premise depends on full-attention targets, limiting the demonstrated applicability and incremental novelty.

As formulated and evaluated, S³R relies on an expensive full-attention target whose historical KV reads can be reduced during drafting. This is a meaningful setting, but its relevance to increasingly efficient sparse and hybrid-attention targets remains unclear. For example, DeepSeek-V4 employs compressed sparse and heavily compressed attention rather than full-context dense attention throughout [1]. Here, attention sparsity should not be confused with MoE parameter sparsity.

When the target already uses sparse attention, simply using sparse attention for drafting does not establish an additional computational advantage. The authors need to identify a substantially cheaper proposal computation and demonstrate that the remaining draft–verification cost gap is sufficient to offset speculative overhead. The current experiments do not establish such an advantage for sparse or hybrid-attention targets. The concern is not that acceleration is theoretically impossible in these settings, but that the paper’s central efficiency argument has only been demonstrated for full-attention policies.

Moreover, speculative decoding for RL rollouts is already an explored approach, including self-speculative methods such as EfficientRollout [2]. Therefore, applying speculative verification to RL is not itself sufficient novelty. The paper should more clearly distinguish its contribution in group-level analysis and runtime integration from established speculative-decoding techniques, and either demonstrate benefits on sparse/hybrid targets or explicitly narrow its claims.

2. The evaluation omits a critical SWA drafting baseline and does not establish the hardware–memory–acceptance trade-off.

The paper does not compare its dynamic Top-K sparse drafter against sliding-window attention (SWA), despite SWA drafting being supported by industrial model designs. In particular, the MiMo-V2-Flash Technical Report explicitly describes lightweight MTP draft blocks using 128-token SWA and reports practical speculative-decoding speedups [3]. SWA is therefore a credible, practically validated alternative, not merely a hypothetical baseline.

SWA has two important systems advantages that the evaluation needs to address.

First, SWA exposes regular, contiguous attention computation and KV access. Compared with dynamic Top-K selection and irregular KV gathering, this can reduce selection overhead and enable more efficient kernels. The authors should measure achieved hardware throughput, memory bandwidth, and MFU where informative, rather than infer execution efficiency from attended-token counts or acceptance rates alone.

Second, SWA offers a concrete KV-storage advantage, not merely a smaller attention read set. SGLang implements separate SWA KV pools and sliding-window eviction, allowing out-of-window physical KV slots to be reclaimed [4]. An SWA-only drafter therefore need not retain its entire historical KV: its required active cache can be bounded by the window size, subject to paging, prefill, and speculative-buffer overhead. This contrasts with S³R’s dynamic historical-token selection over a retained full-context cache. A valid baseline must use actual KV reclamation, not full-history storage with a sliding-window mask.

A full-attention verifier still requires its historical KV, but this does not negate the storage advantage of a separate SWA draft cache. The evaluation must explicitly distinguish shared and separate allocations rather than either dismissing draft-side savings or incorrectly attributing them to the entire system.

At minimum, the authors should compare dynamic Top-K drafting against recent-window and sink-plus-window drafting under the same target, hardware, exact sampler, and execution backend, sweeping window sizes and sparse budgets across context lengths and rollout batch sizes. A same-weight windowed drafter would isolate the attention-selection choice; comparisons with trained SWA/MTP drafters should additionally account for adaptation and synchronization costs.

The comparison should report drafting, selection/gather, verification, KV transfer, physical KV occupancy, supported rollout concurrency, acceptance length, and end-to-end RL throughput. The central question is whether the additional acceptance obtained from dynamic historical-token selection outweighs SWA’s regular execution and bounded draft-cache storage. Without this trade-off study, the paper does not establish that its sparse drafting strategy is preferable to a simpler, industrially validated SWA alternative.

References

[1] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. Section 2.3. https://arxiv.org/abs/2606.19348

[2] EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts. https://arxiv.org/abs/2606.18967

[3] LLM-Core Xiaomi. MiMo-V2-Flash Technical Report. Sections 2.3 and 5. https://arxiv.org/abs/2601.02780

[4] SGLang: SWA KV pools, allocation, and sliding-window eviction. https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/swa_memory_pool.py https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/allocator/swa.py https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/managers/schedule_batch.py

Questions for authors
  1. Token-level vs. response-level importance ratios. Equation (1) already decomposes the learner-to-sparse-behavior ratio into a policy-update term and a sparsity correction, while Appendix A.1 defines WiW_i as the product of the token-level sparsity corrections. Since a complete-response ratio already captures the relative likelihood of different responses, is WiW_i an alternative sequence-level correction or an additional factor for the same mismatch? Please give the exact loss for the corrected-sparse baseline and the idealized response-wise correction, stating which ratios are multiplied, which are fixed weights, and where clipping applies, so that the same mismatch is not counted twice.

  2. Connection between the analysis and the final algorithm. The analysis shows a limitation of response-wise correction, but the final method sidesteps it with standard exact speculative sampling. With a fixed rollout snapshot and independent responses, group-level preservation follows directly from per-response preservation. Which design choice is actually derived from the analysis rather than inherited from standard speculative decoding? Would the rollout algorithm be identical without the importance-weighting derivation, and what does Proposition 2 establish beyond the standard exact-sampling guarantee?

Reviewer_TStcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes S³R, a framework for accelerating RL rollouts using sparse self-speculative decoding. The method uses sparse attention for drafting and full attention for verification to preserve the original policy distribution. It also introduces a KV-aware scheduler to reduce cache offloading overhead. Experiments on Qwen3-1.7B and Qwen3-8B show 1.35–1.39x end-to-end training throughput improvement over autoregressive decoding.

Critical strengths

Improving RL rollout efficiency while preserving the original policy distribution is an important systems problem. The end-to-end training speedups are meaningful.

Critical weaknesses

Limited technical novelty and insufficient baseline comparisons. The core idea of sparse self-speculative decoding has already been studied in SparseSpec [1]. In fact, Appendix B explicitly mentions that the implementation builds on SparseSpec's open-source backend. The main contribution seems to be integrating SparseSpec into the RL rollout pipeline with an AIMD scheduler, which itself is a commonly used scheduling technique. Table 2 shows that AIMD only provides an additional 5.8% throughput improvement. Without a direct comparison against SparseSpec, it's unclear what the actual systems contribution is beyond integrating existing techniques.

Limited applicability to emerging model architectures. Figure 1 heavily relies on full attention being the main decoding bottleneck. However, newer models such as Qwen3-Next, Qwen3.5, and Kimi Linear increasingly use hybrid architectures with only a small fraction of full-attention layers. It's unclear whether the proposed method can still provide meaningful speedups when attention KV-cache reads are no longer the dominant bottleneck. The evaluation only covers full-attention Qwen3 models.

[1] SparseSpec: Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding, MLSys 2026.

Questions for authors
  1. Can the authors compare against SparseSpec under the same RL training setup to isolate the actual contributions beyond the existing sparse self-speculative decoding backend?

  2. Can the authors discuss or evaluate how S³R performs on newer hybrid architectures, where most layers no longer use full attention?

Reviewer_VzCUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Rollout generation is a primary bottleneck in long-context RLVR. The paper makes three claims: sparse attention changes the sampling distribution; reweighting each response cannot recover the dense GRPO update, because its advantage depends on the rest of the group; and verifying sparse drafts with full attention under exact speculative sampling preserves the dense response-group distribution. S³R implements this with a KV-aware scheduler. On Qwen3-1.7B and 8B it reports 1.35–1.39× end-to-end training throughput at comparable benchmark scores.

Critical strengths
  1. The problem is stated clearly, and the main claim is proved. Proposition 1 shows that reweighting each response cannot recover the dense update, and Appendix A.2 gives a three-response counterexample.
  2. The speedups are measured end to end. Training step time falls from 13.9 to 9.9 min on Qwen3-1.7B and from 50.2 to 37.4 min on 8B, on eight H100s, and the code is released.
  3. The applicability condition is stated, and it gets easier as contexts grow. Table 7 puts the break-even acceptance above 100% at 64–128K live KV, at 75.8–77.0% at 256K and at 45.5–46.7% at 512K, so the regime where it pays is the one RLVR is moving toward.
Critical weaknesses
  1. The draft length k is never varied. Eq. 5 makes the speedup a function of k and of the acceptance rate: a longer block amortises verification but wastes more on rejection, a shorter one wastes less but pays more verification per token. The sparse budget is swept over 3–50% on both models (Fig. 12). No sensitivity for k is reported.
  2. The paper is inconsistent about EAGLE3. The main text compares against a frozen one and blames staleness (§6.2); Appendix D.2 retrains it every 15 RL steps, and acceptance barely moves, 2.075 to 2.124, at 8K response length.
Questions for authors
  1. Add an analysis of k: vary the draft length at a fixed sparse budget and report how acceptance and end-to-end speedup each change.
  2. Clarify the frozen and updated EAGLE3. The main text explains the gap by staleness, which Appendix D.2 does not support; the updated version looks like the better baseline for the main comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.