acceptodds
Under review as a conference paper at ICLR 2027

Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a powerful recipe for improving large language models on reasoning tasks, but its training cost is increasingly dominated by long-context rollout generation, making sparse attention as the rollout worker for training dense policy promising. However, in practice, mastering the tradeoff between training stability and efficiency is difficult. In order to study the optimal tradeoff, we first observe that the majority of the tokens are perfectly aligned with dense under highly aggressive sparsity. We then hypothesize that once the tail distribution is constrained by a threshold, we are able to train sparse rollout stably. We prove that our hypothesis holds by introducing a dynamic sparsity scheduling method for controlling the tail distribution constant through generation, and study how the threshold scales with model sizes. Surprisingly, we find that the threshold of 5-percentile mismatch at 0.86 generally works across model sizes and uses cost model analysis for finding the best scheduling for speedup. Empirically, our method generalizes to Qwen3-14B and enables stable training with sparse rollout, while achieving 2.2x, 2.4x, and 2.0x in rollout when training Qwen3-1.7B, Qwen3-4B, and Qwen3-8 B.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Reinforcement Learning, Sparsity, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bFmFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  1. This paper proposes Sparrow, which dynamically adjusts sparse-attention budgets with generation length to control distribution mismatch between the rollout and dense training policies, enabling stable long-context RL training.
  2. Experiments show that targeting a lower-tail acceptance rate of approximately 0.86 preserves performance close to dense-rollout baselines across multiple model sizes and tasks while substantially accelerating rollouts.
Critical strengths
  1. The paper uses controlled experiments to identify lower-tail acceptance rates as a practical stability indicator, enabling faster sparse rollouts while largely preserving dense-policy training performance.
  2. The paper provides extensive empirical validation across model scales and tasks, with clearly stated takeaways and supporting analyses of training stability and practical speedups.
Critical weaknesses
  1. Although the 0.86 target transfers across the evaluated models and tasks, its sensitivity to importance-sampling clipping and other training settings remains unclear. The coarse threshold sweep also does not establish how reliably it separates stable from unstable training.
Questions for authors
  1. What TIS clipping thresholds are used in the main experiments, and how sensitive is the 0.86 acceptance-rate target to these thresholds?
  2. Table 1 reports best-achieved scores for each task. Could you provide results at a common final checkpoint to assess whether sparse rollouts consistently preserve dense-baseline performance?
1 comment
AuthorsOpenReview ↗

Official Response

Thank you for your attention to Sparrow. We are glad that you find our study concrete, our control studies reasonable, and validation extensive.

TIS threshold used and how strict it is

We thank you for raising this question. We use a TIS threshold of 3.0; this choice follows prior work, Jackpot, published at ICLR 2026.

Chen, Z., Liu, H., Zhou, Y., Zheng, H., & Chen, B. (2026, April). Jackpot: Align Actor-Policy Distribution for scalable and stable RL for LLM. In International Conference on Learning Representations (Vol. 2026, pp. 68647-68664).

For generalization, due to compute limitations, we cannot run the full sweep for tail distribution mismatch for different TIS thresholds, each of which requires at least four multi-node H200 runs. Instead, we show here that for a TIS threshold close to 3.0, maintaining the tail distribution mismatch level around 0.86 can generalize and maintain stable training.

We run the Qwen3-1.7B thinking model on coding, following the recipe in Section 3.3 (p. 8)\color{blue}{\text{Section 3.3 (p. 8)}}. Each run uses 2xH200 nodes with a 24K generation length cutoff and TIS thresholds of 2.5 and 3.5. By the time the response is submitted, the training reaches 6,400 training examples, and the training with 0.86 dynamic sparsity is empirically stable.

For the numbers presented in the table below, the EMA smoothing factor is set to 0.99; the training reward, response length, and HumanEval+ curves over the 6,400 training examples are shown in Appendix O (p. 30)\color{blue}{\text{Appendix O (p. 30)}} of the revised paper, where the curves use EMA 0.5 for visualization.

SettingTraining Reward EMA (6,400 examples)Response Length EMA (6,400 examples)HumanEval+ Mean@1LiveCodeBench Mean@4
TIS 3.5 (new)0.4410109960.55480.3933
TIS 3.0 (paper default)0.4422112520.56090.3823
TIS 2.5 (new)0.4355107110.54870.3970

Through this experiment, we show that the identified 0.86 target generalizes to stable training for TIS thresholds in the vicinity of the value of 3.0 we used. Beyond the TIS threshold, the same 0.86 target also transfers across other training settings in our evaluation, including model sizes from Qwen3-1.7B to Qwen3-14B, tasks (mathematics and coding), and model families (Llama-3.2-1B-Instruct), as reported in Sections 3.2–3.3 (pp. 7–8)\color{blue}{\text{Sections 3.2–3.3 (pp. 7–8)}} and Appendix J (p. 25)\color{blue}{\text{Appendix J (p. 25)}}.

On the coarseness of the threshold sweep

We agree that our control study tests a discrete set of targets (0.75, 0.80, 0.86, and 0.92), so it locates the transition between 0.80 and 0.86 rather than pinpointing an exact boundary. Two pieces of evidence support that this separation is reliable rather than incidental. First, the same split holds independently at all three model sizes: at 1.7B, 4B, and 8B, the 0.80 schedule collapses while the 0.86 schedule trains stably (Figure 3(d–f) (p. 5)\color{blue}{\text{Figure 3(d–f) (p. 5)}}), giving three separate tests of the same boundary. Second, Appendix H (p. 23)\color{blue}{\text{Appendix H (p. 23)}} repeats Qwen3-1.7B coding RL with DSparse0.86 across three random seeds, and all runs remain stable, indicating that stability at 0.86 is not the result of a favorable seed. Consistent with this resolution, we have revised Takeaway 1 in Section 2.2 (p. 5)\color{blue}{\text{Takeaway 1 in Section 2.2 (p. 5)}} to describe approximately 0.86 as an empirical operating target rather than a universal stability boundary. Due to limited compute availability, we leave a fine-grained sweep between 0.80 and 0.86, together with repeated seeds on the unstable side, to future work.

Final checkpoint

We select the checkpoint at the same number of training examples for both the dense baseline and the Dynamic Sparsity run. Under this common-checkpoint rule, Dynamic Sparsity rollout matches dense rollout across model sizes, while retaining the 2.2×, 2.4×, and 2.1× rollout speedups for Qwen3-1.7B, 4B, and 8B, respectively.

ModelAIME2025 Mean@16AIME2024 Mean@16AIME2026 Mean@16AMC 2022–2023 Mean@4AMC2024 Mean@4
Qwen3-1.7B (Checkpoint 42K)0.41310.51940.44510.78210.6870
Qwen3-1.7B DS (Checkpoint 42K)0.40000.52910.43950.80120.6722
Qwen3-4B (Checkpoint 44K)0.71040.74790.70200.78880.9126
Qwen3-4B DS (Checkpoint 44K)0.71880.72500.68330.78880.9186
Qwen3-8B (Checkpoint 32K)0.67700.74480.67910.76110.8206
Qwen3-8B DS (Checkpoint 32K)0.68750.76670.66870.78330.8411
Reviewer_6kzMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Sparrow, a framework that speeds up long-context reinforcement learning for LLMs through sparse rollouts. It uses the 5th-percentile acceptance rate to measure the gap between sparse and dense policies, and adjusts the KV cache budget based on context length to keep training stable. In addition, DistillSparse uses on-policy distillation to bring the sparse policy closer to the dense policy. Experiments show that Sparrow speeds up rollout generation while keeping task performance largely unchanged.

Critical strengths
  1. Important and practical problem. Improving rollout efficiency while maintaining training stability is a practically significant problem, and the paper addresses the substantial rollout cost of long-context reinforcement learning for LLMs.
  2. A useful approach to stable sparse rollouts. The 5th-percentile acceptance rate helps identify large gaps between sparse and dense policies. The dynamic sparsity scheduling method also adjusts the KV cache budget based on context length, which helps balance speed and stability.
  3. Extensive experiments. The paper provides extensive experiments across multiple model sizes and tasks. The results show consistent rollout speedups with comparable performance, and further validate the effectiveness of dynamic sparsity scheduling and DistillSparse under different configurations.
Critical weaknesses
  1. Limited justification for the stability metric and threshold. The choice of the 5th-percentile acceptance rate and the empirical threshold of approximately 0.86 are primarily supported by experiments. The paper does not fully explain why this metric and threshold can predict training stability in other settings.
  2. Incomplete analysis of efficiency and scalability. Calibration and profiling introduce additional costs, while the reported speedup varies across model sizes. The paper mainly tests DistillSparse on a smaller model, and more experiments are needed to show how well it scales to larger models and whether its extra training cost is worthwhile.
Questions for authors

see above

3 comments
AuthorsOpenReview ↗

Official Response (Part 1)

We thank the reviewer for finding our motivation sound, our approach useful, and our experimentation extensive.

On the empirical 5-percentile and threshold justification

We thank the reviewer for raising this concern. Our central finding concerns lower-tail actor–policy mismatch: average acceptance rates can obscure a minority of strongly misaligned token positions associated with training degradation. The 5th percentile is a practical summary of this tail behavior, rather than a uniquely privileged statistic.

Appendix H.1 (p. 23)\color{blue}{\text{Appendix H.1 (p. 23)}} compares the 5th, 20th, and 30th percentiles for the same sparsity schedules, reproduced below. These statistics exhibit an empirical correspondence and consistently preserve the ordering of the evaluated schedules. For example, the schedule targeting a 5th-percentile acceptance rate of approximately 0.86 has 20th-percentile values of approximately 0.94–0.96 and 30th-percentile values of approximately 0.97–0.98 across the reported generation-length intervals. Thus, the numerical target depends on the percentile used to summarize alignment; 0.86 cannot be applied unchanged to other percentiles. We use the 5th percentile as a convenient, sensitive, and reliable statistic for estimating the lower-tail distribution mismatch, while avoiding the most extreme quantiles, whose estimates can have high variance.

Generation lengthDSparse₀.₈₆: 5th20th30thDSparse₀.₉₂: 5th20th30th
0–2K0.860.960.980.930.980.99
2–4K0.860.950.980.930.970.99
4–6K0.860.950.980.930.970.99
6–8K0.840.940.970.920.970.99
8–10K0.850.940.980.910.970.99
10–12K0.850.950.980.920.970.99
12–14K0.860.950.980.920.970.99
14–16K0.860.960.980.910.970.99
16–18K0.860.960.980.910.970.99
18–20K0.860.960.980.920.980.99

Regarding transfer, our evidence extends beyond the initial Qwen3-1.7B/4B/8B mathematics experiments to Qwen3-14B, coding RL, and Llama-3.2-1B-Instruct, as reported in Sections 3.2–3.3 (pp. 7–8)\color{blue}{\text{Sections 3.2–3.3 (pp. 7–8)}} and Appendix J (p. 25)\color{blue}{\text{Appendix J (p. 25)}}. These experiments support the usefulness of the acceptance-rate target across the evaluated settings, suggesting that our identified value should be a reliable starting point for practitioners when tuning rollout efficiency for their own RL training runs.

In the revised paper, following the reviewer's feedback, we put more emphasis on clarifying the distinction between lower-tail mismatch and the particular percentile used to summarize it in Section 1 (p. 3)\color{blue}{\text{Section 1 (p. 3)}} and Section 2.1 (p. 4)\color{blue}{\text{Section 2.1 (p. 4)}}. We have also revised the takeaway in Section 2.2 (p. 5)\color{blue}{\text{Section 2.2 (p. 5)}} to describe approximately 0.86 as an empirical operating target rather than a universal stability boundary. As discussed in Appendices J.3 and K.1 (pp. 25–26)\color{blue}{\text{Appendices J.3 and K.1 (pp. 25–26)}}, applying this target to a new setting requires profiling representative rollouts and validating training stability; the required KV-budget schedule remains specific to the model and setting.

AuthorsOpenReview ↗

Official Response (Part 2)

Profiling overhead and lack of scalability evidence of DistillSparse

We thank the reviewer for highlighting these limitations.

Calibration and profiling overhead. The calibration required by our method is a small, one-time setup cost per model and setting (Appendix K.1 (p. 26)\color{blue}{\text{Appendix K.1 (p. 26)}}). It generates sparse rollouts from 720 randomly sampled AIME problems and prefills these trajectories through the dense model to estimate acceptance rates; the full procedure takes approximately 30 minutes, and because the estimates stabilize after 64 trajectories, calibration can be reduced to approximately 10 minutes on a single H200 node. Moreover, the calibrated schedule can be reused across tasks: in Section 3.3 (p. 8)\color{blue}{\text{Section 3.3 (p. 8)}} (details in Appendix J.2 (p. 25)\color{blue}{\text{Appendix J.2 (p. 25)}}), we apply the schedules calibrated on mathematics directly to coding RL without any coding-specific recalibration, and training remains stable with performance on par with dense rollout. Calibration is therefore a one-time effort that serves multiple use cases, rather than a cost paid for every training run. The rollout speedups we report are measured end-to-end with wall-clock timing rather than inferred from the analytical cost model, and we have clarified in Appendix K.1 (p. 26)\color{blue}{\text{Appendix K.1 (p. 26)}} of the revised paper that this calibration cost should be counted separately when assessing the overall benefit for a particular training run.

Speedup across model sizes. The speedup varies across model sizes because larger models require a larger KV budget to keep the 5th-percentile acceptance rate at 0.86 (Figure 4(c) (p. 6)\color{blue}{\text{Figure 4(c) (p. 6)}}); a larger budget means more KV entries to load per decoding step, so dynamic sparsity alone yields 2.1× for Qwen3-8B compared with 2.4× for Qwen3-4B. This is also where combining dynamic sparsity with KV FP8 quantization helps: at 8B, KV FP8 with DSparse 0.88 reaches the same 0.86 target with a 2.54× rollout speedup (Table 10 (p. 24)\color{blue}{\text{Table 10 (p. 24)}}), exceeding either technique alone, and our new 8B training run shows that this combination trains stably and matches dense-rollout performance (Appendix P (p. 31)\color{blue}{\text{Appendix P (p. 31)}}).

Scope and scalability of DistillSparse. We agree that the current experiments do not establish how DistillSparse scales to larger models or whether its additional training cost is justified by the resulting rollout savings. We have revised the paper to present DistillSparse primarily as a proof-of-concept ablation probing the role of lower-tail alignment and providing further evidence for the identified threshold for lower-tail distribution mismatch. Specifically, distillation improves sparse–dense alignment, allowing a more aggressive sparsity configuration to reach approximately the same acceptance-rate target while maintaining stable training in the evaluated setting.

We have narrowed the corresponding claims and made these limitations explicit in our modified version of Section 3.5 (p. 9)\color{blue}{\text{Section 3.5 (p. 9)}}. Due to limited compute availability, we have not conducted an 8B DistillSparse experiment for this revision. We will consider adding this experiment in the next revision, subject to compute availability, together with an accounting of the additional distillation cost and the rollout savings needed to offset it.

Thank you for taking the time to address my concerns. As most of my questions have been resolved, I will maintain my rating.

Reviewer_TGxmOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose using lower tail per-token acceptance rates as an indicator of whether sparse rollouts will lead to a performance collapse for an LLM during reinforcement learning. They use a 5-percentile cutoff and increase the sparsity budget as generation length increases, ensuring that the 5th-percentile acceptance rate remains above a particular threshold. By doing so, they are able to achieve speedups between 2.1-2.4x across different Qwen3 models when training them on mathematical reasoning tasks. The authors also provide results on coding tasks that show that evaluation scores remain steady after sparsity scheduling and show that combining sparsity scheduling with KV cache quantization may lead to even better rollout speedups on some models.

Critical strengths
  1. I think the authors make a strong case for measuring lower tail statistics over average acceptance rates by demonstrating the poor predictive power of the latter. Their finding that the tail per-token acceptance rate is a good predictor of sparse-rollout degradation is very useful for practitioners studying its detection.
  2. The authors also show that their approach leads to sizeable speedups in generating rollouts during LLM post-training with long generation lengths while maintaining stability across a wide variety of benchmarks. Their heuristic, which increases the sparsity budget with the generation length (which correlates with the 5th-percentile acceptance rate), is highly practical and can improve RL post-training wall-clock times given the sizeable generation speedup it produces.
Critical weaknesses
  1. The best reported speedup is achieved by Qwen3-8B (2.54x speedup) when combined with KV cache quantization, in Appendix I. However, it is unknown if the combined speedup maintained performance comparable to the original dense model when evaluated on the paper's benchmarks, or whether performance degraded alongside the generation speedup. Given that evaluations are restricted to 3 models from the same model family, and that combining with KV-cache quantization does not improve generation speedups for 1.7B and 4B models, I think the evidence for the claim in Section 3.4 (that the speedup of the combined technique exceeds simple dynamic sparsity as model size increases) is not well-supported and extrapolates from a single data point on generation speedup without providing degradation results.
  2. While the authors train tasks by having a maximum generation length of 37K tokens, the paper does not currently describe how often the full generation length was utilized by the model during rollouts, and how often the sparsity scheduling approach increased the sparsity budget. Thus, it is possible (although unlikely) that performance stability is maintained in the reported experiments without actually increasing sparsity budgets frequently during training. Some additional context on how many tokens the model responses actually used and how often the sparsity budget was increased would better contextualize the core claims of the paper, and help attribute the performance stability to sparsity scheduling.
Questions for authors
  1. Do the authors have an explanation for why the tail distribution mismatch is preserved when the sparsity budget is increased with generation length, beyond the generation length- 5th-percentile acceptance rate correlation?
  2. Do the authors have any insight into the performance of the Qwen3 4B and 8B models on the evaluated benchmarks when combined with KV cache quantization, and if they exhibit similar stability to the 1.7B model?
  3. How often does the model use the full 37K generation length for generating a rollout during training? What are the median generation lengths for generated rollouts for each task in your setting?
3 comments
AuthorsOpenReview ↗

Official Response (Part 1)

We sincerely thank the reviewer for the attention to our work. We are glad the reviewer finds our work insightful, agrees with our core observations, and appreciates the speedup we present. We will try our best to resolve the concerns raised by the reviewer.

Lack of performance numbers for KV FP8 combined with dynamic sparsity rollout

We thank the reviewer for raising this concern. We present a control study on Qwen3-8B showing that, using the proposed scheme, maintaining the tail distribution mismatch level at 0.86 with KV FP8 and dynamic sparsity at 0.88 trains stably and matches dense-rollout performance. Due to the lack of H200 nodes, we instead run the same settings presented in the paper on 2x GB300 nodes for both the dense rollout and the combined quantization with dynamic sparsity, with batch size 128 and sequence length cutoff of 37K.

By the time of the response upload, the training passes 10,000 training examples and is still stable. We present the training reward curve and the response length curve in Appendix P (p. 31)\color{blue}{\text{Appendix P (p. 31)}} of the updated paper, and the numbers below are also reported in Table 17 (p. 31)\color{blue}{\text{Table 17 (p. 31)}}.

MethodTraining Reward EMA (10K examples)Response Length EMA (10K examples)AIME2025 Mean@16AIME2026 Mean@16
Dense Baseline0.7286130980.69370.6875
KV FP8 + DSparse 0.880.7268138720.70410.6937

Following the reviewer's suggestions, we have narrowed the claim in the revised Section 3.4 (p. 8)\color{blue}{\text{Section 3.4 (p. 8)}} (and the caption of Figure 5(b) (p. 8)\color{blue}{\text{Figure 5(b) (p. 8)}}): we now state explicitly that, at 8B in our evaluation, the speedup of KV FP8 combined with DSparse 0.88 surpasses dynamic sparsity alone, rather than describing a general scaling trend, and we point to the new 8B training results in Appendix P (p. 31)\color{blue}{\text{Appendix P (p. 31)}}. Regarding 4B, the combination is not the faster option at that scale: dynamic sparsity alone achieves a 2.41× rollout speedup versus 2.13× for KV FP8 combined with DSparse 0.88 (Table 10 (p. 24)\color{blue}{\text{Table 10 (p. 24)}}), so we prioritized training the 8B configuration, where the combination yields the largest speedup. We will consider adding the 4B training run and its downstream evaluation in a future revision, subject to compute availability.

Concern about the rollout sequence length distribution

We thank the reviewer for raising the concern about generation length. We use a Qwen3-1.7B thinking model checkpoint trained on more than 60K training examples to measure generation length statistics, sampling 128 prompts with 8 responses each (128x8).

StatisticPolaris (128x8)AIME25 (128x8)
Mean20233.1623297.94
Median20075.523434
Min10764372
Max3700037000
Cutoff Ratio4.2%11.23%

From the above, we can see that the mean and median are very close in value, so the length distribution is not skewed by a few very long responses: most rollouts are genuinely long. Although we did not save the checkpoints for the 4B and 8B thinking runs, we logged their summary statistics (mean, min, max, and cutoff ratio).

ModelMeanMinMaxCutoff Ratio
Qwen3-4B (at 30K examples)15905997370002.5%
Qwen3-8B (at 30K examples)164771549370003.4%

Without the per-response lengths, we cannot recover their medians. From the above table, although the 4B and 8B models generate relatively shorter responses than the 1.7B model, they still generate roughly 16K tokens on average, confirming very long generation.

These statistics also show how frequently dynamic sparsity scheduling is exercised during training. The schedule re-selects the KV budget at every 2K-token bin boundary, and the selected budget is non-decreasing in generation length (it can plateau where a larger budget is not needed to meet the target, e.g., for Qwen3-1.7B between 4K and 8K context; Appendix K.2 (p. 26)\color{blue}{\text{Appendix K.2 (p. 26)}}). With a median length of about 20K tokens, a typical Qwen3-1.7B rollout spans about 10 of these bins, so its budget is re-selected roughly ten times and ends well above its starting value; even the 4B and 8B runs average about 16K tokens, or about 8 bins. In addition, 2.5–4.2% of training rollouts (Polaris for 1.7B, and the 4B and 8B runs) and 11.2% of AIME25 rollouts reach the 37K cutoff, where the schedule operates at its largest budgets. Scheduling is therefore not a rarely triggered safeguard: nearly every rollout passes through multiple budget levels, and the stability we report is obtained with the schedule active throughout generation.

AuthorsOpenReview ↗

Official Response (Part 2)

Why we have to increase the KV sparsity budget as the sequence length increases to maintain the same tail distribution gap

We thank the reviewer for raising the core question about the design of dynamic sparsity. Because of causal attention, when decoding the nth token, the model needs information from the past (n-1) tokens stored in the KV cache. Thus, as n increases, the tail distribution gap between the fixed sparsity KV budget approximation and the original dense distribution at position n would increase. Therefore, to maintain the same tail distribution gap, we increase the KV budget.

Our method, presented in Section 2.2 (p. 4)\color{blue}{\text{Section 2.2 (p. 4)}}, addresses how to increase the KV budget to maintain the tail distribution, and based on that, we present the Dynamic Sparsity technique.

To answer the question more directly: the tail mismatch is preserved largely by construction. For each 2K-token bin, we profile 18 candidate KV budgets on representative rollouts and select the budget whose 5th-percentile acceptance rate meets the target in that bin (Section 2.2 (p. 4)\color{blue}{\text{Section 2.2 (p. 4)}}); the schedule therefore holds the bin-level tail statistic near the target throughout generation, rather than relying on a fixed budget whose alignment happens to correlate with length. Why the required budget must grow has an intuitive mechanism. A fixed budget BB retains a shrinking fraction B/nB/n of the context as the position nn grows, so a growing share of the attention mass falls on evicted tokens, and the sparse output distribution drifts further from the dense one at the positions where that evicted mass matters most, which are precisely the lower-tail tokens. Figure 3(a–c) (p. 5)\color{blue}{\text{Figure 3(a–c) (p. 5)}} shows this directly: under every fixed sparsity configuration, the 5th-percentile acceptance rate falls steadily with sequence length, while the mean acceptance rate barely changes (Section 2.1 (p. 4)\color{blue}{\text{Section 2.1 (p. 4)}}). Increasing the budget with length counteracts this loss of coverage. Consistent with this view, Appendix K.2 (p. 26)\color{blue}{\text{Appendix K.2 (p. 26)}} shows that the extra budget needed per 1K tokens diminishes at longer contexts, suggesting that attention mass on distant tokens becomes increasingly concentrated on a small set of positions that the sparse selection retains. We present this as an explanation of the observed trend.

On presentation

We also take the reviewer's concern about readability seriously. In the uploaded revised paper, we have added clarifications to key parts of the paper (all highlighted in blue): the motivation for lower-tail statistics and the role of the 5th percentile in Section 1 (p. 3)\color{blue}{\text{Section 1 (p. 3)}} and Section 2.1 (p. 4)\color{blue}{\text{Section 2.1 (p. 4)}}, the interpretation of the 0.86 target in Takeaway 1 (p. 5)\color{blue}{\text{Takeaway 1 (p. 5)}}, the scope of the combined-technique and DistillSparse claims in Section 3.4 (p. 8)\color{blue}{\text{Section 3.4 (p. 8)}} and Section 3.5 (p. 9)\color{blue}{\text{Section 3.5 (p. 9)}}, and the calibration cost in Appendix K.1 (p. 26)\color{blue}{\text{Appendix K.1 (p. 26)}}. We will continue to improve the writing and organization in future versions, following the reviewer's feedback.

I thank the authors for their response - all my concerns have been addressed satisfactorily. To further improve the paper, I suggest incorporating models from other families as well in subsequent versions. I recommend this paper for acceptance, and I have updated my score to 4. Good luck with the submission!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary to help the work of AC

We sincerely thank the AC and all reviewers for their time and constructive engagement during the discussion period. To help the AC better review the full conversation, we present the following summary.

Main concerns and how we addressed them (all revisions are highlighted in blue in the revised paper):

  • KV FP8 combined with dynamic sparsity (TGxm). New Qwen3-8B training with KV FP8 + DSparse 0.88 is stable and matches dense rollout on training reward and AIME (Appendix P (p. 31)\color{blue}{\text{Appendix P (p. 31)}}); the Section 3.4 claim is narrowed to the evaluated 8B setting.
  • Justification of the 5th-percentile metric and the 0.86 threshold (6kzM, bFmF). The 5th percentile is a practical summary of lower-tail mismatch whose ordering agrees with other percentiles, and 0.86 is an empirical operating target that holds across 1.7B, 4B, 8B, and seeds, and transfers to Qwen3-14B, coding RL, and Llama.
  • Robustness to TIS clipping and evaluation at a common checkpoint (bFmF). New runs with TIS thresholds of 2.5 and 3.5 remain stable at the 0.86 target (Appendix O (p. 30)\color{blue}{\text{Appendix O (p. 30)}}), and dynamic sparsity matches dense rollout at a common checkpoint across all three model sizes.
  • Calibration cost and speedup across model sizes (6kzM). Calibration is a one-time cost of about 10–30 minutes on one H200 node and is reusable across tasks; the lower 8B speedup reflects a larger required KV budget, where combining with KV FP8 reaches 2.54×.
  • Rollout lengths and the mechanism of dynamic sparsity (TGxm). Rollouts are long (median about 20K tokens for 1.7B), so scheduling is active throughout generation, and per-bin calibration preserves the lower-tail target by construction.

Status of the discussion. Reviewer TGxm confirmed that all of their concerns have been addressed satisfactorily, recommended acceptance, and raised their score from 3 to 4. Reviewer 6kzM indicated that most of their questions have been resolved and maintained their positive rating of 3. All three reviewers now recommend acceptance. Reviewer bFmF has not yet responded; we would greatly appreciate it if Reviewer bFmF could let us know whether our new TIS-threshold experiments, threshold-sweep discussion, and common-checkpoint results address their concerns.

Finally, we respectfully note that beyond the rollout speedups, the paper contributes a simple and broadly applicable diagnostic for RL with approximate rollouts: lower-tail acceptance-rate statistics, together with an empirical operating threshold that transfers across model sizes, tasks, and model families. As rollout cost and actor–policy mismatch become central challenges in large-scale LLM RL, we believe these findings can offer practical guidance to the broader community, and we would be grateful if the AC could help this work reach the wider audience that may benefit from it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.