Reinforcement learning with verifiable rewards (RLVR) has become a powerful recipe for improving large language models on reasoning tasks, but its training cost is increasingly dominated by long-context rollout generation, making sparse attention as the rollout worker for training dense policy promising. However, in practice, mastering the tradeoff between training stability and efficiency is difficult. In order to study the optimal tradeoff, we first observe that the majority of the tokens are perfectly aligned with dense under highly aggressive sparsity. We then hypothesize that once the tail distribution is constrained by a threshold, we are able to train sparse rollout stably. We prove that our hypothesis holds by introducing a dynamic sparsity scheduling method for controlling the tail distribution constant through generation, and study how the threshold scales with model sizes. Surprisingly, we find that the threshold of 5-percentile mismatch at 0.86 generally works across model sizes and uses cost model analysis for finding the best scheduling for speedup. Empirically, our method generalizes to Qwen3-14B and enables stable training with sparse rollout, while achieving 2.2x, 2.4x, and 2.0x in rollout when training Qwen3-1.7B, Qwen3-4B, and Qwen3-8 B.
Official Response
Thank you for your attention to Sparrow. We are glad that you find our study concrete, our control studies reasonable, and validation extensive.
TIS threshold used and how strict it is
We thank you for raising this question. We use a TIS threshold of 3.0; this choice follows prior work, Jackpot, published at ICLR 2026.
Chen, Z., Liu, H., Zhou, Y., Zheng, H., & Chen, B. (2026, April). Jackpot: Align Actor-Policy Distribution for scalable and stable RL for LLM. In International Conference on Learning Representations (Vol. 2026, pp. 68647-68664).
For generalization, due to compute limitations, we cannot run the full sweep for tail distribution mismatch for different TIS thresholds, each of which requires at least four multi-node H200 runs. Instead, we show here that for a TIS threshold close to 3.0, maintaining the tail distribution mismatch level around 0.86 can generalize and maintain stable training.
We run the Qwen3-1.7B thinking model on coding, following the recipe in Section 3.3 (p. 8). Each run uses 2xH200 nodes with a 24K generation length cutoff and TIS thresholds of 2.5 and 3.5. By the time the response is submitted, the training reaches 6,400 training examples, and the training with 0.86 dynamic sparsity is empirically stable.
For the numbers presented in the table below, the EMA smoothing factor is set to 0.99; the training reward, response length, and HumanEval+ curves over the 6,400 training examples are shown in Appendix O (p. 30) of the revised paper, where the curves use EMA 0.5 for visualization.
Through this experiment, we show that the identified 0.86 target generalizes to stable training for TIS thresholds in the vicinity of the value of 3.0 we used. Beyond the TIS threshold, the same 0.86 target also transfers across other training settings in our evaluation, including model sizes from Qwen3-1.7B to Qwen3-14B, tasks (mathematics and coding), and model families (Llama-3.2-1B-Instruct), as reported in Sections 3.2–3.3 (pp. 7–8) and Appendix J (p. 25).
On the coarseness of the threshold sweep
We agree that our control study tests a discrete set of targets (0.75, 0.80, 0.86, and 0.92), so it locates the transition between 0.80 and 0.86 rather than pinpointing an exact boundary. Two pieces of evidence support that this separation is reliable rather than incidental. First, the same split holds independently at all three model sizes: at 1.7B, 4B, and 8B, the 0.80 schedule collapses while the 0.86 schedule trains stably (Figure 3(d–f) (p. 5)), giving three separate tests of the same boundary. Second, Appendix H (p. 23) repeats Qwen3-1.7B coding RL with DSparse0.86 across three random seeds, and all runs remain stable, indicating that stability at 0.86 is not the result of a favorable seed. Consistent with this resolution, we have revised Takeaway 1 in Section 2.2 (p. 5) to describe approximately 0.86 as an empirical operating target rather than a universal stability boundary. Due to limited compute availability, we leave a fine-grained sweep between 0.80 and 0.86, together with repeated seeds on the unstable side, to future work.
Final checkpoint
We select the checkpoint at the same number of training examples for both the dense baseline and the Dynamic Sparsity run. Under this common-checkpoint rule, Dynamic Sparsity rollout matches dense rollout across model sizes, while retaining the 2.2×, 2.4×, and 2.1× rollout speedups for Qwen3-1.7B, 4B, and 8B, respectively.