acceptodds
Under review as a conference paper at ICLR 2027

TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts

Abstract

Large-scale rollouts have become a core component of modern LLM systems, spanning reinforcement learning (RL) post-training, on-policy distillation, and sampling-heavy evaluation. Unlike online serving, which typically targets request-level latency and throughput, synchronous rollouts must collect a complete batch before proceeding. A few long-tail generations can therefore dominate an entire rollout step, and uniform routing can slow these generations further by placing them in high-concurrency decoding batches. We present TailSieve, a partial-rollout-guided framework that jointly controls tail routing and replica allocation. Motivated by the observed persistence of prompt-level tail tendencies across policy updates, TailSieve uses partial-rollout completion status as a training-free signal for identifying candidate tail groups. Selected prompts are combined with newly sampled prompts, and all responses are generated from scratch under the current policy. A hierarchical controller uses response-work history and a serving model: a fast inner loop adjusts the number of isolated groups to balance completion times, while a slower outer loop adapts how many replicas serve these groups. The resulting low-concurrency execution also enables route-specialized speculative decoding with MTP or DFlash. Across the evaluated configurations, TailSieve achieves up to 1.67× speedup with routing alone and 2.59× with routing and speculation combined, both over uniform group routing.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Rollouts, Tail-Aware Routing, Reinforcement Learning Post-Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4R6AOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TailSieve, a framework for reducing straggler-induced delays in synchronous LLM rollouts. The authors formulate rollout scheduling as a makespan minimization problem under concurrency-dependent decoding efficiency and show that isolating a small number of long-tail requests can approximate the offline optimal assignment.

Building on the observation that prompt-level response-length tendencies persist across policy updates, TailSieve identifies candidate tail groups from unfinished partial rollouts and regenerates them under the current policy. A hierarchical controller jointly adjusts the number of isolated tail groups and their allocated replicas. The framework additionally supports route-specialized speculative decoding using MTP and DFlash.

The evaluation includes an offline routing analysis, cross-checkpoint tail-prediction studies, and vLLM experiments across five Qwen model configurations and two workloads on eight H100 GPUs. The reported speedups reach 1.67× with routing alone and 2.59× with routing and speculative decoding, relative to uniform non-speculative routing. GRPO experiments on two smaller models show reduced rollout-generation time with broadly comparable training rewards and downstream accuracy.

Critical strengths

1. Well-motivated routing formulation and effective tail-isolation strategy.

The paper provides a clear analysis of how concurrency-dependent decoding efficiency interacts with long-tail response lengths. The exact offline routing formulation and the observation that top-k isolation achieves within 4% of the oracle optimum across 100 sampled instances provide useful justification for the proposed design. The resulting approach is conceptually simple and does not require a separately trained length predictor.

2. Promising empirical performance across models and workloads.

The evaluation covers multiple model scales and architectures, mathematical and coding workloads, and both standard and speculative decoding. The cross-checkpoint analysis provides evidence that prompt-level tail behavior is persistent, while the routing experiments demonstrate meaningful speedups. The additional GRPO experiments provide preliminary evidence that these efficiency improvements can be obtained without substantial degradation in training outcomes.

Critical weaknesses

1. The evaluation does not sufficiently disentangle tail-aware routing from partial-rollout selection and cross-round deferral.

Under TailSieve, the bulk pool launches NN fresh groups but only accepts the first N−qN-q completed groups, while the remaining qq accepted groups come from regenerated tail prompts. In contrast, the uniform-routing baseline processes all NN groups to completion. Consequently, the two approaches differ not only in replica assignment but also in their completion and selection mechanisms.

This raises an important attribution concern: how much of the reported improvement comes from isolating long-tail requests, rather than deferring unfinished bulk groups across rollout steps? A baseline using the same partial-rollout cutoff, carryover policy, and regeneration mechanism, but without tail-aware routing, would be necessary to isolate the contribution of the proposed routing strategy. This directly affects the interpretation of the main results in Table 1.

2. The evidence for end-to-end practical benefits remains limited.

Most step-wise performance comparisons are conducted after the allocation controller has converged, leaving the amortized cost of initialization and adaptation insufficiently characterized. Furthermore, speculative-decoding experiments use fixed replica splits because runtime reconfiguration introduces additional costs, limiting validation of the fully adaptive design with speculation.

The headline 2.59× speedup is measured against uniform routing without speculative decoding. Appendix F.1 shows that the additional gain over uniform routing with the same speculative decoder is substantially smaller (approximately 1.05–1.34×). Finally, the GRPO experiments measure rollout-generation time rather than complete training wall-clock time. These limitations make the overall efficiency benefit in realistic, dynamically evolving training workloads less conclusive.

Questions for authors
  1. Isolating the routing contribution: Could the authors provide a comparison against a uniform-routing baseline that uses exactly the same partial-rollout cutoff, unfinished-group carryover, and regeneration policy as TailSieve? This would clarify how much of the improvement comes specifically from tail isolation and adaptive replica allocation.

  2. End-to-end efficiency: What is the cumulative training wall-clock improvement when accounting for controller warm-up, allocation transitions, speculative-decoding reconfiguration, and optimization time? In particular, can the authors quantify the benefit of the fully adaptive controller when speculative decoding is enabled?

  3. Cross-round reliability and training-data distribution: How accurately do unfinished groups predict the longest groups in subsequent rollout steps under changing policies, particularly on coding workloads? Can the authors quantify the residual selection bias discussed in Appendix G.5 and assess whether the accepted prompt/reward distributions remain comparable to uniform routing?

3 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer 4R6A (W1, Q1)

[W1/Q1 — Distinguishing routing from cross-round deferral]

This is a great suggestion! Holding deferral fixed gives us a clearer measure of what routing adds. We therefore add a uniform-routing control with the same cutoff, carryover, and regeneration mechanism, together with random-selection controls. We will include these ablation results in the updated version

We separate routing from cross-round deferral in two parts. (1) We describe the four combinations of selection and routing, including the matched Uniform control. (2) We report the contribution of tail isolation when the deferral policy is held fixed.

(1) Matched controls isolate the effect of routing while holding the deferral policy fixed.

We evaluate two selection policies (random or unfinished groups) and two routing policies (Uniform or isolation) across five models on DeepScaleR and KodCode. Random selects the q fresh groups to defer uniformly at random before generation, regardless of their completion times. Uniform distributes fresh and carried groups across all replicas without separating them into dedicated pools. Unfinished and Isolation follow TailSieve’s completion-based deferral and bulk/tail routing mechanisms, respectively (see §3, “A mixed tail–bulk pipeline”).

(2) TailSieve is the fastest configuration across all five models on both workloads, with up to 1.283× speedup over Uniform deferral.

DeepScaleR

ModelRandom + UniformRandom + isolationUnfinished + UniformUnfinished + isolation (TailSieve)
Qwen3.5-35B-A3B0.998×0.968×1.060×1.346×
Qwen3.5-4B1.059×0.967×1.013×1.233×
Qwen3.5-2B1.060×1.100×1.065×1.180×
Qwen3-30B-A3B0.942×0.862×0.872×1.112×
Qwen3-4B1.047×1.085×1.059×1.254×

KodCode

ModelRandom + UniformRandom + isolationUnfinished + UniformUnfinished + isolation (TailSieve)
Qwen3.5-35B-A3B1.235×0.952×1.302×1.670×
Qwen3.5-4B1.081×0.846×1.222×1.403×
Qwen3.5-2B1.033×0.971×1.104×1.214×
Qwen3-30B-A3B1.153×1.155×1.140×1.342×
Qwen3-4B1.186×0.950×1.218×1.442×

Deferral does improve performance in several settings, but provides no consistent speed-up. Random isolation provides no consistent benefit, and brings degeneration at some settings. With the same unfinished-group deferral policy, adding tail isolation yields a further speedup over Unfinished + Uniform of 1.108–1.276× on DeepScaleR and 1.100–1.283× on KodCode. These results indicates that tail identification and isolation are two coupled mechanisms .

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer 4R6A (W2, Q2)

[W2/Q2 — Fully adaptive control and gains with speculative decoding]

There may be some misunderstanding about the use of fully adaptive controller. If we’ve misread your question, we’d be happy to discuss it further. Here, we understand the reviewer’s “fully adaptive controller” to refer to enabling both the inner loop for tail-quota adaptation and the outer loop for replica allocation. We clarify that our RL experiments already use the fully adaptive controller. We additionally report (1) an end-to-end ablation with speculative decoding enabled RL experiments. We then (2) compare per-step rollout times with Uniform using the same speculative decoder, (3) explain why the additional routing gains are smaller when speculative decoding is enabled and (4) report end-to-end timing breakdown.

(1) With speculative decoding enabled, TailSieve provides up to 1.115× additional end-to-end speedup.

We tested two ablation settings with speculative decoding: both control loops off and only the inner loop active. The results are as follows, all speedups reported relative to both loops off.

ModelConfigurationRollout (s)E2E (s)
2BBoth loops off1,704.293,592.87
2BInner-loop on1,524.75 (1.118×)3,348.76 (1.073×)
2BTailSieve1,410.89 (1.208×)3,221.43 (1.115×)
4BBoth loops off1,720.804,096.48
4BInner-loop on1,676.85 (1.026×)3,977.44 (1.030×)
4BTailSieve1,565.65 (1.099×)3,848.02 (1.065×)

(2) TailSieve speeds up rollouts by up to 1.517× over Uniform with the same speculative decoder.

We also evaluate step-wise speed-up for a convergence setting. We report speedups over Uniform with the same speculative decoder.

WorkloadModelDecoderTailSieve (fixed replica)TailSieve
DeepScaleRQwen3.5-2BMTP29.02 s (1.329×)28.76 s (1.341×)
DeepScaleRQwen3.5-35B-A3BDFlash68.91 s (1.107×)63.54 s (1.200×)
DeepScaleRQwen3.5-35B-A3BMTP62.87 s (1.152×)61.42 s (1.179×)
DeepScaleRQwen3.5-4BDFlash47.79 s (1.119×)47.79 s (1.119×)
DeepScaleRQwen3.5-4BMTP41.39 s (1.163×)39.53 s (1.218×)
KodCodeQwen3.5-2BMTP24.26 s (1.274×)24.26 s (1.274×)
KodCodeQwen3.5-35B-A3BDFlash37.07 s (1.117×)30.71 s (1.349×)
KodCodeQwen3.5-35B-A3BMTP34.09 s (1.232×)29.78 s (1.411×)
KodCodeQwen3.5-4BDFlash33.45 s (1.335×)29.43 s (1.517×)
KodCodeQwen3.5-4BMTP29.84 s (1.053×)29.84 s (1.053×)

(3) TailSieve complements speculative decoding by targeting the remaining long tail.

As for the smaller gain relative to speculative decoding, we attribute this to speculative decoding’s tendency to “compress” the long tail: it provides greater acceleration at smaller active batch sizes, leaving less room for routing to improve the makespan. However, our approach still provides consistent gains over various workloads. We expect TailSieve to help most when a small number of long generations still determine the rollout makespan, even with speculative decoding enabled.

(4) TailSieve achieves up to 1.271× end-to-end speedup over Uniform.

We also report the RL timing breakdown. Allocation transitions are already included in the rollout times.

ModelConfigurationRollout (s)Training (s)Initial warm-up (s)E2E (s)E2E speedup
2BUniform3,580.311,884.90—5,465.21—
2BTailSieve2,474.591,782.6044.294,301.481.271×
4BUniform3,242.912,275.20—5,518.11—
4BTailSieve2,866.152,094.85120.065,081.061.086×
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer 4R6A (Q3)

[Q3 — Cross-round reliability and accepted-data distributions]

We appreciate the reviewer’s suggestion! We address your questions in two parts. (1) We evaluate tail identification across checkpoints and across rounds during RL training. (2) We quantify the observed differences in accepted data relative to Uniform.

(1) Tail identification remains reliable across checkpoints and during actual RL training.

Following the same protocol as our math evaluation, we evaluate KodCode coding prompts. We define the source checkpoint’s longest prompt groups as the tail set and use target-checkpoint group-mean lengths as ranking scores. Adjacent-checkpoint AUC ranges from 0.895 to 0.960 for Qwen3.5-2B and 0.930 to 0.997 for Qwen3.5-4B, well above the random baseline of 0.5. These results support a persistent prompt-level length signal on coding workloads.

We next evaluate unfinished groups from each update to their regenerated responses during actual RL training process. We measure the probability that a carried group has a longer group-mean response length than a non-carried group. These results show that unfinished groups retain a strong relative-length signal after policy update.

ModelUpdate transitionsCarry groupsMean carry-length AUC
Qwen3.5-2B293780.9093
Qwen3.5-4B29970.9311

A minor concern might be: what if partial-rollout guided signals fails to predict the longest groups. An important property is that partial rollout also tolerates missed tail predictions: the bulk pool accepts the first N − q of N groups, so its completion time is determined by the (1 − q/N) completion-time quantile rather than the longest stragglers.

(2) Against Uniform, the CDF gaps are below 5.3% for accepted lengths and below 2.08% for rewards.

We quantify the observed distributional differences relative to Uniform: Appendix F.3 already reports maximum empirical CDF gaps below 5.3 percentage points for mean response lengths, and we will additionally report accepted reward distributions in the revised version. Accepted prompt/reward distributions remain comparable to uniform routing. To address residual selection bias, we quantify the observed distributional differences from Uniform using the maximum empirical CDF gaps for accepted response lengths and rewards.

Regarding accepted prompt difficulty, unfinished prompts are carried forward and included in the next training update, with exactly one-step delay, so deferral does not remove harder prompts from training.

Maximum reward-CDF gap: Uniform vs. TailSieve2B4B
Correctness reward, excluding length penalties2.08 pp0.99 pp
Training reward, including length penalties1.07 pp0.61 pp
Reviewer_emyjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TailSieve, a framework that aims to speed up synchronous LLM rollouts, where a few very long responses can slow down the entire training step.

The main idea is to use unfinished prompt groups from previous rounds to identify likely long-tail requests. TailSieve then uses a hierarchical controller to adjust how many requests go to the tail pool and how many GPU replicas are assigned to it. The method also works with MTP and DFlash speculative decoding.

Experiments on several Qwen models and math and coding tasks show up to 1.67× speedup from routing alone, and up to 2.59× when combined with speculative decoding, compared to uniform routing without speculation. The GRPO experiments also show faster rollout generation with similar training quality.

Critical strengths

S1. A simple and practical solution to a real rollout bottleneck.

I like that TailSieve uses unfinished prompts from previous rounds to identify possible long-tail requests, without training an extra length prediction model. The hierarchical controller also adjusts the tail workload and GPU replica allocation to reduce waiting time. Since all training responses are generated under the current policy, the method avoids off-policy issues from reusing old trajectories. Overall, I think the design is straightforward and practical.

S2. Good experimental results showing clear performance improvements.

The authors test TailSieve on different models and both math and coding tasks, showing clear improvements in rollout speed. The ablation studies also support the benefit of adaptive replica allocation. I also like that the paper tests different rollout sizes and combines TailSieve with MTP/DFlash speculative decoding. More importantly, the GRPO experiments show faster rollout generation without noticeably affecting rewards, response lengths, or downstream accuracy. I think these results make the method more convincing.

Critical weaknesses

W1. Limited evaluation of full training efficiency and scalability.

The paper shows improvements in rollout speed, but I would like to see how much time is actually saved in the full RL training process, including optimization. Also, all experiments were run on a single 8 x H100 server. I'm not sure whether the same speedup would hold for larger or multi-node setups.

W2. More validation is needed for tail identification and training data distributions.

The cross-checkpoint tail identification results are mainly based on math tasks. I would like to see whether the same findings hold for coding tasks. Also, Appendix G discusses possible distribution bias, but I think a direct comparison of accepted prompt difficulty and reward distributions between TailSieve and the baseline would help clarify this concern.

Questions for authors

Q1. About the full RL timing. I would like to see the full RL training wall-clock time, including optimization and warm-up. This would help me better understand how much speedup TailSieve actually brings to the overall training process.

Q2. Could the authors provide cross-checkpoint AUC results on coding tasks, or directly evaluate cross-round tail identification accuracy? I would also like to see a comparison of the accepted prompt difficulty and reward distributions between TailSieve and the baseline, to check whether there is any distribution shift.

3 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer emyj (W1, Q1)

[W1/Q1 — End-to-end training efficiency and multi-node scaling]

We thank the reviewer for raising the question of how TailSieve performs under practical RL training conditions. We address the implementation and scalability questions in two parts. (1) We report the single-node timing breakdown. (2) We are working on the multi-node evaluation and expect to report the results by the end of this week. Thanks again for suggesting us to evaluate TailSieve at a larger scale!

(1) TailSieve delivers an end-to-end RL speedup of up to 1.271× over Uniform.

Here, we compare end-to-end wall-clock times for the Uniform baseline and TailSieve. For TailSieve's training, we simply group responses of similar lengths into microbatches. We will include this end-to-end timing comparison in the updated version.

ModelTime componentUniform (s)TailSieve (s)Speedup
Qwen3.5-2BRollout3,580.312,474.591.447×
Qwen3.5-2BTraining1,884.901,782.601.057×
Qwen3.5-2BInitial warm-up round-44.29-
Qwen3.5-2BE2E5,465.214,301.481.271×
Qwen3.5-4BRollout3,242.912,866.151.131×
Qwen3.5-4BTraining2,275.202,094.851.086×
Qwen3.5-4BInitial warm-up round-120.06-
Qwen3.5-4BE2E5,518.115,081.061.086×

(2) Multi-node evaluation is underway, with results expected by the end of this week.

We are currently running multi-node experiments to evaluate how TailSieve’s rollout gains carry over across nodes. We really appreciate the reviewer’s suggestion to evaluate TailSieve at a larger scale, and we expect to report the results by the end of this week. : ) We look forward to discussing these results with you!

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer emyj (W2, Q2)

[W2/Q2 — Tail identification and accepted-data distributions]

We appreciate the reviewer’s suggestion! We address your questions in two parts. (1) We evaluate tail identification across checkpoints and across rounds during RL training. (2) We quantify the observed differences in accepted data relative to Uniform.

(1) Tail identification remains reliable across checkpoints and during actual RL training.

Following the same protocol as our math evaluation, we evaluate KodCode coding prompts. We define the source checkpoint’s longest prompt groups as the tail set and use target-checkpoint group-mean lengths as ranking scores. Adjacent-checkpoint AUC ranges from 0.895 to 0.960 for Qwen3.5-2B and 0.930 to 0.997 for Qwen3.5-4B, well above the random baseline of 0.5. These results support a persistent prompt-level length signal on coding workloads.

We next evaluate unfinished groups from each update to their regenerated responses during actual RL training process. We measure the probability that a carried group has a longer group-mean response length than a non-carried group. These results show that unfinished groups retain a strong relative-length signal after policy update.

ModelUpdate transitionsCarry groupsMean carry-length AUC
Qwen3.5-2B293780.9093
Qwen3.5-4B29970.9311

A minor concern might be: what if partial-rollout guided signals fails to predict the longest groups. An important property is that partial rollout also tolerates missed tail predictions: the bulk pool accepts the first N − q of N groups, so its completion time is determined by the (1 − q/N) completion-time quantile rather than the longest stragglers.

(2) Against Uniform, the CDF gaps are below 5.3% for accepted lengths and below 2.08% for rewards.

We quantify the observed distributional differences relative to Uniform: Appendix F.3 already reports maximum empirical CDF gaps below 5.3 percentage points for mean response lengths, and we will additionally report accepted reward distributions in the revised version. Accepted prompt/reward distributions remain comparable to uniform routing.

Regarding accepted prompt difficulty, unfinished prompts are carried forward and included in the next training update, with exactly one-step delay, so deferral does not remove harder prompts from training.

Maximum reward-CDF gap: Uniform vs. TailSieve2B4B
Correctness reward, excluding length penalties2.08 pp0.99 pp
Training reward, including length penalties1.07 pp0.61 pp
AuthorsOpenReview ↗

Thanks & Update Response to Reviewer emyj (W1, Q1 Update 11.13)

Update 11.13 (Multi-node RL Results):

We are excited to share our latest multi-node RL results! We ran synchronous GRPO with verl and vLLM on 16 H100 GPUs across two nodes linked by a RoCE v2 network. We use 128 prompt groups per update, doubled size compared to single node experiments.

(1) Multi-node RL achieves up to 1.8× end-to-end speedup.

Following the same timing protocol as above, we first report end-to-end training time, including TailSieve’s initial warm-up and training steps. The end-to-end speedup is larger than the rollout speedup. We attribute this in part to length bucketing: with more responses per update, we can form microbatches with more similar response lengths, reducing padding during training.

ModelTime componentUniform (s)TailSieve (s)Speedup
Qwen3.5-2BRollout2,395.342,194.511.1×
Qwen3.5-2BInitial warm-up round–11.53–
Qwen3.5-2BE2E7,618.694,297.251.8×
Qwen3.5-4BRollout3,415.332,944.741.2×
Qwen3.5-4BInitial warm-up round–88.19–
Qwen3.5-4BE2E8,258.086,158.751.3×

(2) Zero rollout-state transfer overhead for tail routing.

TailSieve naturally fits multi-node deployments. Although we use a global controller, coordinating tail routing across nodes requires only lightweight metadata—prompt identities. Each destination replica regenerates the responses under the current policy, so tail routing requires no cross-node transfer of partial responses or KV caches.

(3) Downstream accuracy remains comparable after multi-node RL.

We evaluate the step-30 checkpoints using the same protocol as the paper. GSM8K and MATH-500 use greedy pass@1. AIME results are mean pass@10 over three evaluation seeds.

ModelMethodGSM8K pass@1MATH-500 pass@1AIME24 pass@10AIME25 pass@10
Qwen3.5-2BBase model9.40%13.40%33.33%26.67%
Qwen3.5-2BUniform75.44% (+66.03)76.20% (+62.80)55.56% (+22.22)38.89% (+12.22)
Qwen3.5-2BTailSieve74.60% (+65.20)75.20% (+61.80)57.78% (+24.44)36.67% (+10.00)
Qwen3.5-4BBase model89.61%84.40%90.00%82.22%
Qwen3.5-4BUniform89.46% (−0.15)89.00% (+4.60)90.00% (+0.00)81.11% (−1.11)
Qwen3.5-4BTailSieve89.23% (−0.38)89.40% (+5.00)90.00% (+0.00)81.11% (−1.11)
Reviewer_7fXTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to route requests across replicas in synchronous LLM rollouts, where a step completes only when the slowest generation finishes, so a few long-tail responses dominate the step makespan. The authors first analyze an offline oracle that knows all response lengths in advance and find that the optimum follows a "tail-isolation-with-balancing" structure, with a simple top-k isolation policy coming within 4% of the optimal makespan on their sampled cases. Based on this, they propose TailSieve, which uses partial-rollout completion status as a training-free signal to identify tail-prone prompt groups, carries only prompt IDs into the next round, and regenerates their responses from scratch under the current policy.

Critical strengths
  • The problem formulation and the offline analysis give the system a principled basis
  • The tail signal is cheap and keeps training on-policy
  • The evaluation is broad for a systems paper of this scope
Critical weaknesses
  • All results come from a single node with eight H100 GPUs running a customized vLLM development build with local Qwen3.5 and DFlash patches. There is no multi-node, heterogeneous-replica, or standard RL-framework (e.g., verl-style) evaluation, so it is unclear how much of the benefit survives the communication and scheduling conditions of production RL training.

  • The baselines are routing-only adaptations rather than the actual systems they are named after. The StreamRL-style and Seer-style oracles are re-implemented with historical length estimates on the same vLLM executor, while systems that reorganize rollout rounds or reuse trajectories, such as RollPacker, are explicitly placed out of scope.

Questions for authors
  • How many rollout steps and how much wall-clock time does the hierarchical controller need to converge from a uniform-routing cold start ?
  • Why is there no end-to-end comparison with a real long-tail system such as RollPacker on the same hardware and workloads? If such a comparison is infeasible, can you argue more directly why the historical-length oracle adaptations in App. A.2 are fair stand-ins for those systems rather than weaker variants of them?
5 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer 7fXT (W1)

[W1 — RL framework and multi-node evaluation]

We thank the reviewer for raising the question of how TailSieve performs under practical RL training conditions. We address the implementation and scalability questions in two parts. (1) We clarify the verl and vLLM setup and report the single-node timing breakdown. (2) We are working on the multi-node evaluation and expect to report the results by the end of this week. Thanks again for suggesting us to evaluate TailSieve at a larger scale!

(1) TailSieve delivers an end-to-end RL speedup of up to 1.271× over Uniform in verl.

We clarify that our RL training experiments use verl, with vLLM as the rollout backend. In the revision, we will document the verl version and its integration with vLLM. Here, we compare end-to-end wall-clock times for the Uniform baseline and TailSieve. For TailSieve's training, we simply group responses of similar lengths into microbatches. We will include this end-to-end timing comparison in the updated version.

ModelTime componentUniform (s)TailSieve (s)Speedup
Qwen3.5-2BRollout3,580.312,474.591.447×
Qwen3.5-2BTraining1,884.901,782.601.057×
Qwen3.5-2BInitial warm-up round-44.29-
Qwen3.5-2BE2E5,465.214,301.481.271×
Qwen3.5-4BRollout3,242.912,866.151.131×
Qwen3.5-4BTraining2,275.202,094.851.086×
Qwen3.5-4BInitial warm-up round-120.06-
Qwen3.5-4BE2E5,518.115,081.061.086×

(2) Multi-node evaluation is underway, with results expected by the end of this week.

We are currently running multi-node experiments to evaluate how TailSieve’s rollout gains carry over across nodes. We really appreciate the reviewer’s suggestion to evaluate TailSieve at a larger scale, and we expect to report the results by the end of this week. : ) We look forward to discussing these results with you!

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer 7fXT (W2, Q2)

[W2/Q2 — Routing-style baselines and comparison with RollPacker]

We appreciate the reviewer’s suggestion to clarify our baseline comparisons and include an evaluation with a complete rollout system. We address these concerns in two parts. (1) We explain the rationale for comparing TailSieve with the Seer-style and StreamRL-style routing baselines. (2) We report a direct comparison with the official RollPacker implementation.

(1) TailSieve outperforms both routing baselines even when they are given oracle signal by up to 1.358×.

Yes, we would have liked to compare TailSieve directly with Seer [1] and StreamRL [2]. But unfortunately, we could not find complete public implementations of either system. For a comprehensive and fair comparison, we dedicatedly reimplemented the part most closely related to our work—their routing policies—and evaluated them on the same rollout backend as TailSieve.

The use of historical length, as we show in Appendix C.3 that prompt identity explains most of the variation in response length, is an even stronger variants of both baselines without relying on a length predictor. Besides, our routing comparison follows Seer’s oracle-based evaluation protocol. We believe the gains over these stronger baselines provide further evidence that TailSieve’s routing policy is effective.

(2) TailSieve achieves up to 3.234× rollout speedup and 2.154× end-to-end speedup over RollPacker.

RollPacker [3] was excluded from our routing-policy comparisons because its approach changes the output-length distribution within each round. However, we appreciate the reviewer’s suggestion to include its implementation in our analysis and provide a detailed comparison with RollPacker. We evaluate the official RollPacker implementation [3] under the same workload settings as our RL experiments, using the same eight-GPU server, prompt sequence and generation sampling parameters.

ModelMetricTailSieve (s)RollPacker (s)Speedup (RP/TS)
Qwen3.5-2BE2E training time4,301.489,266.592.154×
Qwen3.5-2BRollout generation/collection2,474.598,002.043.234×
Qwen3.5-4BE2E training time5,081.066,175.721.215×
Qwen3.5-4BRollout generation/collection2,866.153,708.931.294×

The rollout advantage is larger than the E2E training advantage. Our implementation uses a straightforward synchronous verl-based pipeline, whereas RollPacker overlaps generation and training, helping hide training costs. We also observe a larger rollout-time gap during the longer-response stages of 2B training, suggesting that these workloads remain challenging for RollPacker. We will provide a detailed analysis of this trend in the revised version. We will clarify the scope of the routing-style baselines and include these additional results in the revision.

References

[1] Ruoyu Qin et al. Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning. OSDI, 2026.

[2] Yinmin Zhong et al. StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation. arXiv:2504.15930, 2025.

[3] Wei Gao et al. RollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching. NSDI, 2026.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer 7fXT (Q1)

[Q1 — Controller adaptation from a Uniform-routing cold start]

To answer your question, we ran the controller from a fully Uniform start and additionally report two other initial replica splits.

(1) From a Uniform start, the controller reaches near-converged performance in three rounds, and all three starts converge to a 2:2 split.

From a fully Uniform start, the rollout time came within 10% of the best observed time after three rollout rounds and 393 s. All subsequent adaptation rounds stayed within that range.

Starting from (m, q) = (0, 0), the controller followed (0, 0) → (1, 1) → (1, 2) → (2, 3). All rounds, including both transition rounds, were faster than the initial Uniform rollout. The controller reached its stopping criterion after eight tail rounds and 20.41 minutes, including initialization and transitions. All three runs reach the same 2:2 replica split, with slightly different final quotas, consistent with sampling variability in response lengths and measured completion times.

Initial bulk:tailRounds to convergenceFinal (m, q)TimeSpeedup vs. UniformTime to convergence
3:113(2,4)118.61 s1.267×29.44 min
1:39(2,4)116.57 s1.289×20.73 min
Global Uniform8(2,3)121.10 s1.241×20.41 min

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow up Response to Reviewer 4R6A

Dear Reviewer 4R6A,

Thank you for your updated evaluation and for raising the score, which affirms the contributions of our work. Your thoughtful suggestions helped us better isolate the contribution of tail-aware routing and provide a more complete evaluation of TailSieve’s practical benefits.

Your feedback has been instrumental in enhancing the quality of our manuscript and we sincerely appreciate your time, careful review and support!

Best regards,
The authors

AuthorsOpenReview ↗

General Response

For clarity, we refer to Reviewers 7fXT, emyj, and 4R6A as R1, R2, and R3, respectively.

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions. We are encouraged by their recognition of the practical importance of reducing tail latency in synchronous RL rollouts and the breadth of our evaluation (R1, R2, and R3). In particular, R1 and R3 highlight how the routing formulation and offline analysis motivate tail isolation. R2 and R3 recognize the practicality of identifying likely stragglers without training a separate length predictor, and R1 and R2 note that regenerating responses under the current policy keeps training on-policy. We also thank R2 for highlighting the gains from adaptive replica allocation and the compatibility of tail routing with speculative decoding, and R3 for recognizing the cross-checkpoint evidence that prompt-level tail behavior persists as the policy changes. R2 and R3 found the presentation clear and easy to follow. Finally, we are glad that R1, R2, and R3 all found the paper informative and gained useful insights into how long-tail responses affect synchronous RL rollouts and how tail-aware routing can reduce their impact.

We have addressed each reviewer’s questions in our individual responses. Below, we summarize our core contributions and updated experimental results.

Core Contributions of Our Work

  • Tail isolation approaches the offline optimum. We formulate rollout routing as a step-level makespan minimization problem and show that isolating the top-kk longest requests and balancing load across replicas yields a makespan close to the exact offline optimum.
  • Simple tail identification and two-loop routing speed up math and coding rollouts. TailSieve identifies likely stragglers from partial rollout completion and uses two control loops grounded in our routing analysis to adapt the tail quota and replica split. This design reduces rollout time across both math and coding workloads, whose response-length distributions differ.
  • TailSieve speeds up both step-wise rollouts and end-to-end RL training. In step-wise experiments, routing alone achieves up to 1.67× rollout speedup over Uniform. In end-to-end RL, TailSieve achieves up to 1.271× speedup over Uniform and 2.154× over RollPacker on a single node. In a multi-replica scenario spanning two nodes, the evaluated configurations achieve up to 1.77× end-to-end speedup over Uniform, with comparable downstream accuracy.

Updates of experimental results during rebuttal

  • End-to-end RL. We added single- and two-node RL timings, downstream accuracy results, and a direct comparison with the official RollPacker implementation. We also show how the rollout-time gap changes as responses grow longer during training (§4.3, Tables 5–6; Appendices F.2 and F.5).
  • Routing and controller ablations. We isolate the contribution of tail isolation, compare rollout performance using the same speculative decoder, and ablate the two controller loops during RL with MTP (§4.2, Tables 2 and 4; Appendices F.1 and F.4).
  • Controller convergence. We added a fully Uniform cold start. Rollout time comes within 10% of the best observed steady-state time in three rounds, and all three starts converge to the same replica split (§4.2; Appendix D.4).
  • Tail identification and accepted-data distributions. We evaluated tail identification across KodCode checkpoints and across RL updates, and compared accepted response-length and reward distributions with Uniform (Appendices C.2 and F.3).

We have carefully addressed each individual comment provided by the reviewers and look forward to further discussion! We thank the reviewers again for their constructive feedback and have highlighted the new results and clarifications in blue in the revised manuscript for ease of reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.