acceptodds
Under review as a conference paper at ICLR 2027

AdaRollout: Accelerating Synchronous RLVR with Adaptive Parallelism Switching

Abstract

Reinforcement learning with verifiable rewards (RLVR) relies on large-scale rollout generation, yet its performance is often bottlenecked by synchronous execution that is highly sensitive to long-tail decoding latency. Stochastic decoding produces sequences with widely varying lengths, where a few stragglers can stall batch completion and lead to poor accelerator utilization. This issue is exacerbated under static parallelism: as short sequences complete, the workload drastically shifts from high concurrency to long-tail dominated execution, rendering the initial parallel strategy inefficient. In this paper, we present AdaRollout, a system for state-preserving rollout re-parallelization that dynamically reconfigures the execution topology during rollout without restarting in-flight requests. AdaRollout profiles batch-completion throughput offline and derives a lightweight runtime policy that uses unfinished requests as a proxy for the remaining workload while accounting for reconfiguration cost. It further enables efficient topology reconfiguration through architecture-aware weight resharding, cross-topology KV-cache migration, and pre-instantiated runtime resources. By dynamically reallocating resources to long-tail sequences, AdaRollout achieves substantial end-to-end training speedup over optimal static parallelism, while maintaining consistent training dynamics (reward RMSE ) and downstream performance (accuracy RMSE ).

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Dynamic Parallelism, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7FKqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for switching parallelism based on the generation status, using the number of unfinished requests ntn_t as the only online decision metric to determine whether to switch from DP-dominated to TP-dominated parallelism. It achieves speedups by adapting to substantial variation in rollout generation lengths and the resulting stragglers, with reported P99/P50 ratios of 2.74–4.27 after excluding truncated samples.

Critical strengths
  • Switching parallelism during an ongoing rollout to adapt to changing concurrency is a useful systems idea for mitigating straggler-related inefficiency.
Critical weaknesses
  • Insufficient specification and empirical support for the switching policy. The paper does not adequately explain how the “profiled tail workload,” throughput measurements, and switching costs are combined to obtain the unfinished-request threshold. In particular, it does not provide sufficient empirical evidence that the preferred configuration can be characterized by a single threshold after accounting for switching costs. This limits my ability to assess the justification and reproducibility of a central component of the method.

  • Insufficient comparison with request rebalancing and evaluation of broader parallelism choices. The paper lacks a comparison against a fixed DP8/TP1 baseline with request/KV-cache rebalancing, such as this implementation, making it difficult to assess the additional benefit of topology switching over improved load balancing. The evaluation is also limited to the predefined DP8/TP1 → DP2/TP4 switch and does not establish whether the proposed policy can select and switch among a broader set of parallelism configurations. This leaves the generality of the selection and switching policy insufficiently validated.

Questions for authors

Q1: Does LL denote the generated output length of still-active (unfinished) requests? I find the phrase “remaining output length” very confusing, since it would be difficult to measure, as argued in the paper. I would interpret it as the generated output length of the remaining (i.e., unfinished) requests.

Q2: Based on Q1, why is a threshold n∗n^{\ast} justified? What empirical evidence supports the monotonicity of the preferred parallelism configuration with respect to ntn_t after accounting for switching costs? Please define “profiled tail workload” precisely and provide the corresponding workload information, switching costs, and throughput comparisons at different values of ntn_t. Please show how these measurements lead to the threshold n∗=64n^{\ast}=64, and clarify how the resulting empirical boundary is consistent with the use of min⁡\min in the threshold definition and the runtime condition nt≤n∗n_t\leq n^{\ast}.

Q3: How are throughput and switching costs measured or estimated? Please specify the profiling protocol.

Q4: Please address the following two aspects of the evaluation:

  • Comparison with request rebalancing. Does the static DP8/TP1 baseline already include request/KV-cache rebalancing across DP replicas? If so, please describe the policy and configuration. Otherwise, please compare AdaRollout against a fixed DP8/TP1 baseline with such rebalancing enabled. Please use the same hardware and workload, keep other runtime optimizations comparable, and report total rollout completion time.

  • Selection and switching among more parallelism configurations. Does the proposed profiling- and cost-based policy support selection and switching among more than two parallelism configurations? Please evaluate this capability with a broader candidate set, for example DP8/TP1, DP4/TP2, DP2/TP4, and DP1/TP8 under the same device budget, and explain how the policy determines which configuration to use and when to switch. Please report the total rollout completion time, and compare the results with the reported two-configuration policy (DP8/TP1 → DP2/TP4). If the current method is restricted to two predefined configurations, please clarify this limitation.

Please upload anonymized profiling data so that I can assess the throughput and switching-cost measurements and reproduce the threshold-selection procedure, including the derivation of n∗=64n^{\ast}=64. These materials are necessary for me to assess the empirical justification and reproducibility of the proposed policy. Without adequate supporting evidence and clarification, I remain concerned about the transparency of the method and the reproducibility of its claimed benefits.

Reviewer_GBhAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the long-tail effect in synchronous RL training. It proposes switching from DP to TP when the number of unfinished sequences falls below a threshold. This allows the remaining long-tail requests to utilize more accelerators, speeding up generation. Experiments show that the proposed approach accelerates rollouts while maintaining accuracy.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The motivation, i.e., the mismatch between static parallelism and changing rollout concurrency, is clear.
Critical weaknesses
  1. The evaluation is limited to relatively small, dense models (8B and 14B). Since switching overhead depends on model size and architecture, the net benefits for larger models and MoE architectures remain unclear. The overhead reporting also requires clarification: the DP→TP components sum to 9.3 seconds, whereas the reported subtotal is 8.3 seconds.
  2. Comparisons focus on static parallelism, without directly evaluating against other approaches to mitigating rollout stragglers.
Questions for authors
  1. How does switching overhead scale with model size, particularly for MoE architectures?
Reviewer_GjWNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdaRollout addresses long-tail inefficiency in synchronous RLVR by switching the rollout parallel topology as the number of unfinished requests decreases. It uses an offline-profiled concurrency threshold to transition from a throughput-oriented DP configuration to a latency-oriented TP configuration. The system preserves in-flight decoding through architecture-aware weight resharding, cross-topology KV-cache migration, and pre-instantiated runtime resources. Experiments on Ascend NPUs report full-step speedups of 1.12× and 1.30× for Qwen3-8B and Qwen3-14B, respectively, together with comparisons of training dynamics and switching overhead.

Critical strengths
  1. The system addresses a concrete mismatch between rollout workload and static parallelism. High concurrency favors aggregate throughput, whereas the final few requests can benefit from lower per-request latency. Reconfiguring within an ongoing rollout is a useful approach that preserves synchronous execution without truncating or deferring trajectories. The implementation addresses important practical issues, including GQA-aware weight partitioning, KV-cache remapping, and decoding-state restoration.

  2. The reported full-step speedups indicate that rollout improvements can translate into training efficiency after accounting for other stages. The switching-overhead breakdown and component ablations also help explain why state migration and runtime-resource reuse matter.

Critical weaknesses

see below

Questions for authors

Baseline clarification: Which static TP/DP configurations were evaluated for each workload, and what were their runtimes? Were all configurations tuned under the same hardware and memory constraints?

Policy robustness: How sensitive are speedups to the unfinished-request threshold? Was the same threshold reused across model sizes, generation-length distributions, and training checkpoints? Please also clarify the min definition of the threshold in §3.1 relative to the runtime condition (n_t \leq n^*).

Cost accounting: Are the reported full-step measurements inclusive of both topology transitions, weight redistribution, KV migration, and synchronization? What additional initialization time and memory are required for pre-instantiated resources?

Quality evidence: Are longer or multi-seed training results available? A confidence interval on the performance difference, together with a predefined noninferiority margin, would materially strengthen the claim that topology switching preserves training quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.