acceptodds
Under review as a conference paper at ICLR 2027

Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning (RL) is commonly used to enhance the capabilities of large language models (LLMs) in solving complex multi-turn tasks. During rollout, agentic RL exhibits a workload pattern distinct from that of traditional RL: multi-turn interactions repeatedly introduce prefill requests, causing prefill and decode workloads to coexist throughout rollout. This workload makes prefill-decode (PD) disaggregation, a design widely used for online LLM serving, promising for speeding up agentic RL rollout. This paper shows that PD disaggregation can improve rollout efficiency by (a) mitigating interference among data-parallel attention modules, (b) enabling stage-specific parallelism, and (c) reducing end-to-end trajectory completion time. However, a fixed prefill-decode allocation may fail to sustain these benefits as workload composition varies across tasks and over time. We therefore present Prorsa, an agentic RL framework for prediction-guided prefill-decode resource allocation. Prorsa observes the lifecycle of every trajectory to forecast short-horizon prefill and decode demand. These forecasts then guide worker allocation decisions and trigger role switching of rollout engines, aligning prefill and decode capacity with anticipated workload demand. Across synchronous and asynchronous agentic RL settings, Prorsa achieves end-to-end rollout speedups of 1.11-4.05 over prefill-decode colocation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Reinforcement Learning, Agentic RL, Prefill-Decode Disaggregation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ersVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper studies the rollout phase of agentic RL and argues that prefill-decode disaggregation is a good fit here, because multi-turn interaction and asynchronous sample injection create a continuous stream of prefill requests mixed with decode.
  • Section 3 characterizes the workload on three tasks and names three benefits of PD disaggregation: less interference across data-parallel attention ranks in MoE engines, separate parallelism per stage, and lower inter-token latency for long-tail trajectories.
  • Prorsa adds a workload predictor that uses the trajectory lifecycle to forecast prefill tokens and decode KV need over a 10s window, and a role switcher that flips rollout instances between prefill and decode roles.
  • Implemented on VeRL + SGLang and evaluated on Qwen3-235B, DeepSeek-V3.2 and MiMo-V2.5-Pro on 128 GPUs. Rollout speedup over PD colocation is 1.11–4.05x on the two agentic tasks; on math it is a slowdown in most settings. Prediction WAPE is 5–14%.
Critical strengths
  • The workload characterization is informative. Agentic tasks keep issuing prefill all through a step, the prefill/decode token ratio moves up to 1.74x across steps and a lot within a step, and colocated MoE engines lose a lot of GPU time to cross-DP-rank interference. This is a clear, measured story that people building RL infra can use right away.
  • The experiments are at a scale very few groups can run, and the paper is honest when the method loses (math task in Sectio 5.2).
  • The predictor is simple and sensible, with online bias correction, and the paper actually measures its accuracy instead of assuming it works.
  • The writing and figures are clear and easy to follow.
Critical weaknesses
  • Most of the headline speedup comes from PD disaggregation itself, not from the new idea. The authors list that GLM-5, MiniMax-M2 and RollArt already use PD disaggregation for agentic RL rollout. The new part, prediction-guided role switching, is isolated only in Fig. 7, only for MiMo-V2.5-Pro, and gives 1.0x–1.44x over a static PD split. So the real contribution is a good characterization plus a 0–44% gain on one model, while the abstract reads like 4.05x. The static-PD baseline should be shown for all three models and all tasks.
  • The only baseline is colocation, and that baseline may be weak. Methods that attack the same interference, like in-place PD multiplexing, or scheduling systems like Libra / Heddle / RollPacker / OrchestrRL are never compared. Recent accepted RL-infra-flavored papers (for example the STRONGER-MAS, or RollArt / StreamRL) position against the strongest alternative, not only against the naive one.
  • No evidence that the training outcome is unchanged. Role switching cancels in-flight requests and resumes them on another instance from the persisted KV cache; in async mode the resumed part can run under a newer policy version. The paper never shows a reward curve, a final accuracy, or an end-to-end training time for Prorsa vs colocation, only rollout time. For an ML venue this is a gap.
  • Reproducibility is low: a proprietary 1T model, a 128-GPU cluster, no code release mentioned, and the GPU-idle metric excludes scheduling overhead and cross-DP imbalance - more like an MLSys submission.
Questions for authors
  • Can you show Fig. 7 static PD vs Prorsa for Qwen3-235B and DeepSeek-V3.2 and for all staleness settings, and add at least one dynamic-scheduling baseline (Libra, Heddle or OrchestrRL) and one PD-multiplexing baseline? This would tell us how much of the gain is really from role switching.
  • Does Prorsa change the trained model? Please report training reward / eval accuracy and total time-to-target for at least one task with and without Prorsa, and explain how the cancel-and-resume path interacts with importance ratios when a trajectory continues under a newer weight version.
  • Multi-agent workflows create even more prefill churn. Does the predictor and the switcher extend to that case, or is the current design tied to one policy and one engine type?
1 comment

We sincerely thank you for your thoughtful feedback. We address each question below.

Q1: Static PD and scheduling baselines.

We have conducted additional experiments and added static PD baselines for all models and tasks (Figure 5 in Section 5.2). Prorsa achieves a 1.00–1.88× speedup over static PD, with speedups of up to 1.55× on agentic tasks. We have also added a comparison with a recent-load prediction baseline (Figure 8 in Section 5.4), showing that Prorsa reduces WAPE by a factor of 1.31–2.51 compared with this baseline.

Q2: Learning quality and policy consistency.

We have added score curves in Figure 11 (Section 5.5). Under the same training settings, Prorsa and PD colocation follow similar learning trends and both reach scores of approximately 80 at the end. Specifically, Appendix C explains how cancel-and-resume retains generated prefixes and KV state to maintain model behavior.

Q3: Extension to multi-agent workflows.

We have added a discussion of multi-agent workflows in Section 4.2. For these workflows, the lifecycle abstraction should track agent branches and aggregate their prefill and decode demand. Moreover, heterogeneous models require separate resource pools and model-specific predictions. We believe that Prorsa can also be applied to multi-agent workflows.

Q4: Reproducibility and evaluation details.

For reproducibility, we have added batch sizes, group sizes, sequence and turn limits, and sampling acceptance rates in Appendix D. We compute the fraction of GPU time lost to P/D interference by dividing the corresponding idle GPU time by the total GPU time across all GPUs over the entire rollout.

Reviewer_AJW2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies prefill-decode (PD) disaggregation for agentic reinforcement learning rollout, where multi-turn interactions create prefill requests throughout the rollout phase. They propose PRORSA, a framework that tracks trajectory lifecycles to predict short-horizon prefill demand and decode KV-cache requirements, then dynamically reallocates rollout workers between prefill and decode roles. Implemented on Verl and SGLang, PRORSA is evaluated on three models and three task categories under synchronous and asynchronous rollout settings. The reported results show 1.11×–4.05× rollout speedups over PD colocation on agentic tasks.

Critical strengths
  1. A useful characterization of PD interference in agentic RL. The paper provides a useful analysis of why PD disaggregation can be particularly beneficial for agentic RL workloads. In particular, the discussion of synchronization-induced GPU idleness across data-parallel attention modules helps explain performance bottlenecks that are not fully captured by considering prefill and decode throughput independently. The analysis offers practical insights for designing rollout infrastructure for long-horizon agentic workloads.
  2. A lifecycle-aware resource allocation mechanism. PRORSA leverages trajectory lifecycle signals, including sandbox startup and tool execution, to anticipate upcoming inference demand. This is a natural systems design for agentic RL, where the framework has visibility into events that precede future model requests. The evaluation includes multiple models, task categories, and rollout settings, together with prediction-accuracy and role-switching ablations.
Critical weaknesses
  1. The contribution of adaptive role switching is less substantial than the headline speedups suggest. In the ablation study, dynamic role switching provides only 1.0×–1.44× speedup over static PD disaggregation, and this comparison is reported for MiMo-V2.5-Pro. It would be helpful to establish more clearly how much of the overall improvement is attributable to the proposed prediction-guided mechanism, rather than to PD disaggregation itself, and whether the adaptive policy consistently improves performance across models and workloads.
  2. The scope of the demonstrated benefits could be better characterized. The reported benefits vary substantially across workloads. In particular, the paper reports slowdowns on the math task in most evaluated settings, suggesting that PD disaggregation is not uniformly beneficial. A more systematic characterization of when the proposed approach is beneficial, including the impact of switching overhead and workload characteristics, would strengthen the paper's conclusions about its applicability to agentic RL.
Questions for authors
  1. How much of the end-to-end speedup comes from PD disaggregation itself versus prediction-guided role switching?
  2. What is the end-to-end overhead of switching rollout workers between prefill and decode roles, including any necessary draining, reconfiguration, and KV-cache management?
  3. Given that PD disaggregation slows down the math workload in most evaluated settings, can the authors characterize the workload conditions under which PRORSA is expected to outperform colocation?
1 comment

We sincerely thank you for your positive feedback and thoughtful questions. We address each question below.

Q1: Gains beyond PD disaggregation.

We have conducted additional experiments and added static PD comparisons across all models and tasks to show the effectiveness of prediction-based role switching (Figure 5 in Section 5.2). Across all settings, Prorsa achieves a 1.00–1.88× speedup over static PD, with speedups of up to 1.55× on agentic tasks.

Q2: Switching overhead.

We have added a switching-time CDF (Figure 7 in Section 5.2). Switch duration is measured from the time a worker receives a role-switching command until it resumes serving. The median is approximately 4 seconds, and the 95th percentile is 8 seconds. Appendix C explains how requests resume from retained prefixes and KV state while model weights remain resident during reconfiguration. These costs are included in the reported rollout time.

Q3: When PD disaggregation helps.

We have expanded Section 5.2 to characterize workload applicability. At staleness 8, the average fraction of time that prefill instances are active is only 9.5%–26.0% on the Math task, compared with 62.1%–98.0% on the Coding Agent task. Math leaves dedicated prefill capacity underused during long decoding periods. Therefore, Prorsa is expected to outperform colocation only when prefill requests coexist with decoding, allowing full utilization of both prefill and decode instances.

Reviewer_nux1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PRORSA applies prefill-decode disaggregation to multi-turn agentic RL rollouts and dynamically reallocates workers using forecasts derived from trajectory lifecycle information. The predictor uses sandbox startup, tool execution, reward completion, and queue state to estimate near-term prefill demand and decode KV-cache requirements. Experiments on three large MoE models report rollout speedups over colocated inference, with a separate ablation studying adaptive versus static disaggregation.

Critical strengths
  1. Useful workload characterization at a relevant scale. The analysis identifies a concrete bottleneck: synchronized MoE execution can leave decoding attention ranks waiting for prefill work. Testing multiple large models and synchronous/asynchronous settings makes this observation practically relevant to agentic RL infrastructure.

  2. A feasible source of predictive information. The RL runtime can observe tool execution and trajectory lifecycle states before inference requests arrive. Using these signals for resource allocation is well motivated, and the implementation demonstrates measurable improvements over static disaggregation in some settings.

Critical weaknesses
  1. The main comparison does not isolate the proposed contribution from the established benefits of disaggregation. The headline 1.11×–4.05× speedups are against PD colocation, and Section 5.3 attributes much of the largest gains to removing cross-DP interference. The more relevant comparison for the new adaptive controller appears only for MiMo-V2.5-Pro, where the gain over an optimized static allocation is 1.0×–1.44× and is near zero on the general-agent task (Figure 7). There is no comparison against a reactive queue/KV-occupancy controller or a forecast based only on recent load. Section 6 already cites DOPD and TokenScale [1, 2], which address dynamic PD allocation or proactive capacity adjustment, but does not empirically distinguish lifecycle prediction from these simpler alternatives. I therefore find the evidence for disaggregation substantially stronger than the evidence for PRORSA's specific novelty.

  2. The forecast-to-allocation model is insufficiently validated against actual work and switching costs. Appendix B explicitly counts full prompt lengths, including cached and already-processed tokens, and excludes queued requests whose prefill is not predicted to finish within the horizon. This quantity is not generally the remaining prefill work: a large cached prefix can inflate it, while a backlog of long requests can be excluded. Equation (1) nevertheless converts it to a worker count using measured prefill throughput. Decode capacity is determined solely from KV occupancy, without explicitly modeling generation throughput, KV growth, or reclamation. These may be workable approximations, but prediction WAPE alone does not validate their scheduling consequences. Moreover, Appendix C reports roughly five seconds per role switch against a ten-second forecast horizon, while the allocation objective does not explicitly account for that cost. The paper needs evidence that the controller remains beneficial when cache reuse, workload mix, or switching frequency changes.

[1] Liao et al. DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving.

[2] Lai et al. TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity.

Questions for authors
  1. Novelty and baseline fairness: At identical GPU budgets and stage-specific parallelism, how does lifecycle prediction compare with a reactive queue/KV controller and a recent-load forecast? Can the adaptive-versus-static comparison also be reported for the models producing the largest headline gains?

  2. Workload definition: Is prefill throughput in Equation (1) measured in the same full-prompt units as Appendix B, including cache hits? How does the implementation avoid reducing prefill capacity when queued requests are excluded because their predicted completion lies beyond the horizon? Please clarify the minimum-capacity and backlog safeguards.

  3. Switching overhead: How many switches occur per rollout, how much time and KV traffic do they consume, and are these costs included in all reported timings? What prevents repeated reallocations when demand fluctuates on the same timescale as the approximately five-second switch?

  4. RL-level comparability: Please specify the GPU model/memory, rollout batch and concurrency settings, repetitions, and number of training steps underlying the averages. Under asynchronous rollout, are accepted-trajectory counts, staleness distributions, and drop rates comparable? A reward-versus-wall-clock check would establish whether faster rollout translates into faster learning at comparable quality.

1 comment

We sincerely thank you for the detailed review. We address each concern below.

Q1: Baseline fairness.

We have conducted additional experiments and added the static PD baseline for all models (Figure 5 in Section 5.2), showing that Prorsa consistently outperforms the static PD baseline across various settings. Although static PD can already improve rollout performance on agentic tasks compared with PD colocation, Prorsa provides an additional speedup of up to 1.55×. Moreover, Prorsa can reduce GPU idleness on the math task compared with static PD, improving rollout efficiency. We have also compared the prediction accuracy of Prorsa and a recent-load baseline (Figure 8 in Section 5.4). Compared with this baseline, Prorsa can reduce WAPE by a factor of 1.31–2.51.

Q2: Workload definition and backlog safeguards.

We have clarified in Section 4.2 and Appendix B that prefill demand and throughput use the same full-prompt token-counting convention, including cached prefixes. Cache reuse is reflected in the measured throughput.

We chose to predict full prompt length rather than the number of uncached prefill tokens for two reasons. First, the number of uncached tokens alone cannot capture prefill cost, which also depends on the length of the cached prefix. For example, with the uncached length fixed at 16K tokens, increasing the cached context from 200K to 400K tokens increases prefill time by a factor of 1.5 for MiMo-V2.5-Pro. Because cache-hit rates are relatively stable across requests, full prompt length reflects prefill time more accurately.

Second, full-prompt prediction has lower error. We have added a comparison in Figure 9, which reports WAPEs of 9.31% for full-prompt prediction and 15.95% for uncached-prefill prediction.

Regarding backlog safeguards, we have clarified in Equation (1) and Appendix C that at least one instance of each role is retained. Appendix B includes all queued and in-flight prefill requests in the demand estimate, and only the decode-admission forecast uses the predicted prefill-completion cutoff.

Q3: Switching overhead and frequency.

We have added a switching-time CDF (Figure 7 in Section 5.2) and a role-switching trace (Figure 14 in Appendix C). The trace shows an average of 6.5 switches per step. The median switch duration is approximately 4 seconds, and the 95th percentile is 8 seconds. Switching costs are already included in the end-to-end rollout time.

To avoid repeated reallocations when demand fluctuates, we adopted a cooldown mechanism that enforces a minimum interval of 60 seconds between consecutive role switches. Its description has been added to Appendix C.

Q4: RL-level comparability.

We have provided batch and group sizes, sequence and turn limits, and sampling acceptance rates in Appendix D to support reproducibility. We have also added training score curves for Prorsa and PD colocation in Figure 11 in Section 5.5. Prorsa and PD colocation follow similar learning trends and both reach scores of approximately 80 at the end, supporting comparable learning quality with role switching.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.