acceptodds
Under review as a conference paper at ICLR 2027

One Static Schedule Does Not Fit All: Platform-Aware and Runtime-Adaptive Scheduling for Agentic RL Rollouts

Abstract

Agentic reinforcement learning (RL) is increasingly important for advancing large language models, yet synchronous rollouts suffer from severe long-tail latency and poor resource utilization. Existing systems mitigate this bottleneck with predict-then-schedule policies that estimate trajectory durations and prioritize likely stragglers. However, two challenges limit their effectiveness. First, the optimal schedule is platform-dependent: LLM request latency depends on continuous-batch composition, while tool operations contend for parallel executors. Second, token-count and tool-latency predictions inevitably deviate from realized execution, potentially making a static schedule ineffective or even harmful. We present STARS, a simulator-driven framework for platform-aware and adaptive rollout scheduling. Positioned between the agentic RL framework and execution engines, STARS controls LLM admission and tool dispatch without modifying the training algorithm or underlying engines. STARS profiles the target LLM engine and tool pools offline, then uses simulation-based planning to evaluate the counterfactual effects of candidate scheduling decisions and generate a platform-specialized ranked plan for each rollout batch. The plan accounts for composition-dependent LLM latency and tool-executor parallelism. At runtime, an impact-triggered repair mechanism detects consequential deviations and incrementally reschedules the uncommitted portion. STARS exposes a policy-agnostic interface while providing a concrete simulator-guided policy. Experiments across diverse platform configurations demonstrate an average 1.12X speedup in rollout throughput over state-of-the-art synchronous RL systems.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Machine Learning Systems, Agentic Reinforcement Learning, Rollout Scheduling, Platform-Aware Scheduling, Runtime-Adaptive Scheduling, Simulation-Guided Planning, Straggler Mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KiDtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes STARS, a scheduling layer for synchronous agentic RL rollouts. A platform-profiled simulator creates ranked LLM-admission and tool-dispatch plans using remaining critical-path estimates and cross-resource externalities. An impact-triggered controller repairs the uncommitted plan when observed execution diverges from predictions. Evaluation replays a fixed batch of 192 recorded agent trajectories across resource configurations. The reported average speedup over veRL is 1.09x across the plotted configurations and 1.12x after excluding the severely tool-constrained T=2 configurations; a constructed deviation case shows a further 4.26% makespan reduction from runtime repair.

Critical strengths
  1. The paper identifies a useful systems issue: trajectory priority depends on both LLM batching and downstream tool capacity. The examples in Section 2.3 make the mechanism concrete, and the plan-transfer experiment in Section 5.2.1 provides empirical support beyond the examples: target-specific plans outperform transferred plans, with little benefit when a serialized tool bottleneck removes scheduling flexibility. This helps explain when scheduling changes can matter.

  2. The interface and evaluation include practical considerations. STARS changes admission order without changing the learning algorithm or execution engines, and identical replayed trajectories control for workload differences between schedulers. The paper reports absolute makespans, occupancy, weak-benefit configurations, and planning/repair overheads rather than speedup alone (Appendices B-C). These details make the observed efficiency improvement useful to systems practitioners.

Critical weaknesses
  1. The experiments do not establish the incremental value of the main scheduling mechanism over prediction-based priority scheduling. Section 5.1 compares only against veRL's readiness/FIFO behavior. There is no baseline using the same predictions with remaining-critical-path priority alone, nor an ablation of the LLM/tool externality terms in Section 4.1. The platform-transfer experiment supports configuration dependence, but does not resolve whether the proposed counterfactual scoring is necessary for the reported gains. This leaves the central advance over the prediction-based schedulers motivating the paper insufficiently demonstrated.

  2. Evidence for robust runtime adaptation is substantially narrower than the framework's scope. Appendix A uses one fixed replay batch; Section 5.3 changes one trajectory's suffix and reports repair benefit for one condition. Table 6 covers twelve conditions, but reports repair activity and overhead rather than improvements in repair-versus-frozen makespan. The paper also does not quantify prediction errors or simulator fidelity against live execution, both of which directly determine whether proposed repairs help. Replay is a useful controlled comparison, but the present evidence does not establish reliable benefits under representative prediction errors. These two gaps motivate a weak-reject recommendation on the current evidence.

Questions for authors
  1. With identical forecasts and resource constraints, how does STARS compare with a remaining-critical-path-only scheduler? Can the existing experiments isolate the contribution of the LLM and tool externality terms, and specify the scoring weights and lookahead horizon used?

  2. In the replay experiments, are composition-dependent LLM latencies measured by executing on the backend or produced by the profiled simulator? What held-out validation measures the simulator's error and the historical estimator's error without access to final replay outcomes?

  3. What are the net repair-versus-frozen makespan changes for all twelve Table 6 conditions, and which trigger/acceptance thresholds were used? Do naturally occurring prediction errors show the same benefit as the constructed suffix-extension case? These clarifications would directly affect the assessment of the adaptive component.

1 comment

We thank the reviewer. Repair is a recovery path, not a source of the reported gains — every headline number comes from the initial plan alone, and repair is evaluated separately (Section 5.3).

Q1: CP-only comparison, externality ablation, and score parameters

Event-CP is the remaining-critical-path-only scheduler: same forecasts, same resources, priority by remaining critical path, no candidate-branch counterfactual. At T=4 STARS wins against it in all eight capacities of the simulated sweep (4–64 accelerators), at 1.0717× mean speedup and a 6.54% mean makespan reduction; at (L=4, T=4) the reduction is 5.42%.

Section 4.1 makes all three corrections one-sided: the LLM batch externality sums only positive step-latency increases, tool wait is clipped at zero, and tool externality sums only positive increases in other tool phases' completion times. Each term is identically zero when its state is absent, so each acts in the regime its definition selects. In the 72-run, 27-contrast same-policy study, tool externality is positive in all nine backgrounds (mean 7.48%, median 6.83%), while removing all three corrections — keeping only R(c) — is materially worse: the gain is carried by the tool-externality term. H=8 token steps and three unit weights, fixed across cells.

Q2: Source of LLM latencies and held-out errors

Composition-dependent latencies come from the profile-driven simulator, but nothing it consumes is synthetic: the profile is measured on the target backend and engine, and every token count, tool service time, and dependency in the traces is recorded from real executions on that hardware (Section 5.1). The simulator supplies only their interleaving. The step costs are measured, not fitted — a new accelerator changes the table, not a parameter; only composition is learned. On the 53 requests held out by request ID (111 used for fitting), that model's residual is 18.42% MAPE on request service time, after measured engine queueing is removed.

The historical estimator gets the totals close — on the natural-history batches its predicted decode tokens and tool service land within about 5% and 20% of realized values — but it cannot place that mass per request: per-request error runs in the tens of percent, and above 100% for tool service. A plan built on absolute durations would inherit that spread.

The estimator is tested without access to outcomes: each forecast uses only histories completed before the common cutoff plus the public first-prompt token count.

The planner commits to ranks, not timings (Section 3.3); every arm shares one profile, so the model must respond correctly to composition and the residual error must be common. Section 5.2.1 tests exactly that: a plan built for one configuration loses 6.03% on average and up to 14.2% when transferred, and the advantage disappears at C=(4,1), where a single executor serializes the pool.

The evidence rests on one fixed batch, but not a single workload family: it mixes DevOpsGym, SWE-bench Verified, Terminal-Bench 2.1, and Terminal-Bench Pro.

Q3: Twelve repair conditions, thresholds, and natural errors

The twelve constructed conditions stress-test turning a forecast break into a shorter suffix; positive values mean repair shortens the rollout:

Condition(L=1, T=4)(L=2, T=4)(L=4, T=4)
zero-error0.00%0.00%0.00%
tail-32+0.28%+0.29%+1.41%
tail-113+0.27%+0.28%+3.16%
tail-113-decode8+0.29%+0.22%+4.26%

Protocol: 25% deviation threshold, 120 s coalescing and cooldown, 1 s gain margin, 60 s per-attempt control budget, 64 global attempts, two structural retries, two insertion overrides, fees frozen after calibration. The gradient is the mechanism: repair helps only when the frozen plan leaves room for a different order.

The held-out residual is a timing error, not a ranking error: the model is deliberately flat where the decision cannot matter, and the zero-error rows return exactly 0.00%. Natural-error benefit appears only where a batch contains a recoverable tail, so we report it as an operating envelope rather than a deployment promise. The revision will state the planning–repair separation explicitly in Section 5.1.

Reviewer_Q53bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a platform-aware and runtime-adaptive scheduler (STARS) for synchronous agentic RL rollouts. Several techniques, including platform profiling, simulator-based planning, and selective runtime repair, are integrated into the scheduler. The scheduler jointly considers factors including task duration, how LLM requests affect speed together, and whether tool calls need to wait. Experimental results show that STARS is more efficient than both veRL and the variant where too few tool workers are excluded.

Critical strengths
  • Jointly considering how LLM requests affect speed together and whether tool calls need to wait is well-motivated. The modular design is also compatible with existing rollout frameworks.
  • Identical trajectories are replayed in the comparison between different schedulers. The authors also report the initial planning cost and the configurations where STARS does not help.
Critical weaknesses
  • Stronger baselines are missing. The authors only compare STARS with veRL (which has FIFO/work-conserving scheduling). No prediction-based scheduler is compared. No complicated scheduling heuristic is compared (e.g., the longest remaining path). Moreover, ablation studies do not fully isolate the contribution of LLMs and tools.
  • The practical evidence for runtime adaptivity is not that sufficient. The planning experiment reuses a fixed batch of 192 trajectories, and the runtime-benefit experiment is based on a constructed tail deviation.
Questions for authors
  • Could you compare STARS with more complicated scheduling heuristics (e.g., the longest remaining path)?
  • Are replay makespans measured on execution engines or computed using a simulator?
1 comment

We thank the reviewer. Both questions ask one thing: whether the ordering decision, not a per-candidate priority, is what carries the gain. Section 2.3.1 argues it must be; the added runs confirm it under identical forecasts, resources, and work.

Q1: More complicated scheduling heuristics

We add both. Time-L is the longest-remaining-work rule; Event-CP is the dynamic remaining-critical-path scheduler, which re-prioritizes at every event rather than ranking once from a static estimate, and is therefore the harder of the two to beat.

At T=4 — between the tool-saturated (T≤2) and tool-slack (T≥6) regimes — STARS wins at all three of the paper's LLM capacities:

Configurationvs Time-Lvs Event-CP
(L=1, T=4)12.8%10.8%
(L=2, T=4)12.3%6.6%
(L=4, T=4)6.4%5.4%

The wider simulated sweep tells the same story: over eight LLM capacities (4–64 accelerators) the win holds in all eight cells and peaks at 15.9% against Time-L in the middle of the sweep. At (L=1, T=4) the 12.8% corresponds to 40,116 s → 34,971 s — 86 minutes off a 192-trajectory batch. Over the three paper capacities the mean reduction against Time-L is 10.5%; over all eight swept capacities it is 7.85%. The gain is substantial in many of these cells rather than concentrated at the peak, and the revision will describe that spread alongside the means.

The shape is the mechanism. Section 2.3.1 shows the optimal order reversing across platforms for identical trajectories, so a rule that ranks a candidate by its own remaining path carries a bounded ceiling, and the ceiling is highest where LLM batching and tool contention bind together. Time-L is beaten in every cell of the main sweep, and the win also holds against the fastest baseline we measured in each cell at T=4.

Q2: Source of replay makespans

All replay makespans are computed by the simulator. The traces fix token counts, tool service times, and dependencies, and none of those values is synthetic: each is recorded from real executions of the agent on the target hardware and engine (Section 5.1). The simulator supplies only how they interleave under a given order, resolving admission, batch composition, queueing, and tool dispatch against a latency model profiled on that same backend. Every policy runs on the same world, so the difference between arms is the scheduling decision alone. Section 3.3 is explicit that the planner emits ranked priorities, not a timetable, so the property under test is ordinal and the measured profile supplies the ordinals.

The added comparisons run on the paper's fixed evaluation batch. That batch is not a single workload family — it mixes DevOpsGym, SWE-bench Verified, Terminal-Bench 2.1, and Terminal-Bench Pro — so the margins above are measured across four task distributions rather than one, and the revision will state that composition explicitly.

The grid-wide mean is smaller than the peak because most cells sit outside the band where any order has leverage: at T=2 the tool pool saturates, by T≥6 the contention the terms measure is gone and they switch off. T=4 is the regime the mechanism predicts, and it is what the table reports. Both arms share one world, which is what makes the difference the decision alone, and the latency model is profiled on that backend rather than fitted.

Reviewer_ZRJeOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes STARS, a framework for scheduling model requests and tool calls during agentic RL rollouts. It aims to reduce delays caused by slow tasks and busy resources. STARS uses a simulator to choose an execution order that fits the available hardware. It can change the remaining schedule when tasks do not finish as expected. The authors test it by replaying recorded agent runs and report an average speedup of 1.09× over veRL baseline.

Critical strengths
  1. The paper tries to address an important problem, as slow agent runs can leave expensive computing resources unused. Reducing this waiting time could lower the cost of agentic RL.

  2. The design keeps agent scheduling separate from training, as it aims to change the order of operations without changing the learning algorithm or the underlying execution.

Critical weaknesses
  1. The paper evaluates rollout scheduling via recorded trace replay. It does not demonstrate the method within an actual RL training run. Section 5.1 calls the collected traces a “training dataset”, but the described experiments use them for replay. Replaying recorded runs provides controlled comparisons, but does not show how the method behaves as the policy changes during training.

  2. There is insufficient evidence of the benefits that the proposed method offers over simpler scheduling methods. The experiments compare STARS with the basic scheduling policy of veRL. A straightforward baseline would be to introduce a simple rule that starts tasks with the longest remaining work first, or with the existing schedulers discussed in the paper. The results do not establish how much the proposed method adds (improvements are very modest).

  3. The simulator does not include a claer accuracy check. You should report how closely STARS timing predictions match measured execution across different batch compositions and resource configurations.

  4. The experiments do not show which parts of the method are needed. The scheduling score combines several terms, and the repair process uses several checks. Without providing an ablation, it is unclear whether the added complexity offers gains.

  5. The method is difficult to follow because important choices are described in broad terms. The paper introduces score weights, a simulation horizon, and repair thresholds without clearly defining them, giving their values or explaining how they were selected.

Questions for authors
  1. Does STARS maintain its scheduling gains during actual RL training as the policy changes? Can you provide experimental results across training iterations to support this?

  2. How does STARS compare with a simple scheduler that prioritizes the longest remaining work, using the same predictions and resources?

  3. Which parts of STARS are needed to improve on this simple scheduler baseline? Please provide an ablation of the main STARS components.

  4. How closely do the simulator’s predictions match measured execution times on workloads outside its calibration data? Does it correctly predict which schedule will finish sooner?

  5. Please provide information on the score weights, simulation horizon, and repair thresholds. How were they selected, and were they kept fixed across the experiments?

1 comment

We thank the reviewer. For a fixed batch, forecasts, and platform, a platform-conditioned order beats one derived from trajectory attributes alone.

Q1: Actual RL training and policy drift

The replay is the control, not a substitute for it: Section 5.1 fixes token counts, tool service times, and dependencies while admission order, batch composition, queueing, and cross-resource overlap vary with the policy. In a training run, policy and workload move together, so attributing a speedup to the scheduler needs that control first.

The tail comes from heavy-tailed durations meeting a synchronous barrier (Section 2.1), so a new policy moves the distribution but the barrier still yields a straggler-dominated makespan. Drift enters through the estimator, whose interface (Section 3.2) is swappable.

Q2: Longest-remaining-work baseline

Section 2.3.1 shows why it cannot be platform-correct: identical trajectories have opposite optimal orders at T=1 and T=2, and crossing them costs 13.3% and 11.4%. An optimal order is therefore not a function of trajectory attributes, and a rule that ranks a candidate by its own remaining path — longest-remaining-work, or the dynamic remaining-critical-path rule (Event-CP) — expresses urgency but not the cost an admission imposes on in-flight decoding or later tool arrivals. STARS derives the ranking by simulating the platform's response (Section 3.3), so the same workload yields a different order on a different platform.

Time-L is beaten in every cell of the main sweep at 1.0356× mean speedup; at T=4 the mean reduction is 7.85% across all eight LLM capacities of the simulated sweep (4–64 accelerators). Over many configurations the reduction is substantial, and the revision will report that spread.

Q3: Ablation of the main components

Section 4.1 defines all three corrections as one-sided: the LLM batch externality sums only positive step-latency increases, tool wait is clipped at zero, and tool externality sums only positive increases in other phases' completion times. Each term is identically zero when its state is absent, so each switches on in the regime its definition selects.

Tool externality is structural — executors are shared pools — and positive in all nine backgrounds (mean 7.48%, median 6.83%). Dropping all three corrections — keeping only the remaining-path term R(c) — costs 5.5%–9.9% of makespan.

Q4: Simulator accuracy and schedule ranking

The simulator does not predict an unseen execution; it evaluates a recorded one counterfactually under a different order. It implements vLLM's step-level semantics (max_num_seqs cap, max_num_batched_tokens budget) and replaces only LLM admission and tool dispatch. The latency model is a measured table, not a fitted one: a new model or GPU changes the table, a new workload changes which steps occur.

Because the planner commits to ranks, not timings (Section 3.3), the property needed is ordinal, and every arm shares world, profile, and fees, so residual error is largely common. Section 5.2.1 supplies that ordinal evidence: a plan built for one configuration loses 6.03% on average and up to 14.2% when transferred, and the advantage disappears at C=(4,1), where a single executor serializes the pool.

Coverage follows from the profiled grid: extending it is a profiling run, not a code change, and off-grid steps are reconstructed from measured components.

Q5: Score weights, horizon, and repair thresholds

All parameters were fixed before the comparisons. H=8 is a token-step window: an admission's externality is realized in the steps it shares with the incumbent batch, so the lookahead need only cover the prefill it inserts and the decode steps it perturbs. All three weights are 1, since all terms are in seconds.

Repair is a filter chain: an event filter, a consistency test on the affected plans, a 1 s gain margin, a 60 s per-attempt budget, and installation only at a safe boundary. The 25% threshold and 120 s coalescing window govern how often the chain is entered; 1 s is small against an 8.5-hour makespan and large enough to reject noise-level gains. The 64-attempt limit came from a paired diagnosis at (L=4, T=4) reported in Table 6.

The claim is within-rollout under replay, not training-loop speedup, and drift moves the magnitude of a scheduler's value rather than its sign. Outside T=4 no order has much room, since the remaining-path rule already captures the first-order term. Two text fixes follow: Section 5.1 will name the trace pool behind "training dataset", and the revision will state each parameter's role.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.