acceptodds
Under review as a conference paper at ICLR 2027

Scheduling Mixed RL Rollouts Beyond Prefix Locality

Abstract

Modern reinforcement learning (RL) post-training pipelines for large language models (LLMs) increasingly combine rollout workloads across multiple domains and feedback paradigms. Prefix-aware routing improves inference efficiency through cache reuse and load balancing, but it does not control how heterogeneous rollout sessions compete for KV-cache capacity. When reinforcement learning with verifiable rewards (RLVR), reinforcement learning from human feedback (RLHF), and agentic rollouts share an asynchronous inference service, their distinct sequence structures, interaction patterns, and KV-residency times create substantially different serving demands. Rollout scheduling must account for this heterogeneity without distorting the workload mixture specified by the trainer. We present MISA-T, a routing-layer admission policy for mixed rollout serving. MISA-T combines adaptive session admission, workload-aware KV-capacity allocation, and residency-time-aware KV accounting. In rollout-only ablations on Step3.7 and Qwen3.6-35B-A3B, MISA-T improves rollout throughput over a sweep-tuned cache-aware vLLM Router by 53.3% and 43.6%, respectively, while maintaining high prefix-cache hit rates. In a matched 50-iteration Step3.7 experiment, it increases rollout throughput by 35.6% and reduces mean iteration time by 22.8%, while keeping the consumed workload mixture close to the trainer target and achieving comparable task scores.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
reinforcement learning, LLM post-training, rollout serving, request scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6TdvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies admission control for mixed RL rollout serving, where agentic, RLVR, and RLHF rollouts share an asynchronous inference pool. The authors observe that prefix-aware routing (e.g., vLLM Router's cache-aware policy) decides where a request goes but not how many sessions may occupy an instance, so under high concurrency new sessions evict reusable prefixes and throughput collapses. They propose MISA-T, a router-layer policy with three components: a dynamic global session cap driven by a reference KV footprint and contracted under confirmed prefix-hit degradation; per-class soft caps that split protected KV blocks in proportion to demand x footprint (MISA); and residency-time weighting of that demand, including tool/environment gaps for agent sessions (MISA-T). Rollout-only ablations on Step3.7 (196B-A11B) and Qwen3.6-35B-A3B report +53.3% / +43.6% sample throughput over a sweep-tuned vLLM Router. A 50-iteration end-to-end Step3.7 run reports +35.6% rollout throughput, -22.8% iteration time, a consumed mixture slightly closer to the trainer target (TVD 2.71 vs 4.14 pp), and pass@4 within 0.5 pp on three SWE benchmarks.

Critical strengths

S1. The problem is real and the diagnosis is convincing. The paper correctly identifies that cache-aware placement does not bound KV contention across heterogeneous sessions. The high-load row of Table 3 (prefix hit 92.4% -> 4.5%, throughput -55%) and the Appendix A stress test make the failure mode concrete and motivate admission control as a distinct lever from routing.

S2. The ablation cleanly separates the three mechanisms. Session Admission -> MISA -> MISA-T is evaluated on both models under identical controller settings, so the reader can see that global admission alone recovers +12-19%, class allocation adds ~10%, and residency weighting adds the remainder. This is more disciplined than most serving-systems papers.

S3. Experiments are at production scale with end-to-end validation. A 196B sparse-MoE model on multi-node H200 clusters, a second open model on H100s, an end-to-end RL run that tracks iteration time and consumed mixture, and downstream task scores rather than serving metrics alone. The compatibility check with CPU KV offloading (Sec. 5.5) also shows the method composes with existing memory tiers.

Critical weaknesses

W1. Statistical rigor does not match the size of the claims. Table 3 reports only means over three runs with no variance or confidence intervals; Table 2 and Figure 3 are a single 50-iteration run. Several deltas that carry the narrative are ~10% (MISA vs Session Admission on both models, MISA-T vs MISA on Qwen), which is within plausible run-to-run noise for throughput on a shared cluster. The "<0.5 pp" pass@4 claim is given without the actual numbers or any seed variance.

W2. Two central results are unexplained and partly contradict the stated mechanism. (a) On Qwen3.6-35B-A3B, MISA drops prefix-hit rate from 96.3% to 79.2% while increasing throughput; since Sec. 4.2 says class caps never exceed the global cap, class quotas alone should not add eviction pressure, and the paper offers no account of which mechanism admits the extra evicting sessions. (b) Residency weighting is motivated by tool/environment gaps, which are 6.6% of agent task time on Step3.7 and 22.7% on Qwen (Table 4), yet MISA -> MISA-T gains +24.3% on Step3.7 and only +9.7% on Qwen. The gain is largest where the motivating effect is smallest, suggesting the EWMA is mostly capturing decode-length heterogeneity (RLVR p50 output 47K tokens) rather than tool residency. The paper should disentangle these.

W3. The headline throughput numbers are not interpretable without the consumed mixture. Sample rate counts completed rollout units, and classes differ by orders of magnitude in per-sample cost (Table 1). A policy that drains short RLHF/agent samples faster inflates sample rate without serving the trainer's target. The end-to-end run does report mixture (MISA-T over-serves RLHF by +2.6 pp), but Table 3 -- the source of the +53%/+44% headline -- reports no per-class counts or TVD.

W4. Baselines and experimental conditions are underspecified. The only competitor is vLLM Router with a static cap tuned by a coarse sweep (steps of 32/16); the sweep curve is not shown, and no static per-class cap baseline is included, so "dynamic" vs "class-aware" vs "better static split" is not separated. The outer ceiling given to MISA-T is never stated numerically, although Figure 5 shows it running ~150-400 requests versus ~100-130 for the sweep baseline. Table 5 sets the protected budget to 90% of GPU+CPU blocks, yet Sec. 5.5 claims offloading was enabled with "unchanged" settings despite a 1.3 TB CPU tier. The ~10 controller constants in Table 5 have no sensitivity analysis, so it is unclear whether static-concurrency tuning has been removed or relocated.

Questions for authors
  1. What is the exact outer concurrency ceiling for the admission-controlled variants in each experiment, and did the vLLM Router sweep extend to it?

  2. Why does MISA on Qwen reduce prefix-hit rate to 79.2% when class caps are bounded by the global cap? Which mechanism admits the extra sessions?

  3. Please report per-class completed samples and TVD to target for every row of Table 3.

  4. Can you separate the contribution of tool/environment residency from decode-length residency, e.g., by weighting with model-execution time only?

  5. In Sec. 5.5, is the protected budget recomputed to include CPU blocks, and what is the +35.6% RPM compared against? (It coincides exactly with the end-to-end sample-rate figure.)

Reviewer_ETFhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MISA-T, a routing-layer admission policy for heterogeneous RL rollout serving. The key observation is that prefix-aware routing alone cannot effectively manage KV-cache contention among workloads with different sequence lengths, session structures, and KV residency times.

MISA-T combines adaptive session admission, workload-aware capacity allocation, and residency-time-aware KV accounting. It dynamically regulates admitted sessions and allocates per-class capacity based on outstanding demand, estimated KV footprint, and session residency time, while preserving the workload identities specified by the trainer.

Experiments on Step3.7 and Qwen3.6-35B-A3B demonstrate rollout throughput improvements of 53.3% and 43.6% over sweep-tuned vLLM Router, respectively. A 50-iteration end-to-end RL training experiment further reports a 35.6% improvement in rollout throughput and a 22.8% reduction in iteration time, with comparable downstream task scores.

Critical strengths
  1. A practically important problem with a well-motivated design.

The paper identifies a meaningful limitation of existing prefix-aware routing in heterogeneous RL rollout serving. The distinction between request-level prefix locality and session-level KV-capacity commitments is particularly relevant for agentic workloads. The proposed routing-layer approach is also attractive because it can potentially integrate with existing inference engines without changing their core execution mechanisms.

  1. Substantial empirical improvements with useful component-level ablations.

The reported throughput gains are significant across two model and hardware configurations. The ablations separating global session admission, workload-aware allocation, and residency-time weighting provide evidence that each component contributes to the improvements. The end-to-end training experiment further strengthens the practical relevance beyond isolated serving benchmarks.

Critical weaknesses
  1. Insufficient evidence that throughput gains preserve workload comparability and per-class progress.

The paper emphasizes maintaining the trainer-specified workload mixture, but the rollout-only results do not report per-class completion rates, completed workload mixtures, or waiting-time distributions.

Notably, in Table 3, MISA-T improves sample throughput by 53.3% and 43.6%, whereas decode-token throughput improves by only 26.5% and 18.3%. The ratios suggest approximately 17.5% fewer decoded tokens per completed sample as a throughput-based proxy. While this does not necessarily indicate an unfair comparison, it raises the possibility that different completion-length distributions or workload compositions contribute to the reported sample-throughput gains.

Without additional breakdowns, it is difficult to determine how much of the improvement reflects better execution efficiency versus changes in the workloads completed during the measurement window. This concern is particularly relevant to the paper's mixture-preservation objective.

  1. Limited differentiation from existing resource-aware scheduling approaches and insufficient robustness evaluation.

The underlying block-time accounting principle is closely related to the space-time cost model used in prior systems such as ThunderAgent. Although MISA-T applies this principle to workload-aware admission rather than program-level scheduling, the specific novelty of the allocation strategy could be better established.

The current experiments mainly compare against vLLM Router and the proposed method's own ablations. Comparisons with simpler cost-aware admission or weighted resource-allocation baselines would help clarify the incremental contribution.

Furthermore, the experiments use a limited set of workload configurations. Since the policy depends on estimated footprints, residency times, and feedback from asynchronous serving-state reports, its robustness under shifting workload mixtures and inaccurate estimates remains insufficiently demonstrated.

Questions for authors

Q1. Workload comparability

Were the same rollout units and comparable token workloads processed across scheduling policies in the rollout-only experiments? Could the authors report per-class sample throughput, completed workload mixture, output-token distributions, and backlog statistics? In particular, what explains the substantial difference between the relative sample-throughput and decode-token-throughput improvements in Table 3?

Q2. How sensitive is MISA-T to the frequency of policy weight synchronization?

Since KV caches must be invalidated whenever inference replicas load updated model weights, the effectiveness of session-aware caching may depend strongly on the interval between synchronization events. Could the authors report the synchronization frequency, cache warm-up overhead, and throughput gains under different synchronization intervals?

Reviewer_XBauOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents MISA-T, a routing-layer admission policy for heterogeneous RL rollouts combining agentic, RLVR, and RLHF workloads. It argues that prefix-aware routing alone cannot prevent sessions with different KV-cache footprints and lifetimes from overloading shared serving capacity.

Its contributions are adaptive session admission, workload-specific capacity allocation, and KV-residency-time weighting that includes tool/environment interactions. Together, these mechanisms aim to improve completed-rollout throughput while preserving the trainer’s intended workload mixture.

Validation includes three-run rollout-only ablations on Step3.7 and Qwen3.6-35B-A3B, reporting throughput gains of 53.3% and 43.6% over sweep-tuned vLLM Router. A matched 50-iteration Step3.7 training experiment reports 35.6% higher rollout throughput, 22.8% lower iteration time, and closer agreement with the target mixture. Additional experiments illustrate controller adaptation and compatibility with CPU KV offloading. The authors report pass@4 differences below 0.5 percentage points on three SWE benchmarks, but provide no individual scores. Core performance results are relative, without absolute throughput, timing, or uncertainty estimates, limiting assessment of practical efficiency and statistical robustness

Critical strengths
  • A well-motivated treatment of heterogeneous rollout resource demands. MISA-T combines session admission with workload-specific KV capacity allocation and residency-time accounting, including tool-execution intervals. This addresses a consequential limitation of prefix-aware routing: good placement alone cannot control cache contention or balance capacity across heterogeneous sessions. The resulting design targets an important bottleneck in mixed-workload RL training.
  • Ablations and end-to-end validation connect the design to practical gains. The staged ablations show incremental benefits from admission control, workload-aware allocation, and residency weighting across two models. More consequentially, the matched 50-iteration training experiment reports a 22.8% reduction in iteration time, with a consumed workload mixture closer to the target and reported task scores remaining comparable. This provides evidence that the serving improvements translate into training efficiency rather than merely improving an isolated cache metric.
Critical weaknesses
  • Insufficient evidence that the speedup preserves training quality. The only end-to-end experiment covers 50 iterations on one model. The paper states that pass@4 differences are below 0.5 percentage points on three benchmarks, but provides no actual scores, learning curves, or uncertainty estimates. Because scheduling changes which samples complete and enter training, this leaves a central claim—faster training without compromising learning—insufficiently substantiated.
  • The evaluation does not establish the advantage over closely related scheduling approaches. The main external baseline is cache-aware vLLM Router; the remaining comparisons are internal ablations. There is no direct comparison with program-aware scheduling such as ThunderAgent, despite the acknowledged overlap in space-time resource accounting. Consequently, the experiments demonstrate benefits over placement-only routing but do not establish whether MISA-T provides a meaningful advance over existing approaches that already account for session-level resource demands.
Questions for authors

I strongly recommend that the authors report absolute values alongside relative improvements, as this would help readers assess whether the baselines were properly configured and evaluated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.