acceptodds
Under review as a conference paper at ICLR 2027

ALOER: Action-Level Orchestration of External Resources for Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning (RL) trains large language models (LLMs) through environment interactions and requires resources beyond the training cluster, such as CPUs for code execution and GPUs for reward models. Existing service systems for agentic RL typically rely on static resource provisioning, which mismatches the multi-turn and intermittent pattern of external resource invocations, leading to severe inefficiency. We present ALOER, a unified system for elastic, action-level orchestration of external resources in agentic RL. ALOER utilizes a unified action-level formulation and an elastic scheduling algorithm to minimize action completion time (ACT) while satisfying heterogeneous resource constraints. Further, heterogeneous resource managers are tailored to efficiently support the action-level execution with elastic degree-of-parallelism on heterogeneous resources. Evaluation on well-established agentic RL tasks demonstrates that ALOER improves average ACT by up to 4.3, speeds up RL training steps by up to 1.5, and reduces external resource usage by up to 71.2.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agentic LLM, Reinforcement Learning, Resource Management

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZqQbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ALOER, a system for action-level orchestration of external resources in agentic reinforcement learning. The authors identify two sources of resource over-provisioning: resources reserved throughout individual trajectories despite intermittent tool usage, and resources statically allocated to external services despite fluctuating demand across RL tasks.

To address these inefficiencies, ALOER introduces a unified action formulation that models heterogeneous resource requirements and execution elasticity. It further develops an elastic scheduling algorithm combining greedy eviction, dynamic programming, and approximate action completion time (ACT) estimation. The system also provides specialized resource managers, including allocate-on-execution (AOE) for CPUs and evict-on-execution (EOE) for GPUs, to support fine-grained resource sharing while preserving execution states.

Experiments on AI Coding, DeepSearch, and MOPD demonstrate improvements in action completion time, RL training efficiency, and external resource utilization. The authors report up to 4.3× lower ACT, 1.5× faster RL training steps, and 71.2% lower external resource usage.

Critical strengths
  1. The paper addresses an important practical problem in agentic RL. External resource management is an increasingly important bottleneck in agentic RL. The paper clearly identifies resource over-provisioning caused by intermittent tool usage and provides convincing motivation for finer-grained resource orchestration.

  2. The proposed system provides a comprehensive solution for heterogeneous resources. ALOER combines action-level resource sharing, elastic scheduling, and specialized CPU/GPU resource managers within a unified framework. The overall system design is practical and addresses several important implementation challenges.

  3. The evaluation covers multiple representative workloads, including AI Coding, DeepSearch, and MOPD. The reported improvements in ACT, end-to-end training time, and GPU resource utilization are encouraging.

Critical weaknesses
  1. The ablation studies are insufficient to establish the contribution of each component. ALOER combines several mechanisms, including action-level resource pooling, elastic DoP allocation, greedy eviction, and specialized CPU/GPU resource managers. However, the main scheduling ablation only compares elastic DoP against fixed DoP configurations. The paper does not clearly separate the benefits of resource sharing from those of the proposed scheduling algorithm. In particular, it remains unclear whether the greedy eviction strategy provides substantial improvements over simpler scheduling policies.

  2. The scheduling algorithm relies on restrictive assumptions about action execution. The proposed scheduler assumes that each action has a single key elasticity resource and relies on estimates of execution duration and resource elasticity. However, real agentic workloads often involve unpredictable execution times and actions whose performance depends on multiple resources. The evaluation primarily profiles CPU reward computation and GPU reward-model inference, leaving the effectiveness of the scheduler on more general tool invocations insufficiently demonstrated.

  3. Minimizing average ACT may not align well with end-to-end RL training efficiency. The scheduler optimizes the sum of action completion times, but RL training efficiency can be strongly affected by long-tail trajectories and rollout makespan. The MOPD experiments already show that improvements in average ACT do not necessarily translate into comparable end-to-end speedups. Furthermore, the greedy eviction strategy may delay some actions to improve average ACT, potentially worsening tail latency or straggler effects. The paper lacks a systematic evaluation of this trade-off.

Questions for authors
  1. How well does ALOER scale when more reward models need to share the same GPUs? ALOER relies on caching and restoring reward models to share GPU resources. However, as the number of models increases, frequent model switching may introduce significant overhead. Can the authors evaluate how this affects system performance?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ZqQb

Thank you for recognizing the practical importance and heterogeneous-system design, and for identifying areas that need stronger evidence.

W1 — Component attribution. We agree that the fixed-versus-elastic comparison alone does not establish the benefit of greedy eviction. We therefore add Eager, a simple elastic baseline that replaces greedy eviction with immediate best-effort allocation. Across the batch-size and resource-capacity sweeps (Figure 9 of the revised manuscript), ALOER achieves peak ACT speedups of 2.00×–2.08× over Eager(8–32). These results support the value of the greedy-eviction scheduling mechanism beyond simply enabling elasticity or increasing the DoP cap.

W2 — Execution-model assumptions. Our single-resource elasticity assumption derives from both practical workloads and algorithmic tractability. In our target workloads, actions with well-characterized elasticity typically scale primarily with one resource or a fixed resource bundle. Actions without clear elasticity can still be scheduled using fixed feasible allocations, even without exact execution durations (Appendix B.1). However, future agentic RL workloads may introduce actions with elasticity across multiple resources. Jointly optimizing their allocations under low-latency scheduling requirements is challenging, and we discuss this extension as future work in Section 7.

W3 — Gains of MOPD and individual delays. We correct the MOPD explanation: its rollout is much longer than the single reward computation accelerated by ALOER, whereas AI Coding benefits from accelerating multi-turn environment interactions. Its more substantial resource-saving benefit is evaluated in §5.

Your individual-delay concern is valid: FCFS admission, though avoiding starvation, does not guarantee that every action finishes earlier. Therefore, we add a paired comparison of the same reward action in two experiments:

WorkloadBatch SizeMean-ACT speedupNon-increased ACT
CPU5121.577×99.17%
CPU7681.346×92.02%
GPU5122.512×98.73%
GPU7682.343×95.27%

Across the evaluated workloads and load levels, greedy-based eviction algorithm improves mean ACT while keeping ACT faster for 92.02%–99.17% of reward actions compared with fixed-DoP scheduling. This shows that our algorithm guarantees fairness in the vast majority of cases: in theory, it does not cause starvation of action requests; in practice, it does not slow down individual requests.

Q1 — More models. We agree that more models can increase switching costs even when concurrency is unchanged. Our added trace-driven experiments (Appendix E.3, Table 3) show that increasing model count progressively raises system overhead, which is dominated by model switching. Under heavier load, more frequent model switching not only increases system overhead but also amplifies queueing delays, in contrast to the lighter load. Thus, model count affects ACT under EOE, but the mechanism through which it does so is load-dependent.

MOPD bszModel cntsQueue Dur. (s)Sys. Overhead (s)
204890.0810.201
2048120.8750.329
2048152.5610.444
2048185.0030.548
102490.0130.222
1024120.0160.324
1024150.0340.420
1024180.0560.513
Reviewer_wjp4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ALOER, a resource management system that shifts external resource orchestration in agentic reinforcement learning (RL) from static trajectory/task-level allocation to elastic, action-level granularity. By combining a unified vectorized action formulation, a greedy eviction scheduler with dynamic programming (DP), and specialized resource managers—Allocate-on-Execution (AOE) for CPUs and Evict-on-Execution (EOE) for GPUs—ALOER reclaims idle resources between multi-turn LLM generations and dynamically scales the degree-of-parallelism (DoP). Evaluations across AI Coding, DeepSearch, and Multi-teacher On-Policy Distillation (MOPD) show up to a 4.3x reduction in average action completion time (ACT), a 1.5x speedup in RL training steps, and a 71.2% reduction in external resource usage.

Critical strengths
  1. Clearly identifies and quantifies dual-level (trajectory and task) external resource over-provisioning in agentic RL and effectively solves it via action-level orchestration.

  2. The AOE (via dynamic cgroup/cpuset adjustments while retaining container memory) and EOE (via hierarchical CPU-GPU memory caching and power-of-two chunking) designs practically decouple persistent state from active compute resources.

  3. Evaluated on a realistic testbed (48 Hopper GPU nodes for RL, 15 CPU nodes, and 5 external GPU nodes), demonstrating both system efficiency gains and improved RL training convergence at large batch sizes.

Critical weaknesses

While ALOER reduces external reward ACT in MOPD from 49s to 2s (a 24.5x reduction in reward time), the end-to-end RL step time only improves from 1858s to 1526s (~1.21x speedup). The authors admit that because pipelined rollout duration is dominated by long-tail LLM generation trajectories rather than average ACT, optimizing external resource ACT yields diminishing returns for end-to-end training speed.

Questions for authors
  1. How does ALOER obtain ToriT_{ori} and E(m)E(m) in practice for new tasks, and can the system learn these profiles online during early RL steps rather than relying on advance profiling?

  2. In AI Coding, if several trajectories pinned to the same CPU node simultaneously trigger heavy test suites, can node-level congestion occur while other nodes sit idle, and could container migration mitigate this?

1 comment
AuthorsOpenReview ↗

Response to Reviewer wjp4

Thank you for the positive assessment and the specific questions. We have revised the MOPD explanation and clarified profiling and CPU placement.

W1 — MOPD action gains. We apologize for the misleading wording. MOPD gains in step durations are smaller because its overall rollout is much longer than the single reward computation accelerated by ALOER, whereas AI Coding benefits from accelerating multi-turn environment interactions. For MOPD, the more substantial benefit is external-resource savings: §5.3 shows approximately 71% fewer GPUs at equal ACT.

Q1 — Obtaining and updating profiles. CPU rewards and GPU inference are profiled offline over feasible DoPs, whereas unknown-elasticity actions use fixed feasible allocations without requiring exact execution durations (Appendix B.1). Learning and updating profiles during early training is promising but poses challenges, such as balancing profiling granularity against system efficiency. We discuss this as critical future work in §7.

Q2 — Node-local congestion and migration. Thank you for pointing out this possibility. The current design has tried to mitigate imbalance across nodes: placement considers current resource availability and load, while environment completion/creation allows new placements to adapt throughout a step. However, extreme differences in both trajectory duration and compute demand may still cause persistent hotspots. Snapshot migration could help, but incurs state-transfer, pause, and consistency costs. §7 discusses this trade-off as future work.

Reviewer_LzJfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the over-provisioning of external resources in agentic reinforcement learning (RL) training: code execution, reward and teacher-model inference, and retrieval interfaces consume CPUs, GPUs, and API quotas, respectively, and these invocations lie on the critical path of every rollout. Existing systems statically provision resources per trajectory or per service, which mismatches the multi-turn, intermittent invocation pattern of agentic RL. Through production measurements, the authors find that execution environments are actively accessed for only 47% of their lifetime on average, that the GPU utilization of twelve teacher models averages below 3%, and that invocation volumes across tasks fluctuate by up to three orders of magnitude. Idle resources cannot be reclaimed and reused by other actions, leading to queuing delays, prolonged action completion times, and increased costs. Based on this observation, the paper shifts resource management down to the granularity of individual atomic invocations, uniformly characterizes heterogeneous actions through vectorized resource costs and elasticity mappings, and proposes an online scheduling algorithm that minimizes the sum of action completion times. Specialized management mechanisms are designed for CPUs and GPUs, namely allocate-on-execution and evict-on-execution with on-demand state restoration. The authors evaluate the system on AI Coding, DeepSearch, MOPD, and their combined workload, demonstrating up to a 4.3× improvement in action completion time, a 1.4×–1.5× speedup of training steps, and up to 71.2% reduction in external resource usage, showing that action-level orchestration can convert the idle capacity left by intermittent invocations into effective throughput.

Critical strengths
  1. The paper presents a complete and well-closed narrative: it begins with the observation that existing approaches over-provision external resources, substantiates this with empirical measurements, develops a unified formulation upon which the scheduling algorithm is built, and validates the design through experiments.

  2. The evaluation is conducted on real-world workloads, making the conclusions practically valuable for deployment.

Critical weaknesses
  1. The paper lacks an approximation-ratio analysis for the proposed heuristic scheduling algorithm.

  2. The experiments only compare against static baselines such as Kubernetes and SGLang, without including advanced task-/stage-level orchestration systems such as Mars, MegaFlow, SkyRL-Agent, and ROCK as baselines.

  3. The discussion of the applicability boundaries of the proposed method is insufficient. For instance, the GPU scalability experiments show that SGLang is slightly faster under low concurrency, yet the paper does not clearly delineate the regime in which the method is advantageous.

  4. The font sizes are inconsistent across figures; for example, the font in Figure 6 is too large.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer LzJf

Thank you for the constructive review and for recognizing the motivation and practical value of action-level orchestration. We address your four concerns below.

W1 — Approximation guarantees. We apologize for omitting approximation guarantees from the submitted version, because establishing a constant ratio for a full online scheduler is challenging, and the guarantees depend on resource topology. We add topology-specific local bounds in Appendix B.3. For fixed candidates, a complete DP formulation achieves an approximation ratio of 1 within the modeled CPU/GPU feasible domains. GPU power-of-two discretization has a less-than-2 bound relative to integer DoPs on fully free homogeneous nodes. We also provide instance-dependent continuation certificates.

W2 — Baselines. We apologize for not emphasizing that the submitted evaluation already includes non-static K8S+VPA and ServerlessLLM baselines (§5.3). We initially omitted the suggested environment orchestrators because these systems primarily manage CPU-intensive workloads at the environment-/trajectory-level, as Kubernetes does, without explicit resource allocation and reclamation between actions of one persistent environment. Still, thank you for suggesting a modern-system comparison to make the evaluation more comprehensive: we conduct a native ROCK experiment with execution-request replay under the same resource budget, reporting average ACT in Figure 7 of the revised manuscript.

W3 — Applicability. Thank you for highlighting the applicability boundary. SGLang is slightly faster only at a very-low-concurrency point because ALOER incurs a little restoration overhead. The point is included for evaluation completeness rather than as a cost-efficient RL operating regime, with GPU SM activity lower than 2.4% (Figure 3(a), right). We add this discussion to §5.3.

W4 — Figure readability. We redraw Fig. 6 using the original stage measurements with smaller, consistent typography and clearer normalization/absolute-time labels.

AuthorsOpenReview ↗

Supplement to W2

For your convenience, we report the additional ROCK results here. Relative to ROCK, ALOER reduces mean total ACT per trajectory by 3.31×–5.05× across batch sizes 128–1280 on five nodes, and by 3.80×–5.05× across 5–15 nodes at batch size 1280 (Figure 7 of the revised manuscript). Thank you again for suggesting this comparison, which helps make our evaluation more comprehensive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.