- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ALOER, a system for action-level orchestration of external resources in agentic reinforcement learning. The authors identify two sources of resource over-provisioning: resources reserved throughout individual trajectories despite intermittent tool usage, and resources statically allocated to external services despite fluctuating demand across RL tasks.
To address these inefficiencies, ALOER introduces a unified action formulation that models heterogeneous resource requirements and execution elasticity. It further develops an elastic scheduling algorithm combining greedy eviction, dynamic programming, and approximate action completion time (ACT) estimation. The system also provides specialized resource managers, including allocate-on-execution (AOE) for CPUs and evict-on-execution (EOE) for GPUs, to support fine-grained resource sharing while preserving execution states.
Experiments on AI Coding, DeepSearch, and MOPD demonstrate improvements in action completion time, RL training efficiency, and external resource utilization. The authors report up to 4.3× lower ACT, 1.5× faster RL training steps, and 71.2% lower external resource usage.
Critical strengths
-
The paper addresses an important practical problem in agentic RL. External resource management is an increasingly important bottleneck in agentic RL. The paper clearly identifies resource over-provisioning caused by intermittent tool usage and provides convincing motivation for finer-grained resource orchestration.
-
The proposed system provides a comprehensive solution for heterogeneous resources. ALOER combines action-level resource sharing, elastic scheduling, and specialized CPU/GPU resource managers within a unified framework. The overall system design is practical and addresses several important implementation challenges.
-
The evaluation covers multiple representative workloads, including AI Coding, DeepSearch, and MOPD. The reported improvements in ACT, end-to-end training time, and GPU resource utilization are encouraging.
Critical weaknesses
-
The ablation studies are insufficient to establish the contribution of each component. ALOER combines several mechanisms, including action-level resource pooling, elastic DoP allocation, greedy eviction, and specialized CPU/GPU resource managers. However, the main scheduling ablation only compares elastic DoP against fixed DoP configurations. The paper does not clearly separate the benefits of resource sharing from those of the proposed scheduling algorithm. In particular, it remains unclear whether the greedy eviction strategy provides substantial improvements over simpler scheduling policies.
-
The scheduling algorithm relies on restrictive assumptions about action execution. The proposed scheduler assumes that each action has a single key elasticity resource and relies on estimates of execution duration and resource elasticity. However, real agentic workloads often involve unpredictable execution times and actions whose performance depends on multiple resources. The evaluation primarily profiles CPU reward computation and GPU reward-model inference, leaving the effectiveness of the scheduler on more general tool invocations insufficiently demonstrated.
-
Minimizing average ACT may not align well with end-to-end RL training efficiency. The scheduler optimizes the sum of action completion times, but RL training efficiency can be strongly affected by long-tail trajectories and rollout makespan. The MOPD experiments already show that improvements in average ACT do not necessarily translate into comparable end-to-end speedups. Furthermore, the greedy eviction strategy may delay some actions to improve average ACT, potentially worsening tail latency or straggler effects. The paper lacks a systematic evaluation of this trade-off.
Questions for authors
- How well does ALOER scale when more reward models need to share the same GPUs? ALOER relies on caching and restoring reward models to share GPU resources. However, as the number of models increases, frequent model switching may introduce significant overhead. Can the authors evaluate how this affects system performance?
Response to Reviewer ZqQb
Thank you for recognizing the practical importance and heterogeneous-system design, and for identifying areas that need stronger evidence.
W1 — Component attribution. We agree that the fixed-versus-elastic comparison alone does not establish the benefit of greedy eviction. We therefore add Eager, a simple elastic baseline that replaces greedy eviction with immediate best-effort allocation. Across the batch-size and resource-capacity sweeps (Figure 9 of the revised manuscript), ALOER achieves peak ACT speedups of 2.00×–2.08× over Eager(8–32). These results support the value of the greedy-eviction scheduling mechanism beyond simply enabling elasticity or increasing the DoP cap.
W2 — Execution-model assumptions. Our single-resource elasticity assumption derives from both practical workloads and algorithmic tractability. In our target workloads, actions with well-characterized elasticity typically scale primarily with one resource or a fixed resource bundle. Actions without clear elasticity can still be scheduled using fixed feasible allocations, even without exact execution durations (Appendix B.1). However, future agentic RL workloads may introduce actions with elasticity across multiple resources. Jointly optimizing their allocations under low-latency scheduling requirements is challenging, and we discuss this extension as future work in Section 7.
W3 — Gains of MOPD and individual delays. We correct the MOPD explanation: its rollout is much longer than the single reward computation accelerated by ALOER, whereas AI Coding benefits from accelerating multi-turn environment interactions. Its more substantial resource-saving benefit is evaluated in §5.
Your individual-delay concern is valid: FCFS admission, though avoiding starvation, does not guarantee that every action finishes earlier. Therefore, we add a paired comparison of the same reward action in two experiments:
Across the evaluated workloads and load levels, greedy-based eviction algorithm improves mean ACT while keeping ACT faster for 92.02%–99.17% of reward actions compared with fixed-DoP scheduling. This shows that our algorithm guarantees fairness in the vast majority of cases: in theory, it does not cause starvation of action requests; in practice, it does not slow down individual requests.
Q1 — More models. We agree that more models can increase switching costs even when concurrency is unchanged. Our added trace-driven experiments (Appendix E.3, Table 3) show that increasing model count progressively raises system overhead, which is dominated by model switching. Under heavier load, more frequent model switching not only increases system overhead but also amplifies queueing delays, in contrast to the lighter load. Thus, model count affects ACT under EOE, but the mechanism through which it does so is load-dependent.