- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents AReaL-Hex, a heterogeneity-aware scheduling system for asynchronous RL training over heterogeneous GPUs. The paper formulates the joint optimization of resource allocation, parallel strategies, workload assignment, and data staleness constraints, and proposes a two-phase scheduler combining MILP-based search and graph partitioning. The authors implement the system and evaluate it on LLM RL workloads with 1.5B–14B models across heterogeneous GPU configurations. Experiments show that AReaL-Hex improves training throughput and cost efficiency compared with homogeneous asynchronous RL deployments, demonstrating the effectiveness of heterogeneous resource-aware scheduling.
Critical strengths
- The paper proposes a systematic solution for asynchronous RL training on heterogeneous GPU clusters, addressing an important and practical problem. This work focuses on the deployment challenges of asynchronous RL training in heterogeneous GPU environments and jointly considers rollout generation, model training, communication overhead, and data staleness constraints, rather than optimizing individual stages independently. The problem is highly relevant to the resource utilization challenges in large-scale LLM RL training and addresses the limitation of prior works that mainly optimize inference or training in isolation.
- The proposed two-phase scheduling framework effectively balances optimization quality and system scalability, with experiments demonstrating practical benefits. The authors design a scheduler that combines MILP-based search and graph partitioning to decompose the complex optimization of resource allocation, parallel strategy selection, and workload assignment. This design avoids exhaustive exploration of the large scheduling space while maintaining effective execution performance. Extensive experiments across different model scales, GPU configurations, and baselines demonstrate that AReaL-Hex achieves significant improvements in training throughput and cost efficiency.
Critical weaknesses
- The methodological novelty is mainly reflected in system optimization, while the theoretical analysis of the scheduling approach remains limited. Although the paper formulates heterogeneous asynchronous RL training as a joint scheduling problem and proposes a practical two-phase optimization framework, some components, particularly the graph partitioning stage, rely on heuristic strategies without theoretical guarantees regarding optimality or approximation quality. The effectiveness of the approach is primarily demonstrated through empirical evaluations, while further analysis of solution quality, optimality gaps, and robustness across different hardware topologies and workload distributions is limited.
- The evaluation could be further strengthened to better demonstrate the generality of the proposed approach. The paper demonstrates the effectiveness of AReaL-Hex through experiments across different model scales and GPU configurations, showing improvements over homogeneous asynchronous RL systems. However, the current evaluation focuses mainly on specific hardware settings and workload scenarios, with limited comparisons against alternative heterogeneous scheduling strategies.
Questions for authors
- The proposed scheduling method, particularly the graph partitioning stage, mainly relies on heuristic optimization strategies, but lacks theoretical guarantees regarding the quality of the generated schedules. The authors are encouraged to further analyze the robustness of the approach under different GPU topologies and workload distributions, and provide an analysis of the optimality gap between the obtained solutions and the optimal solutions.
- The paper mainly demonstrates the effectiveness of AReaL-Hex through comparisons with homogeneous asynchronous RL deployments. However, it remains unclear how much of the performance improvement comes from the scheduler itself versus the benefits of heterogeneous GPU resource allocation. The authors are encouraged to provide comparisons with stronger heterogeneous scheduling baselines to more accurately evaluate the effectiveness of the proposed scheduling algorithm.
3 comments
I think my concerns are resolved and I keep my rating.
Thank you very much for reviewing our response. We sincerely appreciate your acknowledgment that our clarifications and additional experiments have resolved your concerns.
In particular, we have provided further evidence regarding the scheduler's near-optimal solution quality, robustness across heterogeneous hardware configurations, and throughput improvements over stronger scheduling baselines. We hope these results further demonstrate the effectiveness and contributions of AReaL-Hex.
Given that your concerns have been resolved, we would be very grateful if you could reconsider your rating and potentially increase it to better reflect the additional evidence and clarifications provided. If there are any remaining reservations regarding the overall assessment of our work, we would sincerely appreciate your further feedback.
Thank you again for your time and consideration.
Thank you for your valuable comments and constructive suggestions. We address your concerns and questions below.
Optimality and robustness across hardware configurations. We evaluate our algorithm against exhaustive search on two 56-GPU pools. Detailed results are provided in Table 18 and summarized as follows:
These results demonstrate that AReaL-Hex maintains near-optimal scheduling quality across the evaluated hardware combinations, achieving 99.74%–100% of the solution quality obtained through exhaustive search.
Scheduling effectiveness across workflows. For GRPO, execution plans generated by AReaL-Hex’s scheduler achieve 1.57×–1.68× the throughput of uniform allocation across 1.5B–14B models (Table 3). For 14B PPO, the plan generated by AReaL-Hex’s scheduler achieves 1.13× the throughput of static H20-for-rollout/H800-for-training allocation on the same pool of 48 H20 and 32 H800 GPUs (Appendix H.1). These results demonstrate the effectiveness of AReaL-Hex’s scheduling plans across GRPO and PPO on mathematical-reasoning workloads.
Scheduling gains on identical GPU pools. On identical GPU pools, AReaL-Hex achieves 1.57×–1.68× the throughput of uniform allocation across 1.5B–14B models (Table 3). For 14B PPO, the plan generated by AReaL-Hex’s scheduler also outperforms static H20-for-rollout/H800-for-training allocation by 1.13× by assigning more GPUs to the rollout bottleneck (Appendix H.1).
Comparison with stronger heterogeneous scheduling baselines. We construct four allocation baselines and compare them on the same simulated 56-GPU pool: 12 RTX 4090, 8 RTX 3090, 8 H20, 12 H100, and 16 A800 GPUs. All policies use identical hardware resources and total cost (Table 5 and Appendix H.1). The results are shown as follows:
The reported gains are averaged across the 1.5B, 7B, and 14B models. We will highlight these baseline definitions and simulation results in our updated draft.
We appreciate the reviewer’s careful assessment and constructive suggestions, which have helped us present the scheduler’s contribution and evaluation more clearly. We hope the clarifications on scheduling quality, hardware diversity, and baseline comparisons address your concerns. We welcome further questions and, if the concerns have been resolved, would be grateful for your consideration of a higher score.