acceptodds
Under review as a conference paper at ICLR 2027

AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs

Abstract

Maximizing the training throughput and cost-efficiency of reinforcement learning (RL) for large language models (LLMs) is essential to democratize this advanced technique. One promising but challenging approach is to deploy such a computational workflow over heterogeneous GPU clusters. Unlike conventional large-scale LLM pretraining, RL training generally decomposes into three coupled stages, i.e., rollout generation, reward computation, and policy/value updates, which exhibit markedly different compute intensities, memory footprints, and communication patterns. Recent research shows that fully asynchronous RL training can disaggregate these stages across disjoint hardware pools without sacrificing training stability, creating a great opportunity for real-world heterogeneous deployment. Towards this end, we present AReaL-Hex, a heterogeneity-aware asynchronous RL training system that effectively schedules how to execute rollout generation and policy model training over heterogeneous GPU clusters while enforcing data staleness bounds. Concretely, AReaL-Hex uses a two-phase scheduler: (i) a constrained search with mixed-integer linear programming to select per-stage parallelization strategies and workload assignments given a resource budget, and (ii) a graph-partitioning step that allocates heterogeneous GPUs and interconnects to maximize end-to-end throughput. Built atop a fully asynchronous RL architecture, AReaL-Hex maps HBM-I/O-bound generation and compute-bound optimization to more cost-efficient resources and balances their producer–consumer interactions to avoid both idleness and stale rollout trajectories. On the mathematical reasoning task with various model scales (1.5B, 7B, and 14B), compared to homogeneous deployments of state-of-the-art asynchronous RL systems: (i) When maintaining the same total budgets, AReaL-Hex delivers up to 1.50x higher training throughput; (ii) When achieving the same training throughput, AReaL-Hex results in up to 1.46x reduction in training cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Heterogeneous RL Training, Asynchronous Reinforcement Learning, GPU Resource Scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_SCZ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents AReaL-Hex, a heterogeneity-aware scheduling system for asynchronous RL training over heterogeneous GPUs. The paper formulates the joint optimization of resource allocation, parallel strategies, workload assignment, and data staleness constraints, and proposes a two-phase scheduler combining MILP-based search and graph partitioning. The authors implement the system and evaluate it on LLM RL workloads with 1.5B–14B models across heterogeneous GPU configurations. Experiments show that AReaL-Hex improves training throughput and cost efficiency compared with homogeneous asynchronous RL deployments, demonstrating the effectiveness of heterogeneous resource-aware scheduling.

Critical strengths
  1. The paper proposes a systematic solution for asynchronous RL training on heterogeneous GPU clusters, addressing an important and practical problem. This work focuses on the deployment challenges of asynchronous RL training in heterogeneous GPU environments and jointly considers rollout generation, model training, communication overhead, and data staleness constraints, rather than optimizing individual stages independently. The problem is highly relevant to the resource utilization challenges in large-scale LLM RL training and addresses the limitation of prior works that mainly optimize inference or training in isolation.
  2. The proposed two-phase scheduling framework effectively balances optimization quality and system scalability, with experiments demonstrating practical benefits. The authors design a scheduler that combines MILP-based search and graph partitioning to decompose the complex optimization of resource allocation, parallel strategy selection, and workload assignment. This design avoids exhaustive exploration of the large scheduling space while maintaining effective execution performance. Extensive experiments across different model scales, GPU configurations, and baselines demonstrate that AReaL-Hex achieves significant improvements in training throughput and cost efficiency.
Critical weaknesses
  1. The methodological novelty is mainly reflected in system optimization, while the theoretical analysis of the scheduling approach remains limited. Although the paper formulates heterogeneous asynchronous RL training as a joint scheduling problem and proposes a practical two-phase optimization framework, some components, particularly the graph partitioning stage, rely on heuristic strategies without theoretical guarantees regarding optimality or approximation quality. The effectiveness of the approach is primarily demonstrated through empirical evaluations, while further analysis of solution quality, optimality gaps, and robustness across different hardware topologies and workload distributions is limited.
  2. The evaluation could be further strengthened to better demonstrate the generality of the proposed approach. The paper demonstrates the effectiveness of AReaL-Hex through experiments across different model scales and GPU configurations, showing improvements over homogeneous asynchronous RL systems. However, the current evaluation focuses mainly on specific hardware settings and workload scenarios, with limited comparisons against alternative heterogeneous scheduling strategies.
Questions for authors
  1. The proposed scheduling method, particularly the graph partitioning stage, mainly relies on heuristic optimization strategies, but lacks theoretical guarantees regarding the quality of the generated schedules. The authors are encouraged to further analyze the robustness of the approach under different GPU topologies and workload distributions, and provide an analysis of the optimality gap between the obtained solutions and the optimal solutions.
  2. The paper mainly demonstrates the effectiveness of AReaL-Hex through comparisons with homogeneous asynchronous RL deployments. However, it remains unclear how much of the performance improvement comes from the scheduler itself versus the benefits of heterogeneous GPU resource allocation. The authors are encouraged to provide comparisons with stronger heterogeneous scheduling baselines to more accurately evaluate the effectiveness of the proposed scheduling algorithm.
3 comments

Thank you for your valuable comments and constructive suggestions. We address your concerns and questions below.

W1 & Q1: What are the scheduler’s optimality gap and guarantees, and how robust is it across hardware configurations and RL workloads?

Optimality and robustness across hardware configurations. We evaluate our algorithm against exhaustive search on two 56-GPU pools. Detailed results are provided in Table 18 and summarized as follows:

ConfigurationSearch TimeNormalized Solution Quality
H20 + H8002.00 min100.00%
RTX 4090, RTX 3090, H20, H100, and A800, simulated4.34 min99.74%

These results demonstrate that AReaL-Hex maintains near-optimal scheduling quality across the evaluated hardware combinations, achieving 99.74%–100% of the solution quality obtained through exhaustive search.

Scheduling effectiveness across workflows. For GRPO, execution plans generated by AReaL-Hex’s scheduler achieve 1.57×–1.68× the throughput of uniform allocation across 1.5B–14B models (Table 3). For 14B PPO, the plan generated by AReaL-Hex’s scheduler achieves 1.13× the throughput of static H20-for-rollout/H800-for-training allocation on the same pool of 48 H20 and 32 H800 GPUs (Appendix H.1). These results demonstrate the effectiveness of AReaL-Hex’s scheduling plans across GRPO and PPO on mathematical-reasoning workloads.

W2 & Q2: How much improvement comes from the scheduler, and how does it compare with stronger heterogeneous scheduling baselines?

Scheduling gains on identical GPU pools. On identical GPU pools, AReaL-Hex achieves 1.57×–1.68× the throughput of uniform allocation across 1.5B–14B models (Table 3). For 14B PPO, the plan generated by AReaL-Hex’s scheduler also outperforms static H20-for-rollout/H800-for-training allocation by 1.13× by assigning more GPUs to the rollout bottleneck (Appendix H.1).

Comparison with stronger heterogeneous scheduling baselines. We construct four allocation baselines and compare them on the same simulated 56-GPU pool: 12 RTX 4090, 8 RTX 3090, 8 H20, 12 H100, and 16 A800 GPUs. All policies use identical hardware resources and total cost (Table 5 and Appendix H.1). The results are shown as follows:

Constructed BaselineScheduling RuleAverage Throughput Gain
Naive allocationSplit each GPU type equally between rollout and training.1.76×
HBM/FLOPS-greedy allocationAssign the three GPU types with the highest HBM bandwidth per dollar to rollout and the remaining types to training.2.10×
Type-level hand tuningEvaluate all 30 nonempty assignments that place each GPU type entirely in rollout or training.1.19×
Count-level hand tuningSearch how many GPUs of each type to assign to each stage, allowing a GPU type to be split between stages.1.15×

The reported gains are averaged across the 1.5B, 7B, and 14B models. We will highlight these baseline definitions and simulation results in our updated draft.

We appreciate the reviewer’s careful assessment and constructive suggestions, which have helped us present the scheduler’s contribution and evaluation more clearly. We hope the clarifications on scheduling quality, hardware diversity, and baseline comparisons address your concerns. We welcome further questions and, if the concerns have been resolved, would be grateful for your consideration of a higher score.

I think my concerns are resolved and I keep my rating.

Thank you very much for reviewing our response. We sincerely appreciate your acknowledgment that our clarifications and additional experiments have resolved your concerns.

In particular, we have provided further evidence regarding the scheduler's near-optimal solution quality, robustness across heterogeneous hardware configurations, and throughput improvements over stronger scheduling baselines. We hope these results further demonstrate the effectiveness and contributions of AReaL-Hex.

Given that your concerns have been resolved, we would be very grateful if you could reconsider your rating and potentially increase it to better reflect the additional evidence and clarifications provided. If there are any remaining reservations regarding the overall assessment of our work, we would sincerely appreciate your further feedback.

Thank you again for your time and consideration.

Reviewer_GzL1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AReaL-HEX, a heterogeneity-aware asynchronous RL training system that jointly schedules rollout generation and policy model training across heterogeneous GPUs under data staleness constraints. Specifically, the authors propose a two-stage scheduling algorithm. The first stage employs constrained search and MILP optimization to determine parallelization strategies and workload assignments for training and rollout generation. The second stage adjusts GPU resource allocation through graph partitioning based on execution costs. The two stages alternate until the execution cost converges.

Critical strengths
  1. The paper analyzes the differences in computational and memory bandwidth requirements across different stages of asynchronous RL, while also considering data staleness constraints and performance coupling between stages. The performance analysis reveals the limitations of homogeneous deployments and motivates the need for joint optimization of resource allocation and execution strategies in heterogeneous GPU environments.
  2. The authors jointly model GPU resource allocation, parallelization strategies, and workload assignment, addressing the interdependencies among these decisions through a two-stage alternating optimization approach. Compared with methods that optimize rollout generation or training separately, this design accounts for both heterogeneous hardware characteristics and data staleness constraints in asynchronous RL.
  3. The authors evaluate the proposed method across different model scales and heterogeneous GPU configurations. Performance breakdowns, ablation studies, and comparisons with existing asynchronous RL systems further demonstrate the effectiveness of the scheduling approach and analyze the impact of different scheduling decisions on system performance.
Critical weaknesses
  1. HetRL has already proposed a joint scheduling approach for RL workloads on heterogeneous GPUs using multi-level search, hybrid SHA-EA search, and ILP optimization. In comparison, AReaL-HEX primarily introduces configurable data staleness constraints and a two-stage alternating optimization approach to determine resource allocation and execution strategies. The additional algorithmic contributions beyond existing methods are therefore not sufficiently substantial.
  2. Although the authors compare AReaL-HEX with HetRL on the same heterogeneous hardware, the end-to-end performance comparison is limited to 1.5B models trained with GRPO. Additional comparisons at larger model scales, such as 14B, would help verify whether the performance advantages of AReaL-HEX generalize across different model sizes.
  3. The evaluation primarily focuses on mathematical reasoning tasks and estimates rollout execution time based on generation length distributions and token throughput. However, in Agentic RL scenarios involving multi-turn tool use, trajectory execution time is also affected by tool invocation overhead, environment interactions, and waiting latency, making it difficult to estimate accurately based on token counts alone. Since AReaL-HEX relies on these estimates to optimize rollout replica configurations, workload assignments, and GPU resource allocation, its applicability to complex Agentic RL workloads may be limited.
Questions for authors

See my comments on weaknesses, in particular, weakness 3.

3 comments

Thank you for your valuable comments and constructive suggestions. We address your concerns and questions below.

W1: The additional contributions beyond HetRL’s joint scheduling of heterogeneous RL workloads are not sufficiently substantial.

We provide additional analysis below to clarify our contributions beyond HetRL.

AReaL-Hex incorporates the staleness bound into scheduling. HetRL jointly optimizes task placement and parallelism without an explicit configurable data staleness bound. AReaL-Hex incorporates η\eta into allocation, replica configuration, and workload assignment, accounting for staleness-induced waiting when balancing rollout and training.

AReaL-Hex reduces scheduling time compared with HetRL’s ILP variant. HetRL uses hierarchical search with SHA-EA or ILP. AReaL-Hex alternates execution-plan search and graph partitioning with iterative γ\gamma refinement. On 24 GPUs, scheduling takes 14.9 seconds versus approximately 1.3 minutes for HetRL’s ILP variant, a 5.2× speedup (Appendix H.2).

AReaL-Hex improves throughput across experiments. AReaL-Hex achieves 1.43× HetRL’s throughput for 1.5B GRPO (Appendix H.2) and 1.49× in the additional 14B GRPO comparison below, using identical hardware within each comparison.

We will integrate the above discussions into our updated draft.

W2: The HetRL comparison is limited to 1.5B GRPO; comparisons at larger model scales, such as 14B, are needed.

We provide an additional comparison between AReaL-Hex and HetRL for 14B GRPO using the same hardware setting as our main 14B experiment: 48 H20 and 8 H800 GPUs. AReaL-Hex achieves 1.49× the throughput of HetRL, as shown below.

SystemModel SizeGPU ConfigurationTraining Throughput (tokens/s)
HetRL14B48 H20 + 8 H8002,181
AReaL-Hex14B48 H20 + 8 H8003,250

We will include this additional comparison in our updated draft.

W3 & Q1: Token-based rollout estimates may not capture tool invocation, environment interaction, and waiting latency in Agentic RL.

Agentic RL costs can be integrated into our scheduling framework. Tool invocation, environment interaction, and waiting latency can be explicitly modeled and incorporated into AReaL-Hex's rollout cost estimates, accounting for their overlap with GPU computation and any shared resource constraints. These extensions refine the cost model while preserving our core scheduling principle: jointly optimizing resource allocation, parallelism, and workload assignment to balance rollout and training under the specified staleness bound. We will clarify this extension in our updated draft.

We sincerely thank the reviewer for the valuable feedback, which has helped us clarify AReaL-Hex’s contribution and strengthen the comparison with HetRL. We hope the clarifications and additional 14B results address your concerns, and we welcome any further questions. If you find these concerns resolved, we would be grateful if you would consider a higher score.

Thank you for the clarifications and additional experiments. However, my concern about agentic RL remains unresolved. The current token-based cost model does not adequately capture dynamic tool interactions, waiting latency, and resource contention. Simply adding these costs as an extra term does not address their dependence on scheduling decisions. Without a concrete formulation or empirical validation, I remain unconvinced of the method's applicability to agentic RL. I will maintain my original score.

Thank you for reviewing our clarifications and additional experiments. We further explain the Agentic RL extension by formulating scheduling-dependent tool interactions, waiting, and contention within our objective, graph partitioning, and execution-plan search.

Scheduling objective and agentic rollout cost. We retain the objective in Eq. (1), min⁡σ,τ,DT,DImax⁡{CT,CI}\min_{\sigma,\tau,\mathcal D_T,\mathcal D_I}\max\{C_T,C_I\}, under the data staleness bound η\eta. Agentic trajectories interleave generation, tool calls, and environment steps, with later generation depending on returned results. Changing replica counts yψy_\psi and workload assignments xψx_\psi changes when tool requests arrive, how they compete for shared resources, and when generation resumes. Accordingly, CRolloutC_{Rollout} would be estimated from this interleaved execution, including waiting and overlap, rather than treating tool latency as a fixed addition to CRewardC_{Reward}.

Graph partitioning and resource refinement. The GPU graph GG and its bandwidth-based partition score can still generate candidate allocations. However, an agentic tool bottleneck can keep CIC_I high even when additional rollout GPUs provide little benefit. We would therefore evaluate candidates from neighboring [γL,γH][\gamma_L,\gamma_H] ranges in both directions and retain the plan with the lowest max⁡{CT,CI}\max\{C_T,C_I\}, using the execution-plan costs defined below. This replaces reliance on CT<CIC_T<C_I alone to discard part of the search range. GPU swaps within a candidate range would also be considered when tool-access costs depend on placement; the corresponding links enter the resource-capacity constraints. The partition topology remains unchanged, while candidate evaluation and resource refinement account for agentic contention.

Training-plan search and shared resources. For each candidate allocation with dedicated tool resources, training-plan search retains its existing structure, using the updated trajectory lengths. If tools share CPU or network resources with training, we would search resource reservations UT,κU_{T,\kappa} and UI,κU_{I,\kappa} alongside GPU allocation, subject to UT,κ+UI,κ≤UκU_{T,\kappa}+U_{I,\kappa}\le U_\kappa, where UκU_\kappa is the total capacity of resource κ\kappa. Training costs are evaluated under UT,κU_{T,\kappa}, and the rollout MILP uses UI,κU_{I,\kappa}. Thus, contention with training affects both CTC_T and CIC_I.

Rollout-plan search and MILP extensions. We retain xψx_\psi, yψy_\psi, and the candidate completion-time bound Θ^\widehat{\Theta}. We divide time into discrete slots. A binary variable uo,ψ,t∈{0,1}u_{o,\psi,t}\in\{0,1\} indicates whether operation oo starts in slot tt for a trajectory assigned to configuration ψ\psi. Service durations and resource demands are fixed inputs to each feasibility query; scheduling determines start times and waiting. We add the following constraints:

  • Operation timing and tool dependencies. Denote operation oo's start and finish times by to−t_o^{-} and to+t_o^{+}, and its service duration under configuration ψ\psi by ℓo,ψ\ell_{o,\psi}. Each operation starts once: ∑ψ,tuo,ψ,t=1\sum_{\psi,t}u_{o,\psi,t}=1, with to−=∑ψ,tt uo,ψ,tt_o^{-}=\sum_{\psi,t}t\,u_{o,\psi,t} and to+=∑ψ,t(t+ℓo,ψ)uo,ψ,tt_o^{+}=\sum_{\psi,t}(t+\ell_{o,\psi})u_{o,\psi,t}. Service duration excludes queueing and can be estimated through profiling. Each generation–tool or tool–generation dependency requires to+≤to′−t_o^{+}\le t_{o'}^{-}.
  • Shared-resource capacity. Let ao,ψ,κa_{o,\psi,\kappa} denote the amount of resource κ\kappa reserved by operation oo under configuration ψ\psi. At every time tt, we require ∑o,ψao,ψ,κ∑t′:t′≤t<t′+ℓo,ψuo,ψ,t′≤Uκ\sum_{o,\psi}a_{o,\psi,\kappa}\sum_{t':t'\le t<t'+\ell_{o,\psi}}u_{o,\psi,t'}\le U_\kappa, using UI,κU_{I,\kappa} as the bound for resources shared with training. This covers CPU workers, tool-service slots, and network bandwidth. Replica execution capacity follows yψy_\psi, and memory constraints include KV caches retained during tool waits.
  • Trajectory assignment and completion. Operations belonging to the same trajectory use the same configuration assignment, and xψx_\psi counts trajectories assigned to ψ\psi. We retain ∑ψxψ=B\sum_\psi x_\psi=B and the GPU-count constraints, and require to+≤Θ^t_o^{+}\le\widehat{\Theta} for every operation. These constraints make waiting depend on both tool readiness and resource availability. Binary search over Θ^\widehat{\Theta} remains applicable because a feasible schedule remains feasible at a larger time bound.

Therefore, AReaL-Hex's overall algorithmic framework remains applicable to Agentic RL with appropriate extensions to cost modeling, rollout constraints, and resource refinement.

We will conduct Agentic RL experiments and report the results in a subsequent revision, as implementation and end-to-end evaluation require more time than remains in the rebuttal period. We would be grateful if you could reconsider your rating and consider raising it in light of this analysis and our additional evidence. Thank you again for your time and feedback.

Comments on the paper

5 comments

Thank you for your valuable comments and constructive suggestions. We address your concerns and questions below.

W1: Simulated and real results are presented together, making the scope of the evidence unclear.

We agree that real experiments and simulations should be distinguished more clearly. The classification is as follows:

  • Real experiments include: end-to-end training throughput (§5.2), stage latency (§5.3), weight-synchronization measurements (§5.3 and Appendix H.4), resource-allocation ablations on H800/H20 clusters (§5.4), and the 14B PPO comparison against static stage allocation (Appendix H.1).
  • Simulation experiments include: comparisons against naive, greedy, type-level, and count-level allocation policies on the five-type GPU pool (§5.4 and Appendix H.1), the selected allocations (Appendix H.1), and solution quality relative to exhaustive search (Appendix H.3).

We will clearly label real experiments and simulations in the updated paper.

W2: The staleness–quality tradeoff is underemphasized; throughput gains can be offset by quality degradation and slower time to target.

Staleness selection is orthogonal to our scheduling design and accommodated through η\eta. The training recipe selects η\eta according to its learning-quality requirements. AReaL-Hex treats this bound as an input and optimizes resource allocation, parallelism, and workload assignment subject to it. Scheduling gains therefore come from more efficient execution under the chosen bound, without requiring greater staleness.

Discussion on scheduling quality. We agree that throughput gains from greater staleness can be offset by lower quality and slower time to target, and we quantify this tradeoff in our paper. Increasing η\eta from 2 to 4 improves mean throughput by 10.4% but reduces mean final reward from 0.917 to 0.880. In the separate time-to-target evaluation, we measure 10.8 hours to reach a reward of 0.75 at η=4\eta=4, compared with 9.4 hours at η=2\eta=2 (Appendix H.6).

Re-optimization maintains scheduling gains without relaxing the staleness bound. Our proposed re-optimization mechanism updates resource allocation and execution plans when response lengths change, running on CPUs alongside training while retaining the chosen η\eta. Using 160 CPU cores, AReaL-Hex generates a schedule in 0.3 minutes for a 56-GPU configuration (Appendix D, Table 7). In our tested 14B workflow on 88 GPUs, response lengths remain stable for about 20 iterations on average, or roughly 600 minutes at 30 minutes per iteration. Over this interval, reconfiguration and relaunch downtime below 36 seconds contributes less than 0.1% interruption overhead. In an additional 1.5B experiment, re-optimization improves throughput from 24,200 to 27,300 tokens/s, a 1.13× gain (Appendix E). This preserves scheduling efficiency within the chosen staleness–quality tradeoff.

We will make this distinction and the supporting results clearer in the updated draft.

W3 & Q2: What performance bound holds under sharp response-length drift, and how are ω\omega, ρ\rho, and re-optimization and relaunch overhead evaluated?

Performance under response-length drift. In an additional 1.5B experiment, the original schedule achieves 24,200 tokens/s after the response-length shift, while re-optimization raises throughput to 27,300 tokens/s (Appendix E, Table 8). The original schedule therefore retains 88.6% of the updated schedule’s throughput, and re-optimization provides a 1.13× gain.

Selection rules for ω\omega and ρ\rho. We select ω\omega to fit within the observed response-length stability period and ρ\rho to distinguish meaningful changes from routine variation. For example, in our 14B workflow on 88 GPUs, response lengths remain stable for about 20 iterations on average. At 30 minutes per iteration, this corresponds to a stability period of approximately 600 minutes.

Re-optimization and relaunch overhead. Training continues under the current schedule while the new schedule is computed. Using 160 CPU cores, AReaL-Hex generates a schedule in 0.3 minutes for a 56-GPU configuration (Appendix D, Table 7). Over 600 minutes of training, reconfiguration and relaunch downtime below 36 seconds contributes less than 0.1% interruption overhead.

Q1: How does graph partitioning align with minimizing max⁡{CT,CI}\max\{C_T,C_I\}, and what guarantees apply?

Graph partitioning aligns with the objective through iterative refinement. For each partition, training-plan search and MILP-based rollout optimization estimate CTC_T and CIC_I. When CI>CTC_I>C_T, we reduce γ\gamma to allocate more resources to rollout; when CT>CIC_T>C_I, we increase γ\gamma to allocate more resources to training. Execution plans are refined after each adjustment, and convergence is checked using max⁡{CT,CI}\max\{C_T,C_I\}. Graph partitioning is therefore guided by the resulting execution costs throughout the iterations, so the combined procedure targets min⁡max⁡{CT,CI}\min\max\{C_T,C_I\}.

Q3: What share of 235B MoE training time is spent on weight updates, and could this dominate at larger scales or on slower networks?

235B MoE weight-update fraction. The maximum weight-update fraction for the 235B MoE is 15.52%; the 16.40% value corresponds to the 14B dense model at η=8\eta=8. More details of weight-synchronization overhead across model scales and staleness bounds are provided in Appendix H.4, Table 20, of our paper. We provide the 235B MoE results as follows:

Staleness Boundη=0\eta=0η=2\eta=2η=4\eta=4η=8\eta=8
235B MoE12.18%13.31%14.44%15.52%

Weight synchronization at larger model scales. Weight synchronization does not dominate in our evaluated larger-model configurations. Larger models require transferring more weights, but rollout inference and model training also become more expensive. Under comparable workload and execution settings, the increase in computation helps maintain a broadly stable weight-update share of total training time. We observe similar weight-update fractions across the larger models (Table 20): at η=4\eta=4, the weight-update shares of total training time are 15.98%, 15.24%, and 14.44% for 7B, 14B, and 235B, respectively; at η=8\eta=8, they are 16.01%, 16.40%, and 15.52%. Thus, the measured ratio remains broadly stable as model size increases across these configurations, rather than growing toward dominance.

Weight synchronization on low-bandwidth interconnects. Our H20–H800 experiments use an inter-machine bandwidth of 1.5 GB/s (§5.1). Weight synchronization accounts for 13.54% of iteration time in the reported breakdown (§5.3); therefore, weight synchronization does not dominate execution at this bandwidth. Our scheduler also includes CUpdateC_{Update} when comparing execution plans.

Dear Reviewer xjyu,

As the discussion period draws to a close, we would greatly appreciate your feedback on our rebuttal. Our response clarifies the distinction between real-GPU experiments and simulations, quantifies the staleness–quality and time-to-target tradeoffs, and explains how re-optimization maintains scheduling gains under response-length changes. We also provide further analysis of objective alignment and weight-synchronization overhead.

We would appreciate knowing whether these clarifications and supporting results address your concerns. If any concerns remain, we would be glad to address them and answer further questions.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.