- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a hierarchical MoE routing control framework for agentic reinforcement learning. It encourages turn-level expert specialization through an operation-aware mutual information objective and promotes token-level routing consistency through selective regularization. Policy-entropy gating and auxiliary gradient scaling are further used to stabilize training, while preserving the original top-k routing architecture.
Experiments on AppWorld and AutomationBench with multiple RL algorithms demonstrate improvements in task performance, routing structure, and inference efficiency.
Critical strengths
-
The method leverages existing operation labels from tool calls to guide native MoE expert routing without predefined expert assignments or inference-time architectural modifications. Combining operation-level specialization with selective token-level consistency is a practical design.
-
The experiments cover two agentic benchmarks and multiple RL algorithms. Ablations examine operation labels, routing-control components, load balancing, and entropy gating. Comparisons with random and application-name labels provide useful evidence for the value of operation-aware supervision.
Critical weaknesses
-
The load-balancing loss in Appendix C.5 differs from the claimed Megatron implementation by a normalization factor of under the same expert-frequency definitions and coefficient convention, corresponding to a factor of 16,384 for 128 experts. Although this difference could be absorbed through coefficient tuning, the actual implementation and coefficient mapping remain unclear. Furthermore, Tables 2 and 3 report different GRPO baselines on AppWorld test-challenge (38.80 vs. 46.28) while reporting the same load-balancing result (45.56). Although the paper distinguishes early exploratory experiments from the main settings, the mapping between these results, training configurations, and checkpoints is insufficiently documented. Consequently, the relative effectiveness of load balancing and the claimed necessity of operation labels remain difficult to assess reliably.
-
The main text describes routing control through an additive objective, whereas Appendix D.5 introduces additional token-wise scaling of auxiliary router gradients based on incoming gradients. Tokens with zero incoming gradients receive no auxiliary updates, making this mechanism a gradient-dependent selection rule rather than merely a loss-coefficient adjustment. Although the implementation is disclosed in the appendix, its contribution is not isolated from the proposed turn-level and token-level routing objectives. Without a controlled ablation, it remains unclear which components are responsible for the reported improvements in performance and training stability.
Questions for authors
-
The load-balancing objective in Appendix C.5 differs from the standard Megatron formulation by a normalization factor of . Could the authors clarify the actual loss implementation, coefficient convention, and corresponding code? Additionally, could they provide a sensitivity analysis of the load-balancing coefficient to establish whether the comparison is robust to the choice of loss scaling and hyperparameters?
-
Tables 2 and 3 report different GRPO test-challenge TGC scores (38.80 and 46.28), while both report the same load-balancing score of 45.56. Could the authors clarify the training configurations, random seeds, and checkpoints associated with these results? In particular, since Table 2 shows that load balancing can improve performance on challenging tasks without operation labels, how is the claim that operation labels are necessary for such improvements justified?
-
Appendix D.5 introduces token-wise gradient scaling based on incoming gradients, making the actual optimization procedure different from the simple additive objective presented in the main text. Could the authors compare training with and without this scaling mechanism while keeping the routing objectives, entropy gate, and other settings fixed? This would help isolate the contributions of the routing regularization objectives and the gradient-dependent update mechanism.
-
Table 1 shows considerable variation in improvements across RL algorithms and tasks, including settings with negative or negligible gains. Could the authors report results across multiple independent training seeds, together with uncertainty estimates, and clarify the checkpoint selection protocol? This would help establish whether the proposed framework consistently benefits different RL algorithms rather than only selected configurations.
2 comments
Q4: Results across Multiple Runs
For the repeated experiments, we add a few repeated runs with random seeds, shown in the table below. The checkpoint selection rule is the same as Table 1, picking the best-performing checkpoints on the dev set. The reported empirical improvement is stable across different runs. We will include the multi-run results to the paper's final version.
| Method | Runs | Normal TGC, mean ± SD | Challenge TGC, mean ± SD |
|---|---|---|---|
| PPO | 3 | 61.82 ± 3.21 | 34.97 ± 2.92 |
| PPO + Ours | 3 | 60.05 ± 1.22 | 37.03 ± 2.42 |
| GRPO | 3 | 70.01 ± 2.20 | 42.27 ± 3.35 |
| GRPO + Ours | 3 | 74.83 ± 1.83 | 48.76 ± 1.81 |
| LOOP | 3 | 66.87 ± 3.02 | 39.77 ± 3.47 |
| LOOP + Ours | 3 | 72.42 ± 1.69 | 49.12 ± 0.72 |
| GiGPO | 3 | 53.57 ± 2.29 | 32.69 ± 1.41 |
| GiGPO + Ours | 3 | 58.73 ± 2.25 | 36.17 ± 1.69 |
Thank you so much for taking time to read the details of our paper, and for the positive assessment. The raised concerns are all helpful and constructive for improving the clarity. If you find the responses below sufficient to address your concerns, we truly appreciate it if you consider raising your score.
Q1: Load-Balancing Implementation
Q1.1: Load-Balancing Objective
There is a typo in Appendix C.5 about the load-balancing loss. We confirm that the code implementation of load-balancing loss is:
cM∑l,efl,epˉl,e,
where c is the "moe_aux_loss_coeff". We directly use the Megatron's load-balancing loss without modification. We also provide a psudo-code block below to help you check the details.
Q1.2: Load-Balancing Coefficients
For the choices of load-balancing coefficient, we provide an additional ablation study below. We test two coefficient magnitudes and also check the combined effect of load-balancing and token-level control. The magnitudes of 1e-4 is the highest level to allow for stable training. We use 1e-4 across all of the load-balancing experiments in the paper.
Q2: Inconsistency of GRPO and Load-Balancing Results
For GRPO baselines, we summarize the origin of number difference below. The differences are either from different checkpoint selections or older experiments. We will update the numerical difference caused by older experiments to align with Table 1. For the different checkpoint selections, we will highlight the settings in each table's caption.
train/step)The load-balancing results are all based on our latest GRPO config, which aligns with Table 1. After we update the GRPO results, the inconsistency between GRPO baseline and load-balancing would be resolved.
Q3: Effect of Injected Gradient Scaling
The introduced gradient scaling trick is important for the training stability, as it prevents the per-step update from being dominated by routing control signals. We show the results with and without gradient scaling in the table below, where we only add the token-level control to test stability. Without the gradient scaling, the training becomes crashed at 1e-6, while adding gradient scaling prevents such crash.