acceptodds
Under review as a conference paper at ICLR 2027

Structuring MoE Expert Selection for Agentic Reinforcement Learning

Abstract

Long-horizon LLM agents are frequently implemented using sparse mixture-of-experts (MoE) models, yet the co-design of agentic behavior and MoE structures remains underexplored. In this work, we comprehensively study the connections between agentic post-training and MoE expert selection. In off-the-shelf MoE models, we observe expert selection exhibits a specialized structure that naturally aligns with agentic trajectories. Specifically, expert routing overlaps more between turns where the agent performs semantically similar operations (e.g., READ, UPDATE) than between turns with differing operations. However, standard RL algorithms ignore this specialization, allowing the MoE routing to go uncontrolled during training, which empirically limit task performance and inference efficiency. To address this, we introduce a hierarchical routing control framework for agentic tasks. We explicitly encourage turn-level expert selections to align with agentic operations while regularizing token-level expert selections to maintain local consistency. To resolve stability issues that arise during post-training with the proposed methods, we further introduce an entropy-gated control mechanism. Overall, our routing control framework achieves over 10-point improvements in success rate on all evaluated benchmarks. These results demonstrate that agentic trajectory structure provides an effective signal for optimizing MoE capacity during RL post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic RL, MoE Models, Expert Usage Structure

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_byCUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a hierarchical MoE routing control framework for agentic reinforcement learning. It encourages turn-level expert specialization through an operation-aware mutual information objective and promotes token-level routing consistency through selective regularization. Policy-entropy gating and auxiliary gradient scaling are further used to stabilize training, while preserving the original top-k routing architecture.

Experiments on AppWorld and AutomationBench with multiple RL algorithms demonstrate improvements in task performance, routing structure, and inference efficiency.

Critical strengths
  1. The method leverages existing operation labels from tool calls to guide native MoE expert routing without predefined expert assignments or inference-time architectural modifications. Combining operation-level specialization with selective token-level consistency is a practical design.

  2. The experiments cover two agentic benchmarks and multiple RL algorithms. Ablations examine operation labels, routing-control components, load balancing, and entropy gating. Comparisons with random and application-name labels provide useful evidence for the value of operation-aware supervision.

Critical weaknesses
  1. The load-balancing loss in Appendix C.5 differs from the claimed Megatron implementation by a normalization factor of M2M^2 under the same expert-frequency definitions and coefficient convention, corresponding to a factor of 16,384 for 128 experts. Although this difference could be absorbed through coefficient tuning, the actual implementation and coefficient mapping remain unclear. Furthermore, Tables 2 and 3 report different GRPO baselines on AppWorld test-challenge (38.80 vs. 46.28) while reporting the same load-balancing result (45.56). Although the paper distinguishes early exploratory experiments from the main settings, the mapping between these results, training configurations, and checkpoints is insufficiently documented. Consequently, the relative effectiveness of load balancing and the claimed necessity of operation labels remain difficult to assess reliably.

  2. The main text describes routing control through an additive objective, whereas Appendix D.5 introduces additional token-wise scaling of auxiliary router gradients based on incoming gradients. Tokens with zero incoming gradients receive no auxiliary updates, making this mechanism a gradient-dependent selection rule rather than merely a loss-coefficient adjustment. Although the implementation is disclosed in the appendix, its contribution is not isolated from the proposed turn-level and token-level routing objectives. Without a controlled ablation, it remains unclear which components are responsible for the reported improvements in performance and training stability.

Questions for authors
  1. The load-balancing objective in Appendix C.5 differs from the standard Megatron formulation by a normalization factor of M2M^2. Could the authors clarify the actual loss implementation, coefficient convention, and corresponding code? Additionally, could they provide a sensitivity analysis of the load-balancing coefficient to establish whether the comparison is robust to the choice of loss scaling and hyperparameters?

  2. Tables 2 and 3 report different GRPO test-challenge TGC scores (38.80 and 46.28), while both report the same load-balancing score of 45.56. Could the authors clarify the training configurations, random seeds, and checkpoints associated with these results? In particular, since Table 2 shows that load balancing can improve performance on challenging tasks without operation labels, how is the claim that operation labels are necessary for such improvements justified?

  3. Appendix D.5 introduces token-wise gradient scaling based on incoming gradients, making the actual optimization procedure different from the simple additive objective presented in the main text. Could the authors compare training with and without this scaling mechanism while keeping the routing objectives, entropy gate, and other settings fixed? This would help isolate the contributions of the routing regularization objectives and the gradient-dependent update mechanism.

  4. Table 1 shows considerable variation in improvements across RL algorithms and tasks, including settings with negative or negligible gains. Could the authors report results across multiple independent training seeds, together with uncertainty estimates, and clarify the checkpoint selection protocol? This would help establish whether the proposed framework consistently benefits different RL algorithms rather than only selected configurations.

2 comments

Thank you so much for taking time to read the details of our paper, and for the positive assessment. The raised concerns are all helpful and constructive for improving the clarity. If you find the responses below sufficient to address your concerns, we truly appreciate it if you consider raising your score.

Q1: Load-Balancing Implementation

Q1.1: Load-Balancing Objective

There is a typo in Appendix C.5 about the load-balancing loss. We confirm that the code implementation of load-balancing loss is:

cM∑l,efl,epˉl,ecM \sum_{l,e} f_{l,e} \bar{p}_{l,e},

where c is the "moe_aux_loss_coeff". We directly use the Megatron's load-balancing loss without modification. We also provide a psudo-code block below to help you check the details.

c = args.moe_aux_loss_coeff
M, k = 128, 8
B = number_of_microbatches
S = tensor_context_parallel_group_size  # 8 in these runs

# Scheduler averages auxiliary gradients over microbatches (CP=1).
MoEAuxLossAutoScaler.set_loss_scale(main_loss_scale / B)

for microbatch in update:
    for layer in moe_layers:
        if c > 0:
            p = softmax(current_router_logits[layer], dim=-1)
            assignments = multi_hot(topk(p, k).indices, M)
            counts = all_reduce_sum(assignments.sum(0)).detach()
            T = p.shape[0] * S

            aux = c * M / (k * T**2) * dot(counts, p.sum(0))
            dispatch_weights = MoEAuxLossAutoScaler.apply(
                dispatch_weights, aux
            )  # forward identity; injects auxiliary gradient backward

            tracker[layer] += detach(aux / c)

    backward(policy_loss)               # auxiliary gradient already attached

logged_aux = sum(tracker) / B            # layer SUM; coefficient removed

Q1.2: Load-Balancing Coefficients

For the choices of load-balancing coefficient, we provide an additional ablation study below. We test two coefficient magnitudes and also check the combined effect of load-balancing and token-level control. The magnitudes of 1e-4 is the highest level to allow for stable training. We use 1e-4 across all of the load-balancing experiments in the paper.

SettingcoefficientLoad CV ↓Normal TGC ↑Challenge TGC ↑
GRPO baseline01.24169.6446.28
Load balancing10−410^{-4}1.15574.4045.56
Load balancing10−310^{-3}0.81246.4327.58
Load balancing + token control10−410^{-4}1.32169.0543.17
Load balancing + token control10−310^{-3}crashedcrashedcrashed

Q2: Inconsistency of GRPO and Load-Balancing Results

For GRPO baselines, we summarize the origin of number difference below. The differences are either from different checkpoint selections or older experiments. We will update the numerical difference caused by older experiments to align with Table 1. For the different checkpoint selections, we will highlight the settings in each table's caption.

Paper table(s)Reported TGC: Normal / Challenge (%)Configuration / experiment contextRecorded seeds: training / rolloutCheckpoint (train/step)Selection rule / clarification
Table 169.6 / 40.5Baseline with explicit operation-label and per-layer routing-logging settings1234 / 42179Highest dev-set TGC. Exact test scores: 69.6429 / 40.5276.
Table 266.70 / 38.80Same configuration as Tables 8 and 101234 / 42199Final checkpoint, rounded to one decimal. At two decimals, the logged values are 66.67 / 38.85.
Table 369.64 / 46.28Older baseline, lacking the explicit operation-label logging1234 / 42199Final checkpoint.
Tables 8 and 1066.67 / 38.85Same baseline run as Table 11234 / 42199Final checkpoint.
Tables 7 and 964.88 / 35.97Older baseline, lacking routing control1234 / 42199Final checkpoint.

The load-balancing results are all based on our latest GRPO config, which aligns with Table 1. After we update the GRPO results, the inconsistency between GRPO baseline and load-balancing would be resolved.

Q3: Effect of Injected Gradient Scaling

The introduced gradient scaling trick is important for the training stability, as it prevents the per-step update from being dominated by routing control signals. We show the results with and without gradient scaling in the table below, where we only add the token-level control to test stability. Without the gradient scaling, the training becomes crashed at 1e-6, while adding gradient scaling prevents such crash.

Token-level coefficientTest Challenge TGC w/o gradient scalingTest Challenge TGC w/ gradient scaling
2e−6crashedcrashed
1e−6crashed32.13%
1e−718.13%42.21%

Q4: Results across Multiple Runs

For the repeated experiments, we add a few repeated runs with random seeds, shown in the table below. The checkpoint selection rule is the same as Table 1, picking the best-performing checkpoints on the dev set. The reported empirical improvement is stable across different runs. We will include the multi-run results to the paper's final version.

MethodRunsNormal TGC, mean ± SDChallenge TGC, mean ± SD
PPO361.82 ± 3.2134.97 ± 2.92
PPO + Ours360.05 ± 1.2237.03 ± 2.42
GRPO370.01 ± 2.2042.27 ± 3.35
GRPO + Ours374.83 ± 1.8348.76 ± 1.81
LOOP366.87 ± 3.0239.77 ± 3.47
LOOP + Ours372.42 ± 1.6949.12 ± 0.72
GiGPO353.57 ± 2.2932.69 ± 1.41
GiGPO + Ours358.73 ± 2.2536.17 ± 1.69
Reviewer_swMrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how agentic reinforcement learning affects expert routing in MoE models and proposes hierarchical routing control. Section 3.1 uses mutual information between operation labels and turn-level expert distributions to encourage similar expert preferences for the same operation. Section 3.2 imposes a local consistency constraint on similar routing patterns between adjacent tokens. Section 3.3 gates auxiliary routing gradients using policy entropy to reduce the risk of training collapse. The authors evaluate task performance, routing changes, and inference throughput on AppWorld and AutomationBench using PPO, GRPO, LOOP, and GiGPO.

Critical strengths
  1. The method uses the operational structure of agent trajectories. Operation labels guide turn-level expert specialization, while similarity between adjacent tokens' expert sets encourages local routing consistency. In Appendix F.1, operation labels outperform random labels and application-name labels, supporting this design choice.

  2. Component analyses support the performance gains. On the AppWorld challenge split, GRPO's TGC increases from 40.5% to 49.2%; on AutomationBench, LOOP's score improves by 11.34 percentage points. The component ablation in Table 10 separates the effects of turn-level and token-level control, and Figure 4 shows that entropy gating can mitigate training collapse. (Table 1, Figure 4, Appendix F.7)

Critical weaknesses
  1. The overall motivation is largely intuitive. The routing constraints follow two assumptions: at the turn level, the same operation should prefer similar experts, motivating mutual information between operation labels and expert distributions; at the token level, routing changes between adjacent tokens within the same field may be noise and reduce efficiency, motivating local consistency constraints. Although the experimental metrics improve, the authors do not sufficiently explain why these two routing constraints are reasonable. (Sections 3.1–3.2)

  2. The operation partitioning is task-specific, limiting generality. Appendix D.6 defines separate operation-partitioning rules for AppWorld and AutomationBench. Suitable partitions may be difficult to design in new settings, and different partitioning schemes may affect whether the method remains effective.

  3. The experiments lack an assessment of randomness. The paper does not report independent training seeds or confidence intervals for the main results, making it difficult to determine whether the gains are consistent.

Questions for authors
  1. The two routing constraints in Sections 3.1–3.2 are mainly motivated by intuition. Could the authors provide additional experiments or theoretical analysis explaining why these designs are reasonable and under what conditions they are effective?

  2. Appendix D.6 defines separate operation-partitioning rules for AppWorld and AutomationBench. Could the authors provide more general principles or guidance for defining operations, to help assess the method's applicability to new settings?

  3. Existing MoE RL work proposes mechanisms such as Vanilla Routing Replay (R2) and Rollout Routing Replay (R3) to stabilize training [1, 2]. Is the proposed routing control compatible with these mechanisms? Could conflicts arise when they are used together?

3 comments

Thank you so much for taking time to read the details of our paper, and for the simplicity of your review. The raised concerns are all helpful and constructive for improving the clarity. If you find the responses below sufficient to address your concerns, we truly appreciate your re-consideration for the score.

Q1: Overall Motivation of Routing Constraints

Thank you for this motivation concern. We agree that the detailed evidences are scattered across the paper, not very easy to understand at the first glance. We summarize the motivation as follows:

  • In turn-level, we aim to specialize the expert selection to each operation label, to fully utilize the capacity of each expert. We observe a default alignment between operation labels and expert distributions. We then try to strengthen this inductive bias via MI loss, to improve the turn-level expert specialization, which has been shown to positively affect the performance [1].

  • In token-level, we aim to reduce the expert switch for inference efficiency. We tested various schemes and find the chosen token-level control scheme can achieve effective routing control and training stability at the same time.

We then list the most important evidences into the table below, and will update the paper to highlight these points.

ComponentQuestionExisting evidence in the paper
Turn-levelWhy using operation labels?Fig. 2(a) and Fig. 8(a) show that organizing turns by operation labels is the best aligned with default routing behavior.
Turn-levelWhy using MI?In Eq. 3.1 - 3.3 and their related paragraphs, we explain the decomposition I(E;Yo)=H(E)−H(E∣Yo)I(E;Y_o)=H(E)-H(E\mid Y_o) and how operation-conditioned log-ratio signals reinforce expert preferences.
Token-levelWhy choosing this form?In Appendix C, we compare a wide range of token-level routing control schemes, and analyze the reason why they are ineffective. We design the token-level control from these analysis.
Token-levelWhy enforcing consistency within the same field?Fig. 2(b) shows that the expert selection similarity within the same field is naturally higher than cross-field.
Token-levelWhy routing consistency improve efficiency, and under what conditions?Appendix E.3 provides a controlled routing experiment, which credits the efficiency bonus to larger expert-level batching. This efficiency bonus is effective under inference batch size larger than 1.

[1] https://arxiv.org/abs/2602.17038

Q2: Operation Partition Is Task-Specific

Our operation labels describe the functional role of an interaction with respect to task-relevant objects and environment state. The same core categories can therefore apply across different interaction interfaces, as illustrated below.

Agentic settingREAD: acquire informationCREATE: introduce a new objectUPDATE: modify an existing objectDELETE: remove an existing object
GUI automation [2, 3]Capture a screenshot to inspect the current interfaceCreate a new document through the applicationChange an existing document’s content or an application settingDelete a document
Web interaction [4, 5]Inspect a product page or retrieve search resultsPost a new commentEdit a saved shipping addressRemove a saved address
Coding agents [6, 7]Inspect source code or read an execution logAdd a new source file or test filePatch an existing functionRemove an obsolete file
Database agents [8, 9]Query records with SELECTInsert a new recordModify an existing recordDelete a record
Calendar assistants [10, 11]Check availability or inspect an eventSchedule a new eventReschedule an existing eventCancel an event

The extraction procedure depends on the interface: API-based environments expose function names and arguments, whereas GUI environments require interpreting the selected control and its intended effect. For example, a click on "View details" corresponds to READ, while a click on "Create event" corresponds to CREATE. Thus, labels are assigned to the operation performed rather than to low-level primitives such as clicking or typing.

[2] https://arxiv.org/abs/2404.07972

[3] https://arxiv.org/abs/2405.14573

[4] https://arxiv.org/abs/2307.13854

[5] https://arxiv.org/abs/2403.07718

[6] https://arxiv.org/abs/2310.06770

[7] https://arxiv.org/abs/2407.16741

[8] https://arxiv.org/abs/2411.07763

[9] https://arxiv.org/abs/2510.05318

[10] https://arxiv.org/abs/2406.13352

[11] https://arxiv.org/abs/2601.11957

Q3: Experiments Lack Randomness Assessment

For the repeated experiments, we add a few repeated runs with random seeds, shown in the table below. The checkpoint selection rule is the same as Table 1, picking the best-performing checkpoints on the dev set. The reported empirical improvement is stable across different runs. We will include the multi-run results to the paper's final version.

MethodRunsNormal TGC, mean ± SDChallenge TGC, mean ± SD
PPO361.82 ± 3.2134.97 ± 2.92
PPO + Ours360.05 ± 1.2237.03 ± 2.42
GRPO370.01 ± 2.2042.27 ± 3.35
GRPO + Ours374.83 ± 1.8348.76 ± 1.81
LOOP366.87 ± 3.0239.77 ± 3.47
LOOP + Ours372.42 ± 1.6949.12 ± 0.72
GiGPO353.57 ± 2.2932.69 ± 1.41
GiGPO + Ours358.73 ± 2.2536.17 ± 1.69

Q4: Existing MoE RL Tricks

We analyze the effect of R3 in Appendix F.3, where we find that R3 is important for the training stability of agentic RL. Therefore, we apply R3 throughout our experiments, used together with all of compared routing control methods.

Thank you for your detailed response. I will increase my score.

Reviewer_8MqVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that pretrained MoE agents already route differently for different agentic operations (READ, CREATE, UPDATE) and more consistently within a field, that standard RL erodes this structure, and proposes a turn level MI objective, a gated token level consistency loss, and an entropy gate on the auxiliary gradients. It reports gains on AppWorld and AutomationBench across PPO, GRPO, LOOP, and GiGPO, plus higher rollout throughput.

Critical strengths
  1. The problem framing is new and well motivated. Jointly designing agentic RL with MoE routing is underexplored, and the Figure 2 analysis showing operation grouped specialization in the pretrained model, together with the label ablation (random vs app name vs operation, Table 6), gives real support for using operation labels rather than an arbitrary grouping.
  2. The design is deployable. It keeps the native top k router and fused inference kernels, injects auxiliary gradients only at training time (Appendix D.5), needs no added parameters, and plugs into four different policy gradient algorithms.
  3. Stability is treated seriously. The paper identifies collapse from naive routing control, proposes the entropy gate, and documents informative negative results: outlier regularization crashing, direct router entropy regularization causing action entropy explosion (Figure 10), and hard fixed routing collapsing likelihood (Table 5).
  4. The efficiency claim comes with a mechanism. The controlled router collapsing experiment (Figure 11) ties the throughput gain to batched expert computation and shows it vanishes at batch size 1, which is more than most routing papers offer. The appendix hyperparameter and threshold sweeps are also thorough.
Critical weaknesses
  1. The reported numbers are not consistent across the paper and no variance is given. The same GRPO baseline on AppWorld appears as 69.6 and 40.5 (Table 1), 66.70 and 38.80 (Table 2), 69.64 and 46.28 (Table 3), 66.67 and 38.85 (Tables 8 and 10), and 64.88 and 35.97 (Tables 7 and 9). The base model appears as 35.1, 34.52, and 38.69 on test normal. Figure 1 labels GRPO as 52.8 in one panel and 57.4 in another, and GRPO plus Ours as 55.3 and 62.2, and the 405 tokens per GPU per second in Figure 1 matches nothing in Figure 6 or Table 10. Evaluation uses a single trajectory per task at temperature 0.7 with no seeds. Since two supposedly identical GRPO runs differ by about 2 points on test normal and up to 7 points on test challenge, most reported gains are within this spread. Please report mean and standard deviation over at least 3 seeds, state exactly which checkpoint selection rule produced each table, and reconcile the figures.
  2. The efficiency mechanism does not match the observed gains. Figure 11 shows that collapsing routing from about 5 shifted experts per token to 0 yields at most about 11 percent throughput. Yet Table 10 reports a 25 percent throughput gain from turn level control alone with shifted experts almost unchanged (4.8393 to 4.8132), and 25 percent for the combination (4.84 to 4.74), while Section 4.4 claims 41 percent. A change of 0.1 shifted experts cannot explain this through the stated mechanism. The likely confound is trajectory length, number of turns, or fewer timed out rollouts. Please report tokens per trajectory, turns per trajectory, and abort rate for each method, and compute throughput at matched trajectory length.
  3. The headline claim is overstated. The abstract states over 10 point improvements on all evaluated benchmarks, but Table 1 shows PPO losing 2.4 points on test normal, GiGPO gaining 0.23 on AutomationBench, and GRPO gaining 5.4 on test normal. In Tables 2 and 3 plain load balancing beats the proposed method on test normal (74.40 vs 72.62), so the advantage is specific to test challenge. GiGPO is also far below GRPO on AppWorld (52.4 vs 69.6), which is unusual for a method designed to improve on GRPO and suggests the baselines are not equally tuned. The claims should be narrowed to what the data supports.
  4. The interaction between components and the gate is unexplained. In Table 10 turn level control alone hurts test normal (66.67 to 62.50) and token level control alone helps little, yet the combination gains 8.3 and 10.4 points, a strongly superadditive effect that needs explanation or more seeds. The stable window for token level control is narrow: 0.000001 collapses and 0.0000002 is used (Table 9). The combined run ends with entropy 0.2331 while the gate threshold is 0.2, so the gate may be closed for much of training. Please report the fraction of steps on which the gate is closed, whether the single component rows in Table 10 use the gate, which gate values are used on AutomationBench, and whether R3 is applied in the main runs given Appendix F.3.
  5. Generality of the operation labels is limited. The labels come from hand written rubrics that depend on HTTP method lookups and API name prefixes (Appendix D.6). It is unclear how to obtain them for web, GUI, or coding agents where actions do not map cleanly to CRUD semantics, and the MI estimate needs at least 8 turns per batch, which already breaks PPO. Only the Qwen family is tested, and Section 3.2 states that the token level design was chosen for Qwen style routers, so the transfer to models with routing bias or sigmoid gating such as the DeepSeek family is untested. Rare classes such as AUTH and DELETE may also make the per operation MI terms noisy.
  6. The specialization evidence is modest in magnitude and the causal story is not established. The within minus cross group cosine gaps are 0.03 to 0.05 on a base similarity of about 0.75, app name grouping produces nearly the same gap, the operation classifier reaches macro F1 of only 0.45 to 0.49 with 5 classes, and the MI values are around 0.008 to 0.010 nats. The cited Wang et al. 2026c argues routing reflects geometry rather than semantics, and turns with the same operation also share surface tokens such as show, get, or search, so a lexical similarity control is needed to rule out that the labels are proxies for vocabulary. Since Appendix D.1 notes the auxiliary gradients also reshape hidden states before the router, the performance gain could be a generic regularization effect; the random label control helps but the test challenge difference there is only 39.33 vs 42.69.
Questions for authors

Please refer to the weaknesses section.

7 comments

Thank you so much for taking time to read the details of our paper. The raised concerns are all helpful and constructive for improving the clarity. If you find the responses below sufficient to address your concerns, we truly appreciate your re-consideration for the score.

Q1: Inconsistent Reported Numbers

Q1.1: GRPO

For GRPO baselines, we summarize the origin of number difference below. The differences are either from different checkpoint selections or older experiments. We will update the numerical difference caused by older experiments to align with Table 1. For the different checkpoint selections, we will highlight the settings in each table's caption.

Paper table(s)Reported TGC: Normal / Challenge (%)Configuration / experiment contextRecorded seeds: training / rolloutCheckpoint (train/step)Selection rule / clarification
Table 169.6 / 40.5Baseline with explicit operation-label and per-layer routing-logging settings1234 / 42179Highest dev-set TGC. Exact test scores: 69.6429 / 40.5276.
Table 266.70 / 38.80Same configuration as Tables 8 and 101234 / 42199Final checkpoint, rounded to one decimal. At two decimals, the logged values are 66.67 / 38.85.
Table 369.64 / 46.28Older baseline, lacking the explicit operation-label logging1234 / 42199Final checkpoint.
Tables 8 and 1066.67 / 38.85Same baseline run as Table 11234 / 42199Final checkpoint.
Tables 7 and 964.88 / 35.97Older baseline, lacking routing control1234 / 42199Final checkpoint.

Q1.2: Base Model

For base model numbers, they come from the step-0 of each GRPO runs. We will also update to align with Table 1.

Q1.3: Fig. 1

For Fig. 1's numbers, there are typos of checkpoint selection in our initial submission. We have updated Fig. 1 to be:

Fig. 1 panelMetricBase modelGRPOGRPO + Ours
Left: expert specializationMean TGC (%)28.455.062.1
Middle: routing consistencyMean TGC (%)28.455.062.1
Right: inference throughputEffective tokens/GPU/s283.1254.9350.7

Q1.4: Mean and STD over Repeated Runs

For the repeated experiments, we add a few repeated runs with random seeds, shown in the table below. The checkpoint selection rule is the same as Table 1, picking the best-performing checkpoints on the dev set. The reported empirical improvement is stable across different runs. We will include the multi-run results to the paper's final version.

MethodRunsNormal TGC, mean ± SDChallenge TGC, mean ± SD
PPO361.82 ± 3.2134.97 ± 2.92
PPO + Ours360.05 ± 1.2237.03 ± 2.42
GRPO370.01 ± 2.2042.27 ± 3.35
GRPO + Ours374.83 ± 1.8348.76 ± 1.81
LOOP366.87 ± 3.0239.77 ± 3.47
LOOP + Ours372.42 ± 1.6949.12 ± 0.72
GiGPO353.57 ± 2.2932.69 ± 1.41
GiGPO + Ours358.73 ± 2.2536.17 ± 1.69

Q2: Efficiency Mechanism

The main story about efficiency improvement is at Fig. 6 and Fig. 15, where the efficiency bonus is observed empirically. We additionally provide Fig. 11 to help understand the efficiency machanism in the extreme case, with fixed routing and fixed trajectories. The results in Fig. 11 is only for understanding the relation between expert switch and inference throughput, not matching any of the real experiments in this paper. We show the real expert switch numbers in Fig. 7(a), where our methods reduce the expert switch from 4.8/8.0 to 4.7/8.0. Given the large number of trajectories and MoE layers, this is already a significant reduction.

Q2.1: Possible Efficiency Confound

To clarify the efficiency bonus, we show the token length, number of turns, and abort rate in the table below. Our method achieve shorter trajectories compared to GRPO, and the efficiency bonus is not from longer sequences.

StepsMethod# effective tokens# turnsRemoved-traj rate
0–49GRPO1,274.1712.915.46%
0–49GRPO + Ours1,293.4912.835.89%
50–99GRPO1,480.8015.086.01%
50–99GRPO + Ours1,562.5515.305.68%
100–199GRPO1,680.2516.535.32%
100–199GRPO + Ours1,675.3115.695.94%

Q3: Overstated Headline Claim

Q3.1: Abstract

We will update the claim to be "up to 12.9 points on AppWorld and 11.34 points on AutomationBench" to ensure clarity.

Q3.2: GiGPO Numbers

We independently implemented GiGPO for AppWorld and AutomationBench, using the original paper as the reference for its hyperparameters. The original paper of GiGPO [1] tuned these hyperparameters on ALFWorld and WebShop. The literature shows that GiGPO’s advantage over GRPO is not uniform across AppWorld settings: for example, [2] reports lower AppWorld Normal success rate for GiGPO than GRPO (their Table 1).

Within each benchmark of our paper, GiGPO and GiGPO + Ours share the same GiGPO implementation and underlying RL hyperparameters. The comparison therefore fairly evaluates the incremental contribution of our routing-control framework under a shared GiGPO configuration.

[1] https://arxiv.org/abs/2505.10978

[2] https://arxiv.org/abs/2610.14636

Q4: Interaction between Components

Q4.1: Table 10

In Table 10, we compared the individual effect of token-level and turn-level controls, and did not use entropy gates. The hyperparameter choices are directly from the optimal setting for "Both", and we did not separately tune the individual turn-level / token-level control's coefficient for a fair comparison. For the best hyperparameters of each individual controls, we show their outcomes in the table below. The results are closer to the "Both" setting but still underperform it.

ControlHyperparametersNormal TGCChallenge TGCMean tokens/GPU/s
Turn-level onlyλMI=2×10−6\lambda_{\mathrm{MI}}=2\times10^{-6}67.86%44.12%241.00
Token-level onlyλtoken=10−7\lambda_{\mathrm{token}}=10^{-7}, threshold 0.50.566.67%42.21%290.81

Q4.2: Token-Level Control Coefficient

For the coefficient of token-level control, we observed that 1e-6 leads to collapse but 2e-7 remains stable. The latter value is half of the former value, and their relative difference is large enough.

Q4.3: Entropy Gate Open Rate

For the entropy gate opening statistics, please refer to the table below. The entropy value continually increase as training proceeds, and thus the open rate is decreasing.

Training stepsTotal stepsGate openOpen rate
0–49503672.0%
50–9950918.0%
100–19910022.0%
Overall: 0–1992004723.5%

Q4.4: AutomationBench Hyperparameters

For AutomationBench, we set the entropy gate to be [0, 0.15]. We will add this setting to the hyperparameter tables.

Q4.5: R3

For R3, as we observe its effect of stabilizing the training, it is applied to all of the experiments in this paper.

Q5: Generality of Operation Labels

Q5.1: How Are the Operation Labels Universal to Other Agentic Tasks?

Our operation labels describe the functional role of an interaction with respect to task-relevant objects and environment state. The same core categories can therefore apply across different interaction interfaces, as illustrated below.

Agentic settingREAD: acquire informationCREATE: introduce a new objectUPDATE: modify an existing objectDELETE: remove an existing object
GUI automation [3, 4]Capture a screenshot to inspect the current interfaceCreate a new document through the applicationChange an existing document’s content or an application settingDelete a document
Web interaction [5, 6]Inspect a product page or retrieve search resultsPost a new commentEdit a saved shipping addressRemove a saved address
Coding agents [7, 8]Inspect source code or read an execution logAdd a new source file or test filePatch an existing functionRemove an obsolete file
Database agents [9, 10]Query records with SELECTInsert a new recordModify an existing recordDelete a record
Calendar assistants [11, 12]Check availability or inspect an eventSchedule a new eventReschedule an existing eventCancel an event

The extraction procedure depends on the interface: API-based environments expose function names and arguments, whereas GUI environments require interpreting the selected control and its intended effect. For example, a click on "View details" corresponds to READ, while a click on "Create event" corresponds to CREATE. Thus, labels are assigned to the operation performed rather than to low-level primitives such as clicking or typing.

[3] https://arxiv.org/abs/2404.07972

[4] https://arxiv.org/abs/2405.14573

[5] https://arxiv.org/abs/2307.13854

[6] https://arxiv.org/abs/2403.07718

[7] https://arxiv.org/abs/2310.06770

[8] https://arxiv.org/abs/2407.16741

[9] https://arxiv.org/abs/2411.07763

[10] https://arxiv.org/abs/2510.05318

[11] https://arxiv.org/abs/2406.13352

[12] https://arxiv.org/abs/2601.11957

Q5.2: 8-Turn Requirement

For the minimum 8-turn requirement in MI estimation, it is over eligible turns in the batch, not 8 turns per operation or necessarily per trajectory. Below the threshold, MI is skipped while ordinary RL continues. The table below shows the real frequencies of each opertion label.

Label# turns in 90 training samples# turns in all 732 test samples
AUTH1161,471
READ86710,100
CREATE1822,531
UPDATE873
DELETE36173
Unlabeled turns110997
Total unique turns1,26914,634

We also directly examine the number of batches that violate this 8-turn requirement in real GRPO + Ours and PPO + Ours experiments. There is exactly 0% of batches that violate this requirement. This concern is not practical.

Q5.3: Model Choices

For the model choices, we acknowledge that the proposed routing control framwork is not reportedly tested on other model families. The scope of MoE routing control framwork is limited to Qwen family.

Comments on the paper

1 comment
Area_Chair_4QmeOpenReview ↗

Please engage with authors

Dear Reviewers,

The author-reviewer discussion will be closed on November 18 (AoE). If you haven’t yet replied to the authors’ latest response, please do so on OpenReview as soon as possible so they have time to follow up. A brief reply indicating which of your concerns have been addressed and what questions remain would be helpful. Thank you for your time and for engaging with the authors.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.