acceptodds
Under review as a conference paper at ICLR 2027

Router-Centric Adaptation for MoE LLMs

Abstract

Existing parameter-efficient fine-tuning (PEFT) methods for Mixture-of-Experts (MoE) LLMs primarily introduce trainable parameters into experts, yet ignore the router. In this paper, we find that, under top- expert selection, models adapted with these methods largely inherit their token-to-expert assignment patterns from pre-training. Based on our empirical observations, we argue that adapting routing preferences is crucial for downstream adaptation. We therefore propose Router-Centric Adaptation (RoCA), a PEFT method that accounts for the routing characteristics of MoE LLMs. Its core strategy is to explore potentially task-relevant experts to help shape task-adaptive routing preferences and improve expert utilization. It further uses routing uncertainty to adaptively inject domain knowledge. Experiments across three recent MoE backbones (Qwen3-30B-A3B, DeepSeek-V2-Lite, and OLMoE-1B-7B) and five domains show that RoCA achieves the highest average downstream performance while using 5–10× fewer trainable parameters than the second-best method, demonstrating its efficiency and effectiveness. RoCA also improves general capabilities after fine-tuning. We hope our work provides a new perspective on PEFT for MoE LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Large Language Models, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_DooaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Router-Centric Adaptation (RoCA) for parameter-efficient fine-tuning of mixture-of-experts large language models. RoCA freezes the original model and adjusts expert selection preferences through a small set of scaling and bias parameters. During training, it randomly explores candidate experts and introduces a trainable domain-specific expert at each MoE layer, using routing entropy to dynamically control its output contribution. At inference, it returns to deterministic top-k expert selection. The central idea is to adapt the use of existing experts while supplementing them with domain-specific capabilities for downstream tasks. Experiments across three MoE backbones and five domains show that RoCA achieves the highest average downstream performance among the compared methods with relatively few trainable parameters, while also improving average general benchmark performance on two backbones.

Critical strengths

1 The authors report means and standard deviations, allowing readers to assess the consistency of different methods across domain-specific fine-tuning settings.

2 The experimental evaluation is reasonably comprehensive. In addition to single-domain fine-tuning, the authors evaluate mixed-domain training, assess general capabilities on two backbone models, and analyze the effect of candidate pool size.

3 The method is described clearly. The paper provides equations for router scaling and bias modulation, the domain-specific expert, candidate expert exploration, and entropy-based gating. It also specifies the main parameter dimensions and initialization procedures and provides accompanying code.

Critical weaknesses

1 Although RoCA and PERFT require relatively few trainable parameters, both introduce additional expert parameters. In contrast, methods such as LoRA can merge their learned updates into the original weights without increasing the deployed model’s total parameter count. It is therefore unclear whether the comparisons adequately account for these differences in architectural capacity and deployment overhead.

2 RoCA exhibits relatively large standard deviations, raising concerns about the consistency of its general capabilities across fine-tuning domains. PERFT also exhibits relatively large standard deviations. Could this shared pattern be attributable to the introduction of additional expert modules? Further analysis is needed to determine whether these modules increase sensitivity to the fine-tuning domain.

3 In Table 3, the full method improves the average score by only approximately 0.13 points over the variant with candidate expert exploration alone. HumanEval performance remains unchanged, while LEDGAR performance decreases by 1.50 points. The paper does not sufficiently discuss this trade-off.

4 Rescaling router logits can change routing entropy without necessarily changing expert rankings. Consequently, the entropy-based gate may function as a learnable mechanism for controlling the contribution of the domain-specific branch. Its interpretation as an indicator of “knowledge demand” therefore requires additional validation.

Questions for authors

see Weaknesses

3 comments

We appreciate the reviewer's careful reading and constructive comments. Below, we address the issues raised and provide additional results. We will revise the manuscript accordingly.

1. Architectural Capacity and Deployment Overhead

We report trainable parameters, additional deployed parameters, training time, and inference costs on Qwen3-30B-A3B, with LoRA weights merged into the backbone before inference.

MethodTrainable params (M)Additional deployed params (M)Training time (h)Inference latency (s/request)Peak inference memory (GiB)
LoRA553.6506.5612.5257.04
RoCA151.01151.012.3013.8357.32

RoCA’s Scale-and-Shift modulation can be folded into the router weights and bias at inference. Its domain expert, however, is an independent nonlinear branch for learning domain-specific knowledge and cannot be merged into the backbone weights. Unlike merged LoRA, RoCA therefore retains additional deployment capacity, amounting to approximately 0.5% of the backbone’s parameter count. The reported measurements use unmerged routing modulation.

RoCA achieves stronger downstream performance with fewer trainable parameters and shorter training time. The measured deployment overhead is modest: inference latency increases moderately, while peak inference memory remains close to that of merged LoRA. Given the performance gains and lower adaptation cost, we consider this additional overhead a reasonable and acceptable trade-off.

2. Sensitivity to the Fine-Tuning Domain

We thank the reviewer for highlighting this potential source of variation. To examine whether added expert modules increase sensitivity to the fine-tuning domain, we compare four settings on Qwen3-30B-A3B:

  • Router modulation only: train only Scale-and-Shift modulation of router logits, without a domain expert or candidate expert exploration.
  • Domain expert only: train only the domain expert, keeping pretrained routing unchanged and using constant injection at α=1\alpha=1.
  • PERFT: the baseline that adds independently routed adapter experts parallel to the pretrained MoE block.
  • RoCA: jointly train router modulation and the domain expert with candidate expert exploration and entropy-guided dynamic knowledge injection.

For each setting, we evaluate five domain-specific checkpoints on general benchmarks. Each benchmark reports mean ± standard deviation across these checkpoints, measuring variation with the fine-tuning domain. PERFT and RoCA results are taken from Table 2; following its convention, Avg. averages the benchmark means and standard deviations.

SettingMMLUARC-CARC-EHellaSwagWinoGrandeAvg.
Router modulation only78.67 ± 0.0757.13 ± 0.3779.93 ± 0.1381.36 ± 0.0573.18 ± 0.2374.06 ± 0.17
Domain expert only78.28 ± 1.4161.88 ± 3.8482.59 ± 3.3481.81 ± 0.3970.83 ± 2.6675.08 ± 2.33
PERFT78.39 ± 0.2560.03 ± 1.2982.30 ± 1.5581.99 ± 0.3271.71 ± 1.4674.88 ± 0.98
RoCA79.33 ± 0.4362.03 ± 1.8783.12 ± 1.3581.54 ± 0.2873.29 ± 0.5175.86 ± 0.89

Added expert modules are associated with greater variation across fine-tuning domains. All three methods with additional expert modules (domain-expert-only adaptation, PERFT, and RoCA) indeed exhibit higher standard deviations across fine-tuning domains than router-modulation-only adaptation on all five general benchmarks. A possible explanation is that the added experts learn more domain-specific knowledge, making adaptation more specialized and general capabilities more dependent on the fine-tuning domain.

Full RoCA mitigates this variation. Compared with domain-expert-only adaptation, RoCA reduces the mean per-benchmark standard deviation from 2.33 to 0.89, while improving the average general-capability score from 75.08 to 75.86. These results indicate that the complete RoCA design mitigates the domain sensitivity associated with domain-expert adaptation while retaining its benefits for model’s general ability.

3. Domain-Dependent Effects of Dynamic Knowledge Injection

To examine how injection strength affects overall performance and LEDGAR, we compare constant and entropy-guided injection using the following settings:

  • Base model: the pretrained model without downstream fine-tuning.
  • Framework + candidate expert exploration: train router modulation and the domain expert with candidate expert exploration, using constant injection at α=1\alpha=1.
  • Framework (α=0.5\alpha=0.5) + candidate expert exploration: use the same configuration with a fixed injection weight of α=0.5\alpha=0.5.
  • RoCA: use entropy-guided dynamic knowledge injection, with all other components and training settings unchanged.

Table 3 already reports the α=1\alpha=1 and RoCA results; we additionally evaluate the α=0.5\alpha=0.5 control. The table below reports single-run results.

SettingMATH-500HumanEvalMedQAEthicsLEDGARAvg.
Base model62.0055.4877.1469.1639.5560.67
Framework + candidate expert exploration68.2073.1780.2082.8969.6274.82
Framework (α=0.5\alpha=0.5) + candidate expert exploration68.4073.7880.4482.9363.5273.81
RoCA69.4073.1780.9183.1768.1274.95

LEDGAR sensitivity to injection strength. Reducing the fixed injection weight from α=1\alpha=1 to α=0.5\alpha=0.5 lowers LEDGAR from 69.62 to 63.52, suggesting that this domain may benefit from stronger domain-expert contributions. RoCA’s entropy-guided gate uses injection weights below 1, which may partly explain its slightly lower LEDGAR score (68.12) relative to constant injection at α=1\alpha=1. Despite this slight decrease relative to constant injection, RoCA still achieves the best LEDGAR performance among all methods compared in Table 1 on Qwen3-30B-A3B, outperforming the strongest baseline, ESFT, by 22.48 points.

Overall performance. RoCA achieves the highest five-task average among these settings at 74.95, exceeding constant injection at both α=1\alpha=1 (74.82) and α=0.5\alpha=0.5 (73.81).

4. Routing Entropy and Knowledge Demand

The domain expert is expected to learn domain-specific knowledge during fine-tuning. We therefore use dynamic knowledge injection to describe adjusting its contribution for each token. Routing entropy serves as a heuristic signal, based on the intuition that supplementary domain knowledge may be helpful when routing preferences among pretrained experts are less clear.

We additionally present a case study on Qwen3-30B-A3B, comparing the top-1 next-token predictions of Base model and Domain expert only under the same reference-solution prefix. Domain expert only trains the added domain expert with the pretrained backbone frozen, router modulation fixed at identity, and constant injection (α=1\alpha=1), without entropy gating or candidate expert exploration.

Consider the following prefix from a MATH-500 reference solution:

An integer is divisible by 18 if and only if the sum of its digits is divisible by ...

The correct reference token is 9, corresponding to the rule requiring a digit sum divisible by 9 and an even last digit.

Percentiles rank the 48-layer mean entropy within each model's distribution over the same 500 MATH-500 reference solutions.

SettingNext-token predictionProbability of reference token 9Entropy percentile
Base model140.57%79.06%
Domain expert only956.02%80.65%

In this example, domain-expert adaptation changes the next-token prediction from 1 to 9 and increases the reference-token probability from 40.57% to 56.02% at a token with relatively high routing entropy. This example offers a qualitative illustration consistent with the intuition behind entropy-guided dynamic knowledge injection.

Dear Reviewer Dooa,

As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.

Thank you.

AC

Reviewer_1UuDOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

RoCA is a parameter-efficient adaptation method for pretrained MoE language models. It combines scale-and-shift modulation of frozen router logits with an additional trainable domain expert. During training, candidate expert exploration allows experts near the top-k boundary to participate in computation. An entropy-based gate controls the domain expert’s contribution. Experiments on three MoE backbones and five domains report strong average downstream performance with fewer trainable parameters than the strongest evaluated baseline, together with general-capability and routing-behavior analyses.

Critical strengths

The paper presents a coherent adaptation pipeline and investigates a practically important aspect of MoE fine-tuning. Candidate expert exploration is simple to understand and preserves the native number of selected pretrained experts, with standard top-k routing restored at inference. The distinction between lightweight routing modulation and additional domain-specific computation is also clearly described in the method.

The evaluation is broader than a single-model demonstration. It covers three backbones, several adaptation domains, and general-capability measurements. The candidate-exploration ablation provides useful evidence that exploration improves the underlying framework, and the routing analysis helps characterize how the adapted model uses its expert pool.

Critical weaknesses
  1. The claim that prior adaptation methods overlook routing needs more careful positioning. Please add and discuss Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs / RoMA (ICLR 2026), R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts (ICML 2025), and Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss (ICLR 2026), Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (NeurIPS 2022) and LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts (ICLR 2026). The novelty should be framed around the proposed combination and its demonstrated benefits, rather than the general idea of routing-aware adaptation.

  2. The experiments need to distinguish routing improvements from the contribution of the added domain expert. A direct experimental comparison with RoMA is important because it is an established router-adaptation method and tests the paper’s central positioning. Its additional regularization and resource requirements should be retained and reported rather than reducing it to ordinary router tuning. More importantly, please evaluate domain-expert-only adaptation with unchanged routing and router modulation without the added domain expert. These are basic diagnostic controls, not separate published baselines. The current “Framework” row already combines both components, and the domain expert accounts for nearly all trainable parameters. Consequently, the current ablation does not establish how much of the initial improvement is specifically attributable to routing adaptation.

  3. Parameter efficiency should be separated from runtime efficiency and training variability. The added domain expert introduces computation even though the native routed-expert count remains fixed. Reporting training time, inference latency, and peak memory would make the efficiency assessment more complete.

Questions for authors

How much improvement is obtained by the domain expert alone and by routing modulation alone?

How does RoCA compare with RoMA under the same downstream training data and evaluation protocol?

Does entropy gating outperform a simple learned gate across repeated runs, and what is the measured inference overhead of the complete method?

And also check the weakness

4 comments

Thank you for the thoughtful review and constructive feedback. We respond to your concerns below and will update the manuscript to improve the positioning, experimental evidence, and presentation.

1. Related work and novelty positioning

RoCA focuses on domain-specific PEFT for pretrained MoE LLMs. We study how to adapt pretrained routing preferences to the target domain while learning additional domain knowledge with a trainable domain expert. Our core strategy is candidate expert exploration to learn task-adaptive routing preferences, complemented by entropy-guided dynamic knowledge injection.

  • RoMA performs router-only post-training with manifold regularization on mixed-task data for broad generalization. RoCA emphasizes domain-specific adaptation, combining routing adaptation with additional capacity for domain-specific learning.
  • R2-T2 adjusts multimodal routing weights at test time using reference examples. RoCA learns adaptation during downstream fine-tuning, without per-example test-time optimization.
  • ERC couples routers and experts through an auxiliary loss during MoE pretraining. RoCA addresses downstream adaptation, keeping pretrained router and expert parameters frozen.
  • SSF applies affine modulation to intermediate features. RoCA applies this established principle to router logits; scale-and-shift itself is not claimed as a standalone novelty.
  • LD-MoLE uses a mixture of LoRA experts for PEFT of dense models. RoCA targets PEFT of pretrained MoE models, adapting selection among their native experts.

Thank you for the suggestion. We have added these comparisons to the Related Work section and refined the manuscript to clarify RoCA’s contribution in the context of prior work on routing-aware adaptation.

2. Comparison with RoMA

Consistent with RoMA’s mixed-task training setting, we add RoMA to our existing mixed-domain experiment in Appendix D (Table 6). Both methods use Qwen3-30B-A3B-Base, the same five-domain corpus of 37.5K training examples, and the same evaluation protocol, with a single checkpoint evaluated on all five benchmarks. Base model denotes the pretrained model without downstream fine-tuning.

We reproduce RoMA using its official implementation, following the paper’s best-performing layer and token choices: updating the routers in the last five layers and computing the routing regularizer at the last prompt token. We retain successful-neighbor construction and manifold regularization, with the released code’s defaults (k=5k=5, σ=0.4\sigma=0.4, λreg=0.8\lambda_{\mathrm{reg}}=0.8). Updating these five router matrices gives 1.31M trainable parameters (5×2048×1285\times2048\times128); all remaining pretrained parameters stay frozen.

MethodMATH-500HumanEvalMedQAEthicsLEDGARAvg.
Base model62.0055.4877.1469.1639.5560.67
RoMA64.2049.3976.5172.1340.1360.47
RoCA67.4076.8380.5282.3464.6774.35

RoMA improves performance on MATH-500, Ethics, and LEDGAR, but shows a small decline on MedQA and a larger drop of 6.09 points on HumanEval. The decline on HumanEval is consistent with the router-modulation-only ablation reported below. RoMA targets broad generalization through mixed-task training. For domains requiring substantial domain-specific knowledge, such as code, its small router-only parameter budget may provide insufficient adaptation capacity. RoCA achieves higher scores on all five benchmarks, exceeding RoMA by 13.88 points on average in this mixed-domain setting.

We also report resource requirements on eight NVIDIA L20Z GPUs, including RoMA’s additional preprocessing.

MethodTrainable paramsPreprocessing time (h)Training time (h)Total time (h)Peak allocated memory per GPU (GiB)
RoMA1.31M14.063.8817.9459.46
RoCA151.01M011.4811.4867.64

Although RoMA uses fewer trainable parameters and has a shorter training stage, running the model over the training corpus to identify correctly answered samples accounts for nearly all of its 14.06-hour preprocessing time. Including preprocessing, its total time reaches 17.94 hours, compared with 11.48 hours for RoCA, which also achieves higher scores across all five benchmarks.

3. Component Contributions

To assess how much of the improvement comes from domain-expert learning and how much from routing adaptation, we compare four settings on Qwen3-30B-A3B that isolate the contributions of the domain expert and router modulation. Here, router modulation refers to Scale-and-Shift modulation. Pretrained parameters remain frozen, and the domain expert, when present, uses constant injection at α=1\alpha=1.

  • Base model: the pretrained model without downstream fine-tuning.
  • Router modulation only: train only the modulation parameters λ\lambda and bb, with deterministic top-kk expert selection and no domain expert.
  • Domain expert only: train only the domain expert, with router modulation fixed at λ=1, b=0\lambda=1,\ b=0 and deterministic top-kk expert selection.
  • Framework: jointly train the domain expert and router modulation parameters, with deterministic top-kk expert selection.
  • RoCA: add candidate expert exploration and entropy-guided dynamic knowledge injection to Framework.
SettingMATH-500HumanEvalMedQAEthicsLEDGARAvg.
Base model62.0055.4877.1469.1639.5560.67
Router modulation only63.0051.2277.2270.2539.0960.16
Domain expert only64.4070.1278.6381.6557.1470.39
Framework63.8070.7379.3482.5759.3471.16
RoCA69.4073.1780.9183.1768.1274.95

The domain expert contributes most of Framework’s initial improvement. Training only the domain expert increases the average from 60.67 to 70.39 (+9.72 points), consistent with the domain expert accounting for the vast majority of trainable parameters.

Router modulation alone provides no average improvement, with HumanEval decreasing from 55.48 to 51.22. Similar to the RoMA results above, this decline on code may reflect insufficient adaptation capacity from the small router-only parameter budget when pretrained experts remain frozen. In RoCA, candidate expert exploration lets experts beyond the deterministic top-k contribute to the task loss, which guides router modulation toward domain-specific routing preferences.

4. Runtime and Memory Overhead

We report the following efficiency results for math-domain adaptation on Qwen3-30B-A3B, with LoRA weights merged into the backbone before inference.

MethodTrainable params (M)Training time (h)Peak training memory (GiB)Inference latency (s/request)Peak inference memory (GiB)
LoRA553.65 (1.000×)6.56 (1.000×)70.86 (1.000×)12.52 (1.000×)57.04 (1.000×)
RoCA151.01 (0.273×)2.30 (0.351×)67.64 (0.955×)13.83 (1.105×)57.32 (1.005×)

RoCA uses only 0.273× the trainable parameters and 0.351× the training time of LoRA, with peak training memory reduced to 0.955×. Inference latency and memory are 1.105× and 1.005× those of merged LoRA, respectively. Given the stronger downstream performance and lower adaptation cost, we consider this inference overhead a reasonable trade-off.

5. Entropy Gating versus a Learned Gate

To evaluate whether entropy-guided injection is more effective than learning injection weights directly from hidden states, we compare RoCA with a simple, generic learned gate on Qwen3-30B-A3B across three training seeds. We replace only the domain-expert injection gate with a linear projection followed by a sigmoid, αℓ(x)=sigmoid⁡(wℓ⊤x+cℓ)\alpha_\ell(x)=\operatorname{sigmoid}(w_\ell^\top x+c_\ell), where xx is the input hidden state of the corresponding MoE layer. It preserves token-level scalar weighting in (0,1)(0,1), with zero-initialized parameters giving an initial gate value of 0.5. Candidate expert exploration and all other components and training settings remain identical to RoCA.

The table reports mean ± standard deviation across seeds.

SettingMATH-500HumanEvalMedQAEthicsLEDGARAvg.
Learned gate67.80 ± 0.9273.98 ± 0.9380.47 ± 0.2082.87 ± 0.1463.99 ± 2.1473.82 ± 0.20
RoCA69.73 ± 0.7674.19 ± 1.2780.75 ± 0.2183.23 ± 0.1369.26 ± 1.0375.43 ± 0.50

Across the three training seeds, entropy gating achieves higher mean scores on all five benchmarks, increasing the five-task average from 73.82 to 75.43 (+1.61 points). The largest gains occur on MATH-500 (+1.93) and LEDGAR (+5.26). These repeated-run results support entropy-guided dynamic knowledge injection over this simple learned-gate alternative under matched training conditions.

Thanks for authors' hard work and the rebuttal shows great efforts. I raised my score from 3 to 4. Good luck.

Thank you for taking the time to reconsider our paper and update your assessment. Your constructive feedback has helped us clarify RoCA’s positioning relative to prior work, better explain the contributions of its components, and make our experimental evaluation more comprehensive. We appreciate the time and care you have devoted to reviewing our paper and engaging with our responses.

Reviewer_CfXxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Router-Centric Adaptation (RoCA), a parameter-efficient fine-tuning method for Mixture-of-Experts language models. Motivated by observations that existing PEFT methods largely preserve pretrained routing patterns, RoCA combines learnable scale-and-shift modulation of router logits with a trainable domain expert, while freezing pretrained parameters. During training, candidate expert exploration samples experts near the top-k selection boundary to adapt routing preferences, while an entropy-based gate controls the domain expert’s contribution to each token.The authors evaluate RoCA on Qwen3-30B-A3B, DeepSeek-V2-Lite, and OLMoE-1B-7B across mathematical reasoning, code generation, medical QA, ethical reasoning, and legal understanding. RoCA achieves the highest average downstream performance among seven PEFT baselines on each backbone, using 67–151M trainable parameters—substantially fewer than ESFT, the strongest baseline on average. Component ablations identify candidate expert exploration as the main contributor to performance gains, and routing analyses show broader expert utilization. Additional evaluations report improved average general benchmark performance after domain adaptation on Qwen and DeepSeek.

Critical strengths

A useful routing-focused contribution supported by targeted ablation. RoCA treats expert selection as a central component of MoE adaptation. Its candidate exploration strategy allows potentially relevant experts beyond the usual top-k choices to contribute during training. In Table 3, adding this strategy to the same trainable framework raises Qwen’s five-domain average from 71.16 to 74.82. Together with the routing analyses, this provides evidence that adapting expert utilization contributes meaningfully to downstream performance, giving the paper a substantive methodological contribution.

Critical weaknesses

1.The central claim about learning routing preferences is insufficiently isolated. The “Framework” and its adapted variants in Table 3 jointly train router modulation and an added domain expert. Although the exploration ablation demonstrates a meaningful gain, there is no otherwise identical control with router modulation fixed. Consequently, the experiments do not distinguish the contribution of explicitly learning routing preferences from adapter learning under stochastic expert selection. This limits support for the paper’s central explanation of why RoCA works. 2.The knowledge-demand interpretation and added value of entropy gating remain weakly supported. Figures 4 and 7 and Appendix H describe token types and injection weights, but do not establish that higher routing entropy predicts greater benefit from the domain expert. Comparisons with constant or learned gates are also absent. In the Qwen ablation, adding entropy gating to exploration improves the average only from 74.82 to 74.95, while reducing LEDGAR from 69.62 to 68.12; repeated-run uncertainty is not reported. Although gating improves the framework without exploration, its benefit in the complete method remains unclear. This weakens the justification for the second claimed contribution and the additional mechanism it introduces.

Questions for authors

1.With the domain expert, exploration strategy, and training budget held fixed, how does performance change when the router modulation parameters remain fixed at their initial values (\lambda=1,\ b=0)? 2.Beyond the qualitative token examples, what evidence shows that higher routing entropy predicts greater benefit from the domain expert? Do you have comparisons against constant or learned gates, and repeated-run uncertainty for the 74.82 versus 74.95 exploration-only/full-method comparison in Table 3?

2 comments

We thank the reviewer for the constructive questions and helpful suggestions. We address each point below and will revise the manuscript to clarify the method and incorporate the additional analyses.

1. Independent Contribution of Router Modulation

To isolate the contribution of learning routing preferences, we compare three settings on Qwen3-30B-A3B:

  • Domain expert only: train only the domain expert, with router modulation fixed at λ=1, b=0\lambda=1,\ b=0 and deterministic top-kk expert selection.
  • Domain expert only + candidate expert exploration: keep router modulation fixed and train the domain expert with candidate expert exploration.
  • Framework + candidate expert exploration: jointly train the domain expert and router modulation parameters using the same candidate expert exploration strategy.
SettingMATH-500HumanEvalMedQAEthicsLEDGARAvg.
Domain expert only64.470.1278.6381.6557.1470.39
Domain expert only + candidate expert exploration66.671.9580.1382.7866.9573.68
Framework + candidate expert exploration68.273.1780.2082.8969.6274.82

The second and third rows directly isolate the contribution of trainable router modulation under candidate expert exploration. Learning the modulation parameters improves all five benchmarks, raising the average from 73.68 to 74.82. This demonstrates an additional benefit from explicitly learning routing preferences beyond domain-expert learning under stochastic expert selection.

The first two rows show that exploration also benefits domain-expert learning when router modulation remains fixed, increasing the average from 70.39 to 73.68. Performance improves as candidate expert exploration and trainable router modulation are added in turn, supporting the complementary contributions of these two components.

2. Routing Entropy and Domain-Expert Benefit

To examine whether higher routing entropy indicates greater benefit from the domain expert, we directly measure how next-token prediction changes when its contribution is removed. We use the Qwen3-30B-A3B Framework checkpoint without entropy gating or candidate expert exploration.

We evaluate 100 MATH-500 problems and report three layers spanning early, middle, and late depths (layers 1, 25, and 36). Within each layer, reference-solution tokens are grouped into four routing-entropy quartiles, from Q1 (lowest entropy) to Q4 (highest entropy), using entropy measured before intervention. We disable the domain-expert contribution at one token and layer at a time and compare the same ground-truth next-token loss. The difference, NLL(off) − NLL(keep), reflects the domain expert’s local contribution to next-token prediction, with larger values indicating greater benefit from retaining it.

Entries report mean [95% confidence interval], with all values in units of 10⁻³. Confidence intervals are obtained by problem-level bootstrapping.

LayerQ1 (0–25%)Q2 (25–50%)Q3 (50–75%)Q4 (75–100%)
1-1.44 [-3.79, 0.64]-1.58 [-4.69, 1.11]0.59 [-1.58, 2.64]0.71 [-0.94, 2.40]
25-1.00 [-3.54, 1.32]-0.10 [-2.54, 2.13]1.28 [-0.24, 2.80]0.39 [-1.09, 1.90]
36-1.91 [-4.12, 0.29]-0.82 [-2.19, 0.56]-0.23 [-1.16, 0.67]-0.07 [-1.08, 0.92]

Across the three reported layers, mean domain-expert benefit shows an overall increasing trend with routing entropy, with Q4 exceeding Q1 in each layer. These results provide encouraging evidence consistent with the motivation for entropy-guided dynamic knowledge injection.

3. Comparison with Constant and Learned Gates

Constant-gate comparison. Table 3 already compares Framework + candidate expert exploration, which uses constant injection at α=1\alpha=1, with RoCA's entropy-guided injection. We additionally evaluate the same configuration with a fixed injection weight of α=0.5\alpha=0.5. Framework denotes trainable router modulation and a domain expert. The table below reports single-run results on Qwen3-30B-A3B.

SettingMATH-500HumanEvalMedQAEthicsLEDGARAvg.
Framework + candidate expert exploration68.273.1780.2082.8969.6274.82
Framework (α=0.5\alpha=0.5) + candidate expert exploration68.473.7880.4482.9363.5273.81
RoCA69.473.1780.9183.1768.1274.95

RoCA achieves a higher five-task average than constant injection at both α=1\alpha=1 and α=0.5\alpha=0.5 (74.95 vs. 74.82 and 73.81, respectively). Entropy-guided dynamic knowledge injection therefore provides better overall performance than constant injection.

Learned-gate comparison. Following the reviewer's suggestion, we compare RoCA with a simple, generic learned token-level gate, implemented as a linear projection followed by a sigmoid. We use αℓ(x)=sigmoid⁡(wℓ⊤x+cℓ)\alpha_\ell(x)=\operatorname{sigmoid}(w_\ell^\top x+c_\ell), where xx is the input hidden state of the corresponding MoE layer. This control preserves the entropy gate's token-level scalar weighting and (0,1)(0,1) output range while learning the weight directly from hidden states. The gate parameters are initialized to zero, giving αℓ(x)=0.5\alpha_\ell(x)=0.5. All other components and training settings, including candidate expert exploration, remain identical to RoCA.

The table below reports mean ± standard deviation over three training seeds.

SettingMATH-500HumanEvalMedQAEthicsLEDGARAvg.
Learned gate67.80 ± 0.9273.98 ± 0.9380.47 ± 0.2082.87 ± 0.1463.99 ± 2.1473.82 ± 0.20
RoCA69.73 ± 0.7674.19 ± 1.2780.75 ± 0.2183.23 ± 0.1369.26 ± 1.0375.43 ± 0.50

RoCA achieves higher mean scores on all five benchmarks, with an average advantage of 1.61 percentage points. The largest gains occur on MATH-500 and LEDGAR. These results favor dynamic knowledge injection guided by routing entropy over this learned-gate alternative under matched training conditions.

4. Repeated-run uncertainty

We evaluate Framework + candidate expert exploration (constant injection at α=1\alpha=1) and full RoCA over three training seeds on Qwen3-30B-A3B. We report mean ± standard deviation for each benchmark.

SettingMATH-500HumanEvalMedQAEthicsLEDGARAvg.
Framework + candidate expert exploration67.87 ± 0.5873.58 ± 0.3580.28 ± 0.0882.81 ± 0.0869.99 ± 0.3374.91 ± 0.12
RoCA69.73 ± 0.7674.19 ± 1.2780.75 ± 0.2183.23 ± 0.1369.26 ± 1.0375.43 ± 0.50

Across seeds, RoCA achieves higher mean scores on four benchmarks and improves the five-task average from 74.91 to 75.43. HumanEval also shows a higher mean, although it was unchanged in the original single-run comparison. The mean LEDGAR score decreases slightly from 69.99 to 69.26.

LEDGAR appears to benefit from stronger domain-expert injection. In the single-run comparison above, reducing the fixed injection weight from α=1\alpha=1 to α=0.5\alpha=0.5 lowers the LEDGAR score from 69.62 to 63.52. RoCA’s entropy-guided gate uses injection weights below 1, which may partly explain its slightly lower LEDGAR score relative to constant injection at α=1\alpha=1. Despite this slight decrease, RoCA still achieves the highest LEDGAR score among the methods compared in Table 1 on Qwen3-30B-A3B, scoring 68.12 and exceeding the strongest baseline, ESFT (45.64), by 22.48 points.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

We thank the reviewers and the Area Chair for their time and thoughtful feedback. We appreciate the reviewers’ recognition of the value of studying expert selection during MoE fine-tuning.

In this work, we find that most PEFT methods largely preserve pretrained routing preferences in our domain-specific fine-tuning experiments. However, downstream domains can differ from pretraining data in both knowledge requirements and input formats, so relying on these preferences may leave relevant experts underused. We introduce Router-Centric Adaptation (RoCA) to adapt routing preferences to the target domain while learning additional domain knowledge. On each of three MoE backbones, RoCA achieves the highest average performance across five domains with fewer trainable parameters.

The reviewers’ suggestions helped us clarify how RoCA relates to prior work, isolate the contribution of router modulation, and evaluate entropy-guided injection across repeated runs. They also prompted us to examine how performance varies across domains and to quantify training and deployment costs.

We are encouraged by the positive follow-up feedback and pleased that the additional experiments helped resolve key concerns raised during the discussion. We thank the reviewers and the Area Chair again for their careful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.