- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Router-Centric Adaptation (RoCA) for parameter-efficient fine-tuning of mixture-of-experts large language models. RoCA freezes the original model and adjusts expert selection preferences through a small set of scaling and bias parameters. During training, it randomly explores candidate experts and introduces a trainable domain-specific expert at each MoE layer, using routing entropy to dynamically control its output contribution. At inference, it returns to deterministic top-k expert selection. The central idea is to adapt the use of existing experts while supplementing them with domain-specific capabilities for downstream tasks. Experiments across three MoE backbones and five domains show that RoCA achieves the highest average downstream performance among the compared methods with relatively few trainable parameters, while also improving average general benchmark performance on two backbones.
Critical strengths
1 The authors report means and standard deviations, allowing readers to assess the consistency of different methods across domain-specific fine-tuning settings.
2 The experimental evaluation is reasonably comprehensive. In addition to single-domain fine-tuning, the authors evaluate mixed-domain training, assess general capabilities on two backbone models, and analyze the effect of candidate pool size.
3 The method is described clearly. The paper provides equations for router scaling and bias modulation, the domain-specific expert, candidate expert exploration, and entropy-based gating. It also specifies the main parameter dimensions and initialization procedures and provides accompanying code.
Critical weaknesses
1 Although RoCA and PERFT require relatively few trainable parameters, both introduce additional expert parameters. In contrast, methods such as LoRA can merge their learned updates into the original weights without increasing the deployed model’s total parameter count. It is therefore unclear whether the comparisons adequately account for these differences in architectural capacity and deployment overhead.
2 RoCA exhibits relatively large standard deviations, raising concerns about the consistency of its general capabilities across fine-tuning domains. PERFT also exhibits relatively large standard deviations. Could this shared pattern be attributable to the introduction of additional expert modules? Further analysis is needed to determine whether these modules increase sensitivity to the fine-tuning domain.
3 In Table 3, the full method improves the average score by only approximately 0.13 points over the variant with candidate expert exploration alone. HumanEval performance remains unchanged, while LEDGAR performance decreases by 1.50 points. The paper does not sufficiently discuss this trade-off.
4 Rescaling router logits can change routing entropy without necessarily changing expert rankings. Consequently, the entropy-based gate may function as a learnable mechanism for controlling the contribution of the domain-specific branch. Its interpretation as an indicator of “knowledge demand” therefore requires additional validation.
Questions for authors
see Weaknesses
3 comments
3. Domain-Dependent Effects of Dynamic Knowledge Injection
To examine how injection strength affects overall performance and LEDGAR, we compare constant and entropy-guided injection using the following settings:
- Base model: the pretrained model without downstream fine-tuning.
- Framework + candidate expert exploration: train router modulation and the domain expert with candidate expert exploration, using constant injection at .
- Framework () + candidate expert exploration: use the same configuration with a fixed injection weight of .
- RoCA: use entropy-guided dynamic knowledge injection, with all other components and training settings unchanged.
Table 3 already reports the and RoCA results; we additionally evaluate the control. The table below reports single-run results.
| Setting | MATH-500 | HumanEval | MedQA | Ethics | LEDGAR | Avg. |
|---|---|---|---|---|---|---|
| Base model | 62.00 | 55.48 | 77.14 | 69.16 | 39.55 | 60.67 |
| Framework + candidate expert exploration | 68.20 | 73.17 | 80.20 | 82.89 | 69.62 | 74.82 |
| Framework () + candidate expert exploration | 68.40 | 73.78 | 80.44 | 82.93 | 63.52 | 73.81 |
| RoCA | 69.40 | 73.17 | 80.91 | 83.17 | 68.12 | 74.95 |
LEDGAR sensitivity to injection strength. Reducing the fixed injection weight from to lowers LEDGAR from 69.62 to 63.52, suggesting that this domain may benefit from stronger domain-expert contributions. RoCA’s entropy-guided gate uses injection weights below 1, which may partly explain its slightly lower LEDGAR score (68.12) relative to constant injection at . Despite this slight decrease relative to constant injection, RoCA still achieves the best LEDGAR performance among all methods compared in Table 1 on Qwen3-30B-A3B, outperforming the strongest baseline, ESFT, by 22.48 points.
Overall performance. RoCA achieves the highest five-task average among these settings at 74.95, exceeding constant injection at both (74.82) and (73.81).
4. Routing Entropy and Knowledge Demand
The domain expert is expected to learn domain-specific knowledge during fine-tuning. We therefore use dynamic knowledge injection to describe adjusting its contribution for each token. Routing entropy serves as a heuristic signal, based on the intuition that supplementary domain knowledge may be helpful when routing preferences among pretrained experts are less clear.
We additionally present a case study on Qwen3-30B-A3B, comparing the top-1 next-token predictions of Base model and Domain expert only under the same reference-solution prefix. Domain expert only trains the added domain expert with the pretrained backbone frozen, router modulation fixed at identity, and constant injection (), without entropy gating or candidate expert exploration.
Consider the following prefix from a MATH-500 reference solution:
An integer is divisible by 18 if and only if the sum of its digits is divisible by ...
The correct reference token is 9, corresponding to the rule requiring a digit sum divisible by 9 and an even last digit.
Percentiles rank the 48-layer mean entropy within each model's distribution over the same 500 MATH-500 reference solutions.
| Setting | Next-token prediction | Probability of reference token 9 | Entropy percentile |
|---|---|---|---|
| Base model | 1 | 40.57% | 79.06% |
| Domain expert only | 9 | 56.02% | 80.65% |
In this example, domain-expert adaptation changes the next-token prediction from 1 to 9 and increases the reference-token probability from 40.57% to 56.02% at a token with relatively high routing entropy. This example offers a qualitative illustration consistent with the intuition behind entropy-guided dynamic knowledge injection.
Dear Reviewer Dooa,
As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.
Thank you.
AC
We appreciate the reviewer's careful reading and constructive comments. Below, we address the issues raised and provide additional results. We will revise the manuscript accordingly.
We report trainable parameters, additional deployed parameters, training time, and inference costs on Qwen3-30B-A3B, with LoRA weights merged into the backbone before inference.
RoCA’s Scale-and-Shift modulation can be folded into the router weights and bias at inference. Its domain expert, however, is an independent nonlinear branch for learning domain-specific knowledge and cannot be merged into the backbone weights. Unlike merged LoRA, RoCA therefore retains additional deployment capacity, amounting to approximately 0.5% of the backbone’s parameter count. The reported measurements use unmerged routing modulation.
RoCA achieves stronger downstream performance with fewer trainable parameters and shorter training time. The measured deployment overhead is modest: inference latency increases moderately, while peak inference memory remains close to that of merged LoRA. Given the performance gains and lower adaptation cost, we consider this additional overhead a reasonable and acceptable trade-off.
We thank the reviewer for highlighting this potential source of variation. To examine whether added expert modules increase sensitivity to the fine-tuning domain, we compare four settings on Qwen3-30B-A3B:
For each setting, we evaluate five domain-specific checkpoints on general benchmarks. Each benchmark reports mean ± standard deviation across these checkpoints, measuring variation with the fine-tuning domain. PERFT and RoCA results are taken from Table 2; following its convention, Avg. averages the benchmark means and standard deviations.
Added expert modules are associated with greater variation across fine-tuning domains. All three methods with additional expert modules (domain-expert-only adaptation, PERFT, and RoCA) indeed exhibit higher standard deviations across fine-tuning domains than router-modulation-only adaptation on all five general benchmarks. A possible explanation is that the added experts learn more domain-specific knowledge, making adaptation more specialized and general capabilities more dependent on the fine-tuning domain.
Full RoCA mitigates this variation. Compared with domain-expert-only adaptation, RoCA reduces the mean per-benchmark standard deviation from 2.33 to 0.89, while improving the average general-capability score from 75.08 to 75.86. These results indicate that the complete RoCA design mitigates the domain sensitivity associated with domain-expert adaptation while retaining its benefits for model’s general ability.