- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RC-GEM, a new mechanism for parameter-efficient fine-tuning of Mixture-of-Experts (MoE) models. The main insight is that standard MoE routers use the same predictions for expert selection and output aggregation, assigning a single weight to each selected expert's entire output. RC-GEM addresses this limitation by preserving the pretrained router and aggregation weights while learning fine-grained, subspace-specific adjustments to expert contributions.
The method achieves competitive downstream fine-tuning performance while largely preserving out-of-distribution (OOD) capabilities that deteriorate under several conventional fine-tuning approaches.
Overall, RC-GEM does not necessarily achieve the best results on any individual metric. Methods that prioritize downstream performance may outperform it at the expense of OOD retention, while other methods may preserve pretrained capabilities better but achieve smaller downstream gains. RC-GEM's main strength is its balance across three objectives: downstream performance, OOD capability retention, and parameter efficiency.
Critical strengths
-
Interesting research direction: Separating expert selection from aggregation is a useful and well-motivated direction. In particular, learning fine-grained aggregation while preserving the pretrained router is an interesting approach to MoE adaptation.
-
Competitive trade-offs: RC-GEM offers a lightweight fine-tuning procedure that achieves competitive results across three important objectives: downstream performance, OOD capability retention, and parameter efficiency.
-
Relevant baselines and compelling results: The paper compares against several strong and relevant fine-tuning methods, including MoE-specific approaches. The results are promising, although I have concerns about the experimental comparisons discussed below.
Critical weaknesses
-
Limited evaluation scope and unclear data splits
The authors fine-tune exclusively on Commonsense-15K, which combines examples from the same eight benchmark families used for downstream evaluation. However, the exact dataset composition, training/validation splits, and separation from evaluation examples are not sufficiently documented.
I would also have liked to see a simpler fine-tuning experiment using a single dataset with clearly defined training and test splits. This would help isolate downstream adaptation performance and establish whether the method generalizes beyond the Commonsense-15K training mixture.
-
Insufficient characterization of the performance–OOD retention–efficiency trade-off
My biggest concern is that the paper does not systematically characterize the trade-off between its three primary objectives. Table 1 and Figures 3–4 provide an initial picture, but the reader must piece together the comparisons.
- ID–OOD trade-off: How do downstream performance and OOD capabilities evolve as a function of training steps? Other methods might achieve a similar balance through earlier stopping.
- Unified comparison: A composite metric or Pareto analysis combining downstream gains, OOD retention, and trainable parameter count would make the advantages of RC-GEM easier to quantify.
-
Sensitivity to baseline hyperparameters
Parameter-efficient methods can be highly sensitive to their architectural configurations. For example, LoRA uses rank 32 in the reported experiments, but smaller ranks may considerably reduce its parameter count while maintaining competitive performance. This could significantly change the comparison in Figure 4. Similarly, ACE achieves strong OOD retention, but its parameter count depends on its adapter ranks and configuration, which are not clearly documented.
I am not claiming that smaller configurations would necessarily perform equally well. However, without parameter-budget sweeps or more detailed hyperparameter reporting, it is difficult to assess whether RC-GEM's advantages are robust across different efficiency budgets.
-
Overcomplicated mathematical notation
While the underlying idea is relatively simple and intuitive, the mathematical presentation is unnecessarily difficult to follow, particularly in Section 5. Several quantities carry four or more subscripts, making the formulation cumbersome and obscuring the main mechanism. I believe the authors could substantially improve the exposition by first presenting the mechanism for a single token and layer, then extending it to the general setting. A more compact notation would help readers better appreciate the contribution.
-
Insufficient discussion of closely related aggregation methods
Learning to combine expert outputs, rather than simply relying on pretrained router weights, has been explored in prior work. In particular, the following papers appear relevant:
While DirMoE is briefly mentioned, the other two papers do not appear to be discussed. I would encourage the authors to clarify how RC-GEM differs from these methods, particularly whether fine-grained subspace modulation provides meaningful advantages over previously proposed expert-level aggregation mechanisms.
Questions for authors
-
Can the authors provide a unified analysis of downstream performance, OOD retention, and parameter efficiency? A composite metric, Pareto analysis, or training-step trade-off curves would help quantify RC-GEM's advantages.
-
How does RC-GEM relate to methods for learning combinations of model outputs or parameters? For example, model merging and learned LoRA adapter combinations explore related forms of aggregation. Could insights from these methods be useful in this setting?
-
How does RC-GEM compare directly against the Residual Aggregation Controller (RAC) introduced in Section 4.2? A parameter-matched comparison would help establish whether fine-grained subspace modulation provides advantages beyond scalar expert reweighting.
2 comments
We next address the remaining five questions below.
Q3: Sensitivity to baseline hyperparameters. Parameter-efficient methods can be highly sensitive to their architectural configurations. For example, LoRA uses rank 32 in the reported experiments, but smaller ranks may considerably reduce its parameter count while maintaining competitive performance. […] Similarly, ACE achieves strong OOD retention, but its parameter count depends on its adapter ranks and configuration, which are not clearly documented.
A3: Thank you very much for pointing this out. We have conducted learning-rate sweeps for the baselines and updated Table 1 accordingly. The corresponding optimization settings are provided in A.10 (Table 16). Moreover, we provide additional rank sweeps for rank-sensitive baselines (LoRA, ACE, and PERFT-E), reporting their performance and trainable parameter counts in A.6 (Tables 11–13).
Q4: Overcomplicated mathematical notation. […] A more compact notation would help readers better appreciate the contribution.
A4: Thank you for this helpful suggestion! We have revised Section 5 to simplify the mathematical presentation.
Q5: Insufficient discussion of closely related aggregation methods […] I would encourage the authors to clarify how RC-GEM differs from these methods, particularly whether fine-grained subspace modulation provides meaningful advantages over previously proposed expert-level aggregation mechanisms.
A5: Thank you very much for recommending these relevant works. We have carefully studied them and incorporated discussions of these methods into the Introduction and Related Work to clarify the distinctions between RC-GEM and prior expert aggregation approaches.
Q6: How does RC-GEM relate to methods for learning combinations of model outputs or parameters? For example, model merging and learned LoRA adapter combinations explore related forms of aggregation. Could insights from these methods be useful in this setting?
A6: Thank you for raising this interesting connection. We agree that model merging, learned LoRA combinations, and RC-GEM share the broad goal of improving model performance by learning how to combine multiple components. However, they differ in their combination mechanisms and integration into the model. Model merging combines parameter updates from different models, while learned LoRA combinations aggregate multiple adapter updates or outputs. In contrast, RC-GEM operates within a single pretrained MoE backbone without replacing or restructuring its original architecture. Instead, it introduces lightweight, trainable modulation modules into the expert aggregation process, while keeping all pretrained parameters frozen. These modules refine selected expert representations through routing-conditioned, subspace-wise modulation.
Among the insights from these approaches, the most relevant to RC-GEM is input-dependent combination: rather than relying on fixed combination weights, the contributions of different components can be adapted according to the input. This is closely related to RC-GEM, which uses router logits as input-dependent signals to generate token-dependent modulation coefficients.
Q7: How does RC-GEM compare directly against the Residual Aggregation Controller (RAC) introduced in Section 4.2? A parameter-matched comparison would help establish whether fine-grained subspace modulation provides advantages beyond scalar expert reweighting.
A7: Thank you for this comment. This question overlaps with Reviewer Kr39's Q1. Please see our response to that question for RAC's experimental settings and comparisons with expert-level scalar reweighting first.
In addition, to directly address the parameter-matched comparison, we compare RAC with RC-GEM at G=1 and r=64, yielding nearly identical trainable parameter ratios (0.0313% vs. 0.0303%). Even under this restricted modulation setting (where all learned directions share a single modulation coefficient, please see our response to Reviewer Qbuf's Q1 for further discussion of fine-grained modulation), RC-GEM achieves an ID score above 73 (Fig. 6(b)), substantially exceeding RAC's best CS score of 69.29 across the tested modulation strengths ε (Table 10).
We sincerely appreciate your positive assessment of our work and your insightful suggestions. We have consolidated the concerns raised in the Critical Weaknesses and Questions for Authors sections into seven questions, which we address individually below.
A1: Thank you for this suggestion. We have clarified the dataset composition and partitioning protocol in the revised manuscript. All methods use the same training and validation splits and are evaluated on official benchmark datasets using open-source evaluation frameworks.
Furthermore, following your suggestion, we conduct an additional experiment using SocialIQA as the sole training dataset. Specifically, we randomly sample 2,000 examples without replacement from its official training split using seed 1234, and independently sample 125 examples from the official dev split for validation. After training, we evaluate the resulting model using the same benchmark suite and evaluation settings as in our main experiments. The results are reported below, demonstrating RC-GEM's effectiveness under a simpler single-dataset training setting.
Evaluation results after SocialIQA-only training (%)
A2: Thank you for this helpful suggestion. We summarize the additional analyses below.
(2.1) Unified ID–OOD–efficiency comparison. Fig. 3 compares ID gains and OOD retention across adaptation methods, while Fig. 4 jointly visualizes ID gains, trainable parameter ratios, and mean OOD retention. RC-GEM achieves a 12.77-point ID gain with only 0.0913% trainable parameters and nearly 100% mean OOD retention. Methods achieving comparable ID gains, including FFT, LoRA, Prefix Tuning, and PERFT-E, exhibit substantially greater OOD degradation, particularly on Math and Code. Moreover, FFT updates nearly all model parameters, while Prefix Tuning uses approximately 5.5× as many trainable parameters as RC-GEM. By contrast, methods with stronger OOD retention, such as Router-FT and RoMA, achieve only 41% and 12% of RC-GEM's ID gain, respectively.
(2.2) Sensitivity to parameter budgets. We provide rank sweeps for LoRA, ACE, and PERFT-E in A.6 (Tables 11–13), examining how different ranks affect ID/OOD performance and trainable parameter counts. Fig. 4 further visualizes these performance–parameter trade-offs for LoRA and PERFT-E.
(2.3) Pareto analysis. We also examined the Pareto frontier across ID performance, mean OOD retention, and trainable parameter ratio. Five methods (FFT, LoRA, Router-FT, RoMA, and RC-GEM) are identified as non-dominated, while Prefix Tuning, ACE, and PERFT-E are dominated. However, all parameter-efficient baselines use relatively small parameter budgets (less than 1% of model parameters are trainable), while their ID gains and OOD retention differ substantially. Consequently, methods such as Router-FT and RoMA remain non-dominated despite achieving much smaller ID gains than RC-GEM. Thus, Pareto dominance alone does not sufficiently distinguish methods with a favorable overall trade-off. To provide a more comprehensive comparison across parameter budgets, we have extended Fig. 4 with LoRA and PERFT-E rank-sweep results, illustrating how ID gains and OOD retention vary with trainable parameter ratios.