acceptodds
Under review as a conference paper at ICLR 2027

RC-GEM: Fine-Grained Expert Aggregation via Routing-Conditioned Subspace Modulation

Abstract

Sparse Mixture-of-Experts (MoE) models use routing scores to both select experts and weight their outputs, assigning each selected expert the same weight across all representation dimensions. We first probe the adaptation potential of this aggregation scheme through two diagnostic studies. On OLMoE, a ground-truth oracle shows that a small redistribution of expert weights (mean conditional KL ) improves the seven-benchmark average by 3.3 points, while a learned residual aggregation controller shows that larger scalar reallocations eventually reduce in-distribution (ID) gains and degrade out-of-distribution (OOD) performance. Motivated by these findings, we propose RC-GEM (Routing-Conditioned Granular Expert Modulation), a parameter-efficient adaptation framework that augments standard expert-level weighting with fine-grained modulation over learned subspaces. RC-GEM preserves pretrained expert selection and aggregation weights, while using routing signals to condition subspace-wise modulation within each selected expert. Across OLMoE and Qwen1.5-MoE, RC-GEM approaches or surpasses full fine-tuning in average ID performance with fewer than trainable parameters, while largely preserving pretrained OOD capabilities. Compared with existing MoE adaptation methods and conventional PEFT baselines, RC-GEM achieves a favorable balance between adaptation performance and generalization, highlighting aggregation granularity as an important direction for efficient MoE adaptation. Our implementation is available at https://anonymous.4open.science/r/RC-GEM-F5F4.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Large Language Models, Parameter-Efficient Fine-Tuning, Expert Routing, Model Adaptation, Out-of-Distribution Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3fh2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RC-GEM, a new mechanism for parameter-efficient fine-tuning of Mixture-of-Experts (MoE) models. The main insight is that standard MoE routers use the same predictions for expert selection and output aggregation, assigning a single weight to each selected expert's entire output. RC-GEM addresses this limitation by preserving the pretrained router and aggregation weights while learning fine-grained, subspace-specific adjustments to expert contributions.

The method achieves competitive downstream fine-tuning performance while largely preserving out-of-distribution (OOD) capabilities that deteriorate under several conventional fine-tuning approaches.

Overall, RC-GEM does not necessarily achieve the best results on any individual metric. Methods that prioritize downstream performance may outperform it at the expense of OOD retention, while other methods may preserve pretrained capabilities better but achieve smaller downstream gains. RC-GEM's main strength is its balance across three objectives: downstream performance, OOD capability retention, and parameter efficiency.

Critical strengths
  1. Interesting research direction: Separating expert selection from aggregation is a useful and well-motivated direction. In particular, learning fine-grained aggregation while preserving the pretrained router is an interesting approach to MoE adaptation.

  2. Competitive trade-offs: RC-GEM offers a lightweight fine-tuning procedure that achieves competitive results across three important objectives: downstream performance, OOD capability retention, and parameter efficiency.

  3. Relevant baselines and compelling results: The paper compares against several strong and relevant fine-tuning methods, including MoE-specific approaches. The results are promising, although I have concerns about the experimental comparisons discussed below.

Critical weaknesses
  1. Limited evaluation scope and unclear data splits

    The authors fine-tune exclusively on Commonsense-15K, which combines examples from the same eight benchmark families used for downstream evaluation. However, the exact dataset composition, training/validation splits, and separation from evaluation examples are not sufficiently documented.

    I would also have liked to see a simpler fine-tuning experiment using a single dataset with clearly defined training and test splits. This would help isolate downstream adaptation performance and establish whether the method generalizes beyond the Commonsense-15K training mixture.

  2. Insufficient characterization of the performance–OOD retention–efficiency trade-off

    My biggest concern is that the paper does not systematically characterize the trade-off between its three primary objectives. Table 1 and Figures 3–4 provide an initial picture, but the reader must piece together the comparisons.

    • ID–OOD trade-off: How do downstream performance and OOD capabilities evolve as a function of training steps? Other methods might achieve a similar balance through earlier stopping.
    • Unified comparison: A composite metric or Pareto analysis combining downstream gains, OOD retention, and trainable parameter count would make the advantages of RC-GEM easier to quantify.
  3. Sensitivity to baseline hyperparameters

    Parameter-efficient methods can be highly sensitive to their architectural configurations. For example, LoRA uses rank 32 in the reported experiments, but smaller ranks may considerably reduce its parameter count while maintaining competitive performance. This could significantly change the comparison in Figure 4. Similarly, ACE achieves strong OOD retention, but its parameter count depends on its adapter ranks and configuration, which are not clearly documented.

    I am not claiming that smaller configurations would necessarily perform equally well. However, without parameter-budget sweeps or more detailed hyperparameter reporting, it is difficult to assess whether RC-GEM's advantages are robust across different efficiency budgets.

  4. Overcomplicated mathematical notation

    While the underlying idea is relatively simple and intuitive, the mathematical presentation is unnecessarily difficult to follow, particularly in Section 5. Several quantities carry four or more subscripts, making the formulation cumbersome and obscuring the main mechanism. I believe the authors could substantially improve the exposition by first presenting the mechanism for a single token and layer, then extending it to the general setting. A more compact notation would help readers better appreciate the contribution.

  5. Insufficient discussion of closely related aggregation methods

    Learning to combine expert outputs, rather than simply relying on pretrained router weights, has been explored in prior work. In particular, the following papers appear relevant:

    While DirMoE is briefly mentioned, the other two papers do not appear to be discussed. I would encourage the authors to clarify how RC-GEM differs from these methods, particularly whether fine-grained subspace modulation provides meaningful advantages over previously proposed expert-level aggregation mechanisms.

Questions for authors
  1. Can the authors provide a unified analysis of downstream performance, OOD retention, and parameter efficiency? A composite metric, Pareto analysis, or training-step trade-off curves would help quantify RC-GEM's advantages.

  2. How does RC-GEM relate to methods for learning combinations of model outputs or parameters? For example, model merging and learned LoRA adapter combinations explore related forms of aggregation. Could insights from these methods be useful in this setting?

  3. How does RC-GEM compare directly against the Residual Aggregation Controller (RAC) introduced in Section 4.2? A parameter-matched comparison would help establish whether fine-grained subspace modulation provides advantages beyond scalar expert reweighting.

2 comments

We sincerely appreciate your positive assessment of our work and your insightful suggestions. We have consolidated the concerns raised in the Critical Weaknesses and Questions for Authors sections into seven questions, which we address individually below.


Q1: Limited evaluation scope and unclear data splits […] However, the exact dataset composition, training/validation splits, and separation from evaluation examples are not sufficiently documented. […] I would also have liked to see a simpler fine-tuning experiment using a single dataset with clearly defined training and test splits.

A1: Thank you for this suggestion. We have clarified the dataset composition and partitioning protocol in the revised manuscript. All methods use the same training and validation splits and are evaluated on official benchmark datasets using open-source evaluation frameworks.

Furthermore, following your suggestion, we conduct an additional experiment using SocialIQA as the sole training dataset. Specifically, we randomly sample 2,000 examples without replacement from its official training split using seed 1234, and independently sample 125 examples from the official dev split for validation. After training, we evaluate the resulting model using the same benchmark suite and evaluation settings as in our main experiments. The results are reported below, demonstrating RC-GEM's effectiveness under a simpler single-dataset training setting.

Evaluation results after SocialIQA-only training (%)

MethodBoolQPIQASIQAHellaSwagWinoG.ARC-EARC-COBQACS Avg.
RC-GEM78.7881.4559.7780.9871.4383.2962.8848.8070.92
MethodGSM8KMATHMath Avg.HumanEvalMBPPCode Avg.MMLU Avg.*
RC-GEM47.9210.0028.9611.5929.8920.7453.67

Q2: Insufficient characterization of the performance–OOD retention–efficiency trade-off […] Can the authors provide a unified analysis of downstream performance, OOD retention, and parameter efficiency? A composite metric, Pareto analysis, or training-step trade-off curves would help quantify RC-GEM's advantages.

A2: Thank you for this helpful suggestion. We summarize the additional analyses below.

(2.1) Unified ID–OOD–efficiency comparison. Fig. 3 compares ID gains and OOD retention across adaptation methods, while Fig. 4 jointly visualizes ID gains, trainable parameter ratios, and mean OOD retention. RC-GEM achieves a 12.77-point ID gain with only 0.0913% trainable parameters and nearly 100% mean OOD retention. Methods achieving comparable ID gains, including FFT, LoRA, Prefix Tuning, and PERFT-E, exhibit substantially greater OOD degradation, particularly on Math and Code. Moreover, FFT updates nearly all model parameters, while Prefix Tuning uses approximately 5.5× as many trainable parameters as RC-GEM. By contrast, methods with stronger OOD retention, such as Router-FT and RoMA, achieve only 41% and 12% of RC-GEM's ID gain, respectively.

(2.2) Sensitivity to parameter budgets. We provide rank sweeps for LoRA, ACE, and PERFT-E in A.6 (Tables 11–13), examining how different ranks affect ID/OOD performance and trainable parameter counts. Fig. 4 further visualizes these performance–parameter trade-offs for LoRA and PERFT-E.

(2.3) Pareto analysis. We also examined the Pareto frontier across ID performance, mean OOD retention, and trainable parameter ratio. Five methods (FFT, LoRA, Router-FT, RoMA, and RC-GEM) are identified as non-dominated, while Prefix Tuning, ACE, and PERFT-E are dominated. However, all parameter-efficient baselines use relatively small parameter budgets (less than 1% of model parameters are trainable), while their ID gains and OOD retention differ substantially. Consequently, methods such as Router-FT and RoMA remain non-dominated despite achieving much smaller ID gains than RC-GEM. Thus, Pareto dominance alone does not sufficiently distinguish methods with a favorable overall trade-off. To provide a more comprehensive comparison across parameter budgets, we have extended Fig. 4 with LoRA and PERFT-E rank-sweep results, illustrating how ID gains and OOD retention vary with trainable parameter ratios.

We next address the remaining five questions below.


Q3: Sensitivity to baseline hyperparameters. Parameter-efficient methods can be highly sensitive to their architectural configurations. For example, LoRA uses rank 32 in the reported experiments, but smaller ranks may considerably reduce its parameter count while maintaining competitive performance. […] Similarly, ACE achieves strong OOD retention, but its parameter count depends on its adapter ranks and configuration, which are not clearly documented.

A3: Thank you very much for pointing this out. We have conducted learning-rate sweeps for the baselines and updated Table 1 accordingly. The corresponding optimization settings are provided in A.10 (Table 16). Moreover, we provide additional rank sweeps for rank-sensitive baselines (LoRA, ACE, and PERFT-E), reporting their performance and trainable parameter counts in A.6 (Tables 11–13).


Q4: Overcomplicated mathematical notation. […] A more compact notation would help readers better appreciate the contribution.

A4: Thank you for this helpful suggestion! We have revised Section 5 to simplify the mathematical presentation.


Q5: Insufficient discussion of closely related aggregation methods […] I would encourage the authors to clarify how RC-GEM differs from these methods, particularly whether fine-grained subspace modulation provides meaningful advantages over previously proposed expert-level aggregation mechanisms.

A5: Thank you very much for recommending these relevant works. We have carefully studied them and incorporated discussions of these methods into the Introduction and Related Work to clarify the distinctions between RC-GEM and prior expert aggregation approaches.


Q6: How does RC-GEM relate to methods for learning combinations of model outputs or parameters? For example, model merging and learned LoRA adapter combinations explore related forms of aggregation. Could insights from these methods be useful in this setting?

A6: Thank you for raising this interesting connection. We agree that model merging, learned LoRA combinations, and RC-GEM share the broad goal of improving model performance by learning how to combine multiple components. However, they differ in their combination mechanisms and integration into the model. Model merging combines parameter updates from different models, while learned LoRA combinations aggregate multiple adapter updates or outputs. In contrast, RC-GEM operates within a single pretrained MoE backbone without replacing or restructuring its original architecture. Instead, it introduces lightweight, trainable modulation modules into the expert aggregation process, while keeping all pretrained parameters frozen. These modules refine selected expert representations through routing-conditioned, subspace-wise modulation.

Among the insights from these approaches, the most relevant to RC-GEM is input-dependent combination: rather than relying on fixed combination weights, the contributions of different components can be adapted according to the input. This is closely related to RC-GEM, which uses router logits as input-dependent signals to generate token-dependent modulation coefficients.


Q7: How does RC-GEM compare directly against the Residual Aggregation Controller (RAC) introduced in Section 4.2? A parameter-matched comparison would help establish whether fine-grained subspace modulation provides advantages beyond scalar expert reweighting.

A7: Thank you for this comment. This question overlaps with Reviewer Kr39's Q1. Please see our response to that question for RAC's experimental settings and comparisons with expert-level scalar reweighting first.

In addition, to directly address the parameter-matched comparison, we compare RAC with RC-GEM at G=1 and r=64, yielding nearly identical trainable parameter ratios (0.0313% vs. 0.0303%). Even under this restricted modulation setting (where all learned directions share a single modulation coefficient, please see our response to Reviewer Qbuf's Q1 for further discussion of fine-grained modulation), RC-GEM achieves an ID score above 73 (Fig. 6(b)), substantially exceeding RAC's best CS score of 69.29 across the tested modulation strengths ε (Table 10).

Reviewer_Kr39OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RC-GEM, a parameter-efficient adaptation method for MoE models that performs routing-conditioned subspace modulation. The method introduces learned subspace bases and weighted centering to control expert contributions. Experiments on OLMoE and Qwen1.5-MoE demonstrate substantial ID improvements while preserving pretrained OOD capabilities with fewer than 0.1% trainable parameters.

Critical strengths
  • RC-GEM provides a simple but structured approach to modulating expert contributions through learned subspaces, with weighted centering preserving the total aggregation weight assigned to each subspace.
  • RC-GEM achieves better ID improvements with fewer than 0.1% trainable parameters, demonstrating a favorable performance-to-parameter trade-off.
Critical weaknesses
  • The core mechanism of RC-GEM is a relatively straightforward extension of expert-level scalar reweighting to groupwise modulation over learned subspaces, with weighted centering. Although ablations support the individual components, the methodological contribution beyond simpler expert-output modulation approaches appears limited. Moreover, the diagnostic studies in Section 4 do not convincingly motivate this particular design. The aggregation oracle in Section 4.1 directly optimizes instance-specific weights using ground-truth evaluation labels, primarily demonstrating in-sample optimization rather than generalizable adaptation. Section 4.2 shows that scalar reweighting (RAC) already improves ID performance while largely preserving OOD performance under moderate perturbations. However, RAC and RC-GEM are evaluated under different training settings (e.g., one vs. two epochs), making a direct comparison difficult. Thus, the experiments do not clearly establish why the proposed learned subspace modulation is necessary or preferable to simpler alternatives.

  • Evidence for a superior ID–OOD trade-off is insufficient. RC-GEM does not consistently outperform LoRA on ID benchmarks, while its OOD preservation advantage over MoE-specific baselines remains less clear. Notably, ACE improves both ID and OOD performance on Qwen1.5-MoE. It is therefore difficult to determine whether RC-GEM achieves a better ID–OOD trade-off or simply operates at a different adaptation strength. While Tables 3 and 5 and Figure 5 provide useful ablations, further analyses, i.e., how representations change, downstream routing behavior, and their relationship to OOD retention, would help explain the proposed mechanism. Sensitivity analyses of key hyperparameters (e.g., subspace rank (r), number of groups (G), and orthogonal-regularization weight) are also missing. Furthermore, Appendix A.8 reports an explicit learning-rate search for RC-GEM but relies on recommended or default configurations for baselines, leaving the comparability of hyperparameter tuning unclear.

Questions for authors
  • What specific advantages does routing-conditioned subspace modulation provide over simpler expert-output modulation approaches? Can the authors better justify the proposed design beyond the diagnostic studies in Section 4?
  • Can the authors demonstrate a better ID–OOD trade-off against strong baselines such as ACE at comparable ID performance levels, and provide further evidence explaining how RC-GEM preserves pretrained OOD capabilities?
  • How sensitive are the results to the subspace configuration and optimization settings?
3 comments

We sincerely appreciate your constructive and insightful comments. We have consolidated the concerns raised in your Critical Weaknesses and Questions for Authors into six questions, which we address below.


Q1: Although ablations support the individual components, the methodological contribution beyond simpler expert-output modulation approaches appears limited. […] However, RAC and RC-GEM are evaluated under different training settings (e.g., one vs. two epochs), making a direct comparison difficult.[…] What specific advantages does routing-conditioned subspace modulation provide over simpler expert-output modulation approaches?

A1: Thank you for this comment. We would like to clarify the role of the oracle and RAC, as well as the advantages of RC-GEM over simpler expert-output modulation approaches, including Router-FT and RAC. We have also added RAC's experimental settings and detailed per-task results in A.3 (Fig. 8 and Table 10). We discuss these points below.

(1.1) Clarifying the role and experimental settings of the diagnostic studies. The oracle and RAC are designed as diagnostic studies to investigate the optimization potential of expert aggregation and its ID–OOD trade-off, rather than as direct baselines for RC-GEM. We therefore did not originally provide their complete experimental settings. To clarify this, we have added the detailed RAC experimental settings and per-task results in A.3 (Fig. 8 and Table 10). In fact, RAC and RC-GEM follow the same training protocol and evaluation procedure. Both are trained for 2 epochs, but RAC is evaluated at epoch 1.0 because its validation loss has already stabilized by this point, whereas RC-GEM uses the epoch-2.0 checkpoint (see Fig. 8 for a comparison of their validation-loss trajectories).

(2.1) Advantages over simpler expert-output modulation approaches. If by simpler expert-output modulation approaches you mean expert-level scalar reweighting that directly adjusts aggregation weights, we have already included Router-FT in Table 1 as a representative approach. Under the same evaluation protocol, Router-FT recovers only 41% of RC-GEM's ID gain, while exhibiting noticeable OOD degradation. We agree that RAC is also a relevant expert-level scalar reweighting approach for comparison, and have therefore added its detailed per-task results across different modulation bounds in Table 10. These results show that RAC exhibits a similar trend, recovering only 39% of RC-GEM's ID gain, with OOD performance deteriorating as the modulation bound increases. In contrast, RC-GEM achieves substantially larger ID gains while largely preserving pretrained OOD capabilities, with a comparable trainable parameter budget (0.04%–0.09%).


Q2: Moreover, the diagnostic studies in Section 4 do not convincingly motivate this particular design. […] Can the authors better justify the proposed design beyond the diagnostic studies in Section 4?

A2: Thank you very much for pointing this out. This concern overlaps with Reviewer Qbuf's Q2. Please see our response to that question for the motivation behind our design.


Q3: Furthermore, A.8 reports an explicit learning-rate search for RC-GEM but relies on recommended or default configurations for baselines, leaving the comparability of hyperparameter tuning unclear.

A3: Thanks for your valuable suggestion! We additionally performed learning-rate sweeps for the other baselines and updated Table 1 wherever the learning rates identified through these sweeps differed from the original settings. The corresponding optimization configurations and selection details are provided in A.10. Moreover, for rank-sensitive baselines (LoRA, ACE, and PERFT-E), we also provide additional rank sweeps in A.6 (Tables 11–13).

We next address Questions 4 and 5 below.


Q4: Evidence for a superior ID–OOD trade-off is insufficient. RC-GEM does not consistently outperform LoRA on ID benchmarks, while its OOD preservation advantage over MoE-specific baselines remains less clear. Notably, ACE improves both ID and OOD performance on Qwen1.5-MoE. […] Can the authors demonstrate a better ID–OOD trade-off against strong baselines such as ACE at comparable ID performance levels?

A4: Thank you for this comment. We have updated Table 1 following the baseline tuning in our response to Q3. The updated ID–OOD comparisons are summarized below.

(4.1) ID–OOD trade-off against adaptation baselines. As shown in Fig. 3, methods achieving ID gains comparable to RC-GEM, including FFT, LoRA, Prefix Tuning, and PERFT-E, exhibit substantially greater OOD degradation, particularly on Math and Code. Moreover, FFT updates nearly all model parameters, while Prefix Tuning uses approximately 5.5× as many trainable parameters as RC-GEM. Conversely, methods with relatively strong OOD retention, such as Router-FT and RoMA, achieve only 41% and 12% of RC-GEM's ID gain, respectively. Fig. 4 further illustrates that RC-GEM achieves a 12.77-point ID gain with small trainable parameters while maintaining nearly 100% mean OOD retention. Regarding ACE, although it exhibits relatively strong OOD retention, its ID performance remains below RC-GEM on both backbones under the updated experimental settings (Table 1).

(4.2) Preservation of pretrained OOD capabilities. Although RC-GEM's OOD performance is slightly below the pretrained model on some benchmarks, it largely preserves the original capabilities. As shown in Fig. 3, RC-GEM retains approximately 98%, 100%, and 105% of the pretrained performance on Math, MMLU, and Code, respectively. Similar results are observed on Qwen1.5-MoE, with approximately 95%, 97%, and 100% retention, respectively.


Q5: Further analyses of how representations change, downstream routing behavior, and their relationship to OOD retention, would help explain the proposed mechanism. […] and provide further evidence explaining how RC-GEM preserves pretrained OOD capabilities?

A5: Thank you for your constructive suggestions. We have added further analyses in A.16 and also draw on the modulation magnitude distribution in Fig. 5 as complementary evidence to address this concern.

(5.1) Representation changes, downstream routing behavior, and OOD preservation. We analyze the local effects and accumulated changes induced by RC-GEM, with LoRA serving as a reference at comparable ID performance, focusing on three aspects:

  • Representation changes: Local and accumulated representation perturbations (Table 24, Fig. 10, and Table 25).
  • Downstream routing behavior: Router-logit deviations and pretrained expert retention (Fig. 10 and Tables 25–27).
  • OOD preservation: Pretrained predictive behavior through next-token distribution consistency (Fig. 11 and Table 28).

Please see A.16 for detailed experimental results and analyses.

(5.2) Modulation magnitude. Fig. 5 shows that most learned modulation coefficients remain small, indicating that RC-GEM generally applies modest subspace-wise adjustments rather than large-scale rescaling.

Taken together, A.16 shows that, at comparable ID performance, RC-GEM better preserves pretrained representations, routing patterns, and predictive behavior on OOD inputs than LoRA. The predominantly small modulation coefficients in Fig. 5 provide complementary evidence of targeted and modest subspace-wise adjustments. These findings help explain how RC-GEM achieves effective ID adaptation while better retaining pretrained OOD capabilities.

Finally, we address the last question.


Q6: Sensitivity analyses of key hyperparameters (e.g., subspace rank (r), number of groups (G), and orthogonal-regularization weight) are also missing. […] How sensitive are the results to the subspace configuration and optimization settings?

A6: Thank you for your valuable suggestion. We have added sensitivity analyses of the subspace configuration and orthogonal regularization in A.12 and A.14, and clarified the optimization configurations in A.10. We address the three aspects below.

(6.1) Sensitivity to subspace rank r. We investigate the effect of varying the subspace rank on ID adaptation and OOD performance in A.14 (Table 21 and Fig. 9).

(6.2) Effect of modulation granularity G. The role of G is closely related to isolating the contribution of fine-grained modulation. This aspect overlaps with Reviewer Qbuf's Q1. Please see our response to that question, where we provide the corresponding theoretical analysis in Section 5.2 and experimental analyses in Section 6.3 and A.11, A.14, and A.15.

(6.3) Sensitivity to orthogonal regularization. We investigate the effect of different orthogonal-regularization weights in A.12 (Table 18) and further compare soft and strict orthogonality in A.14.

Reviewer_QbufOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RC-GEM, a parameter-efficient adaptation method for sparse MoE models. The authors first use an aggregation oracle and a residual aggregation controller (RAC) to demonstrate the adaptation headroom in expert aggregation and the ID–OOD trade-off associated with scalar reallocation. They then introduce routing-conditioned modulation over learned representation subspaces. Experiments on OLMoE and Qwen1.5-MoE show strong adaptation performance with fewer than 0.1% trainable parameters, while largely preserving OOD capabilities.

Critical strengths

The paper presents a clear and logically progressive investigation. The oracle first establishes the potential for improving expert aggregation, while RAC further examines the limitations of scalar reallocation and its ID–OOD trade-off. This provides a coherent motivation for exploring alternative aggregation mechanisms. Empirically, RC-GEM achieves strong results across two MoE backbones with fewer than 0.1% trainable parameters. The ablations on weighted centering, conditioning, and basis learning also provide useful insights into the contributions of individual components.

Critical weaknesses

My main concern is the missing link between the preceding analysis and the choice of fine-grained subspace modulation. The oracle and RAC experiments establish the potential and limitations of expert-level aggregation, but do not directly explain why these observations motivate independently modulating multiple learned subspaces. Although Table 2 shows that a learned basis outperforms fixed or random decompositions, it does not isolate the contribution of finer modulation granularity itself.

Questions for authors

Could the authors provide further subspace-level analysis to better establish the connection between the preceding aggregation diagnostics and the proposed fine-grained modulation?

2 comments

Thank you for your careful reading and thoughtful feedback. We sincerely appreciate your valuable suggestions, which have helped us further improve our work. We address each of your concerns below.


Q1: Although Table 2 shows that a learned basis outperforms fixed or random decompositions, it does not isolate the contribution of finer modulation granularity itself.

A1: Thank you very much for pointing this out. To address this concern, we have taken a closer look at the mechanism underlying fine-grained modulation, refining the theoretical analysis in Section 5.2 and conducting targeted experiments and analyses in Section 6.3 and A.11, A.14, and A.15. We detail these additions below.

(1.1) Refining the theoretical analysis of modulation scaling in learned subspaces.

A closer theoretical examination reveals that fine-grained modulation involves two complementary effects: static directional scaling induced by the learned basis and token-dependent dynamic modulation controlled by the predicted coefficients. To account for their interaction, we refine the previous perturbation bound from

\lesssim 2\epsilon\\|z_{t,e}\\|_2$$ to the more precise bound $$\\|\Delta z_{t,e}\\|_2 \leq 2\epsilon\\|A\\|_2^2\\|z_{t,e}\\|_2,$$ which explicitly accounts for potential amplification by the learned basis. The existing soft orthogonal regularization limits this amplification. The full SVD-based derivation and analysis are provided in **Section 5.2**. **(1.2) Empirically disentangling static scaling and dynamic modulation.** We have added targeted ablations in **Section 6.3** to examine the respective contributions of static directional scaling and dynamic subspace-wise modulation. - **Post-hoc interventions (Table 6):** On the same trained checkpoint, *Averaged Coefficients* retains static scaling but removes subspace-specific dynamic modulation, while *Shuffled Coefficients* preserves the coefficient distribution but disrupts alignment with learned directions. Both substantially reduce ID performance (77.08% to 64.43% and 64.97%, respectively), with limited OOD impact. This shows that static scaling alone is insufficient and that its effectiveness relies on aligned dynamic modulation (**A.15**). - **Dynamic modulation granularity (Fig. 6(a)):** Increasing $G$ from 2 to 32 at fixed $r=128$ improves ID performance from 73.91% to 77.08%, with average OOD performance also improving from 31.80% to 34.33%. Both slightly decline beyond $G=32$, indicating benefits from finer-grained modulation up to an appropriate granularity. - **Static directional scaling under coarse dynamic modulation (Fig. 6(b)):** We set $G=1$, so all learned directions share a single token-dependent coefficient, and vary $r$ to increase the directions available for static scaling. ID performance plateaus around 73.4% beyond $r=16$, below the 77.08% achieved with $G=32$ (although larger $r$ also increases capacity and may improve ID performance). This indicates that static scaling capacity alone cannot explain the ID gains, highlighting the contribution of independently modulating multiple learned subspaces. **(1.3) Isolating dynamic modulation under hard orthogonality.** Building on **(1.2)**, we further isolate the contribution of dynamic modulation by removing static directional scaling through hard orthogonality in **A.14**, As shown in **Table 22**, increasing $G$ from 1 to 32 at $r=128$ improves ID performance from 72.76% to 76.59%, confirming the independent contribution of finer-grained dynamic modulation. However, average OOD performance declines from 34.95% to 29.81%. This suggests that strict orthogonality may also constrain learned subspace geometry, whereas soft orthogonal regularization achieves a more favorable ID–OOD trade-off at finer granularity. **(1.4) Analyzing the learned subspace spectra.** Finally, we analyze the learned basis spectra in **A.11 (Table 17)**. The eigenvalues remain bounded away from zero, indicating no rank collapse, yet deviate from unity, revealing moderate direction-dependent scaling. Moreover, their spectral spreads vary across layers, indicating that the learned bases develop distinct static scaling characteristics across MoE layers. **Overall**, these results show that the gains from fine-grained modulation cannot be explained by static directional scaling alone. Rather, they primarily arise from independently modulating multiple learned subspaces through routing-conditioned coefficients, with their alignment to learned directions playing a critical role. The hard-orthogonality experiments further confirm this dynamic contribution even when static scaling differences are removed. Meanwhile, the learned spectra demonstrate that soft orthogonal regularization preserves moderate, layer-dependent static scaling, providing complementary flexibility and a more favorable ID–OOD trade-off than hard orthogonality.

We next address the second question.


Q2: My main concern is the missing link between the preceding analysis and the choice of fine-grained subspace modulation……Could the authors provide further subspace-level analysis to better establish the connection between the preceding aggregation diagnostics and the proposed fine-grained modulation?

A2: Thank you for raising this important point. We agree that the original manuscript did not sufficiently articulate the connection between the aggregation diagnostics and our design choices. We have therefore clarified this progression in the penultimate paragraph of the Introduction, complementing the subspace-level analyses presented in our response to Q1. We outline the design progression below.

(2.1) From aggregation diagnostics to selective and bounded modulation.

The oracle in Section 4.1 establishes the potential for improving downstream performance through expert aggregation adaptation, while RAC in Section 4.2 shows that excessive scalar reweighting can compromise OOD performance. We further provide per-task results across residual bounds in A.3 (Table 10). These diagnostics suggest two design considerations. First, adaptation should be bounded to keep aggregation weights close to their pretrained values, thereby limiting disruption to pretrained routing behavior. Second, scalar expert-level reweighting applies the same adjustment uniformly across all output dimensions, limiting its ability to distinguish between task-relevant and less relevant components. A more targeted approach could selectively adjust potentially task-relevant components while keeping others closer to their pretrained behavior. This motivates bounded, non-uniform modulation across output dimensions.

(2.2) From coordinate-wise modulation to learned subspaces.

A direct way to realize such non-uniform modulation is to assign independent coefficients to individual output dimensions. However, coordinate-wise modulation requires substantially more modulation-head parameters, approximately 64× as many as RC-GEM (A.1). We therefore explore contiguous coordinate grouping as a parameter-efficient alternative, which constitutes our initial grouping design. As shown in Table 2, it provides modest ID gains. We further provide its detailed per-task evaluation, including OOD results, in A.13 (Table 19), confirming that OOD performance is largely preserved. This demonstrates that bounded, finer-grained modulation can improve ID performance while retaining pretrained OOD capabilities, but its ID gains remain limited, potentially because predefined coordinate partitions do not form meaningful modulation units. This motivates learning adaptive subspaces rather than relying on manually predefined partitions, which require costly task-specific grouping of expert-output dimensions and are less generalizable.

As additional evidence of the resulting preservation of pretrained behavior, A.16 analyzes representation perturbations and routing preservation. At comparable ID performance, RC-GEM introduces smaller accumulated representation and router-logit deviations than LoRA, while largely retaining pretrained expert selections, particularly on OOD inputs. We hope these analyses clarify the rationale behind RC-GEM's key design choices.

Together, this progression explains how the diagnostic findings lead from uniform expert-level reweighting to bounded, fine-grained modulation over learned subspaces.

Comments on the paper

1 comment

We deeply appreciate the insightful and valuable comments provided by all reviewers. We are grateful for your recognition of our work and constructive suggestions, which have helped us further improve the paper.

To address all the feedback, we have strengthened the theoretical analysis, conducted extensive additional experiments and analyses, and incorporated the corresponding revisions into the updated PDF. For ease of review, all revisions are highlighted in blue. We strongly encourage you to consult the updated PDF alongside our responses, as it provides the full context of our revisions.

We have provided point-by-point responses to all reviewers’ comments. We sincerely appreciate your time and thoughtful feedback, and hope our responses adequately address your concerns. Thank you again!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.