acceptodds
Under review as a conference paper at ICLR 2027

LoREAL: Low-Rank Expert-Input Adaptation for Mixture-of-Experts Language Models

Abstract

Mixture-of-Experts (MoE) language models use the same token representation to select experts and to compute their outputs. Downstream adaptation can therefore act on two distinct pathways: routing and expert computation. We propose Low- Rank Expert Adaptation (LREA), which separates these pathways and adapts only the inputs delivered to the experts. At each MoE layer, the frozen routing pathway uses the layer input to select experts and compute mixture weights. In parallel, the adaptive expert-input pathway passes the same input through a shared low-rank residual adapter before sending it to the selected frozen experts. The local adapter thus changes what the experts receive without changing the input used for routing at that layer; earlier adapted layers may still affect later routing decisions. The shared adapter introduces 2dr trainable parameters per layer, with optional per-expert scaling adding N scalars. On OLMoE-1B-7B, LREA outperforms router tuning across continued pretraining and supervised fine-tuning, reduces measured forgetting relative to router tuning, and improves on expert LoRA in most comparisons at comparable parameter budgets. Ablations support adapting the expert-input pathway as an effective alternative to increasing router capacity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
continue pretraining, mixture of experts

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iUFSOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LoREAL proposes parameter-efficient adaptation of pretrained MoE language models by modifying the representations delivered to frozen experts. A shared low-rank nonlinear residual adapter transforms expert inputs, while the local frozen router continues to operate on the representation before that adapter. The adapter’s parameter count is independent of the number of experts, apart from optional expert-specific scaling parameters. Experiments on OLMoE cover continued pretraining and supervised fine-tuning for mathematics, instruction following, and code, with comparisons against router tuning and expert-side LoRA.

Critical strengths

The proposed architecture is simple and potentially useful. Sharing an input adapter across experts avoids multiplying the adaptation budget by the expert count, and separating the local router and expert-input pathways gives a clear architectural hypothesis to investigate. The paper correctly acknowledges that freezing a router does not guarantee unchanged routing throughout the network, because earlier adapted layers can change later router inputs.

The evaluation covers several adaptation settings rather than a single benchmark. The improvements over ordinary router tuning are substantial, and the comparisons against expert LoRA use matched effective parameter budgets. I also appreciate the authors’ transparency about single-run results, incomplete ablations, and limitations of the representation-shift analysis, although some of those limitations need to be reflected more consistently in the main claims.

Critical weaknesses
  1. The relationship to representation-based adaptation is insufficiently developed. Please add and discuss ReFT: Representation Finetuning for Language Models (NeurIPS 2024) and Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, which introduces (IA)³ (NeurIPS 2022). Both are relevant to adapting frozen models through their representations or activations. The discussion should explain what is contributed specifically by sharing the adapter across experts and placing it on the expert-input pathway. The broader adaptation discussion should also include Mixture of LoRA Experts (ICLR 2024), HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (NeurIPS 2024), and LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts (ICLR 2026). I am not requesting experiments with every LoRA-expert method.

  2. The experimental comparisons do not yet fully test the central architectural argument. I would prioritize an experimental comparison with Routing Manifold Alignment (RoMA), which is already cited but is not included among the evaluated baselines. Since the paper argues for expert-input adaptation as an alternative to adapting routing, a regularized router-adaptation method is a more informative comparator than ordinary router tuning alone. A representative comparison with LoReFT or another established representation-adaptation method would also help distinguish the proposed pathway separation from the more general benefit of transforming hidden representations. The paper already includes the important “adapter to both pathways” control. However, this variant obtains slightly lower perplexity than expert-only adaptation in Table 6, and forgetting is not reported for that comparison. Please complete this comparison with retention and routing-change measurements rather than treating expert-only placement as already established to be preferable. The joint router–adapter experiments likewise stop well before the planned training budget and cannot establish the final benefit of freezing the router.

  3. The low-rank distribution-shift claim is not established by the current analysis. This is my main concern. The introduction presents the downstream shift as concentrated in a small number of directions. However, Appendix B states that the reported axes were computed relative to the identity matrix even though the transformed reference covariance is not the identity. It further explains that the reference-corrected matrix requires recomputing the axes and dependent measurements. The appendix also identifies non-independent controls and unexpectedly large reference-control ratios, and explicitly cautions against interpreting the reported axis counts as the intrinsic rank of the shift. Please either recompute this analysis with the appropriate reference covariance and document-disjoint controls, or narrow the main-text claim and present the method primarily as an empirically effective architecture. The adaptation results can remain valuable without claiming that a low-rank distribution shift has been demonstrated.

Questions for authors

Does expert-only adaptation retain an advantage over adapting both pathways when comparing forgetting, routing changes, and repeated runs?

And also see weakness

2 comments

We thank the reviewer for the constructive feedback. We address the three weaknesses and provide additional experimental results below.

W1. The relationship to representation-based adaptation is insufficiently developed. Please discuss ReFT, (IA)³, Mixture of LoRA Experts, HydraLoRA, and LD-MoLE.

We thank the reviewer for highlighting these studies and will expand the related work accordingly. While ReFT and (IA)³ adapt frozen models through representations or activations, LoREAL specifically separates routing and expert-input pathways in pretrained MoE models. The router operates on the original hidden states, while a shared low-rank adapter transforms only expert inputs.

Unlike MoE-LoRA approaches employing multiple adaptation modules, LoREAL shares one adapter across all experts per layer, making its parameter count independent of the number of experts and enabling a larger rank under the same budget. Our contribution is thus the shared, expert-input-specific adaptation architecture, rather than low-rank adaptation itself.

W2. The experimental comparisons do not fully establish the architectural advantages. Comparisons with RoMA and LoReFT are missing, the both-pathways control lacks retention and routing-change measurements, and joint router–adapter training is incomplete.

RoMA. We attempted multiple implementations of RoMA, but the necessary implementation details are not publicly available, preventing a reliable comparison. Beyond standard router tuning, we evaluated nonlinear routing adapters with up to 33× more trainable parameters, which still underperform LoREAL.

LoReFT. LoREAL differs from general representation adaptation by explicitly separating routing and expert-input pathways. Our insertion-position ablation directly examines this distinction by comparing expert-only adaptation with adapting both pathways.

Expert-only vs. Both Pathways. We reran the comparison under matched settings and additionally measured forgetting and top-8 routing drift.

ConfigurationPPL ↓Forgetting ↓Routing Drift ↓
Both pathways6.3661+0.46%0.0450
Expert-only6.3634+0.46%0.0331

Expert-only achieves slightly lower PPL and identical forgetting, with 26.4% lower routing drift, supporting our design choice of excluding the adapter from the local routing pathway.

Joint training. We clarify that the incomplete joint router–adapter experiments are exploratory rather than evidence of final convergence. Our main conclusions rely on completed router-tuning and nonlinear-router comparisons.

W3. The low-rank distribution-shift claim is insufficiently supported due to the use of M instead of D, non-independent controls, large reference-control ratios, and uncertainty about the intrinsic rank.

M vs. D. We recomputed the drift axes using the reference-corrected matrix D=M+I−C0D=M+I-C_0. At layer 8 on Math SFT, spread ratios at axes 1, 8, and 32 are 145×, 21×, and 7.6× using D, compared with 129×, 22×, and 9× using M. Similar qualitative patterns hold across domains.

Data independence. Although the original splitting strategy may allow limited document-level overlap, its actual extent was minimal. Additional experiments using strictly document-disjoint splits yielded consistent results.

Reference controls. Reference-control ratios serve as empirical baselines for relative distributional differences rather than exact unit-ratio expectations. The larger downstream ratios along leading directions indicate pronounced directional discrepancies.

Low-rank interpretation. Our D-based analysis consistently identifies larger discrepancies along a small number of leading directions. We clarify that these observations motivate low-rank adaptation without establishing the intrinsic rank of the shift. Importantly, LoREAL learns its adaptation subspace from the task objective, independently of the diagnostic drift axes.

Q1. Does expert-only adaptation retain an advantage over both-pathway adaptation in forgetting, routing changes, and repeated runs?

Our updated results show slightly lower PPL (6.3634 vs. 6.3661), identical forgetting (+0.46%), and 26.4% lower top-8 routing drift (0.0331 vs. 0.0450) for expert-only adaptation. These results support its routing-preservation advantage, although repeated-run statistical significance has not yet been established.

Thanks, I increase the score from 3 to 4, good luck

Reviewer_4SJZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LoREAL, a parameter-efficient adaptation method for pretrained MoE language models. It freezes the pretrained model and inserts a low-rank nonlinear residual adapter shared across experts within each MoE layer. The local router reads the representation before adaptation, while the selected experts receive the adapted representation. An analysis of pretraining and downstream expert-input distributions motivates this design. Experiments on OLMoE-1B-7B cover continued pretraining and math, instruction, and code fine-tuning, comparing against router tuning and expert-weight LoRA at matched total trainable-parameter budgets. The reported results show improved adaptation and less forgetting relative to router tuning and outperform expert LoRA on most evaluated metrics. Ablations examine router capacity, nonlinearity, adapter position, and joint router training.

Critical strengths
  1. Improved adaptation across several tasks. LoREAL improves over router tuning and outperforms expert LoRA on most reported adaptation metrics. The continued-pretraining results also show less forgetting than router tuning, suggesting that adapting expert inputs is a useful direction.

  2. Ablations help explain the gains. Simply increasing router capacity does not match LoREAL's performance, and removing the nonlinearity preserves part of the improvement. These results suggest that the gains are not explained solely by extra router parameters or the nonlinear activation.

Critical weaknesses
  1. Conceptual overlap with LatentMoE. In the Introduction (lines 90–94), the authors describe LoREAL as “among the first training-based adaptation methods to explicitly separate the routing and expert-input pathways within a pretrained MoE layer.” However, LatentMoE already separates router and expert inputs. Although its pretraining setting differs from LoREAL's adaptation setting, the input-separation principle itself is not new.

  2. Novelty beyond adapters and ReFT. LoREAL applies a shared low-rank nonlinear residual adapter to expert inputs. This resembles standard bottleneck adapters, while ReFT also adapts frozen models through interventions on representations. The paper should clarify what new methodological contribution LoREAL provides beyond placing and sharing such a module within MoE layers.

  3. Input-shift does not well-motivated the proposed study. Observing a low-rank distribution shift does not show that it causes the downstream performance gap or that the learned adapter corrects it. The authors acknowledge that the gains do not demonstrate reversal of the measured drift (§4.2, lines 369–370), but the connection between the analysis and the method remains weak.

  4. Attention-adaptation baselines. LoRA on the preceding attention module can also change the representations delivered to experts, while affecting the inputs seen by both the router and experts. A matched-budget attention-LoRA or attention-input-adapter baseline would help test whether restricting adaptation to the expert branch provides an additional benefit.

  5. Active-parameter mismatch and LoRA tuning. At the 9.44M budget, every token uses LoREAL's shared adapters, whereas expert LoRA activates only 8 of 64 expert adapters, or approximately 1.18M adaptation parameters per token. This eightfold difference makes it difficult to attribute the gains to the proposed design. The comparison is valid for a fixed storage budget, but does not match active adaptation capacity. The SFT comparison also uses only rank-1 expert LoRA. Higher-rank expert LoRA with suitable learning-rate tuning and matched active adaptation budgets should be evaluated.

  6. Inference overhead. The nonlinear adapter adds a sequential computation before the experts, even though it has few parameters. Latency and throughput should be compared with the base model and merged expert LoRA, preferably separating prefill and decoding.

  7. Limited model coverage. All experiments use OLMoE-1B-7B, a single older MoE backbone. Evaluation on more recent models and at least one additional MoE family is needed to assess whether the adaptation and retention gains generalize across architectures.

Questions for authors
  1. Please address the concerns and requested comparisons raised in the weaknesses above.
  2. Appendix C states that the expert-only joint run had reached step 56 and the both-input joint run was interrupted at step 175, versus a planned 1,907 steps. I am curious why were these runs not completed? Could you provide full-training comparisons against the frozen-router baseline after suitable tuning, including an independent router learning-rate search?
4 comments

We thank the reviewer for the feedback.

W1. Conceptual overlap with LatentMoE, which already separates router and expert inputs.

W2. Novelty beyond adapters and ReFT. What is new beyond placing and sharing such a module in MoE layers?

Our claim concerns training-based adaptation of pretrained MoE models and remains accurate. LatentMoE is a pretraining architecture. Its experts operate in a different representation dimension, so it is trained from scratch and does not adapt an existing pretrained MoE. LoREAL keeps the hidden-state dimensionality and the pretrained router and experts unchanged.

Bottleneck adapters and ReFT modify the hidden representation shared by all modules, so the router and experts receive the same adapted input. In LoREAL, the router input stays unchanged and only the expert input is corrected by one shared low-rank module. This separation is specific to MoE layers and is absent from both methods.

W3. Weak motivation from input shift. A low-rank shift does not show that it causes the downstream gap or that the adapter corrects it.

The drift analysis motivates the design. The shift at expert inputs is concentrated along a few directions (Figure 2b), so a low-rank expert-input correction is the natural parameterization. LoREAL is not designed to reverse the measured drift. The correction is learned through the downstream objective (Section 3.2).

W4. Attention-adaptation baselines. Matched-budget attention-LoRA would test whether restricting adaptation to the expert branch helps.

Attention-LoRA adapts the attention sublayer, which is outside the MoE layer and not the focus of our method. The two are orthogonal and can be used together.

We still added matched-budget attention-LoRA (q, k, v, o) on all three SFT corpora (r=8 vs. LoREAL r=32 at 2.10M, and r=36 vs. r=144 at 9.44M). Attention-LoRA shows no large advantage. It is better only on IFEval (≈2 points) and GSM8K at 2.10M (≈1 point). LoREAL is better on HumanEval and MBPP at both budgets (MBPP 28.2 vs. 23.4 at 9.44M) and GSM8K at 9.44M.

Retention clearly favors LoREAL and confirms the benefit of restricting adaptation to the expert branch. Forgetting on out-of-distribution text is lower in every setting (2.10M / 9.44M).

LoREALAttn-LoRA
CodeMix+5.0% / +6.4%+7.0% / +8.6%
Tülu 3+4.3% / +6.2%+5.5% / +7.3%
OMI2+13.0% / +17.1%+20.1% / +27.9%

Routing drift is also consistently smaller (0.05–0.10 vs. 0.08–0.14 of top-8 experts changed).

W5. Active-parameter mismatch and LoRA tuning. Expert LoRA activates only 8 of 64 adapters per token (≈1.18M of 9.44M). Higher-rank expert LoRA with matched active budgets should be evaluated.

The low active fraction is a weakness of expert LoRA itself. At the same total budget, it stores as many trainable parameters and optimizer states as LoREAL, but uses only one eighth of them per token. It also inherits the drawbacks of sparse MoE training. Its computation is fragmented across experts, and LoREAL trains 14% faster on H100 GPUs. Its adapters are subject to the same load imbalance as the experts.

We also added expert LoRA with 8× higher rank to match the per-token active parameters of LoREAL at 9.44M. LoREAL remains on par (MBPP 28.2 vs. 29.1 and IFEval inst-strict 40.9 vs. 42.5) with 8× fewer trainable parameters.

W6. Inference overhead. Latency should be compared with the base model and merged expert LoRA.

An inference comparison is not necessary for our claims. Our focus is adaptation quality, knowledge retention, and training efficiency, not inference optimization. As noted above, LoREAL already trains 14% faster. Inference efficiency also depends on kernel fusion, batching, serving frameworks, and hardware. Without a specified deployment configuration, no general conclusion can be drawn.

W7. Limited model coverage. All experiments use OLMoE-1B-7B only.

OLMoE-1B-7B is a deliberate choice. It is a standard backbone in related work (RoMA, RA-MoE, PERFT). Evaluation on a single model family also has precedent in AoE, Routing-Free MoE, and ESFT. OLMoE is also one of the few MoE models with fully open pretraining data, which our drift analysis and forgetting evaluation require.

Q2. Why were the joint runs in Appendix C not completed? Please provide full-training comparisons with a router learning-rate search.

These runs were early exploratory checks. No claim in the paper depends on them, so we did not continue them. LoREAL is defined with a frozen router (Eq. 10), and no result in Tables 2–6 involves joint training. Joint training doubles the trainable parameters (2.10M to 4.19M) and falls outside our matched-budget protocol. Router tuning also has the largest forgetting in Table 2 (+3.03% vs. +0.60% for LoREAL at the same budget). We will remove the joint-training paragraph in Section 4.3 and Appendix C.

Reviewer_4SJZOpenReview ↗

Response to author‘s response

I thank authors for their additional clarification and experiments. Most of my concerns remain.

I still see substantial overlap between LatentMoE. I also understand that representation drift is the motivation for the proposed method but my concern is that it is not connected with why LoREAL could work.

Attention-LoRA is a direct comparable baseline, which offers anotherway to change the representations that reach the experts. The comparasion with attn-lora does not show clear trend. Meanwhile. higher-rank expert LoRA scores is also better then the proposed method under matcher active parameters. Could you clarify why LoREAL’s MBPP score at 9.44M changed from 20.0 in Table 3 to 28.2 in the rebuttal?

The rason for choosing OLMoE is clear, but generalization to other MoE families remains untested, and inference overhead relative to merged expert LoRA remains unmeasured.

Based on the aboved reasons I would like to keep my score.

We thank the reviewer for the follow-up comments and for acknowledging our additional experiments.

W1. Distinction from LatentMoE and LoREAL’s Specific Contribution

LatentMoE is designed to improve pretraining efficiency through architectural redesign, where the router and experts operate in different representation dimensions, requiring the model to be pretrained from scratch. In contrast, LoREAL focuses on post-training adaptation of existing pretrained MoE models, with the primary goal of maximizing the preservation and reuse of pretrained knowledge. Therefore, we deliberately retain the original representation dimensionality, freeze the pretrained routers and experts, and adapt only the expert-input pathway. These are fundamentally different research objectives and settings.

Our claim is not to be the first to introduce dual-path separation, but to achieve parameter-efficient adaptation of existing pretrained MoE models by separating the routing and expert-input pathways while preserving pretrained knowledge.

W2. Connection Between Representation Drift and LoREAL’s Effectiveness

Representation drift motivates our low-rank design, while LoREAL’s effectiveness is empirically validated through extensive downstream experiments and ablation studies. In particular, Table 5 shows that even linear expert-input adaptation outperforms substantially larger routing adapters, while Table 6 demonstrates that adapting expert inputs alone achieves nearly the same performance as adapting both routing and expert inputs. These results support the effectiveness of expert-input adaptation independently of whether the learned correction reverses the observed drift.

W3. Comparison with Attention-LoRA

Attention-LoRA operates on the attention sublayer, outside the MoE architecture, and is therefore orthogonal to our work and not our main concern. rather than a competing approach. Moreover, our additional experiments show that LoREAL consistently achieves lower forgetting and routing drift than Attention-LoRA, further supporting our objective of preserving pretrained knowledge.

W4. Comparison with Higher-Rank Expert LoRA

Expert LoRA introduces practical challenges in MoE adaptation, including fragmented computation across sparsely activated experts, load-imbalanced adapter updates, and substantial optimizer-state overhead. LoREAL is not designed merely to outperform Expert LoRA, but to provide a more efficient adaptation mechanism that preserves and reuses pretrained expert knowledge. By sharing a single low-rank adapter across experts, LoREAL avoids expert-specific parameter updates and achieves comparable downstream performance with 8× fewer total trainable parameters under matched active-parameter budgets, while reducing training time by 14% on H100 GPUs.

W5. Clarification of MBPP Results

We apologize for the inconsistency. The MBPP score of 20.0 in Table 3 resulted from an error during experimental result collection. The correct score is 28.2, as reported in our rebuttal. We will correct this value in the revised manuscript.

W6. Generalization to Other MoE Families

We conducted additional CodeMix SFT experiments on Granite 3.0 MoE (3b-a800m-base; 32 layers, 40 experts, top-8), with all methods matched at 7.86M trainable parameters. LoREAL achieves the best performance on both HumanEval (38.4) and MBPP (42.8), outperforming Attention-LoRA by 0.6/2.6 points and Expert LoRA by 3.0/1.0 points, respectively. LoREAL also achieves lower forgetting (3.65% vs. 4.78%) and routing drift (3.8% vs. 4.9%) than Attention-LoRA.

W7. Inference overhead

Our work focuses on parameter-efficient downstream adaptation rather than inference optimization. LoREAL already demonstrates a 14% reduction in training time on H100 GPUs. Moreover, the added low-rank adapter accounts for only a tiny fraction of the model’s total parameters, suggesting limited additional inference overhead.

My main concern remains the comparison with expert LoRA at matched active parameters.

LoREAL does not achieve performance advanbtage compared to expert lora, and at inference time, it could be also slower owing to the additional sequential computation introduced, whereas expert LoRA can be merged into the main weight. And this latency cost remains unmeasured.

I also see substantial conceptual overlap with LatentMoE in the separation of router and expert inputs. Given these concerns, I would like to my score at 2.

Reviewer_HKcmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LoREAL is a PEFT method for MoE LLMs in which one LoRA per layer is shared by all 64 experts and applied to each expert's input rather than to its weights, with a frozen router and an optional per-expert scalar gaini. The motivation is a drift analysis projecting adaptation-induced input shifts onto the leading principal axes of the frozen expert-input covariance. Evaluation is continual pretraining on a 1B-token corpus and instruction tuning on OMI2 and Tulu3 with OLMoE-1B-7B, against frozen, router-tuning and expert-LoRA baselines at matched trainable-parameter budgets, with placement and gating-nonlinearity studies.

Critical strengths
  1. The paper is well-written and easy to follow. The method of this paper is very clearly presented with words easy to follow and figures easy to understand.
  2. The nonlinearity and placement studies separate the mechanism. Removing SiLU costs the routing residual 0.2094 PPL at rank=64 and 0.3522 at rank=1024, but the expert-input adapter only 0.0649, so the residual's benefit is substantially a reparameterisation while the adapter's is not.
  3. The parameter does not scale with the number of experts. One shared adapter and 64 scalars per layer mean the cost is set by depth and rank, not by expert count. In addition, the smallest expert-LoRA configuration in this paper is only 9.44M.
Critical weaknesses
  1. The analysis is reported as computed on the wrong matrix. Appendix B suggests that the reported axes were computed from M, not D and substituting D requires recomputing the axes and all dependent measurements. This makes it quite strange and makes me wonder if it is an auto-research work.
  2. The matched-budget comparison does not show advantage. Against expert LoRA, LoREAL differs by 0.0050 PPL at 9.44M, 0.0085 at 18.87M and 0.0054 at 37.75M with the differences of a few thousandths. The comparison does not show advantage, while the one large effect is the 0.53 PPL over router tuning at 2.10M. In addition, when considering SFT, MBPP falls from 24.0 at rank=32 to 20.0 at rank=144, below expert LoRA at rank. A shared adapter should not lose performance as capacity increases.
  3. No ablation study shows that the low-rank matrix is needed rather than a shared additive offset. The adapter adds BA times h to every expert's input. Since the drift argument concerns a shift shared across experts, a per-layer bias h + b should also be tested to show what the real contribution is, and no bias-only variant is reported.
Questions for authors
  1. If the axes are recomputed from D, which qualitative claims still make sence?
  2. Is the MBPP drop from 24.0 (rank=32) to 20.0 (ranks=144) a capacity effect?
  3. The paper notes that freezing router weights does not guarantee unchanged expert loads. Do you measure expert load or routing entropy across depth, and does load balance drift after the first adapted layer?
  4. The adapter is per-layer but shared across the 64 experts in a layer. Is the drift subspace itself shared across layers?
4 comments

We thank the reviewer for the feedback.

W1. The analysis is computed on the wrong matrix. Appendix B says the axes come from M, not D.

Q1. If the axes are recomputed from D, which qualitative claims still hold?

The conclusions do not depend on this choice. M and D differ only by I−C0I - C_0. This term comes solely from the whitening regularization, and its eigenvalues ϵ/(λ+ϵ)\epsilon/(\lambda+\epsilon) are bounded by 1. The matrix only selects the axes. The spread ratio along each axis is then measured directly on held-out data. Appendix B states the use of M explicitly. Recomputing the axes and spread ratios from D gives similar values.

Spread ratio (D)Axis 1Axis 8Axis 32
Math181–21926–498.8–15
Math SFT112–20519–327.3–9.1
FLAN158–58416–435.6–15
Code129–20017–317.1–11
peS2o48–1579–194.2–7.1

The table lists five of the seven domains, and the complete D-based results will be added to the appendix. At layer 8 on Math SFT, the ratios at axes 1, 4, 8, 16, and 32 are 145×, 34×, 21×, 13×, and 7.6× with D, against 129×, 34×, 22×, 14×, and 9× with M. The leading axes still carry far larger spread ratios than later axes, so the shift remains concentrated along a few directions. The coverage result in Section 3.1 uses neither M nor D.

W2. The matched-budget comparison with expert LoRA shows no advantage, and MBPP falls from 24.0 (r=32) to 20.0 (r=144).

Q2. Is the MBPP drop a capacity effect?

Our primary comparison is with router tuning at the same 2.10M budget. LoREAL is 0.53 PPL better with five times less forgetting (0.60% vs. 3.03%). Against expert LoRA in CPT, we claim comparable perplexity, and LoREAL is lower at all three budgets. The advantage over expert LoRA lies in SFT and efficiency. In Table 3, LoREAL is better on eight of nine SFT metrics (e.g., +4.6 GSM8K and +3.6 HumanEval). LoREAL also trains 14% faster on H100 GPUs, partly because expert LoRA fragments its trainable parameters into small GEMMs.

The MBPP drop is not a capacity effect. The r=144 CodeMix entry in Table 3 is a typo made when collecting the results. The correct values are 28.2 MBPP and 21.7 HumanEval, under the same protocol as the other SFT runs. This is above r=32 (24.0) and expert LoRA (21.2). MBPP therefore increases with rank, and LoREAL is better on all nine metrics. We will correct Table 3 and the related text in Section 4.2.

W3. No ablation shows that the low-rank matrix is needed rather than a shared additive offset. A per-layer bias h + b should be tested.

We ran this ablation. Each expert e receives h+beh + b_e, with beb_e initialized to zero and the router input unchanged. This per-expert bias has 2.10M trainable parameters, the same as LoREAL r=32. It contains the shared per-layer bias as a special case (be=bb_e = b). All numbers below come from new runs with matched settings for all methods.

The bias is worse on every metric for all three SFT corpora. Its validation PPL is higher (2.1974 vs. 2.0967 on CodeMix, 2.5435 vs. 2.4498 on Tülu 3, and 1.8943 vs. 1.7505 on OMI2). Its forgetting is larger (+6.5% vs. +5.0%, +5.2% vs. +4.3%, and +19.1% vs. +13.0%). Its task accuracy is lower on every benchmark, by 0.6 to 9.4 points.

A bias applies one fixed shift to all tokens. The LoREAL correction depends on the token representation. The gain therefore comes from the input-dependent low-rank map, not from an additive offset. In the same runs, even the bias matches or outperforms router tuning on every metric (e.g., forgetting +19.1% vs. +26.8% on OMI2), which further supports adapting expert inputs. We will add this ablation to the paper.

Q3. Do you measure expert load or routing entropy across depth, and does load balance drift after the first adapted layer?

Yes. We measured per-layer expert load before and after training on identical held-out tokens (four domains, 2.1M tokens) for the CPT LoREAL r=144 checkpoint. Routing in the first adapted layer is unchanged by construction, since its router reads the original input. In later layers, load changes only slightly and does not accumulate with depth. The share of top-8 assignments that move between experts is about 1.0% in layers 1–3 and 0.5–0.8% in layers 4–15 (mean 0.65%). No expert's load share changes by more than 0.3 percentage points. Pretrained load balance is well preserved.

Q4. The adapter is per-layer but shared across experts. Is the drift subspace shared across layers?

LoREAL does not require this. Each layer has its own adapter and learns its own correction subspace. Our analysis estimates drift directions independently for each layer, expert, and corpus, and makes no claim of cross-layer sharing.

We would like to kindly ask whether our response has adequately addressed your concerns. If any questions remain, we would be happy to provide further clarification.

Thanks for the responses which address some of my concerns and misunderstanding. However, the amount of revisions seem to be major, and there seems no mechanism that we can judge the final form of the paper. I would like to maintain my score.

Thanks for the reply. You mentioned that the paper requires substantial revisions. However, we would like to clarify that the additional D-based analysis (W1) and bias ablation (W3) will only be added to the appendix. The only modification to the main text is the correction of a data collection error in Table 3 and the corresponding description in Section 4.2.

We have already explicitly identified this error, provided the corrected values, and committed to correcting it in our previous response. This is a straightforward reporting error with no ambiguity.

Since you acknowledged that some of your concerns have been addressed, could you please clarify which specific concerns remain unresolved?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.