- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LoREAL proposes parameter-efficient adaptation of pretrained MoE language models by modifying the representations delivered to frozen experts. A shared low-rank nonlinear residual adapter transforms expert inputs, while the local frozen router continues to operate on the representation before that adapter. The adapter’s parameter count is independent of the number of experts, apart from optional expert-specific scaling parameters. Experiments on OLMoE cover continued pretraining and supervised fine-tuning for mathematics, instruction following, and code, with comparisons against router tuning and expert-side LoRA.
Critical strengths
The proposed architecture is simple and potentially useful. Sharing an input adapter across experts avoids multiplying the adaptation budget by the expert count, and separating the local router and expert-input pathways gives a clear architectural hypothesis to investigate. The paper correctly acknowledges that freezing a router does not guarantee unchanged routing throughout the network, because earlier adapted layers can change later router inputs.
The evaluation covers several adaptation settings rather than a single benchmark. The improvements over ordinary router tuning are substantial, and the comparisons against expert LoRA use matched effective parameter budgets. I also appreciate the authors’ transparency about single-run results, incomplete ablations, and limitations of the representation-shift analysis, although some of those limitations need to be reflected more consistently in the main claims.
Critical weaknesses
-
The relationship to representation-based adaptation is insufficiently developed. Please add and discuss ReFT: Representation Finetuning for Language Models (NeurIPS 2024) and Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, which introduces (IA)³ (NeurIPS 2022). Both are relevant to adapting frozen models through their representations or activations. The discussion should explain what is contributed specifically by sharing the adapter across experts and placing it on the expert-input pathway. The broader adaptation discussion should also include Mixture of LoRA Experts (ICLR 2024), HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (NeurIPS 2024), and LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts (ICLR 2026). I am not requesting experiments with every LoRA-expert method.
-
The experimental comparisons do not yet fully test the central architectural argument. I would prioritize an experimental comparison with Routing Manifold Alignment (RoMA), which is already cited but is not included among the evaluated baselines. Since the paper argues for expert-input adaptation as an alternative to adapting routing, a regularized router-adaptation method is a more informative comparator than ordinary router tuning alone. A representative comparison with LoReFT or another established representation-adaptation method would also help distinguish the proposed pathway separation from the more general benefit of transforming hidden representations. The paper already includes the important “adapter to both pathways” control. However, this variant obtains slightly lower perplexity than expert-only adaptation in Table 6, and forgetting is not reported for that comparison. Please complete this comparison with retention and routing-change measurements rather than treating expert-only placement as already established to be preferable. The joint router–adapter experiments likewise stop well before the planned training budget and cannot establish the final benefit of freezing the router.
-
The low-rank distribution-shift claim is not established by the current analysis. This is my main concern. The introduction presents the downstream shift as concentrated in a small number of directions. However, Appendix B states that the reported axes were computed relative to the identity matrix even though the transformed reference covariance is not the identity. It further explains that the reference-corrected matrix requires recomputing the axes and dependent measurements. The appendix also identifies non-independent controls and unexpectedly large reference-control ratios, and explicitly cautions against interpreting the reported axis counts as the intrinsic rank of the shift. Please either recompute this analysis with the appropriate reference covariance and document-disjoint controls, or narrow the main-text claim and present the method primarily as an empirically effective architecture. The adaptation results can remain valuable without claiming that a low-rank distribution shift has been demonstrated.
Questions for authors
Does expert-only adaptation retain an advantage over adapting both pathways when comparing forgetting, routing changes, and repeated runs?
And also see weakness
2 comments
Thanks, I increase the score from 3 to 4, good luck
We thank the reviewer for the constructive feedback. We address the three weaknesses and provide additional experimental results below.
We thank the reviewer for highlighting these studies and will expand the related work accordingly. While ReFT and (IA)³ adapt frozen models through representations or activations, LoREAL specifically separates routing and expert-input pathways in pretrained MoE models. The router operates on the original hidden states, while a shared low-rank adapter transforms only expert inputs.
Unlike MoE-LoRA approaches employing multiple adaptation modules, LoREAL shares one adapter across all experts per layer, making its parameter count independent of the number of experts and enabling a larger rank under the same budget. Our contribution is thus the shared, expert-input-specific adaptation architecture, rather than low-rank adaptation itself.
RoMA. We attempted multiple implementations of RoMA, but the necessary implementation details are not publicly available, preventing a reliable comparison. Beyond standard router tuning, we evaluated nonlinear routing adapters with up to 33× more trainable parameters, which still underperform LoREAL.
LoReFT. LoREAL differs from general representation adaptation by explicitly separating routing and expert-input pathways. Our insertion-position ablation directly examines this distinction by comparing expert-only adaptation with adapting both pathways.
Expert-only vs. Both Pathways. We reran the comparison under matched settings and additionally measured forgetting and top-8 routing drift.
Expert-only achieves slightly lower PPL and identical forgetting, with 26.4% lower routing drift, supporting our design choice of excluding the adapter from the local routing pathway.
Joint training. We clarify that the incomplete joint router–adapter experiments are exploratory rather than evidence of final convergence. Our main conclusions rely on completed router-tuning and nonlinear-router comparisons.
M vs. D. We recomputed the drift axes using the reference-corrected matrix D=M+I−C0. At layer 8 on Math SFT, spread ratios at axes 1, 8, and 32 are 145×, 21×, and 7.6× using D, compared with 129×, 22×, and 9× using M. Similar qualitative patterns hold across domains.
Data independence. Although the original splitting strategy may allow limited document-level overlap, its actual extent was minimal. Additional experiments using strictly document-disjoint splits yielded consistent results.
Reference controls. Reference-control ratios serve as empirical baselines for relative distributional differences rather than exact unit-ratio expectations. The larger downstream ratios along leading directions indicate pronounced directional discrepancies.
Low-rank interpretation. Our D-based analysis consistently identifies larger discrepancies along a small number of leading directions. We clarify that these observations motivate low-rank adaptation without establishing the intrinsic rank of the shift. Importantly, LoREAL learns its adaptation subspace from the task objective, independently of the diagnostic drift axes.
Our updated results show slightly lower PPL (6.3634 vs. 6.3661), identical forgetting (+0.46%), and 26.4% lower top-8 routing drift (0.0331 vs. 0.0450) for expert-only adaptation. These results support its routing-preservation advantage, although repeated-run statistical significance has not yet been established.