acceptodds
Withdrawn from ICLR 2027

EAD-LoRA: Expert-Aware Differential Initialization for MoE-LoRA Frameworks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient fine-tuning (PEFT), particularly Low-Rank Adaptation (LoRA), has become essential for adapting large foundation models to downstream tasks. To enhance representation capabilities, Mixture-of-Experts (MoE) architectures are often combined with LoRA (MoE-LoRA) by employing multiple low-rank adapters. However, existing MoE-LoRA methods uniformly initialize all experts using random distributions, ignoring the intrinsic structure of pretrained models and the anticipated functional divergence among experts. We propose EAD-LoRA, Expert-Aware Differential Initialization for MoE-LoRA frameworks. Unlike uniform initialization, our method uses a brief warm-up phase to collect routing data, then applies Principal Component Analysis (PCA) to initialize each expert's parameters based on its specific input distribution. Extensive experiments demonstrate that EAD-LoRA converges faster and achieves an average accuracy improvement of 1.6% and 5.8% over standard MoE-LoRA on different backbones, while its hybrid variant EAD-LoRA+ brings up to 2.7% and 7.7% improvements.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts LoRA, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e8UYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EAD-LoRA, an expert-aware initialization method for MoE-LoRA frameworks. Instead of randomly initializing all LoRA experts, the method collects expert-specific hidden states during a short forward-only warm-up phase and applies PCA to initialize each expert's low-rank matrices. The authors further introduce EAD-LoRA+, which combines PCA-based and random initialization to balance data-driven specialization and exploration. Experiments on five question-answering benchmarks using Llama-3.2-3B and Llama-2-7B demonstrate improved accuracy and faster convergence compared to existing MoE-LoRA initialization methods.

Critical strengths
  1. Simple and well-motivated method. The proposed expert-aware initialization is intuitive and easy to integrate into existing MoE-LoRA frameworks, without introducing additional inference overhead.

  2. Strong empirical results. The method consistently improves accuracy over several competitive baselines across different backbones and benchmarks. The paper also provides extensive ablation and sensitivity studies to support its effectiveness.

Critical weaknesses
  1. Insufficient isolation of the expert-aware contribution. The ablation studies demonstrate the benefits of PCA-based initialization, but do not establish whether expert-specific initialization is necessary. A comparison with global PCA initialization would better isolate this contribution.

  2. Unclear effectiveness of warm-up routing. Since the router is randomly initialized and remains frozen during warm-up, it is unclear whether the resulting token assignments provide meaningful expert-specific information. To validate this design, it will be good to conduct a comparison with random token-to-expert assignments.

  3. Lack of comparison between PCA and SVD. Appendix L reports a substantial performance gap between PCA and SVD applied to collected activations. Since both methods yield equivalent principal subspaces under consistent preprocessing, the source of this gap requires further clarification.

Questions for authors
  1. Equation (10) normalizes the proposed initialization using the Frobenius norm, which may introduce scale differences compared to the baselines. Could you show whether the improvements persist when all methods use the same initialization norm?

  2. Section 3.4 states that hidden states are collected based on expert routing, while Algorithm 1 selects tokens using gate-score quantiles. Since this does not necessarily correspond to top-k routing assignments, could the authors clarify how tokens are assigned to experts during collection and how the sampling budgets are determined?

Reviewer_ZXYkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EAD-LoRA, an expert-specific initialization method for MoE-LoRA. It first collects hidden states according to expert routing assignments during a short warm-up phase, then applies PCA to initialize each expert's LoRA down-projection matrix while keeping the up-projection matrix zero-initialized. A hybrid variant, EAD-LoRA+, combines PCA components with random initialization. Experiments on two Llama backbones and five QA benchmarks report improved accuracy and convergence compared with several initialization baselines.

Critical strengths
  1. The method is lightweight and potentially easy to integrate into existing MoE-LoRA frameworks. Using expert-specific activation statistics to initialize the down-projection matrices is a reasonable idea. Keeping (B_i=0) also preserves the original pretrained model output at initialization. Importantly, the method does not introduce additional parameters or inference-time computation after initialization.

  2. The paper reports encouraging empirical improvements over several initialization baselines. EAD-LoRA+ achieves 63.7% average accuracy on Llama-3.2-3B, compared with 61.6% for PiSSA, and 67.8% on Llama-2-7B, compared with 64.2% for GOAT. The paper also investigates PCA strength, warm-up duration, and expert configurations. These results suggest that activation-informed initialization is worth studying.

Critical weaknesses
  1. The claimed expert-aware specialization is not adequately established by the warm-up procedure. Section 3.4 states that the 20-step warm-up uses forward passes without gradient updates. This means that the router is not actually learning task-specific expert assignments during this stage. The collected activation subsets are therefore determined by the initial routing mechanism rather than learned expert specialization. It is not clear whether per-expert PCA provides an advantage over global PCA, random routing partitions, or shuffled expert assignments. The expert similarity analysis in Table 10 does not resolve this concern because parameter similarity alone does not demonstrate functional specialization. Additionally, the 20 warm-up steps are counted within the 500-step budget but perform no optimization updates, whereas the baselines perform ordinary training throughout. The claimed equality of optimization steps needs clarification.

  2. Several mathematical and experimental details are inconsistent with the proposed initialization explanation. The most concerning example is Table 12, where PCA initialization achieves 66.7% on ScienceQA but SVD of the collected activations achieves only 62.5%. PCA on centered activation data is mathematically equivalent to computing the right singular vectors of the centered data matrix. Such a large gap therefore requires a precise explanation of differences in preprocessing or implementation. Equation 10 is also questionable: it normalizes the entire initialization matrix to a Frobenius norm of (\sqrt{2/d_{\mathrm{in}}}), whereas a rank-r matrix initialized elementwise with Kaiming variance (2/d_{\mathrm{in}}) has an expected squared Frobenius norm of approximately (2r). These are not equivalent scales. Furthermore, the gradient covariance relation in Section 3.5 compares matrices with generally different dimensions; the input-side covariance would involve (\nabla_W L^T\nabla_W L), not (\nabla_W L\nabla_W L^T). Together, these issues weaken the theoretical motivation and make it difficult to verify which implementation was actually evaluated.

Questions for authors
  1. During the 20-step warm-up, are any router parameters updated? If not, how does the method obtain learned expert-specific distributions rather than partitions induced by the initial router? Please provide a comparison with global PCA shared by all experts and PCA based on randomized or shuffled routing assignments.

  2. Could the authors reconcile the PCA-versus-SVD results in Table 12 and clarify the exact initialization normalization used in the implementation? In particular, were PCA and SVD applied to identically centered and normalized activations, and does the code implement Equation 10 as written? Clarifying the gradient covariance argument would also be important.

Reviewer_vqLWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose an initialization scheme for MoE-LoRA. A 20 step forward only warm up collect hidden states of tokens with high gate values.

On five QA benchmarks with Llama-3.2-3B and Llama-2-7B, the paper reports average gains over Kaiming-initialized MoE-LoRA of 1.6/5.8 pp (EAD-LoRA) and 2.7/7.7 pp (EAD-LoRA+), together with faster loss decrease

Critical strengths

This paper provides a cheap initialization with consistent gains over standard MoE-LoRA initialization. EAD-LoRA+ beats both random-init MoE-LoRA baselines in 19 of 20 comparisons, tying the 20th (Llama-2-7B SIQA vs. Kaiming). On Llama-3.2-3B its five-seed range of average accuracy wins in all five expert/top-k configurations tested. Wall-clock cost is slightly higher than the baseline and keeping Bi=0B_i=0 preserves the pre-trained function at step 0.

Critical weaknesses

What drives the gain is not isolated. As a result, the contribution over existing activation-based LoRA initializations is unclear. EAD-LoRA changes the directions of each expert's A, their scale and the data used to compute them, all at once. Eq. (10) rescales each A to a F norm of sqrt(2/d), so its entries are about 100 times smaller than under the Kaiming law of Eq. (5) that the paper says it follows. Scale clearly matters here: the Kaiming and Gaussian baselines, whose scales differ by only a factor of sqrt(2), differ by 7.1 points on Llama-2-7B CSQA. The expert distance of 0.0310 in Table 10 follows from Eq. (10) alone: two matrices with equal norm s and cosine c are s x sqrt(2(1 - c)) apart, which gives 0.032 at the reported cosine of 0.2276, so it is not evidence of specialization. No run uses normalized random A (alpha = 0 in Eq. 9) or a single PCA basis shared by all experts, and EVA and CorDA are neither cited nor compared.

Questions for authors

On Llama-3.2-3B (five tasks, ≥3 seeds), please add three small controls: (a) α=0\alpha=0 with the Eq. (10) normalization, i.e., random directions at EAD-LoRA's scale; (b) one PCA basis computed from all warm-up tokens and shared by every expert; (c) an existing activation-based initialization (EVA or CorDA) applied to each expert. If EAD-LoRA clearly beats (a) and (b) beyond seed variation, I would consider the expert-aware claim supported and raise my rating; if (b) matches it, the contribution reduces to known activation-based initialization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.