- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EAD-LoRA, an expert-aware initialization method for MoE-LoRA frameworks. Instead of randomly initializing all LoRA experts, the method collects expert-specific hidden states during a short forward-only warm-up phase and applies PCA to initialize each expert's low-rank matrices. The authors further introduce EAD-LoRA+, which combines PCA-based and random initialization to balance data-driven specialization and exploration. Experiments on five question-answering benchmarks using Llama-3.2-3B and Llama-2-7B demonstrate improved accuracy and faster convergence compared to existing MoE-LoRA initialization methods.
Critical strengths
-
Simple and well-motivated method. The proposed expert-aware initialization is intuitive and easy to integrate into existing MoE-LoRA frameworks, without introducing additional inference overhead.
-
Strong empirical results. The method consistently improves accuracy over several competitive baselines across different backbones and benchmarks. The paper also provides extensive ablation and sensitivity studies to support its effectiveness.
Critical weaknesses
-
Insufficient isolation of the expert-aware contribution. The ablation studies demonstrate the benefits of PCA-based initialization, but do not establish whether expert-specific initialization is necessary. A comparison with global PCA initialization would better isolate this contribution.
-
Unclear effectiveness of warm-up routing. Since the router is randomly initialized and remains frozen during warm-up, it is unclear whether the resulting token assignments provide meaningful expert-specific information. To validate this design, it will be good to conduct a comparison with random token-to-expert assignments.
-
Lack of comparison between PCA and SVD. Appendix L reports a substantial performance gap between PCA and SVD applied to collected activations. Since both methods yield equivalent principal subspaces under consistent preprocessing, the source of this gap requires further clarification.
Questions for authors
-
Equation (10) normalizes the proposed initialization using the Frobenius norm, which may introduce scale differences compared to the baselines. Could you show whether the improvements persist when all methods use the same initialization norm?
-
Section 3.4 states that hidden states are collected based on expert routing, while Algorithm 1 selects tokens using gate-score quantiles. Since this does not necessarily correspond to top-k routing assignments, could the authors clarify how tokens are assigned to experts during collection and how the sampling budgets are determined?