- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OD-MoLE to address expert redundancy and weak specialization in LoRA-based Mixture-of-Experts models. The method introduces two regularization terms: output orthogonality to reduce similarity between experts and shared-projection diversity to preserve informative feature directions. The authors provide theoretical analyses to motivate these designs and evaluate the method on eight commonsense reasoning benchmarks using Llama-2-7B and Qwen2.5-7B. The experimental results show modest improvements over existing LoRA-MoE methods.
Critical strengths
- The paper addresses a relevant problem in LoRA-based MoE models, particularly expert redundancy and insufficient specialization.
- The proposed method is relatively simple and combines two complementary regularization objectives. The theoretical analysis in Section 4 provides useful motivation for the design.
- The experiments include comparisons with several existing methods, different routing strategies, ablation studies, and computational cost analysis.
Critical weaknesses
- Theorem 2 requires scale constraints on the shared projection matrix, but Section 5.2 does not enforce them. The authors should normalize matrix A and verify its conditioning.
- Theorem 1 considers a quadratic residual objective, while training uses cross-entropy loss. The authors should clarify this gap and evaluate whether orthogonality improves routing effectiveness.
- Figure 1 and Appendix Figures 2–3 mainly show output separation, providing limited evidence of expert specialization. The authors should analyze routing distributions and expert contributions through ablation.
- Tables 1–2 show small improvements (0.25–0.52 percentage points) with only three seeds. The authors should provide significance tests and more comprehensive regularizer ablations.
- Table 4 shows increased training time for modest accuracy gains. The authors should report inference latency and evaluate the method on more diverse downstream tasks.
Questions for authors
see weeknesses
8 comments
Table Q1.3.2 Norm and singular-value spectrum of A evolve during training (PIQA)
| Setting | Epoch | (mean / max) | Effective rank (mean / max) | Stable rank (mean / max) | Normalized median spectrum (mean / max) |
|---|---|---|---|---|---|
| (no ortho, no div) | 0 | 2.3089 / 2.3194 | 15.9649 / 15.9764 | 14.3419 / 14.6667 | 0.24989 / 0.26392 |
| (no ortho, no div) | 1 | 2.5999 / 2.8697 | 14.6758 / 15.6223 | 6.2983 / 10.2325 | 0.24532 / 0.39706 |
| (no ortho, no div) | 2 | 2.9368 / 3.5961 | 13.3540 / 14.7972 | 4.8435 / 7.9145 | 0.24103 / 0.44940 |
| (no ortho, no div) | 3 | 3.2760 / 4.2558 | 12.1875 / 13.9444 | 4.2116 / 6.5360 | 0.23542 / 0.47673 |
| (no ortho, div) | 0 | 2.3089 / 2.3194 | 15.9649 / 15.9764 | 14.3419 / 14.6667 | 0.24989 / 0.26392 |
| (no ortho, div) | 1 | 2.6088 / 2.8766 | 14.7700 / 15.6293 | 6.4475 / 9.7441 | 0.24589 / 0.40120 |
| (no ortho, div) | 2 | 2.9229 / 3.5557 | 13.7568 / 15.0367 | 5.1794 / 7.9312 | 0.24256 / 0.46390 |
| (no ortho, div) | 3 | 3.3494 / 4.3883 | 12.9154 / 14.4494 | 4.6261 / 7.4242 | 0.23907 / 0.49564 |
Table Q1.3.3 Condition number of during training (PIQA)
| Task | Setting | λ_div | Epoch | Median κ₂(A) | p90 κ₂(A) |
|---|---|---|---|---|---|
| PIQA | (no ortho, no div) | 0 | 0 | 1.1196 | 1.1325 |
| PIQA | (no ortho, no div) | 0 | 1 | 1.8845 | 2.2502 |
| PIQA | (no ortho, no div) | 0 | 2 | 2.3881 | 2.7517 |
| PIQA | (no ortho, no div) | 0 | 3 | 2.8333 | 3.1920 |
| PIQA | (no ortho, div) | 0.001 | 0 | 1.1196 | 1.1325 |
| PIQA | (no ortho, div) | 0.001 | 1 | 1.8560 | 2.2313 |
| PIQA | (no ortho, div) | 0.001 | 2 | 2.2328 | 2.5952 |
| PIQA | (no ortho, div) | 0.001 | 3 | 2.5447 | 2.8726 |
Accuracy with the diversity penalty alone. For completeness, we also ran the raw diversity penalty without output orthogonality under selective routing over three seeds. Table Q1.3.4 reports accuracy as mean ± sample SD.
Table Q1.3.4 Diversity penalty alone vs. no regularizer
| Setting | ComQA | CosQA | SIQA | PIQA | ScQA | BoolQ | ARC-C | OBQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| (no ortho, div) | 84.63 ± 0.54 | 87.16 ± 0.54 | 79.87 ± 0.39 | 89.63 ± 0.77 | 92.21 ± 0.54 | 87.68 ± 0.38 | 88.60 ± 0.18 | 95.33 ± 0.23 | 88.14 ± 0.10 |
| (no ortho, no div) | 84.58 ± 0.21 | 85.72 ± 1.70 | 79.38 ± 0.14 | 89.28 ± 0.76 | 91.75 ± 0.52 | 87.35 ± 0.26 | 88.25 ± 0.42 | 95.20 ± 0.20 | 87.69 ± 0.28 |
In Table Q1.3.5, each seed cell shows (no ortho, no div) / (no ortho, div) accuracy. is (no ortho, div) minus (no ortho, no div) in percentage points, with paired 95% confidence intervals across seeds 40–42.
Table Q1.3.5 Paired per-seed comparison
| Task | Paired seeds | Seed 40 | Seed 41 | Seed 42 | Mean Δ (pp) | 95% CI (pp) |
|---|---|---|---|---|---|---|
| ComQA | 3 | 84.77 / 85.26 | 84.60 / 84.28 | 84.36 / 84.36 | +0.057 | [−0.957, +1.070] |
| CosQA | 3 | 87.67 / 87.14 | 84.57 / 86.63 | 84.93 / 87.71 | +1.437 | [−2.888, +5.761] |
| SIQA | 3 | 79.43 / 79.43 | 79.48 / 80.19 | 79.22 / 79.99 | +0.493 | [−0.571, +1.557] |
| PIQA | 3 | 89.72 / 89.99 | 88.41 / 90.15 | 89.72 / 88.74 | +0.343 | [−3.039, +3.725] |
| ScQA | 3 | 92.30 / 92.82 | 91.70 / 91.79 | 91.26 / 92.02 | +0.457 | [−0.387, +1.300] |
| BoolQ | 3 | 87.06 / 87.28 | 87.40 / 88.04 | 87.58 / 87.71 | +0.330 | [−0.346, +1.006] |
| ARC-C | 3 | 87.97 / 88.40 | 88.74 / 88.74 | 88.05 / 88.65 | +0.343 | [−0.425, +1.112] |
| OBQA | 3 | 95.20 / 95.60 | 95.00 / 95.20 | 95.40 / 95.20 | +0.133 | [−0.626, +0.892] |
| Eight-task macro-average | 3 | 88.02 / 88.24 | 87.49 / 88.13 | 87.57 / 88.05 | +0.449 | [−0.071, +0.970] |
The diversity penalty alone gives a higher macro-average in all three seeds and a positive mean difference on all eight tasks. With three seeds, the interval includes zero, so we treat this as supporting evidence and rely on the conditioning statistics above for the claim that the penalty behaves as Theorem 2 describes.
We have added the arbitrary-scale form as Corollary 5 with its proof in Appendix A.6, and a pointer after Theorem 2 stating that the raw penalty is used as a practical surrogate. The final-epoch conditioning statistics are in Table 5, and the per-epoch spectrum, condition number and normalized-penalty results are in Appendix B.12 Tables 20, 21 and 23. The diversity-only ablation is in Table 3 and Appendix B.10 Table 14.
Q2 Explain loss in Theorem 1, and does orthogonality improve routing effectiveness?
Thank you for pointing out this distinction. In Theorem 1, Equation (7) bounds how much the hidden-layer mixture changes when the routing weights change, with the input and expert outputs held fixed. This geometric bound is independent of the downstream loss. While Equation (8) serves as an example, it establishes recovery of the optimal routing weights in a specific case when $ Q_h$ is the quadratic residual objective. We have clarified this in the paragraph after Theorem 1. We therefore test the predicted change in routing distinguishability directly on models trained with cross-entropy.
We compare two selective routing settings with the diversity penalty disabled (). The (no ortho, no div) uses , and the (ortho, no div) uses . For each evaluated token and attention projection, we hold the two selected experts fixed and transfer routing weight from the higher-weight expert to its partner. We measure , where is the change in routing weights and is the resulting change in the expert mixture. A larger means the same weight perturbation produces a larger change in the mixture output. Table Q2.1 shows that adding orthogonality increases median sensitivity on all eight tasks, and Table Q2.2 shows that the fifth percentile increases on five tasks. This supports improved typical routing distinguishability for the evaluated weight perturbation, with mixed effects in the lower tail.
We have added routing distinguishability in Table 4 in the manuscript, and the fifth percentile is in Appendix B.11 Table 16. The orthogonality only setting is a row in Table 3, and its per-seed results are in Appendix B.10 Table 13. We have added a paragraph after Theorem 1 that separates the geometric bound from the recovery guarantee and states that the latter applies only to .
Q2.1 Median routing sensitivity
| Setting | ComQA | CosQA | SIQA | PIQA | ScQA | BoolQ | ARC-C | OBQA |
|---|---|---|---|---|---|---|---|---|
| (no ortho, no div) | 0.6095 | 3.4440 | 1.7579 | 2.5784 | 3.2301 | 1.3927 | 1.3955 | 1.7856 |
| (ortho, no div) | 0.6730 | 3.5596 | 1.8020 | 2.7048 | 3.2461 | 1.4625 | 1.4118 | 1.8049 |
Q2.2 Fifth percentile of routing sensitivity
| Setting | ComQA | CosQA | SIQA | PIQA | ScQA | BoolQ | ARC-C | OBQA |
|---|---|---|---|---|---|---|---|---|
| (no ortho, no div) | 0.1763 | 0.8533 | 0.4077 | 0.6728 | 0.8323 | 0.3456 | 0.3094 | 0.1972 |
| (ortho, no div) | 0.1858 | 0.8431 | 0.4146 | 0.6768 | 0.7176 | 0.3281 | 0.3188 | 0.2596 |
Q2.3 Effectiveness of orthogonal loss over three random seeds.
| Setting | ComQA | CosQA | SIQA | PIQA | ScQA | BoolQ | ARC-C | OBQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| (no ortho, no div) | 84.58 ± 0.21 | 85.72 ± 1.70 | 79.38 ± 0.14 | 89.28 ± 0.76 | 91.75 ± 0.52 | 87.35 ± 0.26 | 88.25 ± 0.42 | 95.20 ± 0.20 | 87.69 ± 0.28 |
| (ortho, no div) | 84.77 ± 0.64 | 86.65 ± 0.40 | 79.75 ± 0.30 | 89.59 ± 0.27 | 92.37 ± 0.03 | 88.12 ± 0.10 | 88.05 ± 0.23 | 96.27 ± 0.46 | 88.20 ± 0.10 |
| Difference (pp) | +0.19 ± 0.68 | +0.92 ± 1.96 | +0.37 ± 0.21 | +0.31 ± 0.77 | +0.61 ± 0.50 | +0.78 ± 0.35 | −0.20 ± 0.22 | +1.07 ± 0.50 | +0.51 ± 0.19 |
Each seed cell below shows (no ortho, no div) / (ortho, no div) accuracy. is (ortho, no div) minus (no ortho, no div), in percentage points.
| Task | Paired seeds | Seed 40 | Seed 41 | Seed 42 | Mean (pp) | 95% CI (pp) |
|---|---|---|---|---|---|---|
| ComQA | 3 | 84.77 / 85.18 | 84.60 / 84.03 | 84.36 / 85.09 | +0.190 | [−1.493, +1.873] |
| CosQA | 3 | 87.67 / 86.34 | 84.57 / 86.50 | 84.93 / 87.10 | +0.923 | [−3.933, +5.780] |
| SIQA | 3 | 79.43 / 79.63 | 79.48 / 80.09 | 79.22 / 79.53 | +0.373 | [−0.154, +0.900] |
| PIQA | 3 | 89.72 / 89.88 | 88.41 / 89.55 | 89.72 / 89.34 | +0.307 | [−1.607, +2.221] |
| ScQA | 3 | 92.30 / 92.40 | 91.70 / 92.35 | 91.26 / 92.35 | +0.613 | [−0.619, +1.846] |
| BoolQ | 3 | 87.06 / 88.23 | 87.40 / 88.04 | 87.58 / 88.10 | +0.777 | [−0.083, +1.636] |
| ARC-C | 3 | 87.97 / 87.97 | 88.74 / 88.31 | 88.05 / 87.88 | −0.200 | [−0.738, +0.338] |
| OBQA | 3 | 95.20 / 96.80 | 95.00 / 96.00 | 95.40 / 96.00 | +1.067 | [−0.184, +2.317] |
| Eight-task macro-average | 3 | 88.02 / 88.30 | 87.49 / 88.11 | 87.57 / 88.17 | +0.506 | [+0.038, +0.974] |
Q1 Scale normalization and conditioning of the shared projection A
Thank you for raising this. We agree that Theorem 2 is stated under the fixed scale tr(AA⊤)=r, while the penalty in Section 5.2 is applied to the raw matrix. Scale matters because uniformly scaling up A increases the raw sampled log-determinant while leaving the ratios between its singular values, including the condition number, unchanged. A larger raw log-determinant is therefore not by itself evidence of better conditioning. We address this in the following.
1. An arbitrary-scale form of Theorem 2. The trace condition is a normalization and can be removed. The same proof gives, for any full-row-rank A under the same input assumption,
λmin(KA)≥νdet(AA⊤)(∥A∥F2r−1)r−1=νr∥A∥F2e−ηˉ(rr−1)r−1,
where Aˉ=rA/∥A∥F and ηˉ=−logdet(AˉAˉ⊤)≥0. This reduces to Theorem 2 when ∥A∥F2=r and is Corollary 5 in Appendix A.6. The bound factors into a scale term ∥A∥F2/r and a shape term that depends only on the normalized matrix Aˉ. This makes your point explicit: the raw log-determinant can increase through either term, but only the shape term reflects how evenly A spreads its energy across directions, since a uniform rescaling of A can be absorbed by the expert matrices. Items 2 and 3 examine the shape term.
2. Normalizing A. We reran OD-MoLE with the log-determinant computed on Aˉ=max(∥A∥F,10−12)rA, where r is the number of rows of A. We normalize the full matrix first and then sample columns from Aˉ. This satisfies the scale condition of Theorem 2 by construction. Both variants use λdiv=0.001 and λorth=0.001.
The normalized variant reaches comparable average accuracy in both routing settings, so enforcing the scale condition does not change the empirical conclusions.
3. Verifying conditioning. To check that the raw penalty of Section 5.2 is not satisfied by rescaling alone, we track A during training under all-expert composition with output orthogonality disabled, comparing no regularizer (λdiv=0) with the raw penalty (λdiv=0.001). Each run is summarized over its 112 shared projection matrices at initialization and at the end of each epoch. Effective rank, stable rank, the normalized spectrum and the condition number κ2(A) are all invariant to uniform rescaling of A.
Scale. ∥A∥F grows during training in both runs and ends slightly higher with the raw penalty, so part of the penalty's effect is scale, as you anticipated. Shape. The scale-invariant statistics also improve. Effective and stable rank decline more slowly with the penalty on both SIQA and PIQA (Tables Q1.3.1 and Q1.3.2), and the median and p90 condition numbers on PIQA are lower after every training epoch, by about 10% at the end of training (Table Q1.3.3).
Table Q1.3.1 Norm and singular-value spectrum of A evolve during training (SIQA)