acceptodds
Under review as a conference paper at ICLR 2027

Collaboration through Diversity and Specialization via Orthogonality for Improved LoRA-based MoE

Abstract

Combining the Mixture-of-Experts (MoE) paradigm with Low-Rank Adaptation (LoRA) enables parameter-efficient adaptation of large language models (LLMs) to downstream tasks. However, existing LoRA-based MoE approaches often suffer from expert interference and weak specialization, where experts produce correlated outputs, leading to overlapping representations and reduced effective capacity. A key challenge is to balance collaboration and specialization: experts jointly capture task-specific knowledge while reducing redundancy and conflicts across experts. To address this challenge, we propose a principled framework that characterizes expert interactions in a data-dependent manner. We show that redundancy arises from correlated expert outputs and propose an explicit regularization objective that encourages orthogonality, promoting effective specialization and mitigating interference. Furthermore, built on an asymmetric LoRA framework with shared low-rank projections, we introduce a complementary regularization that encourages diversity in the shared representations. Our experiments across diverse benchmarks demonstrate that our method encourages representational specialization and improves task performance while maintaining the efficiency of parameter-efficient fine-tuning. Our code is available at https://anonymous.4open.science/r/OD_MoLE-A5F2.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Low-Rank Adaptation, Large Language Models, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PZFQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OD-MoLE to address expert redundancy and weak specialization in LoRA-based Mixture-of-Experts models. The method introduces two regularization terms: output orthogonality to reduce similarity between experts and shared-projection diversity to preserve informative feature directions. The authors provide theoretical analyses to motivate these designs and evaluate the method on eight commonsense reasoning benchmarks using Llama-2-7B and Qwen2.5-7B. The experimental results show modest improvements over existing LoRA-MoE methods.

Critical strengths
  1. The paper addresses a relevant problem in LoRA-based MoE models, particularly expert redundancy and insufficient specialization.
  2. The proposed method is relatively simple and combines two complementary regularization objectives. The theoretical analysis in Section 4 provides useful motivation for the design.
  3. The experiments include comparisons with several existing methods, different routing strategies, ablation studies, and computational cost analysis.
Critical weaknesses
  1. Theorem 2 requires scale constraints on the shared projection matrix, but Section 5.2 does not enforce them. The authors should normalize matrix A and verify its conditioning.
  2. Theorem 1 considers a quadratic residual objective, while training uses cross-entropy loss. The authors should clarify this gap and evaluate whether orthogonality improves routing effectiveness.
  3. Figure 1 and Appendix Figures 2–3 mainly show output separation, providing limited evidence of expert specialization. The authors should analyze routing distributions and expert contributions through ablation.
  4. Tables 1–2 show small improvements (0.25–0.52 percentage points) with only three seeds. The authors should provide significance tests and more comprehensive regularizer ablations.
  5. Table 4 shows increased training time for modest accuracy gains. The authors should report inference latency and evaluate the method on more diverse downstream tasks.
Questions for authors

see weeknesses

8 comments

Q1 Scale normalization and conditioning of the shared projection AA

Thank you for raising this. We agree that Theorem 2 is stated under the fixed scale tr(AA⊤)=r\mathrm{tr}(AA^\top)=r, while the penalty in Section 5.2 is applied to the raw matrix. Scale matters because uniformly scaling up AA increases the raw sampled log-determinant while leaving the ratios between its singular values, including the condition number, unchanged. A larger raw log-determinant is therefore not by itself evidence of better conditioning. We address this in the following.

1. An arbitrary-scale form of Theorem 2. The trace condition is a normalization and can be removed. The same proof gives, for any full-row-rank AA under the same input assumption,

λmin⁡(KA)≥νdet⁡(AA⊤)(r−1∥A∥F2)r−1=ν∥A∥F2re−ηˉ(r−1r)r−1,\lambda_{\min}(K_A) \ge \nu \det(AA^\top)\left(\frac{r-1}{\lVert A\rVert_{\mathrm{F}}^2}\right)^{r-1} = \nu \frac{\lVert A\rVert_{\mathrm{F}}^2}{r} e^{-\bar\eta}\left(\frac{r-1}{r}\right)^{r-1},

where Aˉ=rA/∥A∥F\bar{A}=\sqrt{r}A/\lVert A\rVert_{\mathrm{F}} and ηˉ=−log⁡det⁡(AˉAˉ⊤)≥0\bar\eta=-\log\det(\bar{A}\bar{A}^\top)\ge 0. This reduces to Theorem 2 when ∥A∥F2=r\|A\|_\mathrm{F}^2=r and is Corollary 5 in Appendix A.6. The bound factors into a scale term ∥A∥F2/r\|A\|_\mathrm{F}^2/r and a shape term that depends only on the normalized matrix Aˉ\bar{A}. This makes your point explicit: the raw log-determinant can increase through either term, but only the shape term reflects how evenly AA spreads its energy across directions, since a uniform rescaling of AA can be absorbed by the expert matrices. Items 2 and 3 examine the shape term.

2. Normalizing AA. We reran OD-MoLE with the log-determinant computed on Aˉ=rAmax⁡(∥A∥F,10−12)\bar{A} = \frac{\sqrt{r}A}{\max(\|A\|_\mathrm{F},10^{-12})}, where rr is the number of rows of AA. We normalize the full matrix first and then sample columns from Aˉ\bar{A}. This satisfies the scale condition of Theorem 2 by construction. Both variants use λdiv=0.001\lambda_{\mathrm{div}}=0.001 and λorth=0.001\lambda_{\mathrm{orth}}=0.001.

Compose all expertsComQACosQASIQAPIQAScQABoolQARC-COBQAAvg.
Normalized logdet85.7588.4180.3590.3291.6586.9488.6595.8088.48
Raw logdet85.8388.5180.1990.2192.0287.0388.4896.0088.53
Route to selective expertsComQACosQASIQAPIQAScQABoolQARC-COBQAAvg.
Normalized logdet84.7787.1479.8989.6192.6387.8388.0595.6088.19
Raw logdet85.1887.8179.1289.9993.6188.2088.5796.4088.61

The normalized variant reaches comparable average accuracy in both routing settings, so enforcing the scale condition does not change the empirical conclusions.

3. Verifying conditioning. To check that the raw penalty of Section 5.2 is not satisfied by rescaling alone, we track AA during training under all-expert composition with output orthogonality disabled, comparing no regularizer (λdiv=0\lambda_{\mathrm{div}}=0) with the raw penalty (λdiv=0.001\lambda_{\mathrm{div}}=0.001). Each run is summarized over its 112 shared projection matrices at initialization and at the end of each epoch. Effective rank, stable rank, the normalized spectrum and the condition number κ2(A)\kappa_2(A) are all invariant to uniform rescaling of AA.

Scale. ∥A∥F\|A\|_\mathrm{F} grows during training in both runs and ends slightly higher with the raw penalty, so part of the penalty's effect is scale, as you anticipated. Shape. The scale-invariant statistics also improve. Effective and stable rank decline more slowly with the penalty on both SIQA and PIQA (Tables Q1.3.1 and Q1.3.2), and the median and p90 condition numbers on PIQA are lower after every training epoch, by about 10% at the end of training (Table Q1.3.3).

Table Q1.3.1 Norm and singular-value spectrum of A evolve during training (SIQA)

SettingEpoch∥A∥F\lVert A\rVert_{\mathrm{F}} (mean / max)Effective rank (mean / max)Stable rank (mean / max)Normalized median spectrum (mean / max)
(no ortho, no div)02.3089 / 2.319415.9649 / 15.976414.3419 / 14.66670.24989 / 0.26392
(no ortho, no div)12.6812 / 3.077514.8210 / 15.59696.8792 / 10.04130.24553 / 0.36576
(no ortho, no div)23.1958 / 4.049213.3581 / 14.70455.2882 / 7.78740.23962 / 0.40654
(no ortho, div)02.3089 / 2.319415.9649 / 15.976414.3419 / 14.66670.24989 / 0.26392
(no ortho, div)12.6955 / 3.080715.0661 / 15.70137.3503 / 10.44300.24635 / 0.38040
(no ortho, div)23.2623 / 4.224014.1296 / 15.14885.9818 / 8.71380.24305 / 0.43637

Table Q1.3.2 Norm and singular-value spectrum of A evolve during training (PIQA)

SettingEpoch∥A∥F\lVert A\rVert_{\mathrm F} (mean / max)Effective rank (mean / max)Stable rank (mean / max)Normalized median spectrum (mean / max)
(no ortho, no div)02.3089 / 2.319415.9649 / 15.976414.3419 / 14.66670.24989 / 0.26392
(no ortho, no div)12.5999 / 2.869714.6758 / 15.62236.2983 / 10.23250.24532 / 0.39706
(no ortho, no div)22.9368 / 3.596113.3540 / 14.79724.8435 / 7.91450.24103 / 0.44940
(no ortho, no div)33.2760 / 4.255812.1875 / 13.94444.2116 / 6.53600.23542 / 0.47673
(no ortho, div)02.3089 / 2.319415.9649 / 15.976414.3419 / 14.66670.24989 / 0.26392
(no ortho, div)12.6088 / 2.876614.7700 / 15.62936.4475 / 9.74410.24589 / 0.40120
(no ortho, div)22.9229 / 3.555713.7568 / 15.03675.1794 / 7.93120.24256 / 0.46390
(no ortho, div)33.3494 / 4.388312.9154 / 14.44944.6261 / 7.42420.23907 / 0.49564

Table Q1.3.3 Condition number of AA during training (PIQA)

TaskSettingλ_divEpochMedian κ₂(A)p90 κ₂(A)
PIQA(no ortho, no div)001.11961.1325
PIQA(no ortho, no div)011.88452.2502
PIQA(no ortho, no div)022.38812.7517
PIQA(no ortho, no div)032.83333.1920
PIQA(no ortho, div)0.00101.11961.1325
PIQA(no ortho, div)0.00111.85602.2313
PIQA(no ortho, div)0.00122.23282.5952
PIQA(no ortho, div)0.00132.54472.8726

Accuracy with the diversity penalty alone. For completeness, we also ran the raw diversity penalty without output orthogonality under selective routing over three seeds. Table Q1.3.4 reports accuracy as mean ± sample SD.

Table Q1.3.4 Diversity penalty alone vs. no regularizer

SettingComQACosQASIQAPIQAScQABoolQARC-COBQAAvg.
(no ortho, div)84.63 ± 0.5487.16 ± 0.5479.87 ± 0.3989.63 ± 0.7792.21 ± 0.5487.68 ± 0.3888.60 ± 0.1895.33 ± 0.2388.14 ± 0.10
(no ortho, no div)84.58 ± 0.2185.72 ± 1.7079.38 ± 0.1489.28 ± 0.7691.75 ± 0.5287.35 ± 0.2688.25 ± 0.4295.20 ± 0.2087.69 ± 0.28

In Table Q1.3.5, each seed cell shows (no ortho, no div) / (no ortho, div) accuracy. Δ\Delta is (no ortho, div) minus (no ortho, no div) in percentage points, with paired tt 95% confidence intervals across seeds 40–42.

Table Q1.3.5 Paired per-seed comparison

TaskPaired seedsSeed 40Seed 41Seed 42Mean Δ (pp)95% CI (pp)
ComQA384.77 / 85.2684.60 / 84.2884.36 / 84.36+0.057[−0.957, +1.070]
CosQA387.67 / 87.1484.57 / 86.6384.93 / 87.71+1.437[−2.888, +5.761]
SIQA379.43 / 79.4379.48 / 80.1979.22 / 79.99+0.493[−0.571, +1.557]
PIQA389.72 / 89.9988.41 / 90.1589.72 / 88.74+0.343[−3.039, +3.725]
ScQA392.30 / 92.8291.70 / 91.7991.26 / 92.02+0.457[−0.387, +1.300]
BoolQ387.06 / 87.2887.40 / 88.0487.58 / 87.71+0.330[−0.346, +1.006]
ARC-C387.97 / 88.4088.74 / 88.7488.05 / 88.65+0.343[−0.425, +1.112]
OBQA395.20 / 95.6095.00 / 95.2095.40 / 95.20+0.133[−0.626, +0.892]
Eight-task macro-average388.02 / 88.2487.49 / 88.1387.57 / 88.05+0.449[−0.071, +0.970]

The diversity penalty alone gives a higher macro-average in all three seeds and a positive mean difference on all eight tasks. With three seeds, the interval includes zero, so we treat this as supporting evidence and rely on the conditioning statistics above for the claim that the penalty behaves as Theorem 2 describes.

We have added the arbitrary-scale form as Corollary 5 with its proof in Appendix A.6, and a pointer after Theorem 2 stating that the raw penalty is used as a practical surrogate. The final-epoch conditioning statistics are in Table 5, and the per-epoch spectrum, condition number and normalized-penalty results are in Appendix B.12 Tables 20, 21 and 23. The diversity-only ablation is in Table 3 and Appendix B.10 Table 14.

Q2 Explain loss in Theorem 1, and does orthogonality improve routing effectiveness?

Thank you for pointing out this distinction. In Theorem 1, Equation (7) bounds how much the hidden-layer mixture changes when the routing weights change, with the input and expert outputs held fixed. This geometric bound is independent of the downstream loss. While Equation (8) serves as an example, it establishes recovery of the optimal routing weights in a specific case when $ Q_h$ is the quadratic residual objective. We have clarified this in the paragraph after Theorem 1. We therefore test the predicted change in routing distinguishability directly on models trained with cross-entropy.

We compare two selective routing settings with the diversity penalty disabled (λdiv=0\lambda_{\mathrm{div}}=0). The (no ortho, no div) uses λorth=0\lambda_{\mathrm{orth}}=0, and the (ortho, no div) uses λorth=0.001\lambda_{\mathrm{orth}}=0.001. For each evaluated token and attention projection, we hold the two selected experts fixed and transfer 0.050.05 routing weight from the higher-weight expert to its partner. We measure S=∥Δr∥2/∥Δg∥2S=\lVert\Delta r\rVert_2/\lVert\Delta g\rVert_2, where Δg\Delta g is the change in routing weights and Δr\Delta r is the resulting change in the expert mixture. A larger SS means the same weight perturbation produces a larger change in the mixture output. Table Q2.1 shows that adding orthogonality increases median sensitivity on all eight tasks, and Table Q2.2 shows that the fifth percentile increases on five tasks. This supports improved typical routing distinguishability for the evaluated weight perturbation, with mixed effects in the lower tail.

We have added routing distinguishability in Table 4 in the manuscript, and the fifth percentile is in Appendix B.11 Table 16. The orthogonality only setting is a row in Table 3, and its per-seed results are in Appendix B.10 Table 13. We have added a paragraph after Theorem 1 that separates the geometric bound from the recovery guarantee and states that the latter applies only to QhQ_h.

Q2.1 Median routing sensitivity SS

SettingComQACosQASIQAPIQAScQABoolQARC-COBQA
(no ortho, no div)0.60953.44401.75792.57843.23011.39271.39551.7856
(ortho, no div)0.67303.55961.80202.70483.24611.46251.41181.8049

Q2.2 Fifth percentile of routing sensitivity SS

SettingComQACosQASIQAPIQAScQABoolQARC-COBQA
(no ortho, no div)0.17630.85330.40770.67280.83230.34560.30940.1972
(ortho, no div)0.18580.84310.41460.67680.71760.32810.31880.2596

Q2.3 Effectiveness of orthogonal loss over three random seeds.

SettingComQACosQASIQAPIQAScQABoolQARC-COBQAAvg.
(no ortho, no div)84.58 ± 0.2185.72 ± 1.7079.38 ± 0.1489.28 ± 0.7691.75 ± 0.5287.35 ± 0.2688.25 ± 0.4295.20 ± 0.2087.69 ± 0.28
(ortho, no div)84.77 ± 0.6486.65 ± 0.4079.75 ± 0.3089.59 ± 0.2792.37 ± 0.0388.12 ± 0.1088.05 ± 0.2396.27 ± 0.4688.20 ± 0.10
Difference (pp)+0.19 ± 0.68+0.92 ± 1.96+0.37 ± 0.21+0.31 ± 0.77+0.61 ± 0.50+0.78 ± 0.35−0.20 ± 0.22+1.07 ± 0.50+0.51 ± 0.19

Each seed cell below shows (no ortho, no div) / (ortho, no div) accuracy. Δ\Delta is (ortho, no div) minus (no ortho, no div), in percentage points.

TaskPaired seedsSeed 40Seed 41Seed 42Mean Δ\Delta (pp)95% CI (pp)
ComQA384.77 / 85.1884.60 / 84.0384.36 / 85.09+0.190[−1.493, +1.873]
CosQA387.67 / 86.3484.57 / 86.5084.93 / 87.10+0.923[−3.933, +5.780]
SIQA379.43 / 79.6379.48 / 80.0979.22 / 79.53+0.373[−0.154, +0.900]
PIQA389.72 / 89.8888.41 / 89.5589.72 / 89.34+0.307[−1.607, +2.221]
ScQA392.30 / 92.4091.70 / 92.3591.26 / 92.35+0.613[−0.619, +1.846]
BoolQ387.06 / 88.2387.40 / 88.0487.58 / 88.10+0.777[−0.083, +1.636]
ARC-C387.97 / 87.9788.74 / 88.3188.05 / 87.88−0.200[−0.738, +0.338]
OBQA395.20 / 96.8095.00 / 96.0095.40 / 96.00+1.067[−0.184, +2.317]
Eight-task macro-average388.02 / 88.3087.49 / 88.1187.57 / 88.17+0.506[+0.038, +0.974]
Reviewer_2cPzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OD-MoLE, a shared-projection LoRA mixture with two regularizers: expert-output orthogonality and shared-projection diversity. The former penalizes squared cosine similarities between expert responses, while the latter uses a sampled log-determinant objective. Conditional theoretical results motivate these objectives, and experiments evaluate commonsense reasoning on two 7B backbones under all-expert composition and selective routing.

Critical strengths
  1. Regularizing expert responses on actual activations is a meaningful approach to functional redundancy, since distinct parameter matrices can still produce similar outputs.

  2. The paper provides explicit theoretical assumptions and evaluates both routing settings, with component ablations and training-cost measurements. The results consistently favor the combined regularization over HydraLoRA on average.

Critical weaknesses
  1. The connection between the theoretical conditions and the implemented regularizers remains incomplete. The routing-recovery result requires nonvanishing expert outputs, which the orthogonality penalty does not enforce. The shared-projection result requires scale control, whereas the sampled log-determinant penalty can improve simply by scaling up A and inversely scaling the expert matrices. Although these limitations are acknowledged, the experiments do not establish whether training improves the relevant conditioning or mainly changes representation scale and geometry.

  2. The empirical gains are modest and do not sufficiently establish useful specialization. Under selective routing, average improvements over HydraLoRA are only 0.25 and 0.36 percentage points on the two backbones. The reported standard deviations do not establish whether these paired gains are reliable. Moreover, output separation in t-SNE does not demonstrate complementary task contributions, and the diversity-only ablation improves the average by just 0.04 points. Broader validation and direct expert-utility analyses are needed to substantiate the claimed benefits.

Questions for authors
  1. How do the norm and singular-value spectrum of A evolve during training? Does a scale-normalized log-determinant regularizer retain the gains, ruling out simple rescaling as the explanation?

  2. Do paired per-seed comparisons support statistically reliable improvements over HydraLoRA, particularly under selective routing?

  3. Beyond output decorrelation, do expert removal or replacement experiments demonstrate complementary predictive contributions? Could the method instead encourage distinct but task-irrelevant output directions?

10 comments

Q1. Nonvanishing expert outputs

We agree that Theorem 1 assumes a positive lower bound on the expert output norms at a fixed input activation. The orthogonality penalty discourages alignment between expert outputs. The theorem’s routing-recovery guarantee is therefore conditional on the norm assumption, which the training objective does not enforce. We have added a paragraph after Theorem 1 and Appendix B.11 to state this explicitly.

To assess whether near-zero outputs occur in practice, we evaluate the final expert norms which are trained under selective routing. For each evaluated token and projection, we compute the minimum expert output norm before multiplying by its routing weight. Table Q1.1 takes the minimum over all eight experts, and Table Q1.2 takes the minimum over the two experts selected by the router. Both (no ortho, no div) and (ortho, no div) settings disable the diversity penalty, and (ortho, no div) setting uses λorth=0.001\lambda_{\mathrm{orth}}=0.001. We report the results in Table Q1.1 and Table Q1.2.

At the 10−810^{-8} threshold, the reported near-zero fraction is 0% on all eight tasks in both settings, including when all eight experts are measured. With orthogonality, the median minimum norm increases on seven of eight tasks in both tables. The fifth percentile increases on five of eight tasks across all experts and on six of eight tasks across the selected pair. The all-expert measurements give direct evidence about the norm assumption on the evaluated inputs and projections at the final model. Since both the control and the orthogonality-only variant have a reported near-zero fraction of 0%, this provides empirical support for the nonvanishing-output assumption on the evaluated activations. The routing-recovery guarantee remains conditional on that assumption. We have added minimum-norm measurements in Appendix B.11 Table 17.

Table Q1.1 Minimum output norm across all eight experts

TaskSettingExperts measuredMedian min normp05 min normNear-zero fractionThreshold
ComQA(no ortho, no div)All 8 experts0.533470.145140.000000%1e-08
ComQA(ortho, no div)All 8 experts0.565700.158140.000000%1e-08
CosQA(no ortho, no div)All 8 experts3.035570.687840.000000%1e-08
CosQA(ortho, no div)All 8 experts3.112370.695510.000000%1e-08
SIQA(no ortho, no div)All 8 experts1.414670.357510.000000%1e-08
SIQA(ortho, no div)All 8 experts1.457120.337290.000000%1e-08
PIQA(no ortho, no div)All 8 experts2.023530.373100.000000%1e-08
PIQA(ortho, no div)All 8 experts2.166790.381580.000000%1e-08
ScQA(no ortho, no div)All 8 experts2.669300.533740.000000%1e-08
ScQA(ortho, no div)All 8 experts2.730030.599870.000000%1e-08
BoolQ(no ortho, no div)All 8 experts1.212610.261640.000000%1e-08
BoolQ(ortho, no div)All 8 experts1.254750.245590.000000%1e-08
ARC-C(no ortho, no div)All 8 experts1.181020.220530.000000%1e-08
ARC-C(ortho, no div)All 8 experts1.218520.211490.000000%1e-08
OBQA(no ortho, no div)All 8 experts1.563220.145240.000000%1e-08
OBQA(ortho, no div)All 8 experts1.457440.201210.000000%1e-08

Table Q1.2. Minimum output norm across the two experts selected by the router

TaskSettingExperts measuredMedian min normp05 min normNear-zero fractionThreshold
ComQA(no ortho, no div)Selected 2 experts0.621980.178330.000000%1e-08
ComQA(ortho, no div)Selected 2 experts0.681190.186760.000000%1e-08
CosQA(no ortho, no div)Selected 2 experts3.449760.816370.000000%1e-08
CosQA(ortho, no div)Selected 2 experts3.550400.829620.000000%1e-08
SIQA(no ortho, no div)Selected 2 experts1.710470.409330.000000%1e-08
SIQA(ortho, no div)Selected 2 experts1.780960.396580.000000%1e-08
PIQA(no ortho, no div)Selected 2 experts2.539550.624340.000000%1e-08
PIQA(ortho, no div)Selected 2 experts2.596350.646880.000000%1e-08
ScQA(no ortho, no div)Selected 2 experts3.174160.814560.000000%1e-08
ScQA(ortho, no div)Selected 2 experts3.179530.675920.000000%1e-08
BoolQ(no ortho, no div)Selected 2 experts1.385940.309580.000000%1e-08
BoolQ(ortho, no div)Selected 2 experts1.475400.325920.000000%1e-08
ARC-C(no ortho, no div)Selected 2 experts1.385060.299560.000000%1e-08
ARC-C(ortho, no div)Selected 2 experts1.400800.302630.000000%1e-08
OBQA(no ortho, no div)Selected 2 experts1.804490.187510.000000%1e-08
OBQA(ortho, no div)Selected 2 experts1.790530.249580.000000%1e-08

Q2 How do the norm and singular-value spectrum of A evolve during training? Does only applying log-determinant regularizer retain the gains?

Thank you for suggesting that we track AA during training. We compare all experts composition routing on SIQA and PIQA with output orthogonality disabled, where (no ortho, no div) uses λdiv=0\lambda_{\mathrm{div}}=0, and (no ortho, div) uses λdiv=0.001\lambda_{\mathrm{div}}=0.001. At initialization and each epoch end, we summarize the Frobenius norm, effective rank, and normalized singular value across 112 shared projection matrices per run.

On both tasks, the Frobenius norm and the maximum of the normalized median singular-value spectrum increase during training, while effective rank and the mean of that spectrum decline more slowly with raw log-det than in (no ortho, no div). Effective rank and the normalized spectrum are invariant to uniform rescaling, indicating that the regularizer changes the shape of the spectrum.

Table Q2.1 Norm and singular-value spectrum of A evolve during training (SIQA)

settingEpoch∥A∥F\lVert A\rVert_{\mathrm{F}} (mean / max)Effective rank (mean / max)Stable rank (mean / max)Normalized median spectrum (mean / max)
(no ortho, no div)02.3089 / 2.319415.9649 / 15.976414.3419 / 14.66670.24989 / 0.26392
(no ortho, no div)12.6812 / 3.077514.8210 / 15.59696.8792 / 10.04130.24553 / 0.36576
(no ortho, no div)23.1958 / 4.049213.3581 / 14.70455.2882 / 7.78740.23962 / 0.40654
(no ortho, div)02.3089 / 2.319415.9649 / 15.976414.3419 / 14.66670.24989 / 0.26392
(no ortho, div)12.6955 / 3.080715.0661 / 15.70137.3503 / 10.44300.24635 / 0.38040
(no ortho, div)23.2623 / 4.224014.1296 / 15.14885.9818 / 8.71380.24305 / 0.43637

Table Q2.2 Norm and singular-value spectrum of A evolve during training (PIQA)

settingEpoch∥A∥F\lVert A\rVert_{\mathrm{F}} (mean / max)Effective rank (mean / max)Stable rank (mean / max)Normalized median spectrum (mean / max)
(no ortho, no div)02.3089 / 2.319415.9649 / 15.976414.3419 / 14.66670.24989 / 0.26392
(no ortho, no div)12.5999 / 2.869714.6758 / 15.62236.2983 / 10.23250.24532 / 0.39706
(no ortho, no div)22.9368 / 3.596113.3540 / 14.79724.8435 / 7.91450.24103 / 0.44940
(no ortho, no div)33.2760 / 4.255812.1875 / 13.94444.2116 / 6.53600.23542 / 0.47673
(no ortho, div)02.3089 / 2.319415.9649 / 15.976414.3419 / 14.66670.24989 / 0.26392
(no ortho, div)12.6088 / 2.876614.7700 / 15.62936.4475 / 9.74410.24589 / 0.40120
(no ortho, div)22.9229 / 3.555713.7568 / 15.03675.1794 / 7.93120.24256 / 0.46390
(no ortho, div)33.3494 / 4.388312.9154 / 14.44944.6261 / 7.42420.23907 / 0.49564

We ran OD-MoLE with only diversity loss applied under selective routing over three random seeds and compared with no diversity loss and no orthogonal loss settings. Each cell reports accuracy as mean ± sample SD in Table Q2.3. Across eight tasks, applying log-det alone raises average accuracy from 87.69% to 88.14%. These comparisons show that applying log-det alone gives a higher average accuracy in all three seeds, although the 95% interval [−0.071, +0.970] includes zero.

Table Q2.3 Log-determinant regularizer ablation

SettingComQACosQASIQAPIQAScQABoolQARC-COBQAAvg.
(no ortho, div)84.63 ± 0.5487.16 ± 0.5479.87 ± 0.3989.63 ± 0.7792.21 ± 0.5487.68 ± 0.3888.60 ± 0.1895.33 ± 0.2388.14 ± 0.10
(no ortho, no div)84.58 ± 0.2185.72 ± 1.7079.38 ± 0.1489.28 ± 0.7691.75 ± 0.5287.35 ± 0.2688.25 ± 0.4295.20 ± 0.2087.69 ± 0.28

Each seed cell below shows (no ortho, no div) / (no ortho, div) accuracy. Δ\Delta is (no ortho, div) minus (no ortho, no div), in percentage points. The 95% confidence intervals are paired tt intervals across seeds 40–42.

TaskPaired seedsSeed 40Seed 41Seed 42Mean Δ (pp)95% CI (pp)
ComQA384.77 / 85.2684.60 / 84.2884.36 / 84.36+0.057[−0.957, +1.070]
CosQA387.67 / 87.1484.57 / 86.6384.93 / 87.71+1.437[−2.888, +5.761]
SIQA379.43 / 79.4379.48 / 80.1979.22 / 79.99+0.493[−0.571, +1.557]
PIQA389.72 / 89.9988.41 / 90.1589.72 / 88.74+0.343[−3.039, +3.725]
ScQA392.30 / 92.8291.70 / 91.7991.26 / 92.02+0.457[−0.387, +1.300]
BoolQ387.06 / 87.2887.40 / 88.0487.58 / 87.71+0.330[−0.346, +1.006]
ARC-C387.97 / 88.4088.74 / 88.7488.05 / 88.65+0.343[−0.425, +1.112]
OBQA395.20 / 95.6095.00 / 95.2095.40 / 95.20+0.133[−0.626, +0.892]
Eight-task macro-average388.02 / 88.2487.49 / 88.1387.57 / 88.05+0.449[−0.071, +0.970]

We have added final-epoch spectrum in Table 5 in the manuscript, and the per-epoch results in Appendix B.12 Tables 20 and 21. The log-determinant only setting is a row in the regularizer ablation in Table 3, with per-seed results in Appendix B.10 Table 14.

Q3 Paired per-seed comparisons with HydraLoRA

Thank you for asking about this pair per-seed comparison. We reran OD-MoLE and HydraLoRA using Qwen2.5-7B model under the same setting in selective routing with 8 experts per submodule and top-2 gating over five random seeds. Each seed cell reports HydraLoRA / OD-MoLE accuracy.

TaskPaired seedsSeed 40Seed 41Seed 42Seed 43Seed 44Mean Δ\Delta (pp)95% CI (pp)
ComQA584.77 / 85.1884.60 / 84.6884.36 / 84.6885.67 / 86.0085.09 / 84.85+0.180[−0.149, +0.509]
CosQA587.67 / 87.8184.57 / 85.7684.93 / 85.7184.03 / 84.1483.89 / 84.53+0.572[+0.006, +1.138]
SIQA579.43 / 79.1279.48 / 79.8979.22 / 80.2579.12 / 80.1978.35 / 79.48+0.666[−0.102, +1.434]
PIQA589.72 / 89.9988.41 / 89.3489.72 / 90.3789.66 / 88.8589.83 / 90.37+0.316[−0.519, +1.151]
ScQA592.30 / 93.6191.70 / 92.8291.26 / 92.2692.96 / 92.9691.98 / 91.93+0.676[−0.131, +1.483]
BoolQ587.06 / 88.2087.40 / 87.9287.58 / 88.5087.86 / 86.2188.01 / 87.89+0.162[−1.229, +1.553]
ARC-C587.97 / 88.5788.74 / 88.9988.05 / 88.4887.54 / 88.0587.97 / 88.31+0.426[+0.255, +0.597]
OBQA595.20 / 96.4095.00 / 95.4095.40 / 95.8095.20 / 95.2096.20 / 96.20+0.400[−0.208, +1.008]
Eight-task macro-average588.015 / 88.61087.4875 / 88.10087.565 / 88.2562587.755 / 87.70087.665 / 87.945+0.425[+0.039, +0.811]

where Δ\Delta is OD-MoLE minus HydraLoRA in percentage points. The 95% intervals use a paired t calculation across the five seed differences. For the macro-average, we first average across the eight tasks within each seed. OD-MoLE improves the macro-average in four of five seeds, with a mean paired gain of 0.425 pp and a 95% confidence interval of [0.039, 0.811] pp. We have added this paired per-seed comparison in Appendix B.9 Table 12.

Reviewer_Ui7FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OD-MoLE to address expert redundancy and insufficient specialization in LoRA-based MoE. It introduces output orthogonality and shared-projection diversity regularization to encourage distinct expert representations. Theoretical analyses and experiments demonstrate improved expert specialization and downstream task performance.

Critical strengths

The idea of encouraging expert specialization by directly regularizing expert outputs is commendable.

The theoretical analyses are solid and provide useful insights into why the two regularization objectives can be effective.

The proposed method is simple and practical and the experiments are fairly comprehensive.

Critical weaknesses

The orthogonality loss is applied to every pair of experts, including those that are rarely or never selected together. I'm not sure why these experts need to be orthogonal if their outputs are almost never mixed. This could force them to change their outputs for no clear benefit, and might even get in the way of what each expert is trying to learn. It would be useful to compare this with a loss applied only to experts that are selected together.

I would also like to see more direct evidence that the two regularizers do what the paper claims. For example, looking at the singular values or effective rank of the shared projection could show whether feature collapse is actually reduced. The authors could also look at routing patterns, or check how much the mixture output changes when routing weights are perturbed, to see whether routing distinguishability really improves.

Questions for authors

See weaknesses

5 comments

Q1 Orthogonal loss computed by every pair of experts vs. selected experts

Thank you so much for suggesting this comparison for our OD-MoLE. We agree that comparing with the orthogonal loss computed by selected experts can help better understand the mechanism. We examine this both empirically and theoretically.

Empirical We reran the selective routing setting at top-2 routing with 8 experts over three random seeds and set diversity loss coefficient as λdiv=0\lambda_{\mathrm{div}}=0 in both settings, and the only change is which expert pairs compute the orthogonal loss. The all-pairs uses every expert pair, while the selected-pair uses the experts selected by the router.

SettingComQACosQASIQAPIQAScQABoolQARC-COBQAAvg.
Selected pairs84.50 ± 0.6087.53 ± 1.0879.58 ± 0.5389.30 ± 0.5692.52 ± 0.4087.97 ± 0.6488.31 ± 0.7695.80 ± 0.4088.19 ± 0.19
All pairs84.77 ± 0.6486.65 ± 0.4079.75 ± 0.3089.59 ± 0.2792.37 ± 0.0388.12 ± 0.1088.05 ± 0.2396.27 ± 0.4688.20 ± 0.10

Each seed cell below shows all pairs / selected pairs accuracy. Δ\Delta is selected pairs minus all pairs, with unadjusted paired tt confidence intervals across the three seeds.

TaskPaired seedsSeed 40Seed 41Seed 42Mean Δ (pp)95% CI (pp)
ComQA385.18 / 85.1884.03 / 84.0385.09 / 84.28−0.270[−1.432, +0.892]
CosQA386.34 / 87.9486.50 / 86.3087.10 / 88.34+0.880[−1.486, +3.246]
SIQA379.63 / 79.3280.09 / 79.2279.53 / 80.19−0.173[−2.096, +1.750]
PIQA389.88 / 89.9389.55 / 89.1289.34 / 88.85−0.290[−1.025, +0.445]
ScQA392.40 / 92.9192.35 / 92.5492.35 / 92.12+0.157[−0.765, +1.079]
BoolQ388.23 / 87.2888.04 / 88.1088.10 / 88.53−0.153[−1.928, +1.621]
ARC-C387.97 / 87.4688.31 / 88.9187.88 / 88.57+0.260[−1.400, +1.920]
OBQA396.80 / 95.4096.00 / 95.8096.00 / 96.20−0.467[−2.535, +1.602]
Eight-task macro-average388.30 / 88.1888.11 / 88.0088.17 / 88.39−0.007[−0.477, +0.463]

The all-pairs loss and selected-pairs loss are nearly identical in mean macro-accuracy across these seeds. It shows that penalizing pairs that the router does not select together does not hurt accuracy. We have added "selected pairs" as a row in Table 3 in the revised manuscript and the per-seed comparison is in Appendix B.10 Table 15.

Theoretical. We have extended Theorem 1 to top-KK routing in Appendix A.2 in the revised manuscript, with a pointer in the paragraph after Theorem 1. Two routing vectors with different top-KK supports together involve up to 2K2K experts, so separating their mixture outputs depends on expert pairs that are never selected together. The new corollary bounds this separation by the mean squared cosine over all expert pairs, which is the quantity the all-pairs loss penalizes, whereas the selected-pairs loss gives no such bound. The theory therefore supports penalizing all pairs.

Q2 More direct evidence on two regularizers

Thank you for suggesting more direct evidence on two regularizers in our OD-MoLE. We added two comparisons of the shared projection spectrum and of how the mixture output responds to changes in routing weights.

1. Shared projection diversity We track the stable rank and singular values of shared projection AA at initialization and after each epoch on tasks. We compare all experts composition routing on SIQA and PIQA with output orthogonality disabled, where the unregularized run uses λdiv=0\lambda_{\mathrm{div}}=0 and the regularized runs use λdiv∈{0.001,0.01,0.1,1}\lambda_{\mathrm{div}}\in\{0.001,0.01,0.1,1\}. Table Q2.1 and Table Q2.2 summarize 112 shared projection matrices per run. We report the mean and maximum stable rank across matrices and the mean and maximum of the per-index median raw singular-value spectrum. Mean stable rank decreases during training in every setting. Since stable rank ∥A∥F2/σmax(A)2\|A\|_{F}^2/\sigma_{max}(A)^2 is invariant to uniform rescaling, these differences show that the regularized projections retain more spectral energy relative to their leading singular direction.

Table Q2.1 Stable rank and singular values of the shared projection (SIQA)

Training settingλ_divEpochStable rank (mean / max)Raw singular values (mean / max)Accuracy
(no ortho, no div)0014.3419 / 14.66670.57691 / 0.60918—
(no ortho, no div)016.8792 / 10.04130.63116 / 0.84267—
(no ortho, no div)025.2882 / 7.78740.70820 / 1.0809480.04
(no ortho, div)0.001014.3419 / 14.66670.57691 / 0.60918—
(no ortho, div)0.00117.3503 / 10.44300.66042 / 0.98135—
(no ortho, div)0.00125.9818 / 8.71380.76706 / 1.3041680.14
(no ortho, div)0.01014.3419 / 14.66670.57691 / 0.60918—
(no ortho, div)0.0117.3689 / 10.30360.66121 / 0.98441—
(no ortho, div)0.0126.0256 / 8.91370.76219 / 1.2764679.94
(no ortho, div)0.1014.3419 / 14.66670.57691 / 0.60918—
(no ortho, div)0.117.6340 / 10.45750.65619 / 0.96472—
(no ortho, div)0.126.1871 / 9.05680.75281 / 1.2454080.30
(no ortho, div)1014.3419 / 14.66670.57691 / 0.60918—
(no ortho, div)119.3228 / 13.55190.65409 / 1.01162—
(no ortho, div)127.3886 / 11.24980.77154 / 1.4028380.09

Table Q2.2 Stable rank and singular values of the shared projection (PIQA)

Settingλ_divEpochStable rank (mean / max)Raw singular values (mean / max)Accuracy
(no ortho, no div)0014.3419 / 14.66670.57691 / 0.60918—
(no ortho, no div)016.2983 / 10.23250.61463 / 0.87025—
(no ortho, no div)024.8435 / 7.91450.65948 / 1.06231—
(no ortho, no div)034.2116 / 6.53600.71079 / 1.2341289.72
(no ortho, div)0.001014.3419 / 14.66670.57691 / 0.60918—
(no ortho, div)0.00116.4475 / 9.74410.63666 / 1.02917—
(no ortho, div)0.00125.1794 / 7.93120.70210 / 1.29522—
(no ortho, div)0.00134.6261 / 7.42420.77307 / 1.5250190.21
(no ortho, div)0.01014.3419 / 14.66670.57691 / 0.60918—
(no ortho, div)0.0116.4905 / 9.98420.63627 / 1.02461—
(no ortho, div)0.0125.1903 / 8.21460.70090 / 1.28552—
(no ortho, div)0.0134.6340 / 7.57140.77097 / 1.5144189.77
(no ortho, div)0.1014.3419 / 14.66670.57691 / 0.60918—
(no ortho, div)0.116.7319 / 10.10870.63318 / 1.00900—
(no ortho, div)0.125.3769 / 8.19620.69520 / 1.26985—
(no ortho, div)0.134.7589 / 7.44880.76204 / 1.4807089.66
(no ortho, div)1014.3419 / 14.66670.57691 / 0.60918—
(no ortho, div)118.4478 / 14.49270.63380 / 1.03975—
(no ortho, div)126.8457 / 12.72000.70015 / 1.34947—
(no ortho, div)135.8482 / 10.70280.77909 / 1.6340789.61

2. Routing distinguishability We tested how much the expert mixture changes when we perturb the routing weights in selective routing (8 experts with top-2 routing). At each evaluated token and submodule, we fix the two selected experts and transfer 0.05 routing weight from the higher-weight expert to its partner. We measured the local sensitivity S=∥Δr∥2/∥Δg∥2S=\lVert\Delta r\rVert_2/\lVert\Delta g\rVert_2, where Δg\Delta g is the routing-weight perturbation and Δr\Delta r is the resulting change in the mixture output. The comparison between (no ortho, no div) and (ortho, no div) isolates the output penalty. We report the median and fifth percentile of SS, with squared cosine similarity.

Adding the orthogonality penalty increases median sensitivity SS and decreases mean squared cosine similarity between the selected expert outputs on all eight tasks. The fifth percentile of SS increases on five tasks and decreases on three, indicating mixed effects among the least sensitive cases. Since SS depends on both expert-output norms and directions, the decrease in squared cosine similarity separately shows that the selected expert outputs became less aligned on average. Thus, these results support improved local routing distinguishability.

TaskSettingMedian SSp05 SSMean squared cosine
ComQA(no ortho, no div)0.60950.17630.0457
ComQA(ortho, no div)0.67300.18580.0378
CosQA(no ortho, no div)3.44400.85330.0195
CosQA(ortho, no div)3.55960.84310.0164
SIQA(no ortho, no div)1.75790.40770.0374
SIQA(ortho, no div)1.80200.41460.0321
PIQA(no ortho, no div)2.57840.67280.0292
PIQA(ortho, no div)2.70480.67680.0242
ScQA(no ortho, no div)3.23010.83230.0208
ScQA(ortho, no div)3.24610.71760.0193
BoolQ(no ortho, no div)1.39270.34560.0245
BoolQ(ortho, no div)1.46250.32810.0233
ARC-C(no ortho, no div)1.39550.30940.0475
ARC-C(ortho, no div)1.41180.31880.0430
OBQA(no ortho, no div)1.78560.19720.0361
OBQA(ortho, no div)1.80490.25960.0317

The conclusion is that both regularizers change the quantity they are designed to control. We have added a new "Mechanisms of the Orthogonality and Diversity Regularizers" paragraph in Section 6.2, where Table 4 is for routing distinguishability and Table 5 is for the spectrum of AA. The fifth-percentile sensitivity is in Appendix B.11 Table 16, and the per-epoch spectrum, condition number and λdiv\lambda_{\mathrm{div}} sweep are in Appendix B.12 Tables 20–22.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.