acceptodds
Under review as a conference paper at ICLR 2027

Disentangled Mixture of Rank-1 Experts for Low-Rank Adaptation

Abstract

The uniform rank allocation in Low-Rank Adaptation (LoRA) overlooks heterogeneous learning dynamics across modules and inherent semantic differences across inputs. We propose Disentangled Mixture of Rank- Experts (DMoRE), a principled framework for input-dependent rank allocation. To this end, we develop dynamic self-routing to activate a variable number of rank- components per input without an external router, along with gradient-derived budget estimation to avoid potentially misaligned manual budget specification. We further propose co-activation-aware rank disentanglement that encourages frequently co-activated components to learn distinct transformations. Extensive experiments on diverse language and vision tasks demonstrate that DMoRE consistently outperforms competitive LoRA baselines with negligible additional parameter overhead. It also exhibits structured rank allocation patterns at multiple granularities and reliably translates additional component activations into effective rank capacity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
PEFT, LoRA, MoE

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_inEfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DMoRE for input-dependent rank allocation in LoRA. Rank-1 components are activated by thresholded self-routing, the rank budget is estimated from the effective dimensionality of the gradient, and a co-activation-aware disentanglement regularizer encourages distinct transformations. Experiments on GLUE, natural language reasoning, and image classification show that DMoRE outperforms several LoRA variants with a similar parameter count.

Critical strengths
  1. Studying input-dependent rank allocation for PEFT is a reasonable direction, and evaluating the method across NLU, reasoning, and vision benchmarks provides broad task coverage.

  2. The rank-utilization analysis relating activation count to effective rank is a useful diagnostic for checking whether extra component activations actually add capacity.

Critical weaknesses
  1. In Table 1, why is the performance of almost all MoE-LoRA methods even worse than original LoRA? This contradicts previous work and may result from improper hyperparameter tuning. The same issue also appears in Table 2 (e.g. AdaLoRA drops to 18.48 on GSM8K vs. 34.49 for LoRA). Besides, the improvements of DMoRE seem very marginal on many tasks (e.g., MNLI, QNLI, QQP, and SST-2) on both backbones. These issues make the empirical comparison much less convincing.

  2. Several core claims are not well supported by the paper's own results and analysis:

    • Neglected inference overhead: The abstract and introduction emphasize that LoRA introduces no inference latency because updates can be merged into the base weights. However, DMoRE uses input-dependent gating gDMoRE(x)g_{\text{DMoRE}}(x), so its weights cannot be merged at inference time. Table 4 shows DMoRE increases per-token inference latency by 2.65×2.65\times (0.45 ms vs. 0.17 ms for LoRA) and training time by 1.49×1.49\times, which undermines a primary advantage of LoRA.

    • Disconnected rank disentanglement contribution: Although "Disentangled" is the centerpiece of the title and Section 3.1, Table 5c shows that adding LrankL_{\text{rank}} over bare PΛQP\Lambda Q changes average accuracy by only +0.07+0.07 (61.04 vs. 61.11), and varying ηrank\eta_{\text{rank}} across two orders of magnitude in Table 5d (from 10010^0 to 10−210^{-2}) yields virtually identical scores (61.10, 61.11, 61.13). This suggests that the proposed disentanglement regularizer contributes almost nothing to downstream task performance.

    • Scale-sensitivity in self-routing: Routing by ∣ΛQx∣>τ|\Lambda Q x| > \tau while regularizing P⊤P≈IP^\top P \approx I and QQ⊤≈IQ Q^\top \approx I creates a coupling issue: the model can scale down ∥pi∥\lVert p_i \rVert and inflate ∣λi∣|\lambda_i| to keep the rank-1 update λipiqi⊤\lambda_i p_i q_i^\top unchanged while arbitrarily manipulating the routing score ∣λiqi⊤x∣|\lambda_i q_i^\top x|.

Questions for authors
  1. In Table 1, HydraLoRA, LD-MoLE, SMoRA, and FlyLoRA all underperform vanilla LoRA on both RoBERTa-B and DeBERTa-v3-L (and similarly in Table 2). How were these baselines tuned?

  2. In Table 5c and 5d, removing LrankL_{\text{rank}} or changing ηrank\eta_{\text{rank}} by 100×100\times barely affects accuracy (≤0.09%\le 0.09\% difference). How can the authors justify claiming co-activation-aware rank disentanglement as a primary contribution when its empirical effect is negligible?

  3. Because gDMoRE(x)g_{\text{DMoRE}}(x) is input-dependent, DMoRE cannot be merged into WW, leading to 2.65×2.65\times higher per-token inference latency than LoRA in Table 4. Given the marginal gains on many benchmarks (and degradation on MMLU), why is this unmergeable design preferable to mergeable LoRA variants in practice?

  4. In Eq. (5) and Eq. (8), LrankL_{\text{rank}} only constrains PP and QQ toward unit-norm columns/rows, leaving Λ\Lambda unconstrained. What prevents the optimizer from rescaling (λi,pi)→(cλi,pi/c)(\lambda_i, p_i) \to (c\lambda_i, p_i/c) to bypass the threshold τi\tau_i without changing the effective transformation λipiqi⊤\lambda_i p_i q_i^\top?

5 comments

We thank the reviewer for the insightful feedback.


W1: Performance Drop on NLU. We compare our NLU setup with the closest experimental settings reported in the mentioned MoE-LoRA papers. None reports single-task GLUE fine-tuning with encoder-only backbones such as RoBERTa-B. Their reported advantages therefore need not generalize to our NLU setting.

MethodSingle-task LearningTraining DataEvaluation DataBackbone
HydraLoRA✗FLAN-v2BBHdecoder-only LLMs
LD-MoLE✓CoLA, RTECoLA, RTEdecoder-only LLMs
SMoRA✗FLAN-v2FLAN-v2decoder-only LLMs
FlyLoRA----
DMoRE (ours)✓CoLA, MNLI, MRPC, QNLI, QQP, SST-2, STS-BCoLA, MNLI, MRPC, QNLI, QQP, SST-2, STS-BRoBERTa-B, DeBERTa-v3-L

W1: Performance Drop on NLR. We compare our NLR setup against the closest experimental settings reported in the referenced MoE-LoRA studies. To ensure a comprehensive and up-to-date comparison, we extend the evaluation to seven datasets using recent Llama and Qwen backbones. Tab. 2 shows that the mentioned MoE-LoRA methods show dataset- and backbone-dependent relative performance gain. For instance, on Qwen3-8B, HydraLoRA and FlyLoRA outperform LoRA on average over benchmarks evaluated in their original papers, but such gain does not persist across the full range of tasks. Moreover, although LD-MoLE adopts a similar experimental setup, its original experiment does not compare against vanilla LoRA, leaving its advantage over this baseline unestablished.

MethodSingle-task LearningShared Training DataShared Evaluation DataShared BackbonesOther Backbones
HydraLoRA✓GSM8K, CodeAlpacaMMLU, GSM8K, HE-Llama-2-7B
LD-MoLE✓ARC-C, HSARC-C, HSLlama-3.2-3BLlama-3.1-8B, Qwen3-1.7B
SMoRA✗GSM8K and CodeAlpaca mixedMMLU, GSM8K, HE-Llama-2-7B
FlyLoRA✓MMLU, GSM8K, CodeAlpaca-20KMMLU, GSM8K, HE-Llama-3.1-8B, Qwen-2.5-7B
DMoRE (ours)✓ARC-C, BoolQ, GSM8K, HS, MMLU, WG, CodeAlpaca-20KARC-C, BoolQ, GSM8K, HS, HE, MMLU, WGLlama-3.2-3B, Qwen3-8B-

W1: Marginal Performance. Performance gains vary on individual tasks due to differences in dataset size and task difficulty. DMoRE consistently achieves the highest performance across multiple seeds. Also, it exceeds the strongest competing baseline while using approximately the same number of trainable parameters as LoRA.

W1 & Q1: Hyperparameter Tuning. We adopt a common training protocol adapted from [1] for controlled comparisons. We use method-specific hyperparameters from their papers or official codebases for each baseline (L257, App. B). To further assess whether DMoRE's advantage persists under method- and task-specific tuning, we conduct an additional validation-based search using the same candidate learning rates for all methods on each backbone (i.e., {1e-3, 5e-4, 1e-4, 5e-5} on RoBERTa-B for NLU and {5e-4, 1e-4, 5e-5, 1e-5} on Llama-3.2-3B for NLR). DMoRE retains the highest average performance in both evaluated settings.

MethodCoLAMNLIMRPCQNLIQQPSST-2STSBAvg.
LoRA81.6985.4889.7091.0986.3694.0490.4388.40
HydraLoRA81.3086.0089.4992.4986.7194.3889.1788.51
LD-MoLE82.8985.3990.0792.1386.5894.6190.3788.86
SMoRA83.7085.3289.4992.2886.4994.2790.1888.82
FlyLoRA81.5084.0890.4690.7985.0193.4689.4287.82
DMoRE83.3286.0990.0592.4987.1194.6190.5289.17
MethodARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
LoRA52.0588.9636.8578.9924.3955.9883.9860.17
HydraLoRA54.7888.3242.3878.9725.2055.8883.2161.25
LD-MoLE54.2789.2740.7978.9725.0056.0882.8761.04
SMoRA54.3589.3040.5679.3225.6155.7083.4361.18
FlyLoRA52.6589.5440.7178.6224.3956.0282.4860.63
DMoRE54.4489.6042.9179.0025.4155.8884.0661.61

W2 & Q3: Inference Latency. In Tab. 4, we profile methods without runtime optimization to enable controlled comparisons and minimize optimization-induced confounding effects. To better reflect practical deployment, we additionally measure per-token latency (ms) with commonly used inference optimizations, including KV caching, compilation, and CUDA Graphs. With these general-purpose optimizations alone, DMoRE already approaches the latency of merged LoRA, supporting its practical deployment. Specialized kernels could further reduce the remaining overhead. We leave their development to future work.

MethodEagerOptimizedNLR Avg.
LoRA (merged)0.170.0759.21
DMoRE0.450.0961.13

Q3: Multi-task Learning. DMoRE preserves the multi-task adaptation benefits of MoE-based LoRA variants. To substantiate this practical benefit, we extend the experimental setup in Sec. 4.5 to multi-task learning, constructing a shared training corpus by randomly sampling up to 5,000 training examples per task. DMoRE achieves the highest average performance among the evaluated baselines, including other MoE-based LoRA variants specifically designed for multi-task learning.

MethodARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
LoRA50.0084.5333.0677.8524.3955.2477.8257.56
AdaLoRA48.4678.5325.7875.3120.1254.1273.7253.72
TopLoRA52.9985.9037.1578.4022.5654.9380.3558.90
LD-MoLE54.8685.6035.5678.0224.3954.2480.5159.03
SMoRA54.2785.6736.6977.8925.0055.1677.1958.84
DMoRE53.8486.3939.9577.9223.1755.0780.6659.58

W2 & Q2: Rank Disentanglement. The empirical effects of Lrank\mathcal{L}_{\text{rank}} are:

  • Fig. 1b shows that applying Lrank\mathcal{L}_{\text{rank}} mitigates rank collapse by narrowing the gap between activation counts and realized effective rank (L356).
  • Tab. 5c shows that Lrank\mathcal{L}_{\text{rank}} preserves downstream task performance while global orthogonality constraints lead to performance degradation (L466).

The marginal performance gain in Tab. 5c may be attributed to our orthonormal initialization of Q\boldsymbol{Q}, which satisfies the targeted constraints before training and limits the benefits from further regularization. To examine this, we follow the ablation study setup in Sec. 4.5 and compare the effects of rank disentanglement under Kaiming uniform initialization of Q\boldsymbol{Q}. In this case, applying our rank disentanglement regularization yields a clearer performance gain (+1.84 percentage points), and the resulting model outperforms the baseline using global orthonormal regularization.

CaseARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
PΛQ\boldsymbol{P}\boldsymbol{\Lambda}\boldsymbol{Q}48.8988.2336.0179.0823.1755.5783.6659.23
PΛQ\boldsymbol{P}\boldsymbol{\Lambda}\boldsymbol{Q} ++ Lortho\mathcal{L}_{\text{ortho}}49.5288.1835.3879.0125.2055.9183.5859.54
PΛQ\boldsymbol{P}\boldsymbol{\Lambda}\boldsymbol{Q} ++ Lrank\mathcal{L}_{\text{rank}} (current)53.7589.3840.9679.0824.8055.8783.6361.07

W2 & Q4: Scale Sensitivity. The unit-norm penalty in Lrank\mathcal{L}_{\text{rank}} discourages the proposed rescaling without guaranteeing exact scale invariance. To investigate the benefit of scale invariance, we compare DMoRE with a routing baseline (∥pi∥2∣λiqi⊤x∣=∥λipiqi⊤x∥2\Vert p_i\Vert_2\vert\lambda_iq_i^\top x\vert =\|\lambda_ip_iq_i^\top x\|_2) that is invariant to the proposed compensating rescaling, following the setup in Sec. 4.5. Applying scale-invariant routing provides no clear performance gain but incurs additional training overhead by requiring ∥pi∥2\Vert p_i\Vert_2 to be recomputed after each update to P\boldsymbol{P}.

RoutingARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
∥pi∥2∣λiqi⊤x∣\Vert p_i\Vert_2\vert\lambda_iq_i^\top x\vert54.6989.4340.9278.6724.5954.7083.6160.95
∣λiqi⊤x∣\vert\lambda_iq_i^\top x\vert (current)54.4989.3141.9878.7025.4154.2483.8261.13

[1] Wang, S., Yu, L., & Li, J. (2024). LoRA-GA: Low-rank adaptation with gradient approximation. Advances in Neural Information Processing Systems, 37, 54905-54931.

Thanks for the extra experiments and clarifications. Even under fairer tuning, DMoRE's performance gains remain marginal, so it's still unclear whether the proposed disentanglement and routing components actually make much difference. I also share Reviewer bamu's concern that the work lacks sufficient novelty and a clear conceptual advantage over existing rank-1 and self-gated MoE-LoRA methods, so I'll keep my original score.

Reviewer_bamuOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Disentangled Mixture of Rank-1 Experts (DMoRE), a framework for input-dependent rank allocation in Low-Rank Adaptation (LoRA). The method decomposes LoRA updates into rank-1 components and introduces three mechanisms: (1) dynamic self-routing, which activates components based on their latent response magnitudes without an external router; (2) gradient-derived budget estimation, which uses the effective rank of a candidate gradient to control average component activation; and (3) co-activation-aware rank disentanglement, which encourages frequently co-activated components to learn distinct transformation directions. Experiments across natural language understanding, reasoning, and image classification demonstrate improvements over several existing LoRA variants.

Critical strengths
  1. Extensive experimental evaluation. The paper evaluates DMoRE across different model architectures, scales, and modalities, including language and vision tasks. The reported results demonstrate relatively consistent improvements over existing LoRA variants, supported by ablation studies and analyses of rank allocation.

  2. Simple and reasonably well-motivated design. The proposed self-routing mechanism reuses the existing low-rank projections without introducing an additional router network. The paper also provides analyses of component utilization, effective rank, and computational overhead, which help characterize the behavior of the proposed method.

Critical weaknesses
  1. Limited methodological novelty and substantial overlap with prior work.

The proposed framework appears to combine several techniques already explored in the literature. In particular, Gated LoRA [1] already introduces rank-1 LoRA components as mini-experts, input-dependent self-gating through down-projected features without an external router, and orthogonality regularization to reduce redundancy. Its ReLU-based gating also supports variable numbers of active components. Similarly, SMoRA [2] has explored rank-wise expert activation, while RaLoRA [3] uses gradient intrinsic dimensionality to guide adaptive rank allocation. The main distinctions of DMoRE are therefore largely confined to its particular budget controller and co-activation-weighted orthogonality constraint. More importantly, the paper does not provide a direct empirical comparison with Gated LoRA, arguably its closest methodological predecessor. Without such comparisons, the novelty and significance of the proposed modifications remain insufficiently established.

[1] Eom, S., Yoon, H. S., Yoon, E., Hasegawa-Johnson, M. A., & Yoo, C. D. (2025). Gated LoRA: Dual-Purpose Projections for Parameter-Efficient Mini-Expert Fine-Tuning. OpenReview, ICLR 2026 submission. https://openreview.net/forum?id=ZiBDVotA7g

[2] Zhao, Z., et al. (2026). Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning. Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2026), pp. 1998–2009. https://doi.org/10.1145/3770854.3780222

[3] Ye, J., He, H., Li, M., Han, F., Chen, T., & Ye, P. (2026). Gradient Intrinsic Dimensionality Alignment: Narrowing the Gap Between Low-Rank Adaptation and Full Fine-Tuning. International Conference on Learning Representations (ICLR 2026). https://proceedings.iclr.cc/paper_files/paper/2026/hash/5b4b967d4222d87fa5b28b6ec7144058-Abstract-Conference.html

  1. Insufficient theoretical justification and empirical validation of the proposed mechanisms.

The paper motivates rank disentanglement by arguing that co-activated rank-1 components receive collinear gradients and therefore tend to learn redundant transformations. However, Eq. (3) only establishes gradient alignment at the induced component-update level, which does not necessarily imply redundancy among the actual trainable low-rank factors. The same observation also applies to conventional LoRA, making the proposed failure mode insufficiently justified. Moreover, Table 5c shows that the proposed disentanglement regularizer improves average performance by only 0.07 percentage points (61.04 to 61.11), raising questions about its practical significance. The gradient-derived budget is likewise insufficiently justified as a measure of optimal adaptation capacity, and the fixed-budget comparison does not control for average activation count. Finally, DMoRE introduces approximately 2.65× the inference latency of standard LoRA, further weakening its claimed efficiency advantages.

Questions for authors
  1. Given the substantial architectural similarities between DMoRE and Gated LoRA [1], could the authors clarify the fundamental methodological novelty and provide a direct empirical comparison?

  2. How does the collinearity of induced component gradients in Eq. (3) establish that the actual learned low-rank factors become redundant? Can the authors provide stronger theoretical or empirical evidence for the proposed rank-collapse phenomenon?

  3. Given the marginal performance improvement from the proposed rank disentanglement regularizer, what evidence supports its importance? Additionally, can the authors compare gradient-derived budget estimation against fixed budgets under matched average activation counts?

  4. Considering the increased inference latency and inability to merge the adapter weights, what practical computational advantages does the proposed dynamic rank activation provide?

[1] Eom, S., Yoon, H. S., Yoon, E., Hasegawa-Johnson, M. A., & Yoo, C. D. (2025). Gated LoRA: Dual-Purpose Projections for Parameter-Efficient Mini-Expert Fine-Tuning. OpenReview, ICLR 2026 submission. https://openreview.net/forum?id=ZiBDVotA7g

4 comments

We thank the reviewer for the thoughtful feedback.


W1 & Q1: Novelty. DMoRE explicitly builds on the rank-wise MoE-LoRA formulation, motivated by its inherent potential for input-dependent rank allocation (L36-44). None of the cited methods fully realizes this potential. Our contribution lies in identifying the obstacles to effective input-dependent rank allocation and jointly addressing them through targeted designs, as detailed below.

MethodVariable Component ActivationAdaptive Budget EstimationRank DisentanglementBalanced Component Utilization
SMoRA✗✗✗✓
RaLoRA✗✓✗✗
Gated LoRA✓✗?✗
DMoRE (Ours)✓✓✓✓
  • vs. SMoRA: SMoRA relies on an explicit router to activate a fixed number of rank-11 components per input, making it inherently incapable of input-dependent rank allocation (L139). In contrast, DMoRE realizes it through self-routing with adaptive budget estimation. Co-activation-aware disentanglement further helps translate additional component activations into effective-rank capacity.
  • vs. RaLoRA: RaLoRA assigns module-specific ranks based on gradient intrinsic dimensionality in an input-independent manner, limiting its ability to capture semantic variations across inputs (L102). In contrast, DMoRE enables input-dependent rank allocation independently within each adapted module. This allows module-level rank allocation patterns to emerge naturally without additional architectural modifications or manually designed heuristics (L44, Fig. 3).
  • vs. Gated LoRA: Variable component activation alone does not guarantee effective input-dependent rank allocation because:
    • Manually specified budget: Its ReLU variant relies on a predefined target activation ratio and penalizes over-activation. This may require additional tuning effort to accommodate task-specific adaptation demands. In contrast, DMoRE adaptively estimates the budget and its effectiveness is evaluated in Tab. 5b and further discussed in our response to W3 & Q3.
    • Insufficient rank disentanglement: Its orthogonality regularization acts only on the up-projection B\boldsymbol{B}, leaving redundancy in the down-projection A\boldsymbol{A} unconstrained. Orthogonal output directions alone do not imply that activation counts accurately reflect realized effective rank (L951). Instead, DMoRE regularizes both projections and incorporates component co-activation to preserve the flexibility for task-driven component specialization.
    • Unbalanced component utilization: Gated LoRA provides no explicit load-balancing mechanism. DMoRE instead realizes this through threshold updates. Removing component-wise load balancing while retaining the same rank budget reduces average NLR performance by 0.92 percentage points (Fig. 1c, Sec. 4.4).

W1 & Q1: Empirical Comparison. A direct empirical comparison was omitted because no official implementation of Gated LoRA was publicly available (L897). We now compare DMoRE with our best-effort reimplementation, using its paper-recommended hyperparameters under the setup in Sec. 4.5. DMoRE achieves a higher average performance and outperforms the reimplementation on most evaluated tasks.

MethodARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
Gated LoRA52.0888.1838.7778.7625.4150.6480.9559.25
DMoRE54.4989.3141.9878.7025.4154.2483.8261.13

W2 & Q2: Rank Collapse. Eq. (3) motivates the risk of learning correlated transformations under repeated co-activation (L45, L166). It does not directly establish the redundancy among learned factors. The rank collapse phenomenon is empirically observed in Fig. 1b, where a mismatch between the number of active rank-11 components and the effective rank of the resultant transformation is shown. The collapse is relevant for dynamic rank allocation, where activation counts are used to control transformation capacity. It is not a failure mode of LoRA with a static rank.

W2 & Q3: Rank Disentanglement. The empirical effects of Lrank\mathcal{L}_{\text{rank}} are:

  • Fig. 1b shows that applying Lrank\mathcal{L}_{\text{rank}} mitigates rank collapse by narrowing the gap between activation counts and realized effective rank (L356).
  • Tab. 5c shows that Lrank\mathcal{L}_{\text{rank}} preserves downstream task performance while global orthogonality constraints lead to performance degradation (L466).

The marginal performance gain in Tab. 5c may be attributed to our orthonormal initialization of Q\boldsymbol{Q}, which satisfies the targeted constraints before training and limits the benefits from further regularization. To examine this, we follow the ablation study setup in Sec. 4.5 and compare the effects of rank disentanglement under Kaiming uniform initialization of Q\boldsymbol{Q}. In this case, applying our rank disentanglement regularization yields a clearer performance gain (+1.84 percentage points), and the resulting model outperforms the baseline using global orthonormal regularization.

CaseARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
PΛQ\boldsymbol{P}\boldsymbol{\Lambda}\boldsymbol{Q}48.8988.2336.0179.0823.1755.5783.6659.23
PΛQ\boldsymbol{P}\boldsymbol{\Lambda}\boldsymbol{Q} ++ Lortho\mathcal{L}_{\text{ortho}}49.5288.1835.3879.0125.2055.9183.5859.54
PΛQ\boldsymbol{P}\boldsymbol{\Lambda}\boldsymbol{Q} ++ Lrank\mathcal{L}_{\text{rank}} (current)53.7589.3840.9679.0824.8055.8783.6361.07

W3 & Q3: Budget Estimation. The gradient-derived budget is proposed as a proxy for rank-capacity demand (L255) and we make no claim of optimality. Its effectiveness is evaluated through the budget-estimator ablation in Tab. 5b and the additional fixed-budget comparison below. We construct the fixed-budget baseline with matched average activation counts by (i) running the same DMoRE checkpoints reported as "gradient-based" over the full training set in inference mode; (ii) measuring each adapted module's average activation count; and (iii) training a new DMoRE adapter using these measurements as fixed, module-specific target budgets. Gradient-based budgeting outperforms the fixed-budget baseline on most evaluated tasks and achieves a higher average performance, supporting the benefit of adapting budget estimation during training.

BudgetARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
Fixed + matched50.1488.2635.4579.0822.5655.7183.5659.25
Gradient-based (current)54.4989.3141.9878.7025.4154.2483.8261.13

Q4: Inference Latency. In Tab. 4, we profile methods without runtime optimization to enable controlled comparisons and minimize optimization-induced confounding effects. To better reflect practical deployment, we additionally measure per-token latency (ms) with commonly used inference optimizations, including KV caching, compilation, and CUDA Graphs. With these general-purpose optimizations alone, DMoRE already approaches the latency of merged LoRA, supporting its practical deployment. Specialized kernels could further reduce the remaining overhead. We leave their development to future work.

MethodEagerOptimizedNLR Avg.
LoRA (merged)0.170.0759.21
DMoRE0.450.0961.13

Q4: Multi-task Learning. DMoRE preserves the multi-task adaptation benefits of MoE-based LoRA variants. To substantiate this practical benefit, we extend the experimental setup in Sec. 4.5 to multi-task learning, constructing a shared training corpus by randomly sampling up to 5,000 training examples per task. DMoRE achieves the highest average performance among the evaluated baselines, including other MoE-based LoRA variants specifically designed for multi-task learning.

MethodARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
LoRA50.0084.5333.0677.8524.3955.2477.8257.56
AdaLoRA48.4678.5325.7875.3120.1254.1273.7253.72
TopLoRA52.9985.9037.1578.4022.5654.9380.3558.90
LD-MoLE54.8685.6035.5678.0224.3954.2480.5159.03
SMoRA54.2785.6736.6977.8925.0055.1677.1958.84
DMoRE53.8486.3939.9577.9223.1755.0780.6659.58

Thanks for the detailed reply from authors. I think my concerns have been addressed and would like to raise my score.

Thank you for your constructive feedback and for raising your score. We are glad that our additional discussion has helped clarify the novelty of our work, the multi-task experiments have demonstrated DMoRE’s advantages from multiple perspectives, and the inference-latency analysis has highlighted its practical efficiency. We would be grateful to know if there are any remaining concerns or aspects we could further clarify to strengthen our work.

Reviewer_yMReOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DMoRE, an MoE-LoRA variant that addresses the limitations of the static routing mechanism in the MoE-LoRA setting. Traditional MoE-LoRA assigns a fixed top-k LoRA token to each token, which the authors argue is suboptimal because different inputs and modules require varying capacities. To address this, the authors propose a dynamic routing mechanism.

In this work, DMoRE utilizes a self-routing gating mechanism that allows each rank-1 component to activate itself based on the product of LoRA's down-projection and the input hidden vector, controlled by a learnable threshold, thereby enabling a variable number of active LoRAs per token. To manage the total budget and ensure efficient parameter usage, the framework includes a gradient-derived budget estimation and a specialized disentanglement regularization to prevent rank collapse.

Critical strengths
  1. The shift from a top-k mechanism to a threshold-controlled mechanism allows the model to adapt a variable number of LoRA experts in demand.
  2. By leveraging the internal latent responses as routing signals, the method avoids the computational overhead and training instability associated with adding an auxiliary router network.
  3. This paper presented consistent improvement on both NLU and NLG tasks, and further evaluated on image clasification tasks.
Critical weaknesses
  1. While the paper mentions negligible parameter overhead, the dynamic nature of the architecture means the adapters cannot be merged into the base model weights for deployment. This introduces inevitable latency during inference compared to standard LoRA.
  2. DMoRE introduced multiple losses, trainable parameters, and new hyperparameters to the training framework, making this method less practical than the conventional PEFT approach, such as LoRA.
  3. While the paper explores token-level and module-level patterns, it is less clear how these dynamic patterns evolve over the course of long-context processing or in highly specialized domain-specific tasks.
Questions for authors
  1. The paper mentions zero-initializing the thresholds to ensure all components receive initial gradients. How sensitive is the final convergence to the initial phase before the budget estimation becomes stable?
  2. Given that the disentanglement loss penalizes directional overlap among co-activated experts, how does DMoRE perform in a multi-task setting?
  3. As the activated LoRA increased and saturated as task difficulty increased, will DMoRE eventually become a weight LoRA adaptation rather than a routing-based MoE LoRA? What if the τ\tau keeps zero, but with g(x)g(x) as the weight to control different LoRAs' participation?
3 comments

We thank the reviewer for the constructive feedback.


W1. In Tab. 4, we profile methods without runtime optimization to enable controlled comparisons and minimize optimization-induced confounding effects. To better reflect practical deployment, we additionally measure per-token latency (ms) with commonly used inference optimizations, including KV caching, compilation, and CUDA Graphs. With these general-purpose optimizations alone, DMoRE already approaches the latency of merged LoRA, supporting its practical deployment. Specialized kernels could further reduce the remaining overhead. We leave their development to future work.

MethodEagerOptimizedNLR Avg.
LoRA (merged)0.170.0759.21
DMoRE0.450.0961.13

W2. DMoRE adds one extra loss and two hyperparameters over LoRA, comparable to representative LoRA variants as summarized below. Its parameter overhead over LoRA is only 2r2r per module. Across Tabs. 1-3, DMoRE consistently achieves superior performance using the same method-specific hyperparameters without task-specific tuning. Tabs. 5d-5e further show stable performance across the tested hyperparameter ranges.

MethodExtra LossesExtra Hyperparameters
AdaLoRA17
LD-MoLE25
SMoRA02
DMoRE12

W3. In this work, we focus on developing and validating input-dependent rank allocation. Figs. 1 and 3 characterize the resulting rank allocation behavior at different granularities on different tasks. We acknowledge that the suggested analysis would provide valuable complementary insights. We leave this extension to future work.


Q1. Following the ablation study setup (Sec. 4.5), we assess the sensitivity of final performance to the initial component activation rate by lazy-initializing τ\boldsymbol{\tau} using the first training batch. Across initial activation rates of 25% to 100%, average performance varies by a maximum of 0.07 percentage points, suggesting limited sensitivity. We retain zero initialization of τ\boldsymbol{\tau}, which avoids tuning the initial activation rate and performing batch-dependent calibration.

Initial Activation RateARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
25%54.2789.2341.6279.0424.1955.9084.0361.18
50%54.4789.3341.2779.0423.9855.8583.8261.11
75%54.4989.4141.3279.0123.9855.7583.9561.13
100% (current)54.4989.3141.9878.7025.4154.2483.8261.13

Q2. Following the experimental setup in Sec. 4.5, we examine the performance of DMoRE under multi-task learning and compare it with representative LoRA variants. We construct a shared training corpus by randomly picking up to 5000 training samples from each downstream task. DMoRE achieves the highest average performance, outperforming other MoE-based LoRA baselines that are specifically developed for multi-task learning.

MethodARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
LoRA50.0084.5333.0677.8524.3955.2477.8257.56
AdaLoRA48.4678.5325.7875.3120.1254.1273.7253.72
TopLoRA52.9985.9037.1578.4022.5654.9380.3558.90
LD-MoLE54.8685.6035.5678.0224.3954.2480.5159.03
SMoRA54.2785.6736.6977.8925.0055.1677.1958.84
DMoRE53.8486.3939.9577.9223.1755.0780.6659.58

Q3. If all components remain active for all inputs, DMoRE yields $ \boldsymbol{y}=\boldsymbol{W}\boldsymbol{x}+\frac{\alpha}{r}\boldsymbol{P}\boldsymbol{\Lambda}\boldsymbol{Q}\boldsymbol{x}, $ which is functionally equivalent to an input-independent three-factor LoRA, with the disentanglement loss reducing to global orthogonality regularization. In practice, we do not observe those saturated scenarios: Fig. 3 shows that effective rank does not uniformly saturate at maximum capacity across the evaluated tasks. g(x)\boldsymbol{g}(\boldsymbol{x}) is binary in DMoRE, and replacing it with continuous input-dependent weights would define a different soft-gating variant.

Thanks for your replies. After reading your response, I will keep my rating at 3.

We thank the reviewer for this feedback. We will include those additional results in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.