- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DMoRE for input-dependent rank allocation in LoRA. Rank-1 components are activated by thresholded self-routing, the rank budget is estimated from the effective dimensionality of the gradient, and a co-activation-aware disentanglement regularizer encourages distinct transformations. Experiments on GLUE, natural language reasoning, and image classification show that DMoRE outperforms several LoRA variants with a similar parameter count.
Critical strengths
-
Studying input-dependent rank allocation for PEFT is a reasonable direction, and evaluating the method across NLU, reasoning, and vision benchmarks provides broad task coverage.
-
The rank-utilization analysis relating activation count to effective rank is a useful diagnostic for checking whether extra component activations actually add capacity.
Critical weaknesses
-
In Table 1, why is the performance of almost all MoE-LoRA methods even worse than original LoRA? This contradicts previous work and may result from improper hyperparameter tuning. The same issue also appears in Table 2 (e.g. AdaLoRA drops to 18.48 on GSM8K vs. 34.49 for LoRA). Besides, the improvements of DMoRE seem very marginal on many tasks (e.g., MNLI, QNLI, QQP, and SST-2) on both backbones. These issues make the empirical comparison much less convincing.
-
Several core claims are not well supported by the paper's own results and analysis:
-
Neglected inference overhead: The abstract and introduction emphasize that LoRA introduces no inference latency because updates can be merged into the base weights. However, DMoRE uses input-dependent gating , so its weights cannot be merged at inference time. Table 4 shows DMoRE increases per-token inference latency by (0.45 ms vs. 0.17 ms for LoRA) and training time by , which undermines a primary advantage of LoRA.
-
Disconnected rank disentanglement contribution: Although "Disentangled" is the centerpiece of the title and Section 3.1, Table 5c shows that adding over bare changes average accuracy by only (61.04 vs. 61.11), and varying across two orders of magnitude in Table 5d (from to ) yields virtually identical scores (61.10, 61.11, 61.13). This suggests that the proposed disentanglement regularizer contributes almost nothing to downstream task performance.
-
Scale-sensitivity in self-routing: Routing by while regularizing and creates a coupling issue: the model can scale down and inflate to keep the rank-1 update unchanged while arbitrarily manipulating the routing score .
-
Questions for authors
-
In Table 1, HydraLoRA, LD-MoLE, SMoRA, and FlyLoRA all underperform vanilla LoRA on both RoBERTa-B and DeBERTa-v3-L (and similarly in Table 2). How were these baselines tuned?
-
In Table 5c and 5d, removing or changing by barely affects accuracy ( difference). How can the authors justify claiming co-activation-aware rank disentanglement as a primary contribution when its empirical effect is negligible?
-
Because is input-dependent, DMoRE cannot be merged into , leading to higher per-token inference latency than LoRA in Table 4. Given the marginal gains on many benchmarks (and degradation on MMLU), why is this unmergeable design preferable to mergeable LoRA variants in practice?
-
In Eq. (5) and Eq. (8), only constrains and toward unit-norm columns/rows, leaving unconstrained. What prevents the optimizer from rescaling to bypass the threshold without changing the effective transformation ?
5 comments
W2 & Q3: Inference Latency. In Tab. 4, we profile methods without runtime optimization to enable controlled comparisons and minimize optimization-induced confounding effects. To better reflect practical deployment, we additionally measure per-token latency (ms) with commonly used inference optimizations, including KV caching, compilation, and CUDA Graphs. With these general-purpose optimizations alone, DMoRE already approaches the latency of merged LoRA, supporting its practical deployment. Specialized kernels could further reduce the remaining overhead. We leave their development to future work.
| Method | Eager | Optimized | NLR Avg. |
|---|---|---|---|
| LoRA (merged) | 0.17 | 0.07 | 59.21 |
| DMoRE | 0.45 | 0.09 | 61.13 |
Q3: Multi-task Learning. DMoRE preserves the multi-task adaptation benefits of MoE-based LoRA variants. To substantiate this practical benefit, we extend the experimental setup in Sec. 4.5 to multi-task learning, constructing a shared training corpus by randomly sampling up to 5,000 training examples per task. DMoRE achieves the highest average performance among the evaluated baselines, including other MoE-based LoRA variants specifically designed for multi-task learning.
| Method | ARC-C | BoolQ | GSM8K | HellaSwag | HumanEval | MMLU | WinoGrande | Avg. |
|---|---|---|---|---|---|---|---|---|
| LoRA | 50.00 | 84.53 | 33.06 | 77.85 | 24.39 | 55.24 | 77.82 | 57.56 |
| AdaLoRA | 48.46 | 78.53 | 25.78 | 75.31 | 20.12 | 54.12 | 73.72 | 53.72 |
| TopLoRA | 52.99 | 85.90 | 37.15 | 78.40 | 22.56 | 54.93 | 80.35 | 58.90 |
| LD-MoLE | 54.86 | 85.60 | 35.56 | 78.02 | 24.39 | 54.24 | 80.51 | 59.03 |
| SMoRA | 54.27 | 85.67 | 36.69 | 77.89 | 25.00 | 55.16 | 77.19 | 58.84 |
| DMoRE | 53.84 | 86.39 | 39.95 | 77.92 | 23.17 | 55.07 | 80.66 | 59.58 |
W2 & Q2: Rank Disentanglement. The empirical effects of are:
- Fig. 1b shows that applying mitigates rank collapse by narrowing the gap between activation counts and realized effective rank (L356).
- Tab. 5c shows that preserves downstream task performance while global orthogonality constraints lead to performance degradation (L466).
The marginal performance gain in Tab. 5c may be attributed to our orthonormal initialization of , which satisfies the targeted constraints before training and limits the benefits from further regularization. To examine this, we follow the ablation study setup in Sec. 4.5 and compare the effects of rank disentanglement under Kaiming uniform initialization of . In this case, applying our rank disentanglement regularization yields a clearer performance gain (+1.84 percentage points), and the resulting model outperforms the baseline using global orthonormal regularization.
| Case | ARC-C | BoolQ | GSM8K | HellaSwag | HumanEval | MMLU | WinoGrande | Avg. |
|---|---|---|---|---|---|---|---|---|
| 48.89 | 88.23 | 36.01 | 79.08 | 23.17 | 55.57 | 83.66 | 59.23 | |
| 49.52 | 88.18 | 35.38 | 79.01 | 25.20 | 55.91 | 83.58 | 59.54 | |
| (current) | 53.75 | 89.38 | 40.96 | 79.08 | 24.80 | 55.87 | 83.63 | 61.07 |
W2 & Q4: Scale Sensitivity. The unit-norm penalty in discourages the proposed rescaling without guaranteeing exact scale invariance. To investigate the benefit of scale invariance, we compare DMoRE with a routing baseline () that is invariant to the proposed compensating rescaling, following the setup in Sec. 4.5. Applying scale-invariant routing provides no clear performance gain but incurs additional training overhead by requiring to be recomputed after each update to .
| Routing | ARC-C | BoolQ | GSM8K | HellaSwag | HumanEval | MMLU | WinoGrande | Avg. |
|---|---|---|---|---|---|---|---|---|
| 54.69 | 89.43 | 40.92 | 78.67 | 24.59 | 54.70 | 83.61 | 60.95 | |
| (current) | 54.49 | 89.31 | 41.98 | 78.70 | 25.41 | 54.24 | 83.82 | 61.13 |
[1] Wang, S., Yu, L., & Li, J. (2024). LoRA-GA: Low-rank adaptation with gradient approximation. Advances in Neural Information Processing Systems, 37, 54905-54931.
Thanks for the extra experiments and clarifications. Even under fairer tuning, DMoRE's performance gains remain marginal, so it's still unclear whether the proposed disentanglement and routing components actually make much difference. I also share Reviewer bamu's concern that the work lacks sufficient novelty and a clear conceptual advantage over existing rank-1 and self-gated MoE-LoRA methods, so I'll keep my original score.
We thank the reviewer for the insightful feedback.
W1: Performance Drop on NLU. We compare our NLU setup with the closest experimental settings reported in the mentioned MoE-LoRA papers. None reports single-task GLUE fine-tuning with encoder-only backbones such as RoBERTa-B. Their reported advantages therefore need not generalize to our NLU setting.
W1: Performance Drop on NLR. We compare our NLR setup against the closest experimental settings reported in the referenced MoE-LoRA studies. To ensure a comprehensive and up-to-date comparison, we extend the evaluation to seven datasets using recent Llama and Qwen backbones. Tab. 2 shows that the mentioned MoE-LoRA methods show dataset- and backbone-dependent relative performance gain. For instance, on Qwen3-8B, HydraLoRA and FlyLoRA outperform LoRA on average over benchmarks evaluated in their original papers, but such gain does not persist across the full range of tasks. Moreover, although LD-MoLE adopts a similar experimental setup, its original experiment does not compare against vanilla LoRA, leaving its advantage over this baseline unestablished.
W1: Marginal Performance. Performance gains vary on individual tasks due to differences in dataset size and task difficulty. DMoRE consistently achieves the highest performance across multiple seeds. Also, it exceeds the strongest competing baseline while using approximately the same number of trainable parameters as LoRA.
W1 & Q1: Hyperparameter Tuning. We adopt a common training protocol adapted from [1] for controlled comparisons. We use method-specific hyperparameters from their papers or official codebases for each baseline (L257, App. B). To further assess whether DMoRE's advantage persists under method- and task-specific tuning, we conduct an additional validation-based search using the same candidate learning rates for all methods on each backbone (i.e., {1e-3, 5e-4, 1e-4, 5e-5} on RoBERTa-B for NLU and {5e-4, 1e-4, 5e-5, 1e-5} on Llama-3.2-3B for NLR). DMoRE retains the highest average performance in both evaluated settings.