- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
DimLoRA introduces dimension-level LoRA experts for multi-task adaptation. It first groups and reorders FFN neurons, then assigns each expert to a subset of the intermediate dimensions rather than allowing every expert to adapt the entire FFN. The method combines this localized architecture with task-aware sparse routing and a mutual-information regularizer intended to encourage task specialization while maintaining balanced expert utilization. Experiments on LLaMA3.1-8B and Qwen3-14B-Base evaluate multi-task performance, parameter efficiency, runtime, and continual-learning behavior.
Critical strengths
The dimension-level expert design is intuitive and addresses a clear efficiency problem in LoRA-MoE adaptation. The paper provides a concrete parameter-count argument, and the neuron permutation preserves the underlying FFN computation before adaptation. I find the distinction between adapting localized FFN dimensions and introducing multiple full-layer LoRA modules useful.
The experiments show meaningful effort. They include two backbones and several relevant LoRA-MoE baselines, including HydraLoRA, MTL-LoRA, and SMoRA. The component ablations, alternative grouping strategies, and runtime measurements help explain the practical trade-offs. It is also useful that the paper evaluates a version without task-aware routing and reports that the dimension-level architecture remains competitive in that setting.
Critical weaknesses
-
Related work should more clearly separate localized adaptation from existing routing and specialization mechanisms. Please add and discuss HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA Experts (ICLR 2025), Hybrid Routing for a Mixture of LoRA Experts / HotMoE (AAAI 2026), and Mixture-of-Subspaces in Low-Rank Adaptation (EMNLP 2024). These are relevant to task-aware routing, expert specialization, and finer-grained adaptation. The discussion should also include Mixture of LoRA Experts (ICLR 2024) and LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts (ICLR 2026). The distinctive contribution here appears to be localization to grouped FFN dimensions; that should be distinguished from improvements obtained through learned routing or adapter composition.
-
Targeted experimental controls are needed to isolate the benefit of dimension-level experts. I would prioritize an experimental comparison with one recent task-aware LoRA-expert method, such as HMoRA or HotMoE, under a comparable parameter budget. This would help determine whether the gains remain against stronger routing and specialization mechanisms, rather than only against earlier LoRA-MoE designs. A particularly informative diagnostic would use full-layer LoRA experts with the same task-aware router and mutual-information objective as DimLoRA, adjusting ranks to match the trainable-parameter budget. This is a proposed control rather than a published method. It would isolate the architectural contribution more directly than removing individual components from DimLoRA. The existing ablations are useful, but they do not fully answer this comparison.
Questions for authors
1, Can you double check the normalization used in the mutual-information regularizer? I feel a bit confuse there so if you could make it clear it will be good.
2, Does dimension-level adaptation remain advantageous when full-layer experts use the same router and regularizer at a matched parameter budget?
3, Can you provide a recent task-aware routing comparison, uncertainty estimates, and clarification of task-identity availability?
4, See weakness
3 comments
Rebuttal from Authors of Submission26360 to Reviewer 5pAe (2/2)
(4) Uncertainty Estimates
To evaluate the robustness of DimLoRA, we conduct additional experiments using three independent random seeds on LLaMA-3.1-8B with our multi-domain benchmark. We select standard LoRA, MoLORA (the strongest baseline in our original comparison), and SMoRA (a representative fine-grained LoRA-MoE baseline) for comparison. The following table reports the mean accuracy and standard deviation (Mean ± SD) across three runs.
| Method | Genaral | Math | Code | Law | Medicine | Avg. | %Param | #A | #B |
|---|---|---|---|---|---|---|---|---|---|
| LoRA | 64.57 ± 0.34 | 58.38 ± 0.65 | 35.77 ± 3.73 | 43.00 ± 1.87 | 69.18 ± 0.91 | 54.18 ± 0.49 | 0.700 | 1 | 1 |
| MoLORA | 65.63 ± 0.79 | 58.88 ± 0.32 | 39.84 ± 1.96 | 43.37 ± 1.76 | 70.23 ± 1.15 | 55.59 ± 0.30 | 0.736 | 4 | 4 |
| SMoRA | 65.29 ± 0.18 | 58.68 ± 0.50 | 39.43 ± 0.70 | 43.25 ± 1.68 | 70.19 ± 0.71 | 55.37 ± 0.46 | 0.493 | 16 | 16 |
| DimLoRA | 65.54 ± 0.25 | 59.21 ± 0.38 | 41.66 ± 0.93 | 43.99 ± 0.57 | 71.61 ± 1.46 | 56.40 ± 0.29 | 0.329 | 4 | 4 |
The results show that DimLoRA achieves the highest mean accuracy (56.40%) and the lowest observed standard deviation (0.29) in overall performance among the evaluated methods. In contrast, standard LoRA exhibits greater variability across runs, particularly on the Code domain. These results provide additional evidence that the performance advantages of DimLoRA are consistent across different random seeds.
(5) Clarification of Task-Identity Availability
DimLoRA employs a task-aware routing mechanism that introduces task-specific biases into conventional Top-k routing, encouraging stronger associations between tasks and LoRA experts. This configuration assumes that task identities are available when task-aware routing is applied.
However, the dimension-level expert architecture does not inherently depend on task identities. When task identity is unavailable, the task-specific routing bias can be removed, reducing the mechanism to conventional Top-k routing. We have already evaluated this task-agnostic routing variant in Table 3. The results show that DimLoRA remains competitive with existing LoRA-MoE baselines, demonstrating that the benefits of dimension-level experts are not restricted to a specific routing mechanism.
(6) Expanded Related Work
We appreciate the reviewer's suggestions regarding recent developments in LoRA-MoE adaptation. Due to space constraints, our original Related Work section primarily focused on approaches closely related to expert architecture design. We agree that a broader discussion would help distinguish architectural contributions from improvements in routing and expert specialization.
In the revised manuscript, we will expand this discussion to include HMoRA (ICLR 2025), HotMoE (AAAI 2026), MoSLoRA (EMNLP 2024), Mixture of LoRA Experts (ICLR 2024), and LD-MoLE (ICLR 2026). We will explicitly distinguish DimLoRA's dimension-level expert architecture from the routing and specialization mechanisms introduced by these works.
We sincerely appreciate the reviewer's constructive feedback, which has helped us strengthen both the experimental evaluation and the presentation of our contributions. We hope that these additional results and clarifications adequately address your questions. We would welcome any further feedback or suggestions.
Dear Reviewer 5pAe,
As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.
Thank you.
AC
Rebuttal from Authors of Submission26360 to Reviewer 5pAe (1/2)
We sincerely thank the reviewer for the positive assessment of our architectural design and experimental evaluation, as well as the constructive suggestions for further strengthening our work. We address your questions and concerns below.
(1) Clarification of Mutual Information Regularization
We thank the reviewer for pointing out this ambiguity. In our implementation, the MI regularizer is computed using the softmax probabilities before TopK selection. This ensures that the expert probabilities sum to one, without requiring renormalization after TopK selection. We confirm that this formulation was used in all reported experiments and will clarify this implementation detail in the revised manuscript.
(2) Comparison with Full-Layer LoRA Experts
To isolate the architectural contribution of dimension-level experts, we introduce a full-layer variant, DimLoRA-Full, which uses the same task-aware routing mechanism and MI regularization as DimLoRA. To obtain comparable trainable-parameter budgets, we set the full-layer experts to rank r=3 and scaling factor α=6, while keeping the total number of experts and the number of activated experts identical to DimLoRA. We evaluate both methods on LLaMA3.1-8B using our multi-domain benchmark.
DimLoRA improves average accuracy by 1.14 percentage points over the full-layer variant, with a comparable parameter budget (0.329% vs. 0.300%). Since both methods use the same routing and regularization mechanisms, this comparison provides direct evidence supporting the effectiveness of dimension-level adaptation.
(3) Comparison with Recent Task-Aware Routing Methods
We appreciate the reviewer's suggestions regarding recent task-aware LoRA-MoE methods. We prioritize HotMoE over HMoRA because HMoRA's trainable Transformer-based Task Encoder introduces substantial parameter overhead (approximately 3.90% even with a lightweight configuration).
We adopt the official HotMoE implementation and follow its training methodology. To make the comparison more comparable, we use four experts with Top-2 activation and reduce the LoRA rank to 4. Both methods are evaluated on identical multi-domain datasets under the same training budget.
The original HotMoE experiments were conducted on Qwen2-1.5B with an auxiliary loss coefficient of γ=0.01. However, directly applying this coefficient to our larger backbones results in substantial performance degradation on certain domains. We therefore reduce γ to better adapt HotMoE to our larger model scales. Although this adjustment improves overall performance, substantial performance disparities across domains persist.
DimLoRA outperforms the best-performing HotMoE configurations by 4.18 and 4.44 percentage points with fewer trainable parameters on LLaMA-3.1-8B and Qwen3-14B-Base, respectively. Nevertheless, HotMoE's hybrid routing and DimLoRA's fine-grained expert design are complementary and could potentially be combined for further improvements.