- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper re-examines the advantage of mixtures of LoRA experts over standard LoRA in multi-task learning. It interprets mixture structure partly as an implicit learning-rate multiplier. It then proposes PrismLoRA, which partitions adapter rank into task-group-owned blocks with different learning rates.
Critical strengths
The motivation itself is somewhat interesting. Authors use plenty of figures and tables to confirm the proposition.
The vision improvements are meaningful within the evaluated setting. The grouping and learning-rate ablations also make the empirical result more interpretable than a comparison of final accuracies alone.
Critical weaknesses
The interpretation of learning rate as a task-specific gain is useful, but the paper draws stronger conclusions than the available justification supports. The scalar Bayesian argument does not, by itself, establish the claimed relationship for nonlinear LoRA adapters trained with AdamW. Similarly, similar optimal learning rates do not necessarily imply that tasks should share parameters: their gradients and transfer relationships can still differ.
In addition, I wonder whether the deployment of separating learning rate could really benefit the training of MoE-LoRA. With sufficient training resources, will the paradigm in the paper finally present a same convergence to normal training?
Questions for authors
Please see weaknesses.
Authors have attached a link for code. However, the link is blank and does not contain any effective implementation. It is hard to validate the effectiveness of PrismLoRA at this time.
It seems the texts have been decorated by AI assistants. Some equations, e.g., Eq.1-2, have extremely simple forms. Meanwhile, some expressions in the manuscript just present simple calculations and do not convey enough valid information. I suggest authors refining the overall structure and avoiding manually making the expressions more complex by mathematical equations.
4 comments
Part 2/2 - Reply to Reviewer dtDb
W2. Compare with longer training.
Sure! We add additional exps to directly compare tuned LoRA vs. PrismLoRA at 20 and 100 epochs (new Table 11 in App. H). PrismLoRA still outperforms well-tuned LoRA by around +4 points:
| 20 epochs | 100 epochs | |
|---|---|---|
| Tuned LoRA | 80.33 | 80.01 |
| PrismLoRA | 84.33 | 84.20 |
VTAB-6, seed 42, each method at its validation-selected lr. Reproducible script: tables/rebuttal_long_training.sh.
-
Why doesn't more training close the gap? When using single lr to serve all six distributions, we find more training only moves the trade-off to other distributions. Specifically, we did per-task analysis in table below. Tuned LoRA selects a smaller lr (4.6e-4 → 1.5e-4) on 100 epochs, and CLEVR-Count recovers (+9.4), but CIFAR-100 and SVHN drop (−7.3 & −5.2). PrismLoRA keeps all three (last row).
lr CLEVR-Count CIFAR-100 SVHN Tuned LoRA, 20 epochs 4.6e-4 68.41 68.53 85.20 Tuned LoRA, 100 epochs 1.5e-4 77.83 61.24 79.96 PrismLoRA, 100 epochs per group 78.99 71.12 89.68
Q. The code link.
Soooory! We believe it's a syncing issue of Anonymous GitHub (we just found we need to click "force update" in Anonymous GitHub to sync code!) Actually we are quite proud of our well-organized code!
Pls find in https://anonymous.4open.science/r/PrismLoRA-182E tables/ and figures/. We prepared per-table (all rows) and per-figure (all data points!) reproducible script (also, reproduce this rebuttal is in tables/rebuttal_*.sh). A single run (e.g., ViT-B/16) takes only ~5min on one RTX PRO 6000 GPU. Could you take another look at the link? Thanks!
Q. Improve Texts.
Great! Thanks. We have revised and tighten the text descriptions (changes in blue).
- For AI usage & declaration. As declared in the AI Use Statement (after Sec. 7), honestly, we polished our texts & illustration together with Opus 5.5, and also used agents to double-check hyper-parameter configs. We used agents to implement part of exp deployment scripts. However, all the original ideas, findings and analysis are proposed and decided by ourselves. We take full responsibility for them.
Thank you for the response. I will maintain my original rating.
Great! Deeply thankful for your recognition and efforts.
We will keep polishing our manuscript, and happy to answer any further questions!
Part 1/2 - Reply to Reviewer dtDb
Great! Thank you for finding our motivation interesting and recognize our ask-why analysis! Your point (especially gradients & transfer relationships among tasks) is quite meaningful, and we enjoyed exploring this! (see W1b)!
W1a. The scalar Bayesian argument.
Thanks for figure out this confusion! We have updated our manuscript (texts in blue, next to Eq. 1) to make it more clear!
The scalar form (Eqs. 1–2 you mentioned) is a compact form. The full derivation is in App. A, and it actually covers nonlinear LoRA under AdamW! Extensive experiments support this claim. Specifically,
MoE structure = lr multiplier. 1) We theoretically prove this, in exact form. A constant scale on a block (e.g., a gate weight) equals an lr multiplier under Adam, regardless model, data or loss, and extends to AdamW (Prop. 2 in App. A.1). 2) We empirically verify it, i.e., in Fig. 11a, when LoRA's output is scaled by c, its best lr moves by c−0.49 (theory prediction =c−0.50, App. A.1), and the gap among the 8 methods drops 0.92 avg acc points (1.74 → 0.82, −53%) after dividing each lr by its multiplier (Fig. 2).
Learning rate sets the gain. We derive this (for linearized model at the pretrained weights), and extended to Adam per coordinate (App. A.3–A.4). For non-linear models, we empirically show its consequence still stands (tasks need different gains): same CLEVR images at lr 1e-3, the counting task loses 25.62 points, but the distance task loses only 3.42 (Fig. 3).
W1b. Similar best learning rates ≠ tasks should share parameters.
Totally agree! Thanks for pointing out this confusion. PrismLoRA does not assume it, and we revised the text in the updated manuscript (Intro, Sec. 4.3 & Sec. 5.2, in blue). Specifically, in PrismLoRA a similar lr only makes two distributions candidates to share a block. Whether they really share is checked, not assumed (3 seeds, new Table 10 in App. H):
Similar lr: sharing helps. The group num is selected on val set, and "one block per distribution" (i.e., no sharing) is among the candidates (Table 4). At the same param budget:
*Same total params as PrismLoRA (rank 11 each).
More params can't replace sharing. Rank-67 LoRA per distribution (~6x params) still worse than PrismLoRA:
Same images, different labels: own blocks. For distributions that share images but not labels (e.g., CLEVR-Count & CLEVR-Dist), one shared block would have to give two different answers for the same input, so their gradients differ, just as you said. They get their own blocks (Sec. 5.2):
Notes: every single LoRA is at its own tuned lr. All rows re-run on RTX PRO 6000 for this rebuttal (original Table 1 PrismLoRA's 84.42 ± 0.25 was evaled on RTX 5090). Reproducible scripts for W1b:
tables/rebuttal_independent_lora.sh&tables/rebuttal_struct8_ablation.shin Anonymous Code.I.e., sharing helps when learning rates are similar (+1.1), and just as you said, it stops helping when gradients conflict (+4.3 with own blocks)!