acceptodds
Under review as a conference paper at ICLR 2027

PrismLoRA: Learning Rate as Structure in Mixtures of LoRA Experts

Abstract

Mixtures of LoRA experts (MoE-LoRA) are reported to outperform LoRA in multi-task learning. However, they outperform LoRA at a shared learning rate, not at LoRA's finely tuned one. We show that MoE-LoRA structures mainly act as hidden learning-rate multipliers, i.e., finely tuned learning rates remove 89% (vision) and 96% (language) of the accuracy difference between methods. We further show that tasks need different learning rates, which one shared multiplier cannot provide. For instance, on the same smallNORB images, the camera's azimuth needs a large learning rate and its elevation a small one. We explain these differences with a Bayesian update, where the learning rate sets the gain (i.e., how far training moves the pretrained model toward the labels). Because a confident prior needs a small gain, we measure this confidence with frozen features and predict before training which tasks need a small learning rate. We confirm every prediction after training. We also show that tasks should share parameters only when they need the same learning rate. We therefore propose PrismLoRA, which splits the LoRA rank into blocks at different learning rates and lets each task use the blocks near its optimal learning rate. PrismLoRA sets each task's blocks automatically from frozen features and validation curves. PrismLoRA outperforms all eight tuned baselines with matched parameters. Specifically, PrismLoRA improves over the strongest baseline by 2.63 points on VTAB-6 and 1.35 on Structured-8, and ranks first on language tasks. How much to trust the prior should therefore be set per task by the model's structure.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter efficient fine-tuning, MoE-LoRA, domain adaptation, multi-task learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dtDbOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper re-examines the advantage of mixtures of LoRA experts over standard LoRA in multi-task learning. It interprets mixture structure partly as an implicit learning-rate multiplier. It then proposes PrismLoRA, which partitions adapter rank into task-group-owned blocks with different learning rates.

Critical strengths

The motivation itself is somewhat interesting. Authors use plenty of figures and tables to confirm the proposition.

The vision improvements are meaningful within the evaluated setting. The grouping and learning-rate ablations also make the empirical result more interpretable than a comparison of final accuracies alone.

Critical weaknesses

The interpretation of learning rate as a task-specific gain is useful, but the paper draws stronger conclusions than the available justification supports. The scalar Bayesian argument does not, by itself, establish the claimed relationship for nonlinear LoRA adapters trained with AdamW. Similarly, similar optimal learning rates do not necessarily imply that tasks should share parameters: their gradients and transfer relationships can still differ.

In addition, I wonder whether the deployment of separating learning rate could really benefit the training of MoE-LoRA. With sufficient training resources, will the paradigm in the paper finally present a same convergence to normal training?

Questions for authors

Please see weaknesses.

Authors have attached a link for code. However, the link is blank and does not contain any effective implementation. It is hard to validate the effectiveness of PrismLoRA at this time.

It seems the texts have been decorated by AI assistants. Some equations, e.g., Eq.1-2, have extremely simple forms. Meanwhile, some expressions in the manuscript just present simple calculations and do not convey enough valid information. I suggest authors refining the overall structure and avoiding manually making the expressions more complex by mathematical equations.

4 comments
AuthorsOpenReview ↗

Part 1/2 - Reply to Reviewer dtDb

Great! Thank you for finding our motivation interesting and recognize our ask-why analysis! Your point (especially gradients & transfer relationships among tasks) is quite meaningful, and we enjoyed exploring this! (see W1b)!

W1a. The scalar Bayesian argument.

Thanks for figure out this confusion! We have updated our manuscript (texts in blue, next to Eq. 1) to make it more clear!

The scalar form (Eqs. 1–2 you mentioned) is a compact form. The full derivation is in App. A, and it actually covers nonlinear LoRA under AdamW! Extensive experiments support this claim. Specifically,

  • MoE structure = lr multiplier. 1) We theoretically prove this, in exact form. A constant scale on a block (e.g., a gate weight) equals an lr multiplier under Adam, regardless model, data or loss, and extends to AdamW (Prop. 2 in App. A.1). 2) We empirically verify it, i.e., in Fig. 11a, when LoRA's output is scaled by cc, its best lr moves by c−0.49c^{-0.49} (theory prediction =c−0.50=c^{-0.50}, App. A.1), and the gap among the 8 methods drops 0.92 avg acc points (1.74 → 0.82, −53%) after dividing each lr by its multiplier (Fig. 2).

  • Learning rate sets the gain. We derive this (for linearized model at the pretrained weights), and extended to Adam per coordinate (App. A.3–A.4). For non-linear models, we empirically show its consequence still stands (tasks need different gains): same CLEVR images at lr 1e-3, the counting task loses 25.62 points, but the distance task loses only 3.42 (Fig. 3).

W1b. Similar best learning rates ≠ tasks should share parameters.

Totally agree! Thanks for pointing out this confusion. PrismLoRA does not assume it, and we revised the text in the updated manuscript (Intro, Sec. 4.3 & Sec. 5.2, in blue). Specifically, in PrismLoRA a similar lr only makes two distributions candidates to share a block. Whether they really share is checked, not assumed (3 seeds, new Table 10 in App. H):

  1. Similar lr: sharing helps. The group num is selected on val set, and "one block per distribution" (i.e., no sharing) is among the candidates (Table 4). At the same param budget:

    VTAB-6Mean acc.
    No sharing: one LoRA per distribution*83.36 ± 0.20
    PrismLoRA: sharing inside one learning-rate group84.50 ± 0.18

    *Same total params as PrismLoRA (rank 11 each).

  2. More params can't replace sharing. Rank-67 LoRA per distribution (~6x params) still worse than PrismLoRA:

    VTAB-6ParamsMean acc.
    No sharing: one rank-67 LoRA per distribution59.28M81.92 ± 0.30
    PrismLoRA9.73M84.50 ± 0.18
  3. Same images, different labels: own blocks. For distributions that share images but not labels (e.g., CLEVR-Count & CLEVR-Dist), one shared block would have to give two different answers for the same input, so their gradients differ, just as you said. They get their own blocks (Sec. 5.2):

    Structured-8Mean acc.
    Same-image pairs share a block53.42 ± 0.71
    Same-image pairs get own blocks57.76 ± 0.34

Notes: every single LoRA is at its own tuned lr. All rows re-run on RTX PRO 6000 for this rebuttal (original Table 1 PrismLoRA's 84.42 ± 0.25 was evaled on RTX 5090). Reproducible scripts for W1b: tables/rebuttal_independent_lora.sh & tables/rebuttal_struct8_ablation.sh in Anonymous Code.

I.e., sharing helps when learning rates are similar (+1.1), and just as you said, it stops helping when gradients conflict (+4.3 with own blocks)!

AuthorsOpenReview ↗

Part 2/2 - Reply to Reviewer dtDb

W2. Compare with longer training.

Sure! We add additional exps to directly compare tuned LoRA vs. PrismLoRA at 20 and 100 epochs (new Table 11 in App. H). PrismLoRA still outperforms well-tuned LoRA by around +4 points:

20 epochs100 epochs
Tuned LoRA80.3380.01
PrismLoRA84.3384.20

VTAB-6, seed 42, each method at its validation-selected lr. Reproducible script: tables/rebuttal_long_training.sh.

  • Why doesn't more training close the gap? When using single lr to serve all six distributions, we find more training only moves the trade-off to other distributions. Specifically, we did per-task analysis in table below. Tuned LoRA selects a smaller lr (4.6e-4 → 1.5e-4) on 100 epochs, and CLEVR-Count recovers (+9.4), but CIFAR-100 and SVHN drop (−7.3 & −5.2). PrismLoRA keeps all three (last row).

    lrCLEVR-CountCIFAR-100SVHN
    Tuned LoRA, 20 epochs4.6e-468.4168.5385.20
    Tuned LoRA, 100 epochs1.5e-477.8361.2479.96
    PrismLoRA, 100 epochsper group78.9971.1289.68

Q. The code link.

Soooory! We believe it's a syncing issue of Anonymous GitHub (we just found we need to click "force update" in Anonymous GitHub to sync code!) Actually we are quite proud of our well-organized code!

Pls find in https://anonymous.4open.science/r/PrismLoRA-182E tables/ and figures/. We prepared per-table (all rows) and per-figure (all data points!) reproducible script (also, reproduce this rebuttal is in tables/rebuttal_*.sh). A single run (e.g., ViT-B/16) takes only ~5min on one RTX PRO 6000 GPU. Could you take another look at the link? Thanks!

Q. Improve Texts.

Great! Thanks. We have revised and tighten the text descriptions (changes in blue).

  • For AI usage & declaration. As declared in the AI Use Statement (after Sec. 7), honestly, we polished our texts & illustration together with Opus 5.5, and also used agents to double-check hyper-parameter configs. We used agents to implement part of exp deployment scripts. However, all the original ideas, findings and analysis are proposed and decided by ourselves. We take full responsibility for them.

Thank you for the response. I will maintain my original rating.

Great! Deeply thankful for your recognition and efforts.

We will keep polishing our manuscript, and happy to answer any further questions!

Reviewer_tvWmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies why MoE-LoRA often performs better than standard LoRA in multi-task learning. The authors find that many MoE-LoRA designs effectively change the learning rate. After tuning the learning rate separately for each method, much of the original performance gap disappears. The paper also shows that different tasks prefer different learning rates, and tasks with very different update strengths may not benefit from fully shared parameters. Based on this observation, the authors propose PrismLoRA. It divides LoRA into several blocks with different learning rates and decides parameter sharing based on the preferred learning rate of each task. Experiments on VTAB-6, Structured-8, and language tasks show better overall performance than several tuned LoRA and MoE-LoRA baselines.

Critical strengths
  1. I like the main observation of this paper. The learning-rate analysis is interesting and well supported.
  2. The same-input, different-label experiment makes the task differences quite clear.
Critical weaknesses
  1. The MTL-LoRA search stops at the edge of the learning-rate range, so the comparison is not fully convincing.
  2. PrismLoRA relies on known task identity. It is unclear how it works when task labels are unavailable or distributions are mixed.
Questions for authors
  1. In the language experiments, why was the learning-rate grid for MTL-LoRA not expanded further? After a wider search, how large is the gap between MTL-LoRA and PrismLoRA?
  2. How does PrismLoRA work when the task or distribution identity is unknown? Can the model select blocks automatically based on the input?
  3. LoRA-Mixer [1] also uses multiple LoRA experts to control parameter sharing across tasks. What is the key difference between PrismLoRA and this type of routing-based method, and what is the practical advantage of PrismLoRA? [1] LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
2 comments
AuthorsOpenReview ↗

Part 1/2 - Reply to Reviewer tvWm

We appreciate ur meaningful input! Great you like our key observations and the same-input exps!!

W1 / Q1. lr grid search for MTL-LoRA.

Good catch! We extended MTL-LoRA's grid and re-ran the language benchmark w/ 3 seeds. We updated Table 1 (under "language", in blue). Specifically,

  1. Full lr grid of MTL-LoRA. We extend search grid and select lr=1e-4 (not at edge anymore).

    MTL-LoRA lr5.5e-51e-41.7e-43e-4
    Validation (seed 42)71.7372.5172.4070.61
    Test (seed 42)68.1669.3569.9268.79
  2. Aft full grid search, MTL-LoRA approx PrismLoRA. On language, on val set, PrismLoRA selects G = 1 (only one group). In this case, the PrismLoRA equals a well-tuned LoRA (Prop. 1(i) in Sec. 5.1). The best MoE-LoRA baseline (HydraLoRA, 69.39) is only +0.09 above it, within the seed noise (±0.20). This is consistent with our key finding in Fig. 2c that, once lr is tuned, MoE structure itself bring no gain on language.

    MethodSelected lrTest, 3 seeds
    PrismLoRA (= tuned LoRA, G = 1)1.7e-469.30 ± 0.20
    MTL-LoRA (grid extended)1e-469.10 ± 0.46
    MoLoRA3e-469.35 ± 0.34
    HydraLoRA3e-469.39 ± 0.12
    HydraLoRA (task gate)3e-468.87 ± 0.11
    MOELoRA3e-469.11 ± 0.19
    Polytropon3e-468.45 ± 0.15
    MHR3e-468.38 ± 0.23

Notes: for a fully matched comparison, all methods did not use the eval-time scale. We report this scale as a separate analysis, for LoRA & MTL-LoRA methods alike (App. F.3). Reproducible script: tables/rebuttal_language.sh in Anonymous Code.

W2 / Q2. Under task-unknown scenario

Good idea! PrismLoRA uses task ID in two places: a) inference, and b) tuning & training.

  • For inference, we show that task ID is not a necessary, e.g., a simple task ID classifier on PrismLoRA works well (only -0.2 points in avg acc, we have updated in updated manuscripts, App. H).
  • For tuning & training, task ID is must, as in common multi-task LoRA.

We ran small tests (shown below) without any re-train and append results in App. H, Table 12, in blue.

  1. Inference: known vs. predicted task ID differs by only 0.20 points. We design an small test to explore this. 1) We learn a linear classifier on frozen ViT (using task IDs from train set) to predict task ID from input img (99.74% acc. on test, all six tasks mixed). 2) We route each img to its task ID's selected blocks & head on PrismLoRA. Results show that PrismLoRA drops only 0.20 avg accuracy, still significantly higher than LoRA by ~4 (84.31 vs 80.35):

    VTAB-6Known task IDPredicted task IDChange
    LoRA80.54 ± 0.2380.35 ± 0.22−0.19
    PrismLoRA84.51 ± 0.1984.31 ± 0.19−0.20

    3 seeds, full test set, no re-training. LoRA also needs the task ID to pick its head. New rebuttal runs are on RTX PRO 6000 (Table 1 was on RTX 5090), so LoRA is 80.54 here vs. 80.80 in Table 1. Reproducible script: tables/rebuttal_unknown_identity.sh.

  2. Tuning & training: task ID is common in multi-task LoRA, not extra requirement of PrismLoRA. Most baselines need it too.

    • 5 of the 7 MoE-LoRA baselines (HydraLoRA with a task gate, MOELoRA, MTL-LoRA, Polytropon and MHR) take task ID as input, so Table 1 is a matched comparison.
    • On language, the tuned PrismLoRA model takes no task ID (G=1, no task head).
    • Certain task must feed task ID, agnostic of method. E.g., CLEVR-Count & CLEVR-Dist use identical images across different tasks, i.e., "how many objects?" vs "how far is the nearest one?" for identical training image.
AuthorsOpenReview ↗

Part 2/2 - Reply to Reviewer tvWm

Q3. Compare with LoRA-Mixer.

Thanks for figuring out this important & good work! We compare with LoRA-Mixer in updated Related Work (Sec. 2, in blue). Key differences:

  1. What decides the sharing. LoRA-Mixer learns a token-level router (e.g., via a routing loss), while PrismLoRA decides the sharing before training. Tasks with close optimal lr (read from the val sweep) share one block (details in Sec. 5.2). We empirically compared with the learned token routers (MoLoRA & HydraLoRA, in Table 1), and found that their routers carry only ~4% of the task information (4.13% & 4.07% of the task entropy, VTAB-6, Fig. 14 in App. C.5).
  2. Difference between experts. In routed MoE all experts train at single global optimal lr. In PrismLoRA, tasks are grouped by lr and each block was trained at its own lr. This contributes the most improvement (+3.09 out of the +3.62 avg acc gain on VTAB-6: +1.89 from grouping w/ one re-selected lr, +1.20 more from per-group lr, Table 2).
  3. Practical advantage. PrismLoRA have no additional router nor inference overhead. Each distribution's update can be merged into the foundation model weights, as in LoRA.
Reviewer_UBwcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper argues that the reported gains of mixtures of LoRA experts (MoE-LoRA) over LoRA in multi-task fine-tuning are largely an artifact of comparing methods at one shared learning rate. Under Adam, structural elements such as gates and rank rescale the effective learning rate, so each MoE-LoRA method trains like LoRA at a hidden multiple of its learning rate. Tuning each method at its own learning rate removes 89% (VTAB-6) and 96% (language) of the between-method variance. The authors then show that tasks need different learning rates, even on the same inputs. They interpret the learning rate as setting a gain K=P/(P+R)K = P/(P+R), a Kalman gain, that determines how far training moves the pretrained model. If frozen features already predict the labels, the prior is confident, so the task needs a small gain. Because a shared weight moves by the summed updates of all tasks that read it, different gains require owned parameters. Their main contribution is PrismLoRA, which splits the LoRA rank into blocks trained at different learning rates and assigns each task to the blocks near its preferred rate. The grouping is set from frozen features and the learning-rate sweep that every baseline already runs. Empirical experiments with ViT-B/16 show improvements on VTAB-6 and Structured-8 over the strongest tuned baseline at matched parameters, and they show modest improvement with LLaMA-2-7B on language tasks.

Critical strengths

S1: The central methodological point is well demonstrated. Most of the differences between MoE-LoRA methods at a shared learning rate reflect a hidden learning-rate effect rather than architectural differences. Tuning each method’s own rate removes 89% of the between-method variance on vision and 96% on language, and rescaling each rate by a fitted multiplier makes the curves nearly coincide (Fig. 2, Sec. 4.1). The paper argues that published gains were reported at a shared rate with LoRA untuned, so this result suggests that much of the field’s evidence for MoE-LoRA structure reflects an uncontrolled confound. This extends the finding that tuned PEFT variants perform similarly (Lee et al., 2026; He et al., 2026) to multi-task MoE-LoRA. It also supplies a quantified mechanism, not just a reminder to tune baselines, and so gives a more reliable basis for comparing multi-task PEFT methods.

S2: The label-dependence experiments are clean and directly support the paper’s premise. Holding the images fixed and changing only the labels changes the best learning rate: on smallNORB, azimuth and elevation peak at different rates, and on CLEVR a high rate costs counting 25.62 points but distance only 3.42 (Fig. 3). Tests on shared parameters (Sec. 4.3) show that tasks sharing a block share one gain, so per-task rates on a shared LoRA underperform giving each task its own block. This shows that the need for different learning rates is a property of the tasks, not of the images, which motivates PrismLoRA well. The controlled design (same images, different labels) isolates the label set as the cause, so the finding should be useful beyond this paper’s method.

Critical weaknesses

W1: PrismLoRA appears to get more tuning than the baselines, and the language result is not well supported. PrismLoRA validates several choices beyond the learning rate: the number of groups GG, a global shift of ±1\pm 1 grid step, head learning rates, and an evaluation-time update scale φd\varphi_d (Sec. 5.2). For the baselines, Sec. 6.1 says only that each learning rate is selected on validation. On language, PrismLoRA selects G=1G = 1, so nearly all of its gain over LoRA appears to come from φd\varphi_d applied to three datasets. The result uses a single seed with no variance, and the margin over the strongest baseline, MTL-LoRA, is small. MTL-LoRA’s learning rate also sits at the lower edge of the grid, which was not extended (Table 1 footnote). The claim that PrismLoRA outperforms all eight tuned baselines, and ranks first on language, may therefore reflect extra tuning freedom rather than the method. This matters especially because the paper’s own thesis is that unequal tuning inflates reported gains. The vision margins are larger and appear more robust, but the paper does not show that the baselines received the same options there either.

W2: The ablation does not cleanly isolate learning rate as the structure, particularly on Structured-8. On Structured-8, the first ablation step (“grouping, one learning rate”) accounts for most of the total gain over LoRA, and its footnote says it includes owned parameters for the three same-image task pairs (Table 2). The step that varies learning rates across groups adds only a small fraction of the gain. The paper suggests owned copies matter substantially, but the main text never separates them from grouping or the re-selected global rate. On VTAB-6 the evidence is stronger: the same groups add little at LoRA’s own rate but more with per-group rates (Sec. 6.3). But its first step also combines grouping with a re-selected rate. One of the paper’s central claims is that learning rate acts as structure. This weakens the case for that explanation, particularly on Structured-8, since the gain may come from task-specific parameters, which the paper credits to existing MoE-LoRA methods.

Questions for authors

Note: the appendix was not included in the PDF I reviewed, and the anonymous repository was empty at the time of review despite being referenced in the main text. Some of the questions below may therefore already be answered there. If so, I would welcome a brief summary of the relevant details in your response.

Q1: Matched tuning between PrismLoRA and the baselines. Following up on W1: on language, PrismLoRA selects G=1G = 1, so its difference from tuned LoRA appears to come from the evaluation-time scale φd\varphi_d (the main text does not report the global shift selected there). Could the authors (i) report which of GG, the shift, φd\varphi_d and the head learning rate were selected on language, (ii) apply φd\varphi_d to tuned LoRA and to MTL-LoRA, whose best rate also sits at the lower grid edge (Table 1 footnote), and (iii) report the language results over several seeds? This would show whether the language ranking holds under equal tuning.

Q2: How much of the Structured-8 gain comes from learning rates rather than owned parameters? Table 2 attributes most of PrismLoRA’s improvement over LoRA on Structured-8 (50.16 → 57.93) to the “grouping, one learning rate” row (+6.74), whose footnote says it includes separate copies for the three same-image pairs. Per-group learning rates add +0.96. This fits the paper’s account that methods with per-distribution parameters already capture much of that first step (MTL-LoRA reaches 56.58), which suggests that the learning-rate steps are what separate PrismLoRA from the strongest baseline. Table 2 leaves the Structured-8 entry blank for “grouping at LoRA’s learning rate,” which adds little on VTAB-6 (+0.27). Could the authors fill that in and add a row with owned copies alone? This would show how much of the learning-rate thesis rests on the per-group-rate step.

Q3: Per-task training length versus per-task learning rate. Sec. 3 argues that TT steps at rate η\eta behave like ridge regression toward μ0\mu_0 with precision 1/(ηT)1/(\eta T), so a smaller step budget and a smaller learning rate both reduce the effective gain. Within a run, all distributions share one epoch budget (Sec. 6.1). Sec. 4.2 shows that optima coincide when training does not fit, but per-task budgets are not tested. If per-epoch validation accuracy was logged, could the authors report per-distribution early-stopped checkpoints for the shared LoRA at its tuned rate, and training accuracy for the VTAB runs? This would show whether per-task early stopping already recovers part of the gain.

Q4: What does the frozen-feature prior add beyond the validation sweep? Eq. 6 combines a frozen-feature prior with the argmax of LoRA’s validation curve through a Kalman weight, plus an asymmetry term with κ=0.3\kappa = 0.3. The prior is supported by 12 correct calls (Sec. 1), but on Structured-8 the rule flags only CLEVR-Count (Sec. 4.2). It is also described for ordinal labels, and each VTAB-6 distribution has 200 validation images. Could the authors compare positions set from the sweep alone against the fused positions, and say how κ\kappa and the 70% explained-variance threshold were chosen? This would show whether the Bayesian component does real work beyond the sweep that every baseline already runs.

Q5: Missing reference point. Independent per-task LoRA, one model per task and each tuned separately, is the standard VTAB-1k protocol (Sec. 2), but the main text does not report it as a reference for PrismLoRA’s joint model. Sec. 4.2’s finding that per-distribution best learning rates beat the best shared rate by 2.02 points on VTAB-6 is related, but it is not a separately trained model per task. Could the authors report independent per-task LoRA, together with its total parameter count? This would show where PrismLoRA sits relative to separate models, as a reference point rather than a strict ceiling, since joint training can also transfer between tasks.

7 comments
AuthorsOpenReview ↗

Part 1/3 - Reply to Reviewer UBwc

We appreciate your careful & constructive comments! It points out key improve rooms for us & we are excited for this! We ran every exp you asked for and updated manuscript (in blue).

BTW in our original submission, Appendix PDF was under "Supplementary Material" (next to "Abstract" on OpenReview). I have now merged appendix into my main submission, pls check! The anonymous code is ready (we just found we need to click "force update" in Anonymous GitHub to sync code!), and I provide per-table/figure reproducible script (even for this rebuttal, at tables/rebuttal_*.sh).

Note: new runs for this rebuttal are on RTX PRO 6000, and Table 1 was on RTX 5090. So numbers differ slightly, e.g., on VTAB-6, LoRA 80.54 vs. 80.80 and PrismLoRA 84.50 vs. 84.42.

W1 / Q1. Fair comparison of PrismLoRA & baselines.

Totally agree! This area's key problem is confounded tuning, especially when network param and data is usually small. We are trying to figure out and tackle them here. Following your suggestion, we did extensive & careful ablations for each design, separately for Vision and Language tasks (in updated manuscript, Table 1 "language" & Sec. 5.2, in blue; code see above). Specifically,

  1. Vision: PrismLoRA robustly outperforms LoRA.

    • We applied the "per-task head lr" design to LoRA (Table 8). PrismLoRA wins clearly.

      VTAB-6Structured-8
      LoRA80.8050.16
      LoRA + same head lr81.6851.08
      PrismLoRA84.4257.93
    • Beyond the lr sweeps that all baselines run, PrismLoRA adds only 11 training runs (App. B.8).

      VTAB-6lr candidatesTraining runs
      All baselines9–1030
      PrismLoRA3 adjacent values+11

      Note: the eval-time scale γd\gamma_d (written φd\varphi_d in your review) is not used on vision (val selects γd\gamma_d = 1, i.e., no scaling, on all 14 vision tasks, Fig. 20).

  2. Language: 1) none of existing MoE-LoRA methods outperforms well-tuned LoRA. 2) PrismLoRA equals tuned LoRA. Specifically,

    • We extend the lr search grid of MTL-LoRA, and ran 3 seeds. No MoE-LoRA exceeds tuned LoRA beyond seed std. This aligns with our first key finding (Introduction, contribution 1; Fig. 2c; per-method language results in Fig. 5c).

      MethodSelected lrTest, 3 seeds
      LoRA (= PrismLoRA, G = 1)1.7e-469.30 ± 0.20
      MoLoRA3e-469.35 ± 0.34
      HydraLoRA3e-469.39 ± 0.12
      HydraLoRA (task gate)3e-468.87 ± 0.11
      MOELoRA3e-469.11 ± 0.19
      MTL-LoRA1e-469.10 ± 0.46
      Polytropon3e-468.45 ± 0.15
      MHR3e-468.38 ± 0.23

      Notes: on the extended grid, we select optimal lr=1e-4 for MTL-LoRA on val (full grid in our reply to Reviewer tvWm, W1/Q1). All compared methods at their own val-selected lr. Reproducible script: tables/rebuttal_language.sh.

    • For your Q1 (i), PrismLoRA on language:

      Groups GGlobal lr shiftHead lrEval-time scale γd\gamma_d
      1nonenone0.75 (GSM8K, HellaSwag & ARC-e)

      The G = 1 means PrismLoRA selects single group on val set. In this case, the PrismLoRA equals well-tuned LoRA. This is the intended behaviour (Sec. 6.2, Table 9).

    • For your Q1 (ii) regarding γd\gamma_d, very good point! We did ablation and found γd\gamma_d improves both LoRA & MTL-LoRA baselines. So in the updated manuscript, we removed it from "our design" and give separate analysis in App. F.3.

      W/o γd\gamma_dW/ γd\gamma_d
      LoRA (= PrismLoRA, G = 1)69.3069.97
      MTL-LoRA69.1069.30
AuthorsOpenReview ↗

Part 2/3 - Reply to Reviewer UBwc

W2 / Q2. Structured-8: lr vs. task-specific params.

Key suggestion! We ablate the two designs of PrismLoRA: A) task-specific params (tasks on the same images each get their own LoRA block), and B) per-group lr (each task group gets its own lr). Results provided below, in Table 2 (last three rows, in blue) and in Table 13 in App. H. Specifically,

  1. The two designs improve LoRA separately. A gains more, and A + B is the best.

    Structured-8W/o AW/ A
    W/o B50.22 ± 0.68 (LoRA)57.12 ± 0.42
    W/ B53.42 ± 0.7157.76 ± 0.34

    Why A gains more on Structured-8: 6 out of 8 tasks are same input paired with different labels (e.g., CLEVR counting vs. CLEVR distance). Single shared block cannot give two different answers, so each task require its own block.

    Notes: you also asked for the blank Structured-8 entry of Table 2 ("grouping at LoRA's lr"). It is 56.29 ± 0.90. Reproducible script: tables/rebuttal_struct8_ablation.sh.

  2. PrismLoRA outperforms the strongest baseline (MTL-LoRA, has A also).

    Structured-8Acc.
    MTL-LoRA56.37 ± 0.61
    A only57.12 ± 0.42
    PrismLoRA57.86 ± 0.24
  3. On VTAB-6, A is not used (no task share input images), so gain only comes from lr: same task groups add +0.27 at LoRA's lr, +1.89 w/ one re-selected lr, and +3.09 w/ per-group lr.

    VTAB-6 (Table 2)Acc.
    LoRA80.80
    Task groups, one lr (LoRA's)81.07
    Task groups, one re-selected lr82.69
    Task groups, per-group lr83.89

Q3. Per-task training vs. per-task lr.

Good idea! We tested two ways of doing this on Vision tasks. Neither is comparable with PrismLoRA. Specifically,

  1. Method 1 use one shared LoRA, assign per-task lr (Sec. 4.3, Table 6). It's worse than splitting LoRA into two separate blocks.

    VTAB-6Acc.
    One shared block, per-task lr & Adam state80.26
    Two separate blocks (CLEVR-Count / other 5 tasks)82.51
  2. Method 2 early stops each task at its optimal epoch (new Table 14, App. H, in blue). This does not close the gap to PrismLoRA.

    • Stopping each task at its own best-val epoch:

      LoRAVTAB-6Structured-8
      Final epoch80.54 ± 0.2450.22 ± 0.68
      Each task's best-val epoch80.56 ± 0.4150.03 ± 0.63
    • Pick best epoch based on test (as an upper bound):

      VTAB-6Structured-8
      LoRA, each task's best-test epoch80.90 ± 0.3450.75 ± 0.64
      PrismLoRA, final epoch84.50 ± 0.1857.86 ± 0.24

      Notes: 3 seeds. Reproducible script: tables/rebuttal_early_stopping.sh.

    Why? Both methods still train a single LoRA block for all tasks, i.e., shared weights force all tasks to be trained at identical speed. Our PrismLoRA gives tasks separate blocks, so each block is tuned at its own speed (Sec. 4.3).

  3. Training acc. of VTAB runs (Q3). LoRA reaches 100% training acc. on every VTAB-6 task at 20 epochs.

AuthorsOpenReview ↗

Part 3/3 - Reply to Reviewer UBwc

Q4. What does the frozen-feature prior add?

Thanks for pointing out this misleading text! We clarify that the frozen-feature prior does not change our results. It is an analysis of why tasks need different lr (Sec. 4.2, App. D.3). We ran two checks and revised the text (Sec. 5.2, App. B.2 & App. D.3, in blue). Specifically,

  1. Val sweep alone vs. sweep + prior: the trained lr is the same. So the configuration of PrismLoRA needs only the val sweep, and the prior serves to explain why tasks need different lr.

    VTAB-6Structured-8
    Weight of the val sweep in Eq. 60.890.73
    Trained lr after adding the priorsamesame
  2. κ\kappa: any value in 0–0.97 gives the same groups. κ\kappa is a safety margin that slightly lowers each lr. κ\kappa = 0.3 was fixed in advance and never tuned.

  3. The 70% threshold is only used in the analysis of which tasks need a small lr (Sec. 4.2, App. D.3). No number in Table 1 depends on it. App. D.3 describes how it was chosen.

Q5. Independent, per-task LoRA.

This is an important comparison, thanks! Reviewer dtDb raised a related point (sharing vs. no sharing), also. We trained separate LoRAs for each task, tune each LoRA to its optimal lr (Table 1 last rows & new Table 10 in App. H, 3 seeds, in blue). Results show that:

  1. Single PrismLoRA outperforms six independent LoRAs.

    VTAB-6ModelsParamsMean acc.
    Independent LoRA, rank 67, 20 ep.659.28M81.92 ± 0.30
    Independent LoRA, rank 67, 100 ep.659.28M82.40 ± 0.13
    Independent LoRA, rank 11, 20 ep.69.73M83.36 ± 0.20
    PrismLoRA19.73M84.50 ± 0.18

    Reproducible script: tables/rebuttal_independent_lora.sh.

    We assume this is because tasks can help each other when they share a block (when within same lr group).

  2. The best lr belongs to the task, not to a joint training. We find that when trained alone, CLEVR-Count still selects the smallest lr and SVHN the largest, as in joint training (Fig. 1a):

    Trained alone (rank 11)CLEVR-CountEuroSAT, dSpritesCIFAR-100, RESISC45SVHN
    Selected lr4.6e-41.5e-32.2e-33.2e-3

We hope these address W1 & W2. Thanks again! Feedback from a reviewer w/ such domain insight & hands-on experience is valuable to us! We appreciate your insights. Hopefully this work and follow-up works could bring an honest & meaningful change to the field.

Comments on the paper

1 comment

Thanks to the AC & all three reviewers! As reviewer also I know the huge effort in this. We treasure it!

  • All three reviewers unanimously gave positive initial ratings and recognized our key finding, solid evidence, and clean controlled exps! After discussion, Reviewer UBwc raised the Soundness score and confirmed our transparency and fair comparison! We are glad to answer any further questions and keep updating our manuscript!

  • All requested exps are done. We revised manuscript accordingly (in blue; appendix is now merged into the PDF). Code is at the anonymous link, one script per table/figure (include this rebuttal, in tables/rebuttal_*.sh).

Happy that all three reviewers recognize our key finding, i.e., MoE-LoRA structure mainly acts as a hidden lr multiplier, and tasks need different lr. The concerns are on the method side. We replied each point (conclusions here for ur convenience!).

Reviewer dtDb

  • W1a (theory for nonlinear LoRA + AdamW): "scale = lr multiplier" is proved in exact-form (Prop. 2) and verified empirically (Fig. 11a).
  • W1b (similar lr ≠ should share): In PrismLoRA, the sharing is decided on val set (Sec. 4.3), and it helps, i.e., PrismLoRA 84.50 vs. 83.36 for per-task LoRA at same param budget (Table 1, new Table 10).
  • W2 (longer training): At 100 epochs, PrismLoRA is still ~4 points above well-tuned LoRA (new Table 11).
  • Q (code & presentation): Code link works now. Figures are improved and text is revised (in blue).

Reviewer tvWm

  • W1/Q1 (lr grid of MTL-LoRA): We extended the lr search grid and ran 3 seeds for all methods. On language, no MoE-LoRA exceeds well-tuned LoRA beyond seed std, and PrismLoRA equals well-tuned LoRA (Table 1, "language").
  • W2/Q2 (unknown task ID): At inference, PrismLoRA does not depend on it. With a predicted task ID, PrismLoRA drops only 0.20 points (new Table 12).
  • Q3 (LoRA-Mixer): LoRA-Mixer learns a router, while PrismLoRA sets the sharing & the lr of each block before training (Related Work).

Reviewer UBwc

  • W1/Q1 (equal tuning): On vision, apply the same head lr to LoRA still worse than PrismLoRA (2.7 & 6.9 avg acc points, Table 8), and re-selecting LoRA's lr with heads fixed adds +0.00 (new Tables 15, 16). On language, the eval-time scale is removed from our method (Sec. 5.2, Table 1).
  • W2/Q2 (Structured-8: lr vs. task-specific params): We ablate two designs in a 2×2 table, i.e., task-specific params +6.90, per-group lr +3.19, both +7.54 (Table 2, new Table 13).
  • Q3 (per-task early stopping): This work-around is not comparable with PrismLoRA. Even with the optimal epoch picked on test set (upper bound), LoRA is 3.6 & 7.1 points lower than PrismLoRA (new Table 14).
  • Q4 (frozen-feature prior): Our design & results do not depend on it, i.e., same trained lr without it (App. B.2). It is an analysis of why tasks need different lr (Sec. 4.2, App. D.3).
  • Q5 (independent per-task LoRA): one PrismLoRA (84.50) outperforms six independent LoRAs (81.92–83.36) (Table 1, new Table 10).

Thanks again for your time!

[Update on 18 Nov] Following the reviewers' suggestions, we further revised the figures and text for clarity. Changes to claims or key statements are in blue; plain writing polish is not marked.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.