acceptodds
Under review as a conference paper at ICLR 2027

JIVEAdapter: A Multi-Task Additive Low-Rank Adapter via Joint and Individual Variation Explained (JIVE)

Abstract

Parameter-efficient fine-tuning adapts pretrained models at a fraction of the cost of full fine-tuning, yet most low-rank adapters are single-task and represent each weight update multiplicatively, leaving no explicit account of what is shared across tasks and what is task-specific. We introduce JIVEAdapter, a multi-task additive low-rank adapter inspired by statistical Joint and Individual Variation Explained (JIVE). JIVEAdapter decomposes every weight update into a Joint structure shared across all tasks plus a per-task Individual structure, penalizes the Individual structures to be near-orthogonal to the Joint so shared and task-specific signal stay interpretable and separated, and allocates rank adaptively across a shared Joint pool and a per-task Individual pool. The Joint is learned once, jointly over a task group or incrementally, one task at a time, then frozen and reused as a prior for new tasks without retraining the shared part. On GLUE and SuperGLUE with DeBERTaV3-base, \method is competitive with strong single-task and multi-task low-rank baselines at a matched per-task effective rank, without extra modules such as MoE, and when a related held-in task exists its frozen Joint serves a held-out task by reusing that task's Individual with only a cheap per-direction scale, otherwise training a small new one.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Adaptation, Multi-task learning, Joint and Individual Variation Explained, Additive Decomposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8ErGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JIVEAdapter introduces a multi-task additive low-rank adapter inspired by Joint and Individual Variation Explained. Each task update is decomposed into a shared Joint component and a near-orthogonal task-specific Individual component, with adaptive rank allocation across the two pools. The learned Joint can be reused for new tasks either by calibrating a related task’s Individual or by training a new small Individual.

Critical strengths
  • The Joint-plus-Individual decomposition is a meaningful and interpretable multi-task adapter design.

  • The empirical coverage is broad. The paper evaluates single-task and multi-task performance, incremental task arrival, held-out reuse, alternative rank estimators, and learned subspace geometry rather than relying on one benchmark table.

Critical weaknesses
  • Almost all reported results use only one fixed random seed. And several reported gains are small—for example, 89.07 versus 88.80 on multi-task GLUE

  • The paper lacks a decisive mechanism ablation against simpler alternatives, such as an additive shared/individual adapter without orthogonality or the same structure with fixed ranks. The geometry plots confirm that the learned spaces are nearly orthogonal, but they do not establish that orthogonality or adaptive allocation causes the performance and transfer gains.

Questions for authors
  • Is the parameter comparison with multi-task baselines fully matched in the relevant deployment setting? The paper reports trainable parameters per task head, but the total number of stored Individual parameters grows with the number of tasks. Please report total trainable and stored parameters for the full eight-task model, as well as inference-time active parameters and training-time computation.

  • How is subspace similarity computed for a truly unseen task before its Individual component has been trained?

  • Can the authors clarify the novelty boundary relative to existing shared-plus-specific and orthogonal LoRA methods? The paper discusses HydraLoRA, MTL-LoRA, O-LoRA, and LoRAHub, but several elements—shared and task-specific components, orthogonal subspaces, and adapter reuse—have prior counterparts.

2 comments
AuthorsOpenReview ↗

"Response to Reviewer 3: Multi-seed results, ablations, parameter counts, similarity for unseen tasks, and novelty"

We thank the reviewer for the constructive comments and address each point.

W1: Seeds and small gains. On the size of the gains. The goal of PEFT methods, including ours, is not to improve accuracy but to match full fine-tuning while training and storing few parameters and forgetting less. GLUE and SuperGLUE are near saturation at this size, so large gaps among strong adapters are not expected (e.g., shared LoRA vs. AdaLoRA differ by 0.40 on GLUE). We claim parity while (i) learning a shared Joint and per-task Individuals at once, (ii) enabling reuse/adaptation for new tasks and continual learning, and (iii) providing interpretability; to our knowledge, no baseline offers all of these.

Table 1. Mean ± SD over 3 seeds (8 tasks each).

MethodGLUE 10kGLUE 20kSuperGLUE 10kSuperGLUE 20k
JIVEAdapter (ours)88.29 ± 0.4088.96 ± 0.1181.11 ± 0.2281.58 ± 0.33
MTL-LoRA88.13 ± 0.3788.80 ± 0.0779.78 ± 0.3779.82 ± 0.92
HydraLoRA88.04 ± 0.1288.66 ± 0.1280.96 ± 0.7781.14 ± 1.09
Shared AdaLoRA88.05 ± 0.1688.52 ± 0.3080.46 ± 0.3380.48 ± 0.59
HydraLoRA (storage-matched)87.68 ± 0.2788.29 ± 0.2278.29 ± 0.3079.30 ± 1.54
Shared LoRA87.58 ± 0.1488.12 ± 0.3279.65 ± 0.3480.09 ± 1.25

JIVEAdapter has the highest mean on both benchmarks, within seed variability, consistent with parity; on SuperGLUE it reaches baseline-level accuracy in half the steps, without MoE-style routing.

W2: Simpler alternatives. JIVEAdapter is simple: ΔWt=J+It\Delta W_t = J + I_t, with no router, experts or mixing weights (unlike HydraLoRA, MTL-LoRA, C-Poly). Orthogonality underpins the framework. As in classical JIVE, ΔWt=J+It\Delta W_t = J + I_t with It⊥JI_t \perp J; orthogonality is not an add-on but part of JIVE, just as orthonormal singular vectors are part of SVD. The projection (Eq. 6) guarantees it instead of relying on optimization, making the Joint/Individual split well defined and every algorithm built on JIVE mathematically sound (Alg. 2, 3, 7, adaptation reuse).

Ablations (no-projection = additive, non-orthogonal; same setup and seeds; fixed rank 4 = average adaptive rank after pruning, i.e. the same budget).

MethodGLUESuperGLUE
JIVEAdapter88.96 ± 0.1181.58 ± 0.33
No-projection88.85 ± 0.0581.14 ± 0.50
Fixed-rank88.53 ± 0.1081.13 ± 0.78

Without the projection, orthogonality is learned only when tasks are separable. On GLUE, 8.6% of each Individual's energy lies in the Joint subspace without the projection vs. 7.9% with it (0.5%). GLUE/SuperGLUE tasks are heterogeneous, so the model separates them on its own (see the t-SNE, Figure 6). For homogeneous, non-separable tasks this is much harder, and the projection is what guarantees it.

The goal of adaptive rank is not only accuracy but finding the best rank dynamically, removing the need for a grid search. Still, because rank changes gradually across layers, matrices and training stages, it is slightly better than the fixed rank, as AdaLoRA is over LoRA. We will add a fixed-rank sweep (2 to 6).

Shared-only adapters (Table 1) are lowest and cannot do reuse or adaptation.

Q1: Parameters and cost. Exact adapter counts (Table 2 of the paper mixed per-task and total counts):

MethodStored (disk)In RAM, forward pass
JIVEAdapter, 1 task1.33M1.33M
LoRA r8 / AdaLoRA1.33M1.33M
JIVEAdapter, 8 tasks5.97M1.33M
MTL-LoRA2.69M2.66M
HydraLoRA2.90M2.90M

8 single-task adapters need 10.62M. We store more on disk than MTL-LoRA/HydraLoRA but use about half in RAM/forward pass. Storage-matched HydraLoRA (rank 17) is worse (Table 1), so our results do not come from storage. Compute is hardware-dependent; for training, we reach SuperGLUE baseline accuracy in half the steps.

Q2: Similarity for an unseen task. The target is held out (leave-one-out; Section 4.1) and never used to learn the Joint or any Individual. Similarity needs no Individual (Task2Vec/GPM-style, Eq. 7): on the frozen W + J, we fit only a temporary linear head, compute loss gradients w.r.t. W over a few training minibatches, and take the top r_I left singular vectors as the signature; held-in tasks are computed the same way. We route to the highest overlap (Eq. 8) and learn only one scale per direction. No validation/test data is used.

Q3: Novelty. We do not claim any single ingredient is new. The novelty is one simple structure, classical JIVE in weight space, that covers multi-task, single-task, transfer (reuse/adaptation), continual learning (10.0–18.4 less forgetting on GLUE, Table 6) and interpretability (similarity, clustering, Alg. 7). Prior methods (e.g., MTLoRA, O-LoRA, InfLoRA) each address one setting. We will add C-Poly, MTLoRA and InfLoRA to related work.

AuthorsOpenReview ↗

Part2 of response

Frozen Joint with only the classifier head trained. We keep the backbone, the frozen Joint and all held-in Individuals fixed, set the target's Individual to zero and train only a new head (same Stage-1 setup, step budget and seed 6 as the runs with an Individual).

TargetHead onlyWith an IndividualDiff.
CB (acc)71.4382.14 (reused MultiRC Individual)+10.71
COPA (acc)88.0091.00 (newly trained Individual)+3.00

On both targets an Individual, reused or newly trained, adds accuracy over the head alone, so the frozen Joint does not carry all the structure a new task needs; the Individual adds the task-specific part. We will run the remaining five targets in the final version.

W1/Q1(a,b): Rank sweep and Joint/Individual split (App. D.3, D.4). Same backbone (DeBERTaV3-base), schedule and seed.

Effective per-task rank (GLUE)4816
Eight-task mean89.1688.9588.82
J/I split at rank 8 (SuperGLUE)2/64/46/2Allocator (Tab. 2)
Eight-task mean80.9282.2682.1482.06

The mean moves only 0.34 points across a fourfold rank change, so the adaptation is intrinsically low rank: the Joint spans the shared directions and each Individual adds only a few residual ones. Extra rank fits noise; small tasks overfit (CoLA 72.66 to 70.16, MRPC 90.44 to 88.97). The split matters: a balanced split is best, and the allocator is within 0.2 of it without searching over the split.

Reviewer_ruK7OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In multi-task PEFT, tasks usually share one low-rank adapter or a common down-projection with task heads. Inspired by JIVE decomposition, authors argue shared and task-specific updates should live in orthogonal subspaces. They propose JIVEAdapter, which sums a shared SVD update JJ and a task-specific update ItI_t projected orthogonal to JJ (Eq. 6). It uses separate rank budgets for JJ and ItI_t, and allows reusing a frozen JJ on new tasks. On DeBERTaV3-base across GLUE and SuperGLUE, it gets 89.44 / 78.65 single-task average (Tab. 1) and 89.07 / 82.06 multi-task average (Tab. 2).

Critical strengths
  1. Simple design. The adapter is simply J+ItJ + I_t (Eq. 1) with 1.33M active parameters per task (Tab. 2), so it is a simple drop-in change.
  2. Competitive GLUE score. It beats HydraLoRA (89.07 vs 88.80, winning 7/8 tasks) and MTL-LoRA (88.42) on GLUE (Tab. 2), and candidly reports negative results (Tab. 6-7).
  3. Frozen-JJ reuse. Training a fresh 0.66M ItI_t on top of frozen JJ gets 81.82 on the 7 held-out SuperGLUE tasks (Fig. 2, vs ~80.5 for the same tasks in Tab. 1).
Critical weaknesses

JIVEAdapter is competitive with HydraLoRA on GLUE (Tab. 2), but experiments do not show "JIVE-style orthogonal projection" drives the gain. Key comparisons also have issues with rank-allocator selection, total parameter count, and single-seed noise.

W1. No ablation isolates orthogonality or the shared+private split (Tab. 2, Fig. 5). Paper never tests removing the orthogonal projection (Eq. 6), setting λ=0\lambda=0, or running a standard non-orthogonal shared+private LoRA at the same rank. So there is no evidence that forcing It⊥JI_t \perp J improves accuracy or transfer. In the π\pi-sweep (Fig. 5), every point keeps JJ, ItI_t, and projection, omitting π=1\pi=1 (fully shared LoRA). Also, Tab. 5 lists hyperparameters for fully-shared LoRA and AdaLoRA, but Tab. 2 omits their scores and only compares against HydraLoRA and MTL-LoRA.

W2. Multi-task table picks the best allocator per benchmark, uses ~2x total parameters, and runs 1 seed (Tab. 2, Tab. 7).

  1. Tab. 2 uses the importance allocator on GLUE (89.07), but switches to the permutation test on SuperGLUE (82.06). If we fix either allocator across both benchmarks, JIVEAdapter loses to HydraLoRA on one of them (permutation gets 88.54 < 88.80 on GLUE; importance gets 81.21 < 82.24 on SuperGLUE; Tab. 7).
  2. Tab. 2 lists 1.33M active parameters per task for JIVEAdapter vs 2.90M for HydraLoRA. However, 2.90M is HydraLoRA's total parameter count across all 8 tasks. JIVEAdapter trains one shared JJ plus eight task-specific ItI_t (rank 4 each), which totals approx 5.97M parameters (~2x HydraLoRA).
  3. All results use 1 seed without error bars. Yet nearly identical effective-rank-8 SuperGLUE runs vary by >1 pt across tables (82.26 in Fig. 5, 81.21 in Tab. 7, 82.06 in Tab. 2; WSC swings from 78.85 to 87.50). This noise is larger than the 0.2-0.3 pt margins over HydraLoRA and AdaLoRA.

W3. Reuse and continual-learning claims lack the needed controls (Fig. 2, Fig. 4, Tab. 6).

  1. Reuse experiments miss basic controls, such as training only the classifier head on frozen W+JW+J, using a random/lowest-overlap donor ItI_t, or training a fresh rank-4 LoRA w/o JJ (Sec. 4.1.3). Also, paper claims higher subspace overlap gives higher transfer accuracy (p9 L434), but gives no within-target evidence (e.g., best vs random donor on the same target task). Across the 7 SuperGLUE tasks, overlap and raw accuracy show no positive relation (Spearman ρ\rho ~ -0.11; Figs. 2, 4). This cross-task comparison mixes task difficulty and metrics, so it is only suggestive.
  2. Continual learning lacks a retention control. Alg. 2 updates JJ and re-truncates old ItI_t without old data, so the claim that earlier tasks are preserved (p8 L407) is not verified. Tab. 6 shows a gap to joint training (MNLI -2.5, QQP -3.5, QNLI -7.8, SST-2 -3.1; CoLA 70.23 vs 45.06), but this gap alone does not show forgetting. Paper reports no before/after scores on the same earlier tasks. Tab. 6 also omits per-task LoRA, frozen JJ + fresh ItI_t, and continual LoRA baselines like O-LoRA [R3] and InfLoRA [R4].

W4. Circular diagnostics and missing related work (Fig. 3, Tab. 3). Near-zero energy of ItI_t inside JJ (Fig. 3) is guaranteed by the projection step in Eq. 6. Cross-task ItI_t overlaps (0.00-0.02) simply match random subspaces in 768-dim space (r/dr/d ~ 0.005). Meanwhile, the proposed η\eta metric (Alg. 3) and task attribution score (Alg. 7) are never reported in any experiment. Also, Tab. 3 misses prior shared+task-specific LoRA methods such as C-Poly (ICLR 2024) [R1] and MTLoRA (CVPR 2024) [R2].

References

[R1] H. Wang, T. Sun, C. Jin, et al. Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning. ICLR 2024. arXiv:2312.03248.

[R2] A. A. Agiza, M. Neseem, S. Reda. MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning. CVPR 2024. arXiv:2403.20320.

[R3] X. Wang, T. Chen, Q. Ge, et al. Orthogonal Subspace Learning for Language Model Continual Learning. Findings of EMNLP 2023. arXiv:2310.14152.

[R4] Y.-S. Liang, W.-J. Li. InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. CVPR 2024. arXiv:2404.00228.

Questions for authors
  1. Core ablations, parameter match & seeds (W1, W2). Over 3 seeds with one fixed allocator, can you report: (a) JIVEAdapter w/o orthogonal projection (Eq. 6) and with λ=0\lambda=0, (b) fully-shared LoRA/AdaLoRA (from Tab. 5), and (c) HydraLoRA / MTL-LoRA scaled to match JIVEAdapter's ~5.97M total parameters?
  2. Reuse & continual-learning controls (W3). In Fig. 2, what do (a) frozen W+JW+J + classifier head only, (b) random donor ItI_t, and (c) fresh rank-4 LoRA w/o JJ score? In Tab. 6, what happens if you simply freeze JJ from Stage 1 and train a fresh ItI_t for each new task?
  3. Unused diagnostics (W4). Did you compute the η\eta ratio (Alg. 3) and attribution scores (Alg. 7) on GLUE/SuperGLUE? Also, does GtG_t in Eq. 7 use gradients at W+JW+J or the learned ItI_t (p6 L297)?
3 comments
AuthorsOpenReview ↗

"Response to Reviewer 2: Orthogonality and split ablations, multi-seed results, reuse and continual-learning controls, and diagnostics"

We thank the reviewer for the constructive comments and address each point.

W1/Q1(a,b): Orthogonality and the split. Orthogonality underpins the framework. As in classical JIVE, ΔWt=J+It\Delta W_t = J + I_t with It⊥JI_t \perp J; orthogonality is not an add-on but part of JIVE, just as orthonormal singular vectors are part of SVD. The projection (Eq. 6) guarantees it instead of relying on optimization, making the Joint/Individual split well defined and every algorithm built on JIVE mathematically sound (Alg. 2, 3, 7, adaptation reuse).

Ablations (same setup and seeds; fixed rank 4 = average adaptive rank after pruning, i.e. the same budget).

MethodGLUESuperGLUE
JIVEAdapter88.96 ± 0.1181.58 ± 0.33
No-projection88.85 ± 0.0581.14 ± 0.50
Fixed-rank88.53 ± 0.1081.13 ± 0.78

Without the projection, orthogonality is learned only when tasks are separable. Just like we never impose orthogonality between Individuals, yet training learns it: the Individuals separate clearly in t-SNE (Fig. 6), since separating them helps fit all tasks. On GLUE, 8.6% of each Individual's energy lies in the Joint subspace without the projection vs. 7.9% with it (≈0.5%). GLUE/SuperGLUE tasks are heterogeneous; for homogeneous, non-separable tasks this is much harder, and the projection is what guarantees it.

The goal of adaptive rank is not only accuracy but finding the best rank dynamically, removing the need for a grid search. Still, because rank changes gradually across layers, matrices and training stages, it is slightly better than the fixed rank, as AdaLoRA is over LoRA. We will add a fixed-rank sweep (2 to 6).

(b) Split. Fully shared LoRA/AdaLoRA (no split) are lowest (Table 1), and without per-task Individuals on a shared Joint there is no reuse, adaptation, continual learning or interpretability.

W2/Q1(c): Allocator, parameters, seeds. We fix the importance-based allocator; single-seed gaps were within seed variability. On the size of the gains. PEFT, including ours, aims not to beat but to match full fine-tuning with few parameters and less forgetting. GLUE/SuperGLUE are near saturation, so large gaps are not expected. We claim parity while (i) learning a shared Joint and per-task Individuals at once, (ii) enabling reuse/adaptation for new tasks and continual learning, and (iii) providing interpretability; to our knowledge, no baseline offers all of these.

We store more on disk but use about half in the forward pass. Storage-matched HydraLoRA is worse (Table 1), so our results do not come from storage.

Table 1. Mean ± SD, 3 seeds.

MethodParams: stored (disk) / used per forward passGLUE 10kGLUE 20kSuperGLUE 10kSuperGLUE 20k
JIVEAdapter (ours)5.97M / 1.33M88.29 ± 0.4088.96 ± 0.1181.11 ± 0.2281.58 ± 0.33
MTL-LoRA2.69M / 2.66M88.13 ± 0.3788.80 ± 0.0779.78 ± 0.3779.82 ± 0.92
HydraLoRA2.90M / 2.90M88.04 ± 0.1288.66 ± 0.1280.96 ± 0.7781.14 ± 1.09
Shared AdaLoRA1.33M / 1.33M88.05 ± 0.1688.52 ± 0.3080.46 ± 0.3380.48 ± 0.59
HydraLoRA r17 (matched)≈6M / ≈6M87.68 ± 0.2788.29 ± 0.2278.29 ± 0.3079.30 ± 1.54
Shared LoRA1.33M / 1.33M87.58 ± 0.1488.12 ± 0.3279.65 ± 0.3480.09 ± 1.25

JIVEAdapter has the highest mean on both, within seed variability (parity); on SuperGLUE it reaches baseline-level accuracy in half the steps, without MoE-style routing.

W3/Q2: Reuse and continual learning. Our claim is per target (a higher-overlap donor transfers better); cross-target correlation mixes difficulty and metrics. A random donor lowers accuracy on every target:

TargetBest donorRandom donorDiff.
BoolQMultiRCWiC−9.63
WSCMultiRCRTE−3.85
MultiRCBoolQWiC−1.40
COPAWiCBoolQ−1.00
RTEMultiRCCB−0.72

Full pairwise, multi-seed comparisons will be in the final version. A frozen Joint plus a fresh Individual per new task is our adaptation mode (Figure 3a, 81.82); old tasks are unchanged by construction. Incremental JIVE reduces forgetting vs. naive sequential training by 10.0–18.4 (GLUE) and 5.3–8.1 (SuperGLUE) points (Table 6).

W4/Q3: Diagnostics. Figure 3 is a consistency check; even without the projection 92% of Individual energy still lies outside the Joint. Near-random overlap between Individuals (0.00–0.02) is intended: shared structure goes into the Joint. Routing instead uses gradient signatures (overlap ≈0.17–0.25). η and the attribution score were proposed as future work; omitted for space, we will add them. g in Eq. 7 is the loss gradient w.r.t. the frozen W at W + J with the target Individual off, over B minibatches; no trained Individual is needed.

Related work. we will add C-Poly, MTLoRA (CVPR'24) and InfLoRA.

AuthorsOpenReview ↗

Part 2 of response

Continuing our response above (W1/Q1(a,b), W3/Q2(a), W4/Q3).

Ablation of the Joint (J) and Individuals (I) (W1/Q1(b), W4/Q3; Alg. 7 removal and swapping). To test the split directly, we remove each component at evaluation time from a trained model. Removing either component lowers the mean: removing the Joint lowers the mean by 15.12 (GLUE) and 7.54 (SuperGLUE) points, and removing a task's own Individual by 14.97 and 17.45. In swapping, every task scores best with its own Individual. Both components are therefore needed, and each Individual carries task-specific information that the Joint does not.

TaskFullNo JNo ITaskFullNo JNo I
MNLI90.2762.4877.62BoolQ85.7881.6268.23
QNLI94.5585.9569.93CB91.8083.0945.98
SST-296.2291.7566.06RTE87.3668.9576.53
QQP89.1532.3781.64MultiRC59.6046.3038.86
RTE89.1775.0975.45WiC72.4170.5359.24
MRPC91.3788.8681.30WSC84.6276.9368.27
CoLA70.0669.4652.36COPA90.0086.0090.00
STS-B92.1185.9788.74ReCoRD87.7785.6272.61
Mean89.1173.99 (−15.12)74.14 (−14.97)Mean82.4274.88 (−7.54)64.97 (−17.45)

No J / No I: Joint / own Individual removed.

W3/Q2(a): Frozen Joint with only the classifier head trained. We keep the backbone, the frozen Joint and all held-in Individuals fixed, set the target's Individual to zero and train only a new head (same Stage-1 setup, step budget and seed 6 as the runs with an Individual).

TargetHead onlyWith an IndividualDiff.
CB (acc)71.4382.14 (reused MultiRC Individual)+10.71
COPA (acc)88.0091.00 (newly trained Individual)+3.00

On both targets an Individual, reused or newly trained, adds accuracy over the head alone, so the frozen Joint does not carry all the structure a new task needs; the Individual adds the task-specific part. We will run the remaining five targets in the final version.

W1/Q1(a,b): Rank sweep and Joint/Individual split (App. D.3, D.4). Same backbone (DeBERTaV3-base), schedule and seed.

Effective per-task rank (GLUE)4816
Eight-task mean89.1688.9588.82
J/I split at rank 8 (SuperGLUE)2/64/46/2Allocator (Tab. 2)
Eight-task mean80.9282.2682.1482.06

The mean moves only 0.34 points across a fourfold rank change, so the adaptation is intrinsically low rank: the Joint spans the shared directions and each Individual adds only a few residual ones. Extra rank fits noise; small tasks overfit (CoLA 72.66 to 70.16, MRPC 90.44 to 88.97). The split matters: a balanced split is best, and the allocator is within 0.2 of it without searching over the split.

Thanks for new exps! Your view is interesting and my major focus is your share/task-specific in structured way. Pity that they are not working as expected.

  • The multi-seed, fixed allocator and the stored/used parameter comments resolve my W2.
  • However, the no-projection ablation is within noise of the full method (88.96 ± 0.11 vs 88.85 ± 0.05 on GLUE; 81.58 ± 0.33 vs 81.14 ± 0.50 on SuperGLUE), and the Individual energy inside the Joint barely changes (8.6% vs 7.9%). I think this means orthogonal split, your key design, does not matter as much as expected. And accuracy is at parity with the baselines.
  • W3/W4 are only partly addressed. Thanks for the random-donor and head-only controls. But continual learning still has no before/after results on old tasks and no comparison with O-LoRA/InfLoRA, and η is still not reported.

I am sorry to reduce my rating to 2.

Reviewer_E63MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JIVEAdapter for multi-task adaptation. It decomposes each task’s low-rank weight update into a shared joint component and a task-specific individual component, applies a projection to keep the two components near-orthogonal, and adaptively allocates rank within the two separate pools. JIVEAdapter performs comparably to the evaluated low-rank adaptation baselines on GLUE and SuperGLUE. Cross-genre experiments on MNLI further demonstrate the potential for reusing the learned joint and individual components.

Critical strengths
  1. The method is intuitive. It explicitly separates parameter updates into shared joint and task-specific individual structures and uses an orthogonal projection to separate their subspaces. This decomposition provides a useful organization for multi-task adaptation. The subspace analysis further illustrates the separation between the learned joint and individual components.
  2. The reuse strategy offers a practical way to adapt to new tasks. It freezes the joint component and either calibrates a borrowed individual component or learns a new one. This allows the model to reuse an existing task pool without retraining the shared component.
  3. The paper presents extensive comparisons with baseline methods. JIVEAdapter achieves competitive results on GLUE and SuperGLUE in both single-task and multi-task adaptation. On MNLI, reusing and lightly calibrating an existing individual component performs nearly as well as training a new one, demonstrating effective reuse.
Critical weaknesses
  1. The performance gains are small and vary across tasks. JIVEAdapter falls behind some baselines on BoolQ and WSC in Table 1, and BoolQ, RTE, and WiC in Table 2. Multiple runs would help show whether the small differences are reliable.
  2. The benefit of the orthogonality constraint is not clearly isolated. Appendix D.3 provides subspace and rank-split analyses, but a matched comparison without the orthogonality constraint would help establish its contribution. Algorithm 7 also describes component removal and swapping, but I could not find the corresponding performance results.
  3. Reusing an existing Individual is less effective across heterogeneous SuperGLUE tasks than across MNLI genres. The paper discusses this limitation, but the benefit of similarity-based donor selection remains unclear. Figures 2 and 4 do not show whether the highest-similarity donor performs better than other donors. Comparing different donor Individuals for the same target task under the same calibration budget would help establish the value of this selection strategy.
Questions for authors
  1. Please report means and standard deviations over multiple seeds for JIVEAdapter and the strongest multi-task baseline.
  2. Could you add an ablation that removes only the projection between the Joint and Individual components? Please also report the component removal and swapping results described in Algorithm 7.
  3. On some MNLI genres, similarity-based reuse outperforms adaptation despite training fewer parameters. How do you explain this result, and does it hold across multiple seeds?
  4. Does selecting the highest-similarity donor perform better than other choices? A comparison of different donor Individuals for the same target task under the same calibration budget would help clarify this.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1: Multi-seed results, projection and adaptive-rank ablations, component removal and swapping, and reuse vs. adaptation

We thank the reviewer for the constructive comments and address each point below.

On the size of the gains. The goal of PEFT methods, including ours, is not to improve accuracy but to match full fine-tuning while training and storing few parameters and forgetting less. GLUE and SuperGLUE are near saturation at this size, so large gaps among strong adapters are not expected (e.g., shared LoRA vs. AdaLoRA differ by 0.40 on GLUE). We claim parity while (i) learning a shared Joint and per-task Individuals at once, (ii) enabling reuse/adaptation for new tasks and continual learning, and (iii) providing interpretability; to our knowledge, no baseline offers all of these.

Table 1. Mean ± SD over 3 seeds (6,42,123).

MethodGLUE 10kGLUE 20kSuperGLUE 10kSuperGLUE 20k
JIVEAdapter (ours)88.29 ± 0.4088.96 ± 0.1181.11 ± 0.2281.58 ± 0.33
MTL-LoRA88.13 ± 0.3788.80 ± 0.0779.78 ± 0.3779.82 ± 0.92
HydraLoRA88.04 ± 0.1288.66 ± 0.1280.96 ± 0.7781.14 ± 1.09
Shared AdaLoRA88.05 ± 0.1688.52 ± 0.3080.46 ± 0.3380.48 ± 0.59
HydraLoRA (storage-matched)87.68 ± 0.2788.29 ± 0.2278.29 ± 0.3079.30 ± 1.54
Shared LoRA87.58 ± 0.1488.12 ± 0.3279.65 ± 0.3480.09 ± 1.25

JIVEAdapter has the highest mean on both benchmarks, within seed variability, consistent with parity; on SuperGLUE it reaches baseline-level accuracy in half the steps, without MoE-style routing.

Orthogonality underpins the framework. As in classical JIVE, ΔWt=J+It\Delta W_t = J + I_t with It⊥JI_t \perp J; orthogonality is not an add-on but part of JIVE, just as orthonormal singular vectors are part of SVD. The projection (Eq. 6) guarantees it instead of relying on optimization, making the Joint/Individual split well defined and every algorithm built on JIVE mathematically sound (Alg. 2, 3, 7, adaptation reuse).

Ablations (same setup and seeds; fixed rank 4 = average adaptive rank after pruning, i.e. the same budget).

MethodGLUESuperGLUE
JIVEAdapter88.96 ± 0.1181.58 ± 0.33
No-projection88.85 ± 0.0581.14 ± 0.50
Fixed-rank88.53 ± 0.1081.13 ± 0.78

Without the projection, orthogonality is learned only when tasks are separable. On GLUE, 8.6% of each Individual's energy lies in the Joint subspace without the projection vs. 7.9% with it (~0.5%). GLUE/SuperGLUE tasks are heterogeneous, so the model separates them on its own (see the t-SNE, Figure 6). For homogeneous, non-separable tasks this is much harder, and the projection is what guarantees it.

The goal of adaptive rank is not only accuracy but finding the best rank dynamically, removing the need for a grid search. Still, because rank changes gradually across layers, matrices and training stages, it is slightly better than the fixed rank, as AdaLoRA is over LoRA. We will add a fixed-rank sweep (2 to 6).

Removal and swapping (Alg. 7), evaluation-time on a trained model: removing either component lowers performance, and in swapping every task scores best with its own Individual. No J / No I: Joint / own Individual removed.

TaskFullNo JNo ITaskFullNo JNo I
MNLI90.2762.4877.62BoolQ85.7881.6268.23
QNLI94.5585.9569.93CB91.8083.0945.98
SST-296.2291.7566.06RTE87.3668.9576.53
QQP89.1532.3781.64MultiRC59.6046.3038.86
RTE89.1775.0975.45WiC72.4170.5359.24
MRPC91.3788.8681.30WSC84.6276.9368.27
CoLA70.0669.4652.36COPA90.0086.0090.00
STS-B92.1185.9788.74ReCoRD87.7785.6272.61
Mean89.1173.99 (−15.12)74.14 (−14.97)Mean82.4274.88 (−7.54)64.97 (−17.45)

Reuse vs. adaptation. We intentionally show a positive and a negative case to show when each fits. Positive (MNLI genres, Figure 3b): reuse matches adaptation (89.33 vs. 89.36); genres share the same task and labels, so the Joint and routed Individual suffice; adaptation trains a fresh Individual and may overfit, while reuse trains only scales (74 vs. 218 tensors); gaps are within sampling error (≈0.7). Negative (heterogeneous SuperGLUE, Figure 3a): adaptation wins. Thus the choice depends on how much of the new task the Joint covers, not on overlap (≈0.17 to 0.25 in both).

Does the most similar donor help? Yes: with everything else fixed, a random donor lowers accuracy on every target, slightly for the same task type and more otherwise.

TargetBest donorRandom donorDiff.
BoolQMultiRCWiC−9.63
WSCMultiRCRTE−3.85
MultiRCBoolQWiC−1.40
COPAWiCBoolQ−1.00
RTEMultiRCCB−0.72

Full pairwise, multi-seed comparisons will be in the final version.

AuthorsOpenReview ↗

Part 2 of response

Rank sweep and Joint/Individual split (App. D.3, D.4). Same backbone (DeBERTaV3-base), schedule and seed.

Effective per-task rank (GLUE)4816
Eight-task mean89.1688.9588.82
J/I split at rank 8 (SuperGLUE)2/64/46/2Allocator (Tab. 2)
Eight-task mean80.9282.2682.1482.06

The mean moves only 0.34 points across a fourfold rank change, so the adaptation is intrinsically low rank: the Joint spans the shared directions and each Individual adds only a few residual ones. Extra rank fits noise; small tasks overfit (CoLA 72.66 to 70.16, MRPC 90.44 to 88.97). The split matters: a balanced split is best, and the allocator is within 0.2 of it without searching over the split.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.