- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
JIVEAdapter introduces a multi-task additive low-rank adapter inspired by Joint and Individual Variation Explained. Each task update is decomposed into a shared Joint component and a near-orthogonal task-specific Individual component, with adaptive rank allocation across the two pools. The learned Joint can be reused for new tasks either by calibrating a related task’s Individual or by training a new small Individual.
Critical strengths
-
The Joint-plus-Individual decomposition is a meaningful and interpretable multi-task adapter design.
-
The empirical coverage is broad. The paper evaluates single-task and multi-task performance, incremental task arrival, held-out reuse, alternative rank estimators, and learned subspace geometry rather than relying on one benchmark table.
Critical weaknesses
-
Almost all reported results use only one fixed random seed. And several reported gains are small—for example, 89.07 versus 88.80 on multi-task GLUE
-
The paper lacks a decisive mechanism ablation against simpler alternatives, such as an additive shared/individual adapter without orthogonality or the same structure with fixed ranks. The geometry plots confirm that the learned spaces are nearly orthogonal, but they do not establish that orthogonality or adaptive allocation causes the performance and transfer gains.
Questions for authors
-
Is the parameter comparison with multi-task baselines fully matched in the relevant deployment setting? The paper reports trainable parameters per task head, but the total number of stored Individual parameters grows with the number of tasks. Please report total trainable and stored parameters for the full eight-task model, as well as inference-time active parameters and training-time computation.
-
How is subspace similarity computed for a truly unseen task before its Individual component has been trained?
-
Can the authors clarify the novelty boundary relative to existing shared-plus-specific and orthogonal LoRA methods? The paper discusses HydraLoRA, MTL-LoRA, O-LoRA, and LoRAHub, but several elements—shared and task-specific components, orthogonal subspaces, and adapter reuse—have prior counterparts.
2 comments
Part2 of response
Frozen Joint with only the classifier head trained. We keep the backbone, the frozen Joint and all held-in Individuals fixed, set the target's Individual to zero and train only a new head (same Stage-1 setup, step budget and seed 6 as the runs with an Individual).
| Target | Head only | With an Individual | Diff. |
|---|---|---|---|
| CB (acc) | 71.43 | 82.14 (reused MultiRC Individual) | +10.71 |
| COPA (acc) | 88.00 | 91.00 (newly trained Individual) | +3.00 |
On both targets an Individual, reused or newly trained, adds accuracy over the head alone, so the frozen Joint does not carry all the structure a new task needs; the Individual adds the task-specific part. We will run the remaining five targets in the final version.
W1/Q1(a,b): Rank sweep and Joint/Individual split (App. D.3, D.4). Same backbone (DeBERTaV3-base), schedule and seed.
| Effective per-task rank (GLUE) | 4 | 8 | 16 |
|---|---|---|---|
| Eight-task mean | 89.16 | 88.95 | 88.82 |
| J/I split at rank 8 (SuperGLUE) | 2/6 | 4/4 | 6/2 | Allocator (Tab. 2) |
|---|---|---|---|---|
| Eight-task mean | 80.92 | 82.26 | 82.14 | 82.06 |
The mean moves only 0.34 points across a fourfold rank change, so the adaptation is intrinsically low rank: the Joint spans the shared directions and each Individual adds only a few residual ones. Extra rank fits noise; small tasks overfit (CoLA 72.66 to 70.16, MRPC 90.44 to 88.97). The split matters: a balanced split is best, and the allocator is within 0.2 of it without searching over the split.
"Response to Reviewer 3: Multi-seed results, ablations, parameter counts, similarity for unseen tasks, and novelty"
We thank the reviewer for the constructive comments and address each point.
W1: Seeds and small gains. On the size of the gains. The goal of PEFT methods, including ours, is not to improve accuracy but to match full fine-tuning while training and storing few parameters and forgetting less. GLUE and SuperGLUE are near saturation at this size, so large gaps among strong adapters are not expected (e.g., shared LoRA vs. AdaLoRA differ by 0.40 on GLUE). We claim parity while (i) learning a shared Joint and per-task Individuals at once, (ii) enabling reuse/adaptation for new tasks and continual learning, and (iii) providing interpretability; to our knowledge, no baseline offers all of these.
Table 1. Mean ± SD over 3 seeds (8 tasks each).
JIVEAdapter has the highest mean on both benchmarks, within seed variability, consistent with parity; on SuperGLUE it reaches baseline-level accuracy in half the steps, without MoE-style routing.
W2: Simpler alternatives. JIVEAdapter is simple: ΔWt=J+It, with no router, experts or mixing weights (unlike HydraLoRA, MTL-LoRA, C-Poly). Orthogonality underpins the framework. As in classical JIVE, ΔWt=J+It with It⊥J; orthogonality is not an add-on but part of JIVE, just as orthonormal singular vectors are part of SVD. The projection (Eq. 6) guarantees it instead of relying on optimization, making the Joint/Individual split well defined and every algorithm built on JIVE mathematically sound (Alg. 2, 3, 7, adaptation reuse).
Ablations (no-projection = additive, non-orthogonal; same setup and seeds; fixed rank 4 = average adaptive rank after pruning, i.e. the same budget).
Without the projection, orthogonality is learned only when tasks are separable. On GLUE, 8.6% of each Individual's energy lies in the Joint subspace without the projection vs. 7.9% with it (0.5%). GLUE/SuperGLUE tasks are heterogeneous, so the model separates them on its own (see the t-SNE, Figure 6). For homogeneous, non-separable tasks this is much harder, and the projection is what guarantees it.
The goal of adaptive rank is not only accuracy but finding the best rank dynamically, removing the need for a grid search. Still, because rank changes gradually across layers, matrices and training stages, it is slightly better than the fixed rank, as AdaLoRA is over LoRA. We will add a fixed-rank sweep (2 to 6).
Shared-only adapters (Table 1) are lowest and cannot do reuse or adaptation.
Q1: Parameters and cost. Exact adapter counts (Table 2 of the paper mixed per-task and total counts):
8 single-task adapters need 10.62M. We store more on disk than MTL-LoRA/HydraLoRA but use about half in RAM/forward pass. Storage-matched HydraLoRA (rank 17) is worse (Table 1), so our results do not come from storage. Compute is hardware-dependent; for training, we reach SuperGLUE baseline accuracy in half the steps.
Q2: Similarity for an unseen task. The target is held out (leave-one-out; Section 4.1) and never used to learn the Joint or any Individual. Similarity needs no Individual (Task2Vec/GPM-style, Eq. 7): on the frozen W + J, we fit only a temporary linear head, compute loss gradients w.r.t. W over a few training minibatches, and take the top r_I left singular vectors as the signature; held-in tasks are computed the same way. We route to the highest overlap (Eq. 8) and learn only one scale per direction. No validation/test data is used.
Q3: Novelty. We do not claim any single ingredient is new. The novelty is one simple structure, classical JIVE in weight space, that covers multi-task, single-task, transfer (reuse/adaptation), continual learning (10.0–18.4 less forgetting on GLUE, Table 6) and interpretability (similarity, clustering, Alg. 7). Prior methods (e.g., MTLoRA, O-LoRA, InfLoRA) each address one setting. We will add C-Poly, MTLoRA and InfLoRA to related work.