acceptodds
Withdrawn from ICLR 2027

PLEAT: Progressive LoRA Expert Aggregation and Transport for Continual Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual learning aims to acquire knowledge from new tasks while retaining previously learned capabilities. One way to pursue this goal is through model merging, which combines the weights of independently trained experts without additional training. A central challenge in continual model merging is to integrate new experts without retaining an expanding expert pool. In resource-constrained settings, merging experts trained with low-rank adaptation (LoRA) must also preserve their compact representation. Existing merging strategies face three limitations in this setting: (1) joint merging requires retaining and revisiting an expanding collection of historical experts; (2) methods designed for fully fine-tuned models typically operate on dense updates, giving up LoRA’s compact representation when applied directly to LoRA experts; and (3) fixed-rank LoRA merging preserves compactness but can become a capacity bottleneck as knowledge accumulates over long task sequences. We propose PLEAT, Progressive LoRA Expert Aggregation and Transport, a training-free approach that exploits the low-rank structure of LoRA experts to maintain a factorized merged update with progressively adjustable capacity. PLEAT incrementally updates a shared input basis under a prescribed rank schedule, aligns each incoming expert to the updated representation, and transports historical coefficients before aggregation. Experiments across multiple backbone models show that PLEAT outperforms a range of existing merging methods, while retaining the parameter-efficient representation of LoRA and enabling continual merging without revisiting historical expert weights.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual model merging, low-rank adaptation, continual learning, parameter-efficient fine-tuning, multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uZaCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes PLEAT a training-free method for continual model merging of LoRA adapters (experts) into one factrized adapter. For each module it maintains a row-orthonormal input basis by incremental SVD over the arriving A factors, with a nondecreasing rank chosen by an energy threshold (or a schedule). Each incoming expert is aligned to the current basis with a Procrustes map, and the accumulated output coefficients are transported to the new basis with a second Procrustes map before the incoming term is added. Experiments on LLaVA-8, DriveLM-7, CLIP-20 (three ViTs) and a 32-task DriveLM repartition (Scene32) report the best suite averages against averaging, task-vector, LoRA-aware merging methods and continual merging baselines.

Critical strengths
  1. Interesting and relevant problem.
  2. Evaluation part with multimodal LoRA experts (LLaVA, DriveLM) as well with CLIP-based commonly used setting.
Critical weaknesses
  1. Contribution 1, "Historical Coordinate Consistency" says that coefficients must be re-expressed when the basis changes. That is a requirement that is introduced by the method because the basis is changed. Not sure if this is the main contribution (first) supported by Table 6 that shows it only with a 2-D toy example.
  2. OPCM is present in Fig. 4 but not always in a table, i.e., Tab. 2. Why? It's seems like a competitive method, as well as NUFILT that is only present in the Appendix.
  3. Looking at the variation of the PLEAT in Tab.9, e.g. it spread for CLIP (1.2) and the results in Tab.4 PLEAT vs other baselines K-Merge 59.9 - 59.1 can be based on margin. Would be better to report std for multiple seed for other methods.
  4. For CLIP in Tab.4 PLEAT is the only method above zero shot, but still far behind "individual LoRA"
  5. Section 3.2 says PLEAT uses λ = 1/t (l. 246; Table 10, "Scale rule 1/t"). But App. C says PLEAT receives 24 Latin-hypercube configurations over scale [.1, 1.0]?

The code is not provided.

Questions for authors
  1. Can authors elaborate more on the motivation and relation in comparison to KnOTS (Stoica et al., 2025) /Core Space (Panariello et al., 2025)? What benefit PLEAT introduces in comparison to online version of KnOTS/Core Space? KnOTS builds a shared basis from an SVD of the stacked LoRA updates and merges in it. Core Space projects LoRA factors into a common reference basis and merges there without loss.
  2. What bolding in the tables mean, e.g. Tab.3, always PLEAT is bolded, while even in the group of continual methods the other present better results.
  3. Where are the definitions of Online SVD and Fixed basis methods?
  4. Why Fig. 4 a single expert is not starting from 100% - what is exactly reference here?
Reviewer_1GAoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PLEAT continually merges incoming LoRA experts by updating a shared input basis through incremental SVD, aligning incoming experts with Procrustes maps, and transporting historical coefficients without revisiting past weights. Experiments on LLaVA, DriveLM, CLIP show performance and resource benefits, while ablations with matched basis trajectories examine the contribution of historical coordinate transport.

Critical strengths
  1. The method receives only the incoming expert and a historical summary, without retaining an expanding expert pool. It also distinguishes the exported adapter from the state needed for future arrivals, giving the resource advantages a clear practical context.
  2. Copying coefficients after a basis rotation changes the update they represent.
Critical weaknesses
  1. Equation (4) replaces the projected input factor with the orthogonal part of its polar decomposition, discarding singular-value scaling, and reconstruction in Equation (5) is exact only under specific conditions. Under the equivalent reparameterization A→cA and B→B/c, the single-expert orthogonal map is essentially unchanged, but the contribution changes with B. Without explicit canonicalization, the result may depend on the chosen LoRA factorization rather than the expert update itself.
  2. The no-transport control demonstrates the need to update historical coordinates, but does not establish that norm-preserving Procrustes is the right choice. Table 6 already shows that least squares yields lower reconstruction error when the subspace changes; their retention performance should also be compared on real tasks. 3.PLEAT exceeds OPCM by only 0.5 points on DriveLM and K-Merge by 0.8 points in average CLIP accuracy; its three-order average in Appendix J is nearly tied with NUFILT. Although the appendix provides PLEAT's seed-level results, it does not adequately report paired intervals across seeds and orders for these key method differences.
  3. Scene3.2's 32 tasks come from the same driving-data setting, and actual ranks in the resource calibration are only 16–18, yielding identical states under the two larger caps. This supports compactness for that stream, but does not establish long-term retention or the quality–cost trade-off when new directions continue to arrive and the rank cap becomes binding.
Questions for authors

see in the weaknesses

Reviewer_CusWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes PLEAT, a training-free method for continual merging of LoRA experts that keeps the merged update in factorized form while allowing its rank to grow. Each module retains a shared input basis and accumulated output coefficients, so the merged model remains a single LoRA adapter. At each arrival, the basis is updated through an incremental SVD, the incoming expert is aligned to it with a Procrustes map, and the historical coefficients are transported into the new basis before aggregation, without revisiting earlier experts. The main conceptual contribution is the observation that an evolving basis leaves stored coefficients in stale coordinates, so history must be updated together with the basis. Experiments on LLaVA, DriveLM, CLIP, and a 32-task driving-scene sequence show modest improvements over joint, LoRA-aware, and continual merging baselines, with bounded memory compared with retaining and replaying all experts.

Critical strengths

S1: The paper identifies and formalizes a failure mode specific to factorized continual merging: when the shared basis evolves, stored coefficients must be mapped into the new coordinates, even at fixed rank (Table 6). The analysis is concrete. Transport preserves the represented update exactly under span containment, and a computable error decomposition separates the history and incoming residuals (Appendix B). Full PLEAT outperforms a matched no-transport ablation on all four suites (Table 5), which supports the claim that updating historical coordinates matters. The issue is not specific to PLEAT’s design: any factorized continual merger that updates a shared basis must address it, which explains why aligning only incoming experts is insufficient.

S2: PLEAT achieves bounded, replay-free continual merging without data or gradients at merge time. Per LoRA module, it retains only the basis, coefficients, and singular values, and it exports a standard LoRA adapter. At 32 Scene32 arrivals, retained state is 156 MiB versus 4,880 MiB when retaining all experts (Sec. 4.4). On ViT-B/32, it is roughly 4–5.5x faster per arrival than DOP and NUFILT, with about half their peak memory and equal or higher accuracy on the seed-11 collection (Table 16). This matters because the practical motivation for continual merging is to avoid a growing expert pool, and PLEAT does so while keeping a deployable single-adapter form and requiring no merge-time optimization.

Critical weaknesses

W1: The experiments do not adequately isolate the choice of transport operator or the contribution of progressive rank growth. The transport controls (Tables 5 and 12, Fig. 16) copy historical coefficients rather than mapping them into the new basis, and the task suites never compare Procrustes with least-squares transport under the same basis and rank trajectory. Appendix B.3 derives the reconstruction trade-off between the two analytically, but the only numerical comparison is the synthetic example in Table 6. The residual diagnostics defined in Appendix F also do not appear to be reported for real experts. For rank, Appendix H shows that capacity matters on CLIP: at the 1/t1/t scale, fixed rank 128 exceeds fixed rank 64 by 2.3 points (Table 13). However, this is a single run (seed 11, O1), and beyond a 0.1–0.2-point difference between the main schedule and constant rank 256, it does not establish a benefit of progressive growth over a constant cap. On CLIP, with rank-8 experts, the requested caps of 64, 128, and 256 at 8, 14, and 20 arrivals meet or exceed the available directions at each milestone, so the cap does not truncate at those points (the paper does not say whether the energy rule also truncates there; see W4). On Scene32, the resource calibration ends at ranks of only 16–18 after 32 arrivals, so caps of 64 and 256 yield identical states (Sec. 4.4). Procrustes transport and progressive capacity are central to the paper’s framing, so it remains unclear how much of the main-suite performance is attributable to each (see Q2, Q3).

W2: The empirical gains are modest in several settings, and their uncertainty is not quantified. On CLIP-20, PLEAT’s cross-backbone average of 59.9 is only 0.3 points above the pretrained reference of 59.6, and on ViT-B/16 it matches the reference (60.3 vs. 60.33, Table 4). At 14 tasks, PLEAT falls below the pretrained reference on all three backbones, as do all compared methods, although the reference uses canonical-order prefixes while the merging methods average three arrival orders. Margins over the strongest reported baselines are 0.9 points on LLaVA (TIES, normalized mean, Table 2), 0.5 on DriveLM (OPCM, Table 3), and 0.8 on CLIP (K-Merge, Table 4). In Fig. 4, which appears to report LLaVA-8, OPCM is within 0.1 points of PLEAT on the expert-relative score. The paper reports no confidence intervals or paired significance tests for these comparisons. Because all methods share the same experts within a seed, the relevant quantity is the paired margin, yet baseline results are not reported per seed. On CLIP, the 0.8-point margin is smaller than PLEAT’s own across-seed range of 59.2–60.4 (Table 9), which does not by itself establish significance but means the margin cannot be assessed without paired results. Finally, in the single-run seed-11, O1 analysis, PLEAT’s fixed early-task cohort accuracy falls to 50.0 at 20 tasks, below NUFILT’s 55.6 (Table 12), although its backward transfer (+2.9) is the highest of the four methods, so the aggregate endpoint does not fully characterize retention. Since the abstract claims that PLEAT outperforms a range of existing merging methods, small margins without uncertainty estimates limit how strongly this can be concluded (see Q6).

W3: The incoming alignment introduces an uncontrolled rescaling of each expert. Proc(At,Qt)\mathrm{Proc}(A_t, Q_t) is not invariant to general factor reparameterizations (BtGt,Gt−1At)(B_t G_t, G_t^{-1} A_t) of Eq. 2 (it is invariant to orthogonal GtG_t, but not to rescalings or other invertible GtG_t), so the magnitude of an expert’s contribution depends on how its scale is split between AtA_t and BtB_t, not on its update BtAtB_t A_t alone. For example, absorbing the loader multiplier into AtA_t rather than BtB_t (Sec. 3.1) would change each expert’s contribution by that factor. In the reported configurations α/r=2\alpha/r = 2 for every expert (Appendix C), so if the loader multiplier is α/r\alpha/r, this particular change would be a global scale, but the same argument applies to any per-expert split of scale between AtA_t and BtB_t. Eq. 21 further shows that the singular values of AtQt⊤A_t Q_t^\top are replaced by 11, so incoming reconstruction is inexact even when AtA_t lies in the span of QtQ_t, unless its rows are orthonormal. The paper notes the scaling term (Appendix B.5) but does not quantify it for real experts. The crossed controls in Appendix F (Eq. 22) toggle Procrustes against a coordinate-copy map EE, so they test whether alignment helps, not how much of that effect is due to the rescaling. Appendix C also states that the exact paired mean has no tuned parameter, whereas PLEAT’s configuration search includes a scale range (see W4), so a difference in effective scale could contribute to the gap as well. No experiment separates the rescaling from the geometric alignment, so part of the reported gains over baselines that apply no such rescaling (e.g., the paired mean) may come from it (see Q1).

W4: Several baseline rows and reported settings are inconsistent across tables, which makes some comparisons hard to interpret:

• “Paired mean” matches “Independent factor mean” (Table 5) on LLaVA and CLIP but not on DriveLM or Scene32, although Eq. 1 shows these are different operators. The “Uniform” row in Table 2 matches neither (Q4).

• OPCM appears in Fig. 4 but is absent from Table 2 (Q5).

• Table 4’s ViT-B/32 20-task values appear to match a single seed and order rather than the stated three-seed, three-order average (Q5).

• The export scale is listed as fixed at 1/t1/t (Table 10), but Appendix C describes a search over a scale range of [0.1,1.0][0.1, 1.0]; it is unclear whether this range is absolute or relative to 1/t1/t.

• CLIP’s rank rule is described as a schedule (Sec. 3.3) but listed with an energy threshold and a single cap (Table 10).

Because these concern the main comparisons and settings, they weaken confidence in the headline results until clarified.

Questions for authors

Q1: Dependence of incoming alignment on the factorization. Proc(At,Qt)\mathrm{Proc}(A_t, Q_t) is not invariant to the general reparameterizations (BtGt,Gt−1At)(B_t G_t, G_t^{-1} A_t) in Eq. 2. For example, PLEAT absorbs the loader multiplier sts_t into BtB_t (Sec. 3.1). Absorbing it into AtA_t instead would change each expert’s contribution by that factor, since Proc(sA,Q)=Proc(A,Q)\mathrm{Proc}(s A, Q) = \mathrm{Proc}(A, Q) for s>0s > 0. (In the reported configurations α/r=2\alpha/r = 2 for every expert, so, if st=α/rs_t = \alpha/r, this particular change is a global scale, the same argument holds for any per-expert split of scale between AtA_t and BtB_t.) Eq. 21 also shows that the singular values of AtQt⊤A_t Q_t^\top are replaced by 11. Could the authors report (i) results with least-squares incoming alignment, Tt=AtQt⊤T_t = A_t Q_t^\top; (ii) results with canonicalized factors, i.e., writing At=LAΣAVA⊤A_t = L_A \Sigma_A V_A^\top and using (BtLAΣA,VA⊤)(B_t L_A \Sigma_A, V_A^\top); and (iii) the singular values of AtQt⊤A_t Q_t^\top on the real experts? This would show how much of PLEAT’s advantage over exact paired merging comes from this implicit rescaling. If CLIP never truncates, and “Paired mean” is the exact paired mean (see Q4) exported with the same 1/t1/t scale, this rescaling could account for much or all of PLEAT’s gap to the paired mean in Table 4.

Q2: Transport operator. The no-transport control copies historical coefficients without mapping them to the new basis (Sec. 4.3). Could the authors (i) compare Procrustes transport against least-squares transport, RtLS=Qt−1Qt⊤R_t^{\mathrm{LS}} = Q_{t-1} Q_t^\top (Sec. 3.2), under the same basis and rank trajectory on the main suites, and (ii) report the diagnostics etine_t^{\mathrm{in}} and ethiste_t^{\mathrm{hist}} defined in Appendix F, along with the bound in Eq. 20, on real experts across arrivals? Appendix B.3 shows that the squared historical residual of Procrustes exceeds that of least squares by tr[Ct−1(I−S)2Ct−1⊤]≥0\mathrm{tr}\left[C_{t-1}(I - S)^2 C_{t-1}^\top\right] \geq 0, where S=(MM⊤)1/2S = (MM^\top)^{1/2} and M=Qt−1Qt⊤M = Q_{t-1} Q_t^\top. Since least squares is never worse on reconstruction, the choice of Procrustes must be justified empirically, by showing that norm preservation improves task performance. Appendix F defines the diagnostics but reports no measurements, and Table 17 uses synthetic matrices. Do the signed values in Fig. 8 correspond to these quantities? If not, what do they show?

Q3: Contribution of progressive rank growth. Table 13 compares fixed caps with the main schedule on a single collection (seed 11, O1), where the main schedule exceeds constant rank 256 by 0.2 points at the 1/t1/t scale (0.1 at 0.8/t0.8/t and 1.2/t1.2/t). On CLIP with rank-8 experts, the requested caps of 64, 128, and 256 at 8, 14, and 20 arrivals meet or exceed the available directions at each milestone (the paper does not say whether the energy rule also truncates there; see W4). In the Scene32 resource calibration, which uses τ=0.98\tau = 0.98 rather than the main configuration’s 0.95, realized ranks are only 16–18 (Sec. 4.4). Could the authors report the realized ranks in the main configurations of all four suites, which currently appear only in the LLaVA-8 and Scene32 resource calibrations (Table 11, Fig. 5), and extend the progressive-versus-constant-cap comparison across seeds and orders, including a suite where the cap binds (e.g., LLaVA-8 with cap 64, Table 11)? This would show how much of the main-suite performance is attributable to progressive capacity rather than to transport and alignment.

Q4: Which merging operator does each baseline row use? Appendix C lists an “exact paired mean” comparison, presumably the “Paired mean” row, i.e., 1t∑iBiAi\frac{1}{t}\sum_i B_i A_i. Its values match Table 5’s “Independent factor mean”, presumably (1t∑iBi)(1t∑jAj)\left(\frac{1}{t}\sum_i B_i\right)\left(\frac{1}{t}\sum_j A_j\right), on LLaVA (51.4) and CLIP (58.6), but not on DriveLM (40.4 vs. 40.5) or Scene32 (52.4 vs. 48.4). Eq. 1 shows these are different operators. Table 2 also reports a “Uniform” row (56.6) under “Factor-wise and dense merging methods”. If it averages dense updates, it should coincide with the paired mean up to compression to the export rank. If it averages factors, it should coincide with the independent factor mean. It matches neither. Could the authors clarify which operator produced each of the “Uniform,” “Paired mean,” and “Independent factor mean” rows, and the “Factor averaging” line in Fig. 4 (whose final value, 75.6, equals the Paired mean’s Rel.% in Table 2)?

Q5: Missing baseline entries and Table 4 averaging. (i) Fig. 4 appears to report LLaVA-8, since PLEAT’s 81.5 matches Table 2’s Rel.%. It shows OPCM at 81.4, yet OPCM is absent from Tables 2 and 8. Could the authors add it to Table 2, including its normalized score, and define the “Online SVD” control (Appendix C) and the “Fixed basis” method in Figs. 4, 14, and 15? (ii) Table 4 states that it averages three training seeds and three arrival orders. However, its ViT-B/32 20-task values for PLEAT (56.1), OPCM (54.4), and DOP (55.3) equal the seed-11, order-O1 entries in Table 15, and for DOP differ from seed 11’s three-order mean (55.17). Could the authors confirm how this column was averaged?

Q6: Uncertainty and value over zero-shot on CLIP. Could the authors report per-seed results for the baselines in Tables 2–4 (or paired confidence intervals), since all methods share the same experts within a seed? On CLIP, PLEAT’s cross-backbone average at 20 tasks is 59.9 vs. 59.6 for the pretrained model (Table 4), and at 14 tasks it is below the pretrained reference on all three backbones (as are all compared methods), although that reference uses O1 prefixes only while methods average three orders. Could the authors compute the reference over the same three orders, and discuss when merging LoRA experts adds value beyond zero-shot in this setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.