acceptodds
Under review as a conference paper at ICLR 2027

SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation

13 views

Abstract

Parameter-efficient fine-tuning enables fast personalization of text-to-image diffusion models, but composing multiple custom concepts remains challenging due to representation interference. Existing modular methods either rely on expensive post-hoc fusion or freeze adaptation subspaces, which limit expressiveness and concept fidelity. To address this trade-off, we propose Sequential regularized LoRA (SeqLoRA), a constrained continual learning framework that jointly optimizes both LoRA factors via bilevel optimization while keeping each new basis orthogonal to all previously learned ones. Theoretically, we establish monotone descent and convergence to a critical point of the constrained problem, and model the residual layer activations as a matrix sub-Gaussian process to derive a high-probability bound on catastrophic forgetting in multi-layer nonlinear networks. Within the feasible subspace, we prove that a data-adapted basis minimizes the residual interference energy identified by this bound, whereas a random frozen basis is suboptimal in expectation. Experiments on Stable Diffusion with up to 101 concepts show that SeqLoRA improves identity preservation over fusion-based methods, attains the lowest cross-concept leakage in multi-concept compositions, requires no fusion step, and scales to concept counts at which fusion runs out of memory.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, Multi-concept customization, Bilevel optimization, Low-rank adaptation, Text-to-image diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_m8MfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the challenge that while synthesizing a single concept is now very convenient, synthesizing multiple concepts can lead to representation interference, the authors propose Sequential Regularized LoRA (SeqLoRA)—a constrained sequential learning framework—by jointly optimizing two LoRA factors through a two-stage optimization process, while ensuring that each new basis remains orthogonal to all previously learned bases. They provide theoretical derivations and proofs and validate the design through relevant experiments.

Critical strengths
  1. The authors’ proposed SeqLoRA uses a two-layer optimization with cross-Hessian correction to jointly learn two LoRA matrices for new concepts, and employs closed-form projection constraints to ensure orthogonality between the new and old bases, enabling the trained concept modules to be combined as needed.
  2. The authors use Theorem 1 to apply existing convergence analysis tools to analyze the optimization properties of the algorithm. They then utilize Theorem 2 to combine existing probabilistic tools with the structure of orthogonal LoRA to derive an upper bound on forgetting, which is used to analyze the sources of concept interference. Furthermore, they analyze the expected advantage of the optimal basis selected based on data covariance relative to a randomly frozen basis within the feasible subspace, providing a theoretical basis for learning LoRA bases from data.
  3. In experiments involving 8–101 concepts on Stable Diffusion v1.5, the authors verified that SeqLoRA achieves the lowest multi-concept leakage among the compared methods, improves identity preservation, and remains operational even when some fusion baselines run out of GPU memory. The authors also analyzed the effects of concept order, LoRA rank, and concept similarity, as well as the computational cost, and preliminarily validated the feasibility of transferring the method to SDXL.
Critical weaknesses
  1. In multi-concept generation, the base model W_0 provides general knowledge, while LoRA performs additional personalization adjustments. SeqLoRA uniformly applies orthogonal constraints to the newly added adaptation subspaces for different concepts to reduce cross-concept interference. However, different concepts may require some shared adaptation information; therefore, it is unclear whether strict subspace isolation would limit beneficial knowledge reuse. We hope to see further explanation on this point.
  2. SeqLoRA primarily protects existing concepts through orthogonal subspaces; however, its uniform, fixed-rank approach does not flexibly allocate capacity based on the needs of different concepts or layers. This is because the requirements of different modules may vary—some may learn effectively with fewer directions, while others may require more. Furthermore, identity preservation is critical for certain layers, whereas increasing capacity in other layers may yield minimal benefits. A uniform allocation may result in suboptimal resource distribution. Therefore, we hope to see further explanations regarding more targeted interference control and adaptive capacity allocation within the same total budget.
  3. While Theorem 1 derives that the update quantities and gradients approach zero and further asserts that the entire parameter sequence converges to a critical point, it is still necessary to prove that the parameters will not drift indefinitely and will indeed eventually converge to a specific parameter combination. However, the fact that the gradients become increasingly small does not in itself guarantee the existence of this parameter combination; therefore, the argument lacks sufficient justification in this regard. It is recommended that the authors supplement the paper with relevant arguments.
Questions for authors
  1. Regarding the beneficial reuse currently restricted by existing constraints, is it possible to allow partial sharing of adaptation directions while maintaining the stability of the original concepts and controlling leakage?
  2. Given the same total number of parameters or total direction budget, would allocating capacity based on the actual needs of concepts and layers yield better learning and retention results than using a uniform, fixed rank?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

We thank the reviewer for the careful reading and thoughtful comments. In the revised paper, we made three changes to address your comments:

  • Knowledge reuse (W1, Q4). Appendix F.4 tests when an earlier adapter helps a new concept and measures the cost of conflicting responses. Section 5 outlines how concepts could share selected input directions.
  • Rank allocation (W2, Q5). Appendix F.5 compares uniform and adaptive ranks under the same budget. Section 5 clarifies how ranks can vary across layers and concepts.
  • Convergence (W3). We clarified the stationarity guarantee in Theorem 1(ii) and its proof. Remark 1 explains when an accumulation point exists and when the full sequence converges to it.

We address each point below. We hope these additions address the reviewer's concerns and that the reviewer will consider raising the score.


W1) Shared adaptation and knowledge reuse. Thank you for raising this question. We added Appendix F.4 to test when an earlier adaptation helps a new concept and when the orthogonality constraint makes the new concept harder to learn.

As in SeqLoRA generally, the first adapter stays frozen while we train the second, but it still contributes to the layer output. If both concepts need the same response to an input feature, the first adapter already provides it and the second does not have to learn it. The difficulty arises only when the second concept needs a different response to that feature. If an input lies entirely in the first adapter's learned span, the new adapter contributes zero at that layer (Proposition 1); for other inputs, it can use the component outside that span to learn a correction.

We tested both cases by holding the target fixed and varying what the first concept had learned (20 paired seeds, Table B). When the two concepts shared the same adaptation, the projected model reached a lower target loss than training the target alone (0.071 versus 0.084) and the same loss as the unconstrained adapter, so the constraint removed none of the benefit. When the concepts needed opposite responses on the same directions, projection increased the final target loss from 0.103 to 0.117 but reduced forgetting of the first concept from 2.00 to 1.77; the unconstrained adapter is better there only because it overwrites the first concept, hence its higher forgetting; the constraint limits such overwriting (Proposition 1), hence the lower forgetting. So the constraint does not block reuse; it only decides, when two concepts disagree on the same directions, that the earlier one is kept.

The cost also grew with the number of earlier concepts. As the number of similar predecessors, each sharing half of the target's adaptation, increased from one to eight, the extra target loss from projection rose from 0.006 to 0.055 (Table 9 of the paper). We now discuss this trade-off in Sections 3.1 and 5 and consider a shared subspace in Q4. For more details, please refer to Appendix F.4 in the revised paper.

Table B: Selected results from Appendix F.4, Table 8 of the revised paper. Target loss after 400 steps and forgetting of the earlier concept are normalized by the corresponding loss under W0W_0. Cost is the projected minus the unconstrained target loss, computed before rounding.

PredecessorTarget loss, unconstrainedTarget loss, projectedCostForgetting, unconstrainedForgetting, projected
None (target alone)0.0840.084———
0% shared0.1000.109+0.0101.211.09
50% shared0.0870.091+0.0040.720.64
100% shared0.0710.071<0.0010.090.09
100%, opposite sign0.1030.117+0.0132.001.77

W2 & Q5) Rank allocation. Thank you for raising this question. Different layers and concepts may need different amounts of capacity. SeqLoRA allows us to choose a rank ri,ℓr_{i,\ell} for each concept ii and layer ℓ\ell before training that concept. The same update then applies as long as the new basis fits in the remaining input subspace, with ∑iri,ℓ≤mℓ\sum_i r_{i,\ell} \le m_\ell. At each layer, the projection protects the span of the earlier input bases. Theorem 7 gives one way to guide the rank choice: the next eigenvalue of the input covariance in the remaining subspace measures how much additional input energy one more direction could capture.

AuthorsOpenReview ↗

Rebuttal by Authors 2/2

We added two comparisons in Appendix F.5. First, we tested 13 preset ways to divide a budget of 12 directions across three layers. With rank-6 teachers at every layer, the uniform split (4,4,4)(4,4,4) gave the lowest mean loss after learning, forgetting, and final loss (Table 10). Uniform rank also performed well when the teacher ranks differed across layers. For example, with teacher ranks (2,4,6)(2,4,6), the uniform split gave lower loss after learning than the matching split in all five seeds (0.0620.062 versus 0.0690.069 on average, Table 11).

Second, we tested the adaptive choice asked about in Q5. For each new concept, we selected the two hidden-layer ranks from its first training minibatch, using either the input activations or the loss gradients. All methods received exactly 12 directions and 1,024 trainable parameters per concept. Across five paired runs, the mean final loss was 1.4221.422 with uniform rank, 1.4501.450 with the activation rule, and 1.4141.414 with the gradient rule. The gradient rule chose (4,4,4)(4,4,4) for 37 of 40 concepts. Its final loss was lower than uniform in two runs, equal in two, and higher in one.

These results support uniform rank as a reasonable default in this small network. They also show how SeqLoRA can use a data-dependent allocation within the same budget. Whether such an allocation improves identity preservation in a diffusion model remains an interesting future research direction.

On Stable Diffusion we have only the uniform-rank ablation (Table 13), where r=4r=4 is best and larger ranks raise leakage. Need-based allocation is possible without changing the method, and we think Theorem 7 gives the right criterion for it; we leave it to future work.


W3) Convergence in Theorem 1. Thank you for the careful reading and pointing this out. Summing Eq. (42) shows that the step lengths are square-summable and hence tend to zero. Convergence of the factors to a single pair requires a further argument.

To address this point, we revised Theorem 1(ii). The proof in Appendix B uses the same descent and gradient estimates as before. We made minor edits to clarify which update they apply to and what follows from them. For the exact projected update, the gradient with respect to AiA_i and the projected gradient with respect to BiB_i both tend to zero. If a subsequence of factor pairs converges, continuity then shows that its limit is critical. This is the conclusion we now state in the theorem.

We have added Remark 1 at the end of Appendix B, which explains when such a point exists and when the entire sequence converges to it. An accumulation point exists whenever the factors remain bounded. We give one concrete way to ensure this: with the optional norm penalty in Eq. (51) and sufficiently small step sizes, the regularized iterates stay in a compact set, and a critical point inside the ball of radius RR is critical for the original problem.

For convergence of the whole sequence, we verify the sufficient-decrease and relative-error bounds using Eqs. (42) and (46)–(49). If the objective used in the updates has the Kurdyka–Łojasiewicz property at an accumulation point, these bounds imply that the step lengths have a finite sum, and the entire sequence converges to that critical point (Attouch, Bolte and Svaiter, 2013, Theorem 2.9). All changes are in red.


Q4) Partial sharing of adaptation directions. Thank you for the question. Beyond the reuse discussed in W1, Section 5 notes one way to share explicitly. Before training, reserve a small subspace S\mathcal S of each layer's input space as a shared part. Every concept may read and write on S\mathcal S, so related concepts can adapt it together. Outside S\mathcal S the method is unchanged: each concept gets its own input directions, orthogonal to those of the others, so Proposition 1 still protects them.

Sharing S\mathcal S means that a later concept can change what an earlier one experiences there, so the total change on S\mathcal S must be kept small, for example by a norm penalty or by clipping its updates. If the accumulated change on S\mathcal S has spectral norm at most τ\tau, the extra change in a layer's output on an earlier concept's inputs XiX_i is at most τ\tau times the part of those inputs that lies in S\mathcal S, τ∥PSXi∥F\tau\lVert P_{\mathcal S}X_i\rVert_F. This is one more term of the kind that Theorem 2 already bounds, so the forgetting bound extends to the shared design, with the size of S\mathcal S and τ\tau setting how much sharing is allowed against how much interference. We leave its evaluation to future work.

AuthorsOpenReview ↗

Rebuttal Follow-up and Clarifications

Dear Reviewer m8Mf,

Thank you for your review. We hope that our rebuttal and revision have answered your initial concerns about the paper. If you have any remaining questions, please do not hesitate to raise them now so that we can clarify them before the author-reviewer period ends.

All the best,

The Authors

Reviewer_m8MfOpenReview ↗

Comments

Thank you for the detailed response and the revisions. The rebuttal addresses my main concerns. On knowledge reuse, the new experiments in Appendix F.4 show that the orthogonality constraint does not remove the benefit when two concepts share the same adaptation, and that the cost arises only when they require conflicting responses, in which case the earlier concept is protected. On rank allocation, Appendix F.5 compares uniform and adaptive ranks under the same budget and supports uniform rank as a reasonable default. On convergence, the revised Theorem 1(ii) and the new Remark 1 now state the result correctly, with stationarity of accumulation points under boundedness and convergence of the whole sequence under the Kurdyka-Łojasiewicz property, which resolves the gap I pointed out.

I note that the new experiments on reuse and rank allocation are conducted on small networks, and that the shared-subspace design and adaptive allocation on diffusion models are left to future work.

After considering the response together with the comments of the other reviewers, I will keep my current rating.

Reviewer_T6TpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The core idea of SeqLoRA is training a new LoRA per task with both factors trainable, freezing earlier LoRAs, keeping the new subspace orthogonal to all earlier ones, and summing adapters at inference without replay.

Critical strengths

None

Critical weaknesses

Limited novelty relative to uncited prior work. The core idea of SeqLoRA (training a new LoRA per task with both factors trainable, freezing earlier LoRAs, keeping the new subspace orthogonal to all earlier ones, and summing adapters at inference without replay) is essentially O-LoRA (Wang et al., 2023, https://arxiv.org/abs/2310.14152). It was later applied to continual customization of text-to-image diffusion models by CIDM (Dong et al., 2024, https://arxiv.org/abs/2410.17594). None of these is cited or compared, which contradicts the claim that "no existing method optimizes both LoRA factors freely while enforcing subspace orthogonality."

Relative to this prior work, SeqLoRA adds two things:

  1. An exact projection instead of a soft penalty, which is a standard gradient-projection step.
  2. A bilevel cross-Hessian update that gives no significant gain over the Continual Alternating (see tables 1 and 2).
Questions for authors
  1. How does SeqLoRA differ conceptually from O-LoRA (Wang et al., 2023, https://arxiv.org/abs/2310.14152) and CIDM (Dong et al., 2024, https://arxiv.org/abs/2410.17594)?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

We thank the reviewer for the comments and for the question on the relation to O-LoRA and CIDM. In response we have:

  • cited O-LoRA and CIDM and positioned SeqLoRA relative to them (revised paper, lines 80–88; Section 3.1, lines 185–188);
  • shown analytically (W1 below) and in a controlled experiment (Appendix F.2 of the revised paper; Table A in our response to reviewer ymsw) that the exact input-side constraint closes the interference channel that an output-side penalty, as in O-LoRA and CIDM, leaves open;
  • added Appendix F.3, isolating what the bilevel update contributes;
  • we will add CIDM as a baseline under our protocol in the final version.

We also use this opportunity to highlight the key contributions of the paper:

  • exact orthogonality of the input-side bases B\mathbf{B}, enforced by a closed-form projection with both LoRA factors trained (reviewer ymsw: simplicity of the mechanism);
  • the analysis of why orthogonality over B\mathbf{B} rather than A\mathbf{A} is what limits cross-concept interference in multi-concept generation;
  • the resulting guarantees: monotone descent and asymptotic stationarity (Theorem 1), residual-only crosstalk (Proposition 1; reviewer ymsw), a forgetting bound (Theorem 2) and the interference-minimizing basis (Theorem 7);
  • summation-based composition of up to 101 concepts without replay or a learned merge (reviewer 6Mab: relevance of the problem);
  • the bilevel update as a refinement (Figure 9 illustrates its superiority).

We answer each point below; all changes are highlighted in red in the revised paper. We hope these additions address the reviewer's concerns and that the reviewer will consider raising the score.


W1 and Q1) Relation to O-LoRA and CIDM. Thank you for the comment. O-LoRA (Wang et al., 2023) and CIDM (Dong et al., 2024) train both LoRA factors with an orthogonality penalty, and we have narrowed our claim to learning both factors under an exact constraint on the input bases. We cite both in the revised version.

Write ΔWj=AjBj⊤\Delta W_j = A_j B_j^\top, with Aj∈Rd×rA_j \in \mathbb{R}^{d \times r} and Bj∈Rm×rB_j \in \mathbb{R}^{m \times r}. O-LoRA's formulation and CIDM's R1R_1 regularize the output factor AjA_j: where the adapter writes. Our method (SeqLoRA) constrains what it reads by requiring Bj⊤Bi=0B_j^\top B_i = 0 for every earlier concept ii.

Why the side matters. On fixed old layer inputs XiX_i, the new adapter contributes Ej→i=Aj(Bj⊤Xi)E_{j\to i} = A_j (B_j^\top X_i). Even exact output orthogonality, Ai⊤Aj=0A_i^\top A_j = 0, gives only Ai⊤Ej→i=0A_i^\top E_{j\to i} = 0. The perturbation avoids the directions in which adapter ii writes, but Bj⊤XiB_j^\top X_i remains unconstrained. For example, take x=Bi=Bj=e1x = B_i = B_j = e_1, Ai=e1A_i = e_1, and Aj=c e2A_j = c\,e_2 for orthogonal unit vectors e1,e2e_1, e_2. Then ΔWjx=c e2\Delta W_j x = c\,e_2: an input entirely in the old basis can still produce an arbitrarily large change.

The same distinction follows from the gradient motivation of O-LoRA. For output coordinate aa, ⟨eax⊤,ΔWj⟩F=ea⊤ΔWjx\langle e_a x^\top, \Delta W_j \rangle_F = e_a^\top \Delta W_j x. Vanishing for all aa and old inputs is equivalent to ΔWjXi=0\Delta W_j X_i = 0; Bj⊤Xi=0B_j^\top X_i = 0 suffices, as in activation-subspace protection (GPM; Saha et al., 2021). By contrast, Ai⊤Aj=0A_i^\top A_j = 0 implies ⟨ΔWi,ΔWj⟩F=0\langle \Delta W_i, \Delta W_j \rangle_F = 0, which need not preserve old outputs.

Let PiP_i project orthogonally onto col⁡(Bi)\operatorname{col}(B_i). SeqLoRA protects the component of each old input in col⁡(Bi)\operatorname{col}(B_i). The exact constraint Bj⊤Bi=0B_j^\top B_i = 0 gives Bj⊤Pi=0B_j^\top P_i = 0. Hence

Ej→i=ΔWjXi=ΔWj(I−Pi)Xi,∥Ej→i∥F2≤∥ΔWj∥22 ∥(I−Pi)Xi∥F2.E_{j\to i} = \Delta W_j X_i = \Delta W_j (I - P_i) X_i, \qquad \lVert E_{j\to i} \rVert_F^2 \le \lVert \Delta W_j \rVert_2^2 \, \lVert (I - P_i) X_i \rVert_F^2 .

Only the residual input contributes to the perturbation. For Σi=E[xx⊤]\Sigma_i = \mathbb{E}[x x^\top], its mean residual energy is E∥(I−Pi)x∥22=tr⁡((I−Pi)Σi)\mathbb{E}\lVert (I - P_i) x \rVert_2^2 = \operatorname{tr}((I - P_i)\Sigma_i). Summing over later adapters gives CiPi=0C_i P_i = 0 for Ci=∑j>iΔWjC_i = \sum_{j > i} \Delta W_j, the identity used in Theorem 2. This also holds for projected alternating updates under the exact input constraint. Output-side orthogonality alone leaves Bj⊤PiXiB_j^\top P_i X_i unconstrained, so the new adapter can still respond to the captured component of an old input.

AuthorsOpenReview ↗

Rebuttal by Authors 2/2

Numerical evaluation (Appendix F.2 of the revised paper). In the small network of Appendix F (eight concepts, five seeds, everything else identical) we project either the output factor A\mathbf{A} or the input basis B\mathbf{B} after every step, for independent and for similar concepts (Table A in our response to reviewer ymsw). Projecting B\mathbf{B} closes the interference channel exactly (zero through each concept's own subspace) and injects 5–11 times less interference than projecting A\mathbf{A}, which even exceeds no constraint at all: it takes away output directions that later concepts need, so their factors grow and read more of the earlier concepts' inputs. Projecting B\mathbf{B} also fits new concepts almost as well as no constraint (0.34 vs. 0.28), while projecting A\mathbf{A} does not (0.49, 0.53), and it forgets less for similar concepts (3.5 vs. 3.9). This tests the side of the projection, not the full methods; the full comparison with CIDM will be added.


W1, point 2) What does the bilevel correction add? SeqLoRA and Continual Alternating use the same input projection. The correction accounts for how a change in B\mathbf{B} affects the next update of A\mathbf{A}.

Effect on learning. In our new paired comparison with Continual Alternating in the revised paper (Appendix F.3, Figure 4 and Table 7), this correction lowered held-out loss by 6% ± 0.4% after five iterations. At K=200K = 200, bilevel reached lower loss, with slightly less forgetting under the bilevel update in all five seeds.

Image generation. In the single training run per method in Table 1, SeqLoRA scored higher on all five mean identity metrics by 0.005–0.012 (DINOv3 was 0.418 vs. 0.406). The standard errors are across the 32 concepts. In Table 2, SeqLoRA gave lower regional leakage (0.203 vs. 0.223, paired Wilcoxon p=0.014p = 0.014) and lower in-region identity (0.347 vs. 0.366), giving similar margins (0.144 vs. 0.143). Orthogonal Adaptation had similar identity (0.344) and higher leakage (0.231). Figure 9 shows examples with matched seeds and layouts.

Cost and choice of K. At the default K = 3, training took about 660 s per concept for SeqLoRA and 454 s for Continual Alternating (Table 4). In Table 16, K=1K = 1 gave similar identity scores to K=3K = 3 and lower mean HPSv3 (8.784 vs. 9.043). Both settings apply the correction. Continual Alternating is the comparison that isolates its contribution.


Reference

[6]. Saha, G., Garg, I., and Roy, K. Gradient Projection Memory for Continual Learning. ICLR, 2021.

AuthorsOpenReview ↗

Rebuttal Follow-up and Clarifications

Dear Reviewer T6Tp,

Thank you for your review. We hope that our rebuttal and revision have answered your initial concerns about the paper. If you have any remaining questions, please do not hesitate to raise them now so that we can clarify them before the author-reviewer period ends.

All the best,

The Authors

Reviewer_ymswOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies continual multi-concept personalization of text-to-image diffusion models, where concepts arrive sequentially and learned LoRA adapters are composed by summation. SeqLoRA optimizes both LoRA factors for each new concept while keeping its basis exactly orthogonal to all previous bases, using a bilevel update with a cross-Hessian term and a closed-form projection. The authors prove monotone descent and stationarity, and derive a forgetting bound governed by residual activation energy, showing that a covariance-aligned basis outperforms a random frozen one. Experiments on Stable Diffusion v1.5 with up to 101 concepts report better single-concept identity preservation and lower cross-concept leakage than fusion-based and frozen-basis baselines.

Critical strengths
  1. A simple, practical mechanism. The closed-form orthogonal projection is easy to implement. Concepts are composed by plain summation, with no fusion step.

  2. Under orthogonality, the crosstalk acts only on the residual activations outside each concept’s basis. The synthetic study (Appendix F) checks this and illustrates why a learned basis forgets less than a random frozen one.

  3. The paper includes the Continual Alternating ablation to isolate the bilevel term, and analyses of concept order, rank, concept similarity and hyperparameters.

Critical weaknesses
  1. Limited novelty; related work is missing. The claim that no prior method optimizes both LoRA factors under subspace orthogonality is overstated: O-LoRA [1] and CIDM [2] already do this, CIDM in the same concept-incremental text-to-image setting. C-LoRA [3], STAMINA [4] and CNS [5] also target continual customization. None is cited or compared.

  2. The cross-Hessian term is not supported by the paper’s own results. Against Continual Alternating, Table 1 differences are within one standard error. In Table 2, identity is lower and the margin is unchanged. Table 10 shows K=1 matching the default, while training cost rises by about 45%. The leakage gain (p=0.014) is confounded with weaker identity. No results over multiple training seeds are reported.

  3. Outdated backbone. All main results use SD v1.5. Current personalization work targets DiT/MMDiT models, where joint text–image attention changes the interference structure. The 4-concept SDXL test, with no baselines, is too limited to establish generality.

[1] X. Wang et al. Orthogonal Subspace Learning for Language Model Continual Learning. Findings of EMNLP, 2023.

[2] J. Dong et al. How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? NeurIPS, 2024.

[3] J. S. Smith et al. Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA. TMLR, 2024.

[4] J. S. Smith et al. Continual Diffusion with STAMINA: STack-And-Mask INcremental Adapters. arXiv, 2023.

[5] Y.-C. Liao et al. Continual Personalization for Diffusion Models. ICCV, 2025.

Questions for authors

Please see 3 weaknesses.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

We thank the reviewer for recognizing the simplicity of the mechanism, the residual-crosstalk identity and the ablations, and for comments that helped us improve the paper. In response we have:

  • cited [1–5], stated our claim precisely as exact orthogonality of the input-side LoRA basis B\mathbf{B}, and positioned SeqLoRA relative to O-LoRA and CIDM in the introduction (lines 80–88) and in Section 3.1 (lines 185–188);
  • shown analytically (W1 below) and in a controlled experiment (Appendix F.2 of the revised paper; Table A below) that the exact input-side constraint closes the interference channel that an output-side penalty, as in O-LoRA and CIDM, leaves open;
  • clarified the role of the bilevel update and added Appendix F.3, which isolates what the cross-Hessian term contributes;
  • started a four-concept comparison on Stable Diffusion 3 Medium (MMDiT) against Continual Alternating and an extended SDXL evaluation with the same baseline; we will post the results in this thread and add them to the revised paper (W3);
  • and we will add CIDM [2] as a baseline under our protocol in the final version.

We answer each point below; all changes are highlighted in red in the revised paper. We hope that these additions address the reviewer's concerns and that the reviewer will consider raising the score.


W1) Relation to prior work. Thank you for the comment. O-LoRA [1] and CIDM [2] train both LoRA factors with an orthogonality penalty, and we have narrowed our claim to learning both factors under an exact constraint on the input bases. We cite [1–5] in the revised version.

Write ΔWj=AjBj⊤\Delta W_j = A_j B_j^\top, with Aj∈Rd×rA_j \in \mathbb{R}^{d \times r} and Bj∈Rm×rB_j \in \mathbb{R}^{m \times r}. O-LoRA's formulation and CIDM's R1R_1 regularize the output factor AjA_j: where the adapter writes. Our method (SeqLoRA) constrains what it reads by requiring Bj⊤Bi=0B_j^\top B_i = 0 for every earlier concept ii.

Why the side matters. On fixed old layer inputs XiX_i, the new adapter contributes Ej→i=Aj(Bj⊤Xi)E_{j\to i} = A_j (B_j^\top X_i). Even exact output orthogonality, Ai⊤Aj=0A_i^\top A_j = 0, gives only Ai⊤Ej→i=0A_i^\top E_{j\to i} = 0. The perturbation avoids the directions in which adapter ii writes, but Bj⊤XiB_j^\top X_i remains unconstrained. For example, take x=Bi=Bj=e1x = B_i = B_j = e_1, Ai=e1A_i = e_1, and Aj=c e2A_j = c\,e_2 for orthogonal unit vectors e1,e2e_1, e_2. Then ΔWjx=c e2\Delta W_j x = c\,e_2: an input entirely in the old basis can still produce an arbitrarily large change.

The same distinction follows from the gradient motivation of O-LoRA. For output coordinate aa, ⟨eax⊤,ΔWj⟩F=ea⊤ΔWjx\langle e_a x^\top, \Delta W_j \rangle_F = e_a^\top \Delta W_j x. Vanishing for all aa and old inputs is equivalent to ΔWjXi=0\Delta W_j X_i = 0; Bj⊤Xi=0B_j^\top X_i = 0 suffices, as in activation-subspace protection [6]. By contrast, Ai⊤Aj=0A_i^\top A_j = 0 implies ⟨ΔWi,ΔWj⟩F=0\langle \Delta W_i, \Delta W_j \rangle_F = 0, which need not preserve old outputs.

Let PiP_i project orthogonally onto col⁡(Bi)\operatorname{col}(B_i). SeqLoRA's constraint gives Bj⊤Pi=0B_j^\top P_i = 0, hence

Ej→i=ΔWj(I−Pi)Xi,∥Ej→i∥F2≤∥ΔWj∥22 ∥(I−Pi)Xi∥F2.E_{j\to i} = \Delta W_j (I - P_i) X_i, \qquad \lVert E_{j\to i} \rVert_F^2 \le \lVert \Delta W_j \rVert_2^2 \, \lVert (I - P_i) X_i \rVert_F^2 .

Only the residual enters this bound. For Σi=E[xx⊤]\Sigma_i = \mathbb{E}[x x^\top], the residual energy per input is E∥(I−Pi)x∥22=tr⁡((I−Pi)Σi)\mathbb{E}\lVert (I - P_i) x \rVert_2^2 = \operatorname{tr}((I - P_i)\Sigma_i). Summing later updates gives CiPi=0C_i P_i = 0 for Ci=∑j>iΔWjC_i = \sum_{j > i} \Delta W_j, the identity used in Theorem 2. It holds for the exact constraint, including projected alternating updates.

Numerical evaluation (Appendix F.2 of the revised paper). In the small network of Appendix F (eight concepts, five seeds, everything else identical) we project either the output factor A\mathbf{A} or the input basis B\mathbf{B} after every step, for independent and for similar concepts (Table A).

Table A: Projecting the output factor A\mathbf{A} versus the input basis B\mathbf{B} in the synthetic setting of Appendix F (eight concepts, five seeds; mean ± standard error). Interference is the crosstalk energy that later adapters inject on an earlier concept; the last column is the part of it that passes through that concept's own subspace col(Bi)\mathrm{col}(\mathbf{B}_i). Lower is better in every column.

ConceptsProjectionLoss after learningForgettingInterference…through captured subspace
Independentnone0.28 ± 0.013.5 ± 0.27.0 ± 0.52.1 ± 0.1
Independentof A\mathbf{A} (output)0.49 ± 0.043.1 ± 0.236.0 ± 2.012.7 ± 1.0
Independentof B\mathbf{B} (input)0.34 ± 0.023.1 ± 0.26.7 ± 0.60.0
Similarnone0.28 ± 0.013.9 ± 0.211.1 ± 0.73.8 ± 0.3
Similarof A\mathbf{A} (output)0.53 ± 0.053.9 ± 0.295.4 ± 7.043.6 ± 3.4
Similarof B\mathbf{B} (input)0.36 ± 0.023.5 ± 0.28.3 ± 0.80.0
AuthorsOpenReview ↗

Rebuttal by Authors 2/2

Projecting B\mathbf{B} closes the interference channel exactly (zero in the last column) and injects 5–11 times less interference than projecting A\mathbf{A}, which even exceeds no constraint at all: it takes away output directions that later concepts need, so their factors grow and read more of the earlier concepts' inputs. Projecting B\mathbf{B} also fits new concepts almost as well as no constraint (0.34 vs. 0.28), while projecting A\mathbf{A} does not (0.49, 0.53), and it forgets less for similar concepts (3.5 vs. 3.9). This tests the side of the projection, not the full methods; we will add a matched comparison with CIDM for image generation in the final version.

Relation to C-LoRA, STAMINA and CNS. These methods limit interference at the level of individual weight entries: C-LoRA [3] penalizes changes to entries that earlier concepts modified, STAMINA [4] confines each concept to a sparse mask of the LoRA residual, and CNS [5] updates only selected "concept neurons". Protecting the entries an earlier concept wrote does not constrain the effect of a new update on that concept's activations, ΔWtXi\Delta \mathbf{W}_t \mathbf{X}_i, which is what SeqLoRA's input-side constraint bounds; none of them provides a forgetting bound. We have added this discussion, together with [1, 2], to the introduction of the revised paper.


W2) What does the bilevel correction add? SeqLoRA and Continual Alternating use the same input projection. The correction accounts for how a change in BB affects the next update of AA.

Effect on learning. In our new paired comparison with Continual Alternating in the revised paper (Appendix F.3, Figure 4 and Table 7), this correction lowered held-out loss by 6% ± 0.4% after five iterations. At K=200K = 200, bilevel reached lower loss, with slightly less forgetting under the bilevel update in all five seeds.

Image generation. In the single training run per method in Table 1, SeqLoRA scored higher on all five mean identity metrics by 0.005–0.012 (DINOv3 was 0.418 vs. 0.406). The standard errors are across the 32 concepts. In Table 2, SeqLoRA gave lower regional leakage (0.203 vs. 0.223, paired Wilcoxon p=0.014p = 0.014) and lower in-region identity (0.347 vs. 0.366), giving similar margins (0.144 vs. 0.143). Orthogonal Adaptation had similar identity (0.344) and higher leakage (0.231). Figure 9 shows examples with matched seeds and layouts.

Cost and choice of KK. At the default K=3K = 3, training took about 660 s per concept for SeqLoRA and 454 s for Continual Alternating (Table 4). In Table 16, K=1K = 1 gave similar identity scores to K=3K = 3 and lower mean HPSv3 (8.784 vs. 9.043). Both settings apply the correction. Continual Alternating is the comparison that isolates its contribution.


W3) Backbone. Thank you for this point. We used Stable Diffusion v1.5 so that the comparison with the baselines, which were developed on it, is like-for-like. Nothing in SeqLoRA is specific to that backbone: the projection and the theory act on each adapted linear layer separately, so Proposition 1 holds for a layer with joint text–image attention as it does for a UNet cross-attention layer. How large the residual interference is on a DiT/MMDiT model is an empirical question, and we agree that the four-concept SDXL test without baselines does not settle it. We are preparing a controlled four-concept comparison on Stable Diffusion 3 Medium (MMDiT) against Continual Alternating, alongside an extended SDXL evaluation with the same baseline; we will post the results in this thread as soon as they are available and add them to the revised paper.


References

[6] Saha, G., Garg, I., and Roy, K. Gradient Projection Memory for Continual Learning. ICLR, 2021.

AuthorsOpenReview ↗

Rebuttal Follow-up and Clarifications

Dear Reviewer ymsw,

Thank you for your review. We hope that our rebuttal and revision have answered your initial concerns about the paper. If you have any remaining questions, please do not hesitate to raise them now so that we can clarify them before the author-reviewer period ends.

All the best,

The Authors

Thank you for the response. The revision narrows the claim to an exact input-side constraint, but the empirical evidence that this matters against O-LoRA and CIDM is a synthetic test of the projection side (Table A), not a comparison of the full methods, and the CIDM baseline is only promised for the final version. On the bilevel term, the rebuttal confirms my concern. Table 1 still comes from a single training run with differences of 0.005–0.012. In Table 2, the margin over Continual Alternating is unchanged (0.144 vs. 0.143). As the authors note in their response to Reviewer 6Mab, Continual Alternating reaches a lower loss at comparable backward-pass budgets in Appendix F.3, while the bilevel term costs about 45% more training time. The DiT/MMDiT and extended SDXL results have also not been provided yet. Therefor, I will keep my score.

Reviewer_6MabOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper SeqLoRA addresses the representation interference challenge in continuous multi-concept personalization in diffusion models. In existing works, the LoRA-based approaches either require some optimization to combine independently trained adapters or use fixed orthogonal bases to reduce the catastrophic forgetting and cross-concept interference. The authors here learn LoRA matrices while keeping the input subspaces of successive LoRA adapters orthogonal. They also elaborate a bilevel approximation that updates both A and B in LoRA factors while accounting for their interactions through cross-Hessian correction. This leads learning concept-specific directions while being orthogonal to earlier adapters. Detailed theoretical analysis on optimization convergence and forget bound was shown in the paper. Experiments on Sdv1.5 show improvement in single-concept identity preservation, decreased cross-concept leakage, and stability upto 101 concepts.

Critical strengths
  1. The paper points out a relevant problem: when multiple concepts are learned together, their combination creates an interference in the model, leading to degraded model performance. Without losing the knowledge of previous adapters, the authors explore how to reduce such cross-talk.

  2. The method SeqLoRA allows subspaces to adapt to each concept rather than keeping it frozen. The combination of orthogonality and bilevel optimization results in achieving the SOTA among the baselines provided.

  3. The paper includes sound theoretical analyses regarding the forget bound and convergence.

  4. The evaluation also shows minimal cross-concept leakage, and combination of adapters without a costly fusion process. Experiments cover concept ordering, LoRA rank and training efficiency as well.

Critical weaknesses
  1. SeqLoRA has higher compute overhead due to more training time per concept than current methods. The paper does not elaborate if the benefit outweighs the training cost, specially with its moderate improvement over Continual Alternating method.

  2. A more detailed analysis of the trade-off between identity preservation vs cross-concept leakage would be useful, since the results suggest that reducing interference might come at the cost of concept fidelity.

  3. The architecture evaluation is quite limited to UNet structures only. DiT evaluation would be appreciated.

Questions for authors
  1. Since a lot of the orthogonality restraints are imposed on B of the LoRA update, does it get restrictive for future updates after large number of updates?

  2. Is there a trade-off between bilevel optimization, training time and performance?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

We thank the reviewer for the careful reading and for highlighting the relevance of the problem, the adaptive subspaces, the theory and the evaluation. In this rebuttal, we have:

  • explained in Section 4.3 what the 45% extra training time buys (lower leakage in compositions; one bilevel iteration gives the same identity at a fraction of the cost) and added Appendix F.3, isolating the correction's effect over alternating updates (W1, Q2);
  • analysed the identity–leakage trade-off: it appears only in compositions, where the margin is preserved and Figure 9 shows the difference, and the rank moves along it (W2);
  • added Appendix F.4 on what the constraint costs a new concept as earlier concepts accumulate, and related it to the capacity bound and the 101-concept results (Q1);
  • started a four-concept comparison on Stable Diffusion 3 Medium (MMDiT) against Continual Alternating and an extended SDXL evaluation with the same baseline; we will post the results in this thread and add them to the revised paper (W3).

All changes are in red in the revised paper. We respond to each point below and hope the reviewer will consider raising the score.


W1 & Q2) Training cost and benefit of the bilevel update. On the same hardware, SeqLoRA takes about 660 seconds per concept, compared with 454 seconds for Continual Alternating. The cross-Hessian correction adds about 3.4 minutes per concept (Table 4). Both methods compose adapters by summation. For comparison, Mix-of-Show and LoRACLR train faster per concept but require 33 and 8 minutes of fusion, respectively, at 32 concepts.

The clearest benefit is lower leakage when several concepts appear in one image. SeqLoRA reduced regional leakage from 0.2230.223 to 0.2030.203 (9%, paired Wilcoxon p=0.014p = 0.014, Table 2). In-region identity was also lower (0.3470.347 versus 0.3660.366), leaving the identity-minus-leakage margin nearly unchanged (0.1440.144 versus 0.1430.143). Figure 9 illustrates the difference: under the same seed and regional conditioning, fewer attributes appear in another concept's region with SeqLoRA. When concepts were generated individually, SeqLoRA scored slightly higher on all five identity metrics (Table 1).

These results favor the extra training time when reducing attribute mixing is the priority. For single-concept identity under a tight budget, Continual Alternating is a competitive choice, and so is SeqLoRA with K=1K = 1, which kept identity scores similar to K=3K = 3 at lower training time, with a lower HPSv3 score (Table 16). Appendix F.3 shows the same trade-off in a small network: the correction lowered loss early at matched iterations, while Continual Alternating reached lower loss at comparable backward-pass budgets. We now state this trade-off in Section 4.3.


W2) Identity preservation versus cross-concept leakage. Compared with Continual Alternating, SeqLoRA has lower leakage in compositions (0.2030.203 vs. 0.2230.223) and lower in-region identity (0.3470.347 vs. 0.3660.366), with the same margin (0.1440.144 vs. 0.1430.143). The rest of Table 2 shows, however, that lower leakage is not simply bought with identity. Orthogonal Adaptation has the same identity as SeqLoRA (0.3440.344 vs. 0.3470.347) but higher leakage (0.2310.231 vs. 0.2030.203), so at equal identity SeqLoRA leaks 12% less. Outside compositions there is no trade-off at all: for single concepts, SeqLoRA has the highest identity on all five metrics (Table 1). Figure 9 shows the difference in images, with fewer attributes of one concept appearing in another concept's region under matched seeds and regional conditioning.

The balance can also be adjusted. In the rank ablation (Appendix I.2, Table 13; 16-concept stream), rank 16 raises identity (0.3220.322 to 0.3380.338) and leakage (0.2000.200 to 0.2190.219) together, and rank 4 gives the highest margin among the tested ranks. New Appendix F.4 explains where the price comes from: the learning cost of the orthogonality constraint is below 0.0010.001 when two concepts need the same adaptation and at most 0.0130.013 otherwise. We now discuss these findings explicitly in Section 4.2.


W3) Evaluation on DiT architectures. Thank you for the suggestion. SeqLoRA does not rely on the UNet. The projection and the bilevel update work on any weight matrix that carries a LoRA adapter, and our theory makes no assumption about the architecture, so it holds for the attention and MLP layers of a DiT as well. To check this in practice, we are preparing a four-concept comparison on Stable Diffusion 3 Medium (MMDiT) against Continual Alternating, together with an extended SDXL evaluation with the same baseline. We will post the results in this thread as soon as they are available and add them to the revised paper.

AuthorsOpenReview ↗

Rebuttal by Authors 2/2

Q1) Does the constraint on BB become restrictive after many concepts? Each time we finish training a concept, we freeze its input basis, and later concepts must learn in the remaining directions. At a layer with input dimension mm, freezing tt mutually orthogonal bases of rank rr leaves m−trm - tr directions; with m=768m = 768 and r=4r = 4, 101 concepts leave 364 directions at that layer. In practice we have not reached the limit: at 101 concepts single-concept identity is no lower than at 32 (DINOv2 0.4310.431 vs. 0.4180.418, CLIP-I 0.7050.705 vs. 0.6930.693; Table 20), and concepts learned late in the stream show no systematic disadvantage (Table 12).

The constraint can matter before the directions run out. Earlier adapters stay active, so a new concept reuses a response they already provide; if it needs a different response on the same input directions, it must learn that correction from the part of its input outside the frozen span. We tested this in Appendix F.4 by holding the target concept fixed and increasing the number of earlier concepts that share half of its adaptation: from one to eight such concepts, the extra normalized target loss from the constraint rose from 0.0060.006 to 0.0550.055 (Table 9). The practical limit thus depends on how many directions remain and on how useful they are for the next concept. Beyond it, the options are a smaller rank for later concepts, which the method allows per layer and concept (Appendix F.5), or a reserved shared subspace (Section 5).

AuthorsOpenReview ↗

Rebuttal Follow-up and Clarifications

Dear Reviewer 6Mab,

Thank you for your review. We hope that our rebuttal and revision have answered your initial concerns about the paper. If you have any remaining questions, please do not hesitate to raise them now so that we can clarify them before the author-reviewer period ends.

All the best,

The Authors

The rebuttal answered most of my questions but I do not yet see the DiT achitecture results in this thread, also the novelty seems to be limited in the work. Hence, I will keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.