- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To address the challenge that while synthesizing a single concept is now very convenient, synthesizing multiple concepts can lead to representation interference, the authors propose Sequential Regularized LoRA (SeqLoRA)—a constrained sequential learning framework—by jointly optimizing two LoRA factors through a two-stage optimization process, while ensuring that each new basis remains orthogonal to all previously learned bases. They provide theoretical derivations and proofs and validate the design through relevant experiments.
Critical strengths
- The authors’ proposed SeqLoRA uses a two-layer optimization with cross-Hessian correction to jointly learn two LoRA matrices for new concepts, and employs closed-form projection constraints to ensure orthogonality between the new and old bases, enabling the trained concept modules to be combined as needed.
- The authors use Theorem 1 to apply existing convergence analysis tools to analyze the optimization properties of the algorithm. They then utilize Theorem 2 to combine existing probabilistic tools with the structure of orthogonal LoRA to derive an upper bound on forgetting, which is used to analyze the sources of concept interference. Furthermore, they analyze the expected advantage of the optimal basis selected based on data covariance relative to a randomly frozen basis within the feasible subspace, providing a theoretical basis for learning LoRA bases from data.
- In experiments involving 8–101 concepts on Stable Diffusion v1.5, the authors verified that SeqLoRA achieves the lowest multi-concept leakage among the compared methods, improves identity preservation, and remains operational even when some fusion baselines run out of GPU memory. The authors also analyzed the effects of concept order, LoRA rank, and concept similarity, as well as the computational cost, and preliminarily validated the feasibility of transferring the method to SDXL.
Critical weaknesses
- In multi-concept generation, the base model W_0 provides general knowledge, while LoRA performs additional personalization adjustments. SeqLoRA uniformly applies orthogonal constraints to the newly added adaptation subspaces for different concepts to reduce cross-concept interference. However, different concepts may require some shared adaptation information; therefore, it is unclear whether strict subspace isolation would limit beneficial knowledge reuse. We hope to see further explanation on this point.
- SeqLoRA primarily protects existing concepts through orthogonal subspaces; however, its uniform, fixed-rank approach does not flexibly allocate capacity based on the needs of different concepts or layers. This is because the requirements of different modules may vary—some may learn effectively with fewer directions, while others may require more. Furthermore, identity preservation is critical for certain layers, whereas increasing capacity in other layers may yield minimal benefits. A uniform allocation may result in suboptimal resource distribution. Therefore, we hope to see further explanations regarding more targeted interference control and adaptive capacity allocation within the same total budget.
- While Theorem 1 derives that the update quantities and gradients approach zero and further asserts that the entire parameter sequence converges to a critical point, it is still necessary to prove that the parameters will not drift indefinitely and will indeed eventually converge to a specific parameter combination. However, the fact that the gradients become increasingly small does not in itself guarantee the existence of this parameter combination; therefore, the argument lacks sufficient justification in this regard. It is recommended that the authors supplement the paper with relevant arguments.
Questions for authors
- Regarding the beneficial reuse currently restricted by existing constraints, is it possible to allow partial sharing of adaptation directions while maintaining the stability of the original concepts and controlling leakage?
- Given the same total number of parameters or total direction budget, would allocating capacity based on the actual needs of concepts and layers yield better learning and retention results than using a uniform, fixed rank?
4 comments
Rebuttal by Authors 2/2
We added two comparisons in Appendix F.5. First, we tested 13 preset ways to divide a budget of 12 directions across three layers. With rank-6 teachers at every layer, the uniform split gave the lowest mean loss after learning, forgetting, and final loss (Table 10). Uniform rank also performed well when the teacher ranks differed across layers. For example, with teacher ranks , the uniform split gave lower loss after learning than the matching split in all five seeds ( versus on average, Table 11).
Second, we tested the adaptive choice asked about in Q5. For each new concept, we selected the two hidden-layer ranks from its first training minibatch, using either the input activations or the loss gradients. All methods received exactly 12 directions and 1,024 trainable parameters per concept. Across five paired runs, the mean final loss was with uniform rank, with the activation rule, and with the gradient rule. The gradient rule chose for 37 of 40 concepts. Its final loss was lower than uniform in two runs, equal in two, and higher in one.
These results support uniform rank as a reasonable default in this small network. They also show how SeqLoRA can use a data-dependent allocation within the same budget. Whether such an allocation improves identity preservation in a diffusion model remains an interesting future research direction.
On Stable Diffusion we have only the uniform-rank ablation (Table 13), where is best and larger ranks raise leakage. Need-based allocation is possible without changing the method, and we think Theorem 7 gives the right criterion for it; we leave it to future work.
W3) Convergence in Theorem 1. Thank you for the careful reading and pointing this out. Summing Eq. (42) shows that the step lengths are square-summable and hence tend to zero. Convergence of the factors to a single pair requires a further argument.
To address this point, we revised Theorem 1(ii). The proof in Appendix B uses the same descent and gradient estimates as before. We made minor edits to clarify which update they apply to and what follows from them. For the exact projected update, the gradient with respect to and the projected gradient with respect to both tend to zero. If a subsequence of factor pairs converges, continuity then shows that its limit is critical. This is the conclusion we now state in the theorem.
We have added Remark 1 at the end of Appendix B, which explains when such a point exists and when the entire sequence converges to it. An accumulation point exists whenever the factors remain bounded. We give one concrete way to ensure this: with the optional norm penalty in Eq. (51) and sufficiently small step sizes, the regularized iterates stay in a compact set, and a critical point inside the ball of radius is critical for the original problem.
For convergence of the whole sequence, we verify the sufficient-decrease and relative-error bounds using Eqs. (42) and (46)–(49). If the objective used in the updates has the Kurdyka–Łojasiewicz property at an accumulation point, these bounds imply that the step lengths have a finite sum, and the entire sequence converges to that critical point (Attouch, Bolte and Svaiter, 2013, Theorem 2.9). All changes are in red.
Q4) Partial sharing of adaptation directions. Thank you for the question. Beyond the reuse discussed in W1, Section 5 notes one way to share explicitly. Before training, reserve a small subspace of each layer's input space as a shared part. Every concept may read and write on , so related concepts can adapt it together. Outside the method is unchanged: each concept gets its own input directions, orthogonal to those of the others, so Proposition 1 still protects them.
Sharing means that a later concept can change what an earlier one experiences there, so the total change on must be kept small, for example by a norm penalty or by clipping its updates. If the accumulated change on has spectral norm at most , the extra change in a layer's output on an earlier concept's inputs is at most times the part of those inputs that lies in , . This is one more term of the kind that Theorem 2 already bounds, so the forgetting bound extends to the shared design, with the size of and setting how much sharing is allowed against how much interference. We leave its evaluation to future work.
Rebuttal Follow-up and Clarifications
Dear Reviewer m8Mf,
Thank you for your review. We hope that our rebuttal and revision have answered your initial concerns about the paper. If you have any remaining questions, please do not hesitate to raise them now so that we can clarify them before the author-reviewer period ends.
All the best,
The Authors
Comments
Thank you for the detailed response and the revisions. The rebuttal addresses my main concerns. On knowledge reuse, the new experiments in Appendix F.4 show that the orthogonality constraint does not remove the benefit when two concepts share the same adaptation, and that the cost arises only when they require conflicting responses, in which case the earlier concept is protected. On rank allocation, Appendix F.5 compares uniform and adaptive ranks under the same budget and supports uniform rank as a reasonable default. On convergence, the revised Theorem 1(ii) and the new Remark 1 now state the result correctly, with stationarity of accumulation points under boundedness and convergence of the whole sequence under the Kurdyka-Łojasiewicz property, which resolves the gap I pointed out.
I note that the new experiments on reuse and rank allocation are conducted on small networks, and that the shared-subspace design and adaptive allocation on diffusion models are left to future work.
After considering the response together with the comments of the other reviewers, I will keep my current rating.
Rebuttal by Authors 1/2
We thank the reviewer for the careful reading and thoughtful comments. In the revised paper, we made three changes to address your comments:
We address each point below. We hope these additions address the reviewer's concerns and that the reviewer will consider raising the score.
W1) Shared adaptation and knowledge reuse. Thank you for raising this question. We added Appendix F.4 to test when an earlier adaptation helps a new concept and when the orthogonality constraint makes the new concept harder to learn.
As in SeqLoRA generally, the first adapter stays frozen while we train the second, but it still contributes to the layer output. If both concepts need the same response to an input feature, the first adapter already provides it and the second does not have to learn it. The difficulty arises only when the second concept needs a different response to that feature. If an input lies entirely in the first adapter's learned span, the new adapter contributes zero at that layer (Proposition 1); for other inputs, it can use the component outside that span to learn a correction.
We tested both cases by holding the target fixed and varying what the first concept had learned (20 paired seeds, Table B). When the two concepts shared the same adaptation, the projected model reached a lower target loss than training the target alone (0.071 versus 0.084) and the same loss as the unconstrained adapter, so the constraint removed none of the benefit. When the concepts needed opposite responses on the same directions, projection increased the final target loss from 0.103 to 0.117 but reduced forgetting of the first concept from 2.00 to 1.77; the unconstrained adapter is better there only because it overwrites the first concept, hence its higher forgetting; the constraint limits such overwriting (Proposition 1), hence the lower forgetting. So the constraint does not block reuse; it only decides, when two concepts disagree on the same directions, that the earlier one is kept.
The cost also grew with the number of earlier concepts. As the number of similar predecessors, each sharing half of the target's adaptation, increased from one to eight, the extra target loss from projection rose from 0.006 to 0.055 (Table 9 of the paper). We now discuss this trade-off in Sections 3.1 and 5 and consider a shared subspace in Q4. For more details, please refer to Appendix F.4 in the revised paper.
Table B: Selected results from Appendix F.4, Table 8 of the revised paper. Target loss after 400 steps and forgetting of the earlier concept are normalized by the corresponding loss under W0. Cost is the projected minus the unconstrained target loss, computed before rounding.
W2 & Q5) Rank allocation. Thank you for raising this question. Different layers and concepts may need different amounts of capacity. SeqLoRA allows us to choose a rank ri,ℓ for each concept i and layer ℓ before training that concept. The same update then applies as long as the new basis fits in the remaining input subspace, with ∑iri,ℓ≤mℓ. At each layer, the projection protects the span of the earlier input bases. Theorem 7 gives one way to guide the rank choice: the next eigenvalue of the input covariance in the remaining subspace measures how much additional input energy one more direction could capture.