acceptodds
Withdrawn from ICLR 2027

CLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual personalization of text-to-image diffusion models requires sequentially acquiring new concepts while retaining previously learned ones. However, existing methods either suffer from catastrophic forgetting or rely on storing additional concept-specific parameters and spatial components, causing their parameter footprint to grow with the concept stream. This limits their ability to scale to long sequences of personalization tasks. We propose a rehearsal-free approach that uses a single fixed-size hypernetwork to continually personalize a frozen diffusion model. Instead of expanding the model as new concepts are acquired, the hypernetwork dynamically produces the concept-specific adaptations required for personalization while preserving previously learned concepts. Our framework further integrates spatial control into the personalization process, allowing users to specify where a personalized concept should appear without introducing additional per-concept components. This formulation enables continual personalization with a parameter footprint that remains independent of the number of learned concepts, aside from compact concept representations. Experiments demonstrate strong retention of previously learned concepts and reliable spatial grounding, matching or improving upon existing methods while scaling effectively to long streams of personalization tasks.

Primary area:
generative models
Keywords:
continual learning, text-to-image diffusion models, personalization, hypernetworks, LoRA, spatial control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7BYUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses continual personalization of text-to-image diffusion models, where concepts are learned one after another without access to earlier data. CLASP replaces per-concept adapters with one fixed-size hypernetwork. The hypernetwork maps a task embedding for each concept to LoRA updates on the cross-attention layers of a frozen backbone. Task embeddings are orthogonalized with Gram–Schmidt, and an output-space regularizer keeps the updates generated for earlier concepts close to their previous values. The same hypernetwork also produces bounding-box-conditioned tokens for a gated attention branch, which controls where the concept is placed. On CIFC with SD-1.5, the authors report higher image alignment than CIDM at matched text alignment, about 4× lower forgetting, constant parameter cost up to 100 concepts, and correct quadrant placement in all tested cases.

Critical strengths
  1. Solid experiments that support the main claims. Results are averaged over several seeds, and the authors define a significance threshold in advance. Forgetting is clearly lower than CIDM and sequential fine-tuning. The generated adapters perform on par with independently trained per-concept LoRAs, and inference cost does not grow with the number of concepts.

  2. Analyses that are useful beyond this method. The ablations show that a generated adapter depends heavily on the output projection, and this holds on both backbones. They also show that attribute words in training captions compete with the adapter for encoding the concept's appearance. Both findings are relevant to other adapter and hypernetwork work.

  3. Appearance and placement come from the same network. Spatial control adds no per-concept parameters. In the reported setting, the concept consistently appears in the requested region, and identity preservation improves when a box is given.

Critical weaknesses
  1. Limited methodological novelty, and the paper's account of prior work is inaccurate. The content module largely follows von Oswald et al. (2020), combined with hypernetwork-generated LoRA updates of the kind used in UnHype (Wójcik et al., 2026). The paper describes UnHype only as a concept-removal method, which understates how close the two architectures are. The placement module combines known components: GLIGEN-style gating (Li et al., 2023), a parallel attention branch similar to IP-Adapter (Ye et al., 2023), and MultiDiffusion-style bootstrapping (Bar-Tal et al., 2023). Gram–Schmidt is presented as part of the design, but the paper's own ablations show no measurable benefit from it. Closely related work on grounded personalization, such as GroundingBooth (Xiong et al., 2024), and on continual personalization, such as ConceptGuard (Guo et al., 2025), is not discussed.

  2. The comparisons are too narrow to support the scalability claim. CIDM (Dong et al., 2024) is the only main baseline, and its results are taken from the original paper rather than reproduced under the same setup. Scaling beyond CIDM's 37-concept limit is a weak reference point, because that limit comes from CIDM's implementation rather than from the problem. SeqLoRA (Parsa et al., 2026) already learns about a hundred concepts sequentially on the same backbone and dataset, and CNS (Liao et al., 2025) also avoids adapter fusion. Both papers are cited but not compared against.

  3. The method depends on several practical assumptions. At inference, the concept index must be supplied, since the model cannot identify the concept from the prompt. Training requires captions with attribute words removed, and the captions for the added concepts are generated by an LLM. Placement training relies on segmentation-based composites, some of which needed manual mask selection.

References not cited in the submission

[1] Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv:2308.06721, 2023.

[2] Xiong, Z., Xiong, W., Shi, J., Zhang, H., Song, Y., and Jacobs, N. GroundingBooth: Grounding text-to-image customization. arXiv:2409.08520, 2024.

[3] Guo, S., et al. ConceptGuard: Continual personalized text-to-image generation with forgetting and confusion mitigation. In CVPR, 2025.

Questions for authors

Please see Critical Weaknesses.

Reviewer_ZrgvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CLASP, which uses a fixed-size hypernetwork to generate concept-specific LoRA updates and placement tokens for continual personalization of a frozen diffusion model with spatial control. Orthogonal task embeddings and output-space regularization mitigate forgetting. Experiments demonstrate strong concept retention and extend the concept sequence to 100 without storing a separate adapter for each concept.

Critical strengths

Addresses a practical storage problem. The shared hypernetwork generates LoRA updates on demand, avoiding a separate adapter for each concept and offering clear storage advantages as the number of concepts grows.

Strong concept retention. At comparable text alignment, the method achieves better image alignment and lower forgetting on SD-1.5, supported by results across three training seeds and longer concept sequences.

Critical weaknesses
  1. CLASP shares SD-LoRA’s goal of scalable, rehearsal-free continual adaptation using LoRA. Although their mechanisms differ, the authors should clarify the specific advantages of hypernetwork-generated updates and compare against an adapted SD-LoRA baseline.

  2. Equation (6) requires iterating over previously learned concepts, so its computational cost grows with the sequence length. A fixed-size hypernetwork does not imply constant overall storage or training costs; these aspects should be clearly distinguished.

  3. The main text primarily evaluates seven object concepts across four quadrants, which is insufficient to establish control over arbitrary bounding-box positions and sizes. Moreover, the reported 96% rate of IoU > 0.5 is calculated only over generations with successful detections. Both the detection rate and localization success rate over all generated images should be reported to account for failures.

Questions for authors
  1. What specific advantages do hypernetwork-generated updates offer over SD-LoRA? Can you compare against an SD-LoRA baseline adapted to the same personalization setting?

  2. How do per-task training time, peak memory, and total storage scale with the number of concepts, including the cost of Equation (6), stored task embeddings, and the hypernetwork snapshot?

  3. How does spatial control perform across varying bounding-box positions, sizes, and aspect ratios beyond the four-quadrant setting? What are the detection rate and the overall localization success rate, counting missed detections as failures?

Reviewer_bkEUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses continual personalization of text-to-image diffusion models: concepts arrive one by one, and the model should learn each new concept without forgetting earlier ones. Instead of storing a LoRA per concept, one fixed-size hypernetwork generates the LoRA for each concept from a task embedding, and a regularizer keeps the LoRAs of earlier concepts from drifting. The same hypernetwork also produces tokens that place the concept inside a user-given bounding box. On the CIFC benchmark with SD-1.5, the authors report better image alignment and lower forgetting than CIDM, and streams of up to 100 concepts.

Critical strengths
  1. The paper is clearly written and easy to follow.
  2. The method beats the main baseline: at CIDM's text alignment it reaches 79.9 IA against 78.0, with lower forgetting than both CIDM and sequential fine-tuning (Figure 4).
  3. Spatial placement works reliably: the concept lands in the requested quadrant in every evaluated generation, while prompt-only and training-free layout baselines do not.
Critical weaknesses
  1. The problem setting is weakly motivated. The paper's premise is that storing one LoRA per concept does not scale. But a LoRA is tiny compared with the backbone, and by Section 5 independently trained LoRAs match the generated ones in quality (78.5 vs 79.1 IA) and never forget. Since the concept is assumed to be known at generation time, the right LoRA can simply be picked by index. Recent training-free methods go further and personalize from reference images with no training at all, e.g. ELITE (Wei et al., ICCV 2023), BLIP-Diffusion (Li et al., NeurIPS 2023), Kosmos-G (Pan et al., ICLR 2024). The paper does not show a realistic scenario where a continually trained hypernetwork is preferable to either option.

  2. Limited practical relevance of the setup. All main results use SD-1.5, and the method relies on UNet cross-attention: the LoRA is masked to the class-word token and the placement branch runs next to text cross-attention. Current models use MMDiT with joint text-image attention, and it is unclear whether the design carries over. A discussion of this, or results on a modern backbone, is needed for the method to be useful in practice.

  3. The 100-concept claim is checked only on the first ten concepts. Figure 6 tracks only the first ten concepts along the stream, so we do not know how well concepts learned late are acquired. Even for these ten, the advantage over CIDM shrinks to four of ten concepts at T=37, and DINO falls from 0.617 to 0.531 by T=100. Please report results for concepts from the second half of the stream.

  4. Missing comparisons with other constant-memory methods. Jha et al. (ICLR 2025) and Liao et al. (ICCV 2025) are cited but not evaluated, and ConceptGuard (Guo & Jin, CVPR 2025) is not cited. One of them on CIFC would show whether the gain comes from the hypernetwork itself.

Overall, I recommend rejection. The paper is well written and beats CIDM, but it does not convince me that the problem it solves matters in practice. A concrete scenario where a bank of LoRAs or a training-free method falls short, plus results for late concepts, would change my view.

Questions for authors
  1. Appendix C reports 26.0M hypernetwork parameters, which Figure 9 equates to about 62 per-concept adapters. For the 10 to 50 concepts studied, what does CLASP offer over storing independent LoRAs and selecting one by index?
  2. What are IA and DINO on the final checkpoint for concepts 41–50 (T=50) and 91–100 (T=100), compared with their scores right after each was learned?
  3. The 4× and 23× ratios divide by 0.0060 ± 0.0031, so the ratio to CIDM ranges from about 2.5× to 8× within one standard deviation, and the regularizer ablation in the same section starts from 0.0094. Could you report absolute differences instead, and state how many runs the CIDM forgetting value is based on?
  4. CIDM's IA and TA in Figure 4 are taken from Dong et al., who generate 50 images for each of 20 prompts, while Appendix C uses ten images per prompt. Is the 79.9 vs 78.0 comparison made with the same prompts and sample count?
  5. How would the class-word masking of K/V and the parallel placement branch be implemented in an MMDiT backbone that has no separate cross-attention?

Minor: according to Appendix C, the 14%, 45% and 154% overheads in the parameter-cost paragraph of Section 5 are generation-time figures; please separate storage from latency there.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.