acceptodds
Withdrawn from ICLR 2027

C2P-VAR: Continual and Compositional Personalization in Visual Autoregressive Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual autoregressive (VAR) models have recently emerged as an efficient paradigm for text-to-image generation, yet their personalization capabilities remain largely limited to static, single-concept settings. In practice, users may continuously introduce new concepts and wish to compose multiple personalized concepts within a single image. Such scenarios pose two fundamental challenges: catastrophic forgetting during sequential personalization and feature interference during multi-concept composition. In this work, we study continual and compositional personalization in VAR models and propose C2P-VAR, a unified framework addressing both challenges. For single-concept continual personalization, we introduce C2PVAR-S, which identifies concept-relevant parameters from gradient magnitudes and dynamically updates their selection during training. To preserve previously learned concepts, C2PVAR-S applies regularization only to parameters shared by the current and historical concepts, thereby reducing unnecessary interference without introducing additional model components. For multi-concept personalization, we further propose C2PVAR-M, which employs parallel global and concept-specific branches with spatially localized feature fusion and logit aggregation to achieve controllable concept placement and reduce feature entanglement. Extensive experiments on continual and multi-concept personalization demonstrate that C2P-VAR consistently outperforms existing baselines in subject fidelity, while maintaining competitive text alignment and introducing negligible storage and inference overhead. Our results establish a unified framework for scalable and controllable personalization of visual autoregressive models.

Primary area:
generative models
Keywords:
Visual autoregressive model, continual personalization, compositional personalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ubyROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates continual personalization learning and multi-concept compositional generation in visual autoregressive (VAR) models, and proposes a unified framework called C2P-VAR, which consists of two modules. C2PVAR-S selects concept-relevant parameters based on gradient magnitudes and dynamically updates the parameter masks. It applies regularization only to important parameters shared between the current task and historical tasks to mitigate catastrophic forgetting. C2PVAR-M introduces global and concept-specific branches during inference and achieves multi-concept composition through spatial feature fusion and logit aggregation. The authors conduct experiments based on Infinity-2B, including continual learning of 15 concepts and multi-concept generation. The experimental results show that the proposed method outperforms existing baselines in subject fidelity while maintaining relatively reasonable text alignment performance.

Critical strengths

The research problem has practical relevance. Existing work on VAR personalization primarily focuses on single-concept training. This paper further considers continual learning and multi-concept composition, broadening the scope of research on personalized generation with VAR.

The experimental results are competitive. In the continual learning experiment involving eight concepts, C2PVAR-S achieves an average DINO score of 68.17, substantially outperforming the strongest baseline at 60.04. The experiments involving 15 concepts also show advantages, indicating that the method is effective in mitigating catastrophic forgetting.

Critical weaknesses

I have some concerns about the necessity of continual personalization in practical personalized generation scenarios. In real-world applications, users typically only need to compose a small number of concepts. For such scenarios, Joint Training can be directly applied, or independent LoRA adapters can be trained for different concepts and composed during inference. Even when new concepts are introduced later, they can be accommodated through retraining or adding independent adapters. Although continual learning avoids repeatedly accessing historical data, the paper does not clearly specify practical constraints, such as the unavailability of historical data or the high cost of retraining. It also lacks sufficient comparisons with Joint Training or Independent LoRA. Therefore, it is difficult to determine what practical advantages the proposed method offers over these simpler alternatives. Furthermore, C2PVAR-S essentially balances the learning of new and previously learned concepts through regularization within a fixed parameter space, rather than directly expanding the model's personalization capacity. As the number of concepts increases, parameter conflicts may become increasingly severe, but the paper does not sufficiently analyze the long-term scalability of the proposed method. For multi-concept composition, C2PVAR-M itself is a relatively independent inference strategy that could theoretically be applied to Joint Training or Independent LoRA. The paper does not demonstrate the necessity of combining these two modules.

The core assumption of C2PVAR-S is that different concepts rely on different subsets of important parameters, and that overlap between these parameters may lead to concept conflicts. However, the paper does not provide analyses of gradient distributions, mask overlap, or gradient similarity across different concepts. Therefore, it is unclear whether different concepts actually exhibit significant differences in parameter selection. More importantly, parameter overlap does not necessarily imply parameter conflict. Even if two concepts rely on the same parameters, their gradient directions may be consistent. The authors directly apply regularization to overlapping important parameters without verifying whether these parameters actually exhibit gradient conflicts. In addition, the paper continuously accumulates historical important parameter masks through the OR operation, but does not analyze how mask coverage and parameter overlap ratios change as the number of tasks increases. If the historical masks eventually cover most of the Cross-Attention parameters, the proposed method may gradually lose the advantages of selective regularization.

Questions for authors

Q1. Could the authors provide analyses of Mask Overlap and Gradient Cosine Similarity across different concepts to verify whether different concepts indeed rely on different parameter subsets and whether shared parameters actually exhibit gradient conflicts? In addition, how does the coverage of historical masks change as the number of tasks increases? Q2. Could the authors provide comparisons with Joint Training and Independent LoRA to demonstrate the practical advantages of continual personalization in terms of generation quality, incremental training cost, and storage overhead? Joint Training could serve as a reference upper bound when all historical data is accessible. Q3. C2PVAR-M introduces additional spatial control and multi-branch fusion mechanisms. Could the authors clarify the multi-concept generation strategies used by each baseline and provide experiments applying C2PVAR-M to other personalization methods, in order to distinguish the improvements brought by personalization training from those brought by inference-time composition strategies? Q4. The authors train Infinity-2B on a single A6000 GPU but do not report the actual Peak GPU Memory. I am particularly curious about the additional GPU memory consumption introduced by the parameter-level masks during training.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

W1: I have some concerns about the necessity of continual personalization in practical personalized-generation scenarios. Even when new concepts are introduced later, they can be accommodated through retraining or by adding independent adapters.

Response: We thank the reviewer for raising this important point. We agree that the original manuscript did not clearly identify the specific constraints motivating continual personalization, such as the unavailability of historical data and the high cost of retraining. We will add this constraint in the final version.

Maintaining a separate adapter for each concept and loading only the corresponding LoRA at inference differs from the continual-learning setting considered in our work. Similar to methods such as C-LoRA and OA, even when a separate LoRA is stored for each concept, different merging strategies are still required to combine concept-specific LoRAs for final single-concept and multi-concept generation.

As the number of concepts increases, independent adapters incur cumulative storage and model-management costs. They require adapter selection for single-concept queries and weight-level or feature-level composition for multi-concept generation. Automatic adapter selection may further require a learned prompt-to-adapter router. These costs are therefore not universally negligible.

By contrast, C2PVAR-S integrates all learned concepts into a single set of weights, avoiding per-concept adapter storage, external adapter selection, and LoRA fusion during single-concept generation. This single-model, fusion-free setting is also consistent with previous continual-personalization methods such as CNS.

Q1: Could the authors provide analyses of mask overlap and gradient cosine similarity across different concepts to verify whether different concepts indeed rely on different parameter subsets and whether shared parameters actually exhibit gradient conflicts? In addition, how does the coverage of historical masks change as the number of tasks increases?

Response: We thank the reviewer for this valuable suggestion. We analyzed mask overlap in the eight-task personalization setting. We used Jaccard similarity to measure mask overlap among all 28 concept pairs formed by the eight concepts. We provide the heatmap in the revised manuscript in Appendix A.9 and four representative concept pairs are shown below.

Concept pairJaccard similarity
dog–dog232.90%
cat–cat226.33%
duck_toy–cat22.33%
cat–teddybear26.24%

The mean Jaccard similarity across all 28 concept pairs is 26.35%, which is lower than the 37.04% parameter coverage of CNS calculated using the same procedure. This result indicates that different concepts rely on distinct but partially shared subsets of sensitive parameters. Our method can therefore identify and protect task-relevant parameters to mitigate catastrophic forgetting during continual learning.

We also analyzed the gradient cosine similarity of overlapping parameters across different concept pairs. The mean gradient cosine similarity over the intersecting parameters is 0.2774, while an average of 40.27% of the shared parameters have nonzero gradients with opposite signs for the two concepts. These results support our hypothesis that partial gradient conflicts exist between different concepts.

Although some shared parameters have positive cosine similarity, aligned updates after removing the regularization term do not necessarily preserve previously learned concepts. Near an optimum for an earlier task, the gradient of that task may be small, while displacement along a sensitive curvature direction can still increase its loss. Our ablation study further shows that removing the regularization term substantially reduces subject fidelity and causes catastrophic forgetting, confirming the effectiveness of the proposed regularization.

We additionally report results for a 15-task sequence in Appendix A.5. Although overlap increases during long-term continual learning, our method still substantially reduces catastrophic forgetting and achieves the best overall performance. Its performance does not noticeably degrade across the 15 tasks.

We further analyzed the coverage of the accumulated historical mask. The results are shown below.

Number of learned conceptsHistorical mask coverage
110.37%
216.19%
318.01%
419.52%
520.25%
621.20%
722.48%
823.79%

Across the evaluated sequence of eight concepts, the accumulated historical mask does not saturate and does not degenerate into an all-encompassing protection region.

AuthorsOpenReview ↗

Official Comment by Authors

Q2: Could the authors provide comparisons with Joint Training and Independent LoRA to demonstrate the practical advantages of continual personalization in terms of generation quality, incremental training cost, and storage overhead? Joint Training could serve as a reference upper bound when all historical data are accessible.

Response: We thank the reviewer for this constructive suggestion. We added comparisons with Joint Training and Independent LoRA. We report the DINO score for each concept and the average DINO score. The results are shown below.

MethodV1 dogV2 DucktoyV3 CatV4 teddybearV5 Dog2V6 DrawingV7 Cat2V8 InkpaintingAvg.
Joint Training81.9166.8178.4465.3365.1843.3661.4433.9162.05
Independent LoRA85.3770.2788.5571.4683.1538.0274.1235.4868.30
Ours81.8969.1782.8675.5080.3641.8078.2135.5768.17

These results show that our method maintains or exceeds the performance of Joint Training, which serves as an upper-bound reference when all historical data are available. Joint Training is affected by similar concepts in the training set. For example, interference between V1 dog and V5 Dog2 reduces its overall performance. Independent LoRA achieves subject fidelity comparable to our method. However, it requires additional storage and repeated adapter loading and composition as user requirements evolve. Our method therefore offers a better balance between generation quality and practical deployment cost.

Q3: C2PVAR-M introduces additional spatial control and multi-branch fusion mechanisms. Could the authors clarify the multi-concept generation strategies used by each baseline and provide experiments applying C2PVAR-M to other personalization methods, in order to distinguish the improvements brought by personalization training from those brought by inference-time composition strategies?

Response: We thank the reviewer for this important suggestion. As stated in the manuscript, for multi-concept customization, we incorporate the region-controllable sampling module from Mix-of-Show into the baseline methods for a fair comparison. Bounding-box masks replace the global cross-attention outputs with the corresponding regional outputs. Outputs in overlapping regions are averaged, while uncovered positions retain the global outputs.

We also applied C2PVAR-M to other personalization methods using the same experimental settings and evaluation protocol as in the manuscript. We report regional subject fidelity measured by DINO. The results are shown below.

BaselineC2PVAR-MRegion-controllable sampling module
C-LoRA57.3252.61
OA61.5856.52
Independent LoRA60.5956.58

Applying C2PVAR-M consistently improves subject fidelity for multi-concept generation across the baseline personalization methods, demonstrating that the proposed inference-time composition strategy provides benefits beyond the personalization training method itself.

Q4: The authors train Infinity-2B on a single A6000 GPU but do not report the actual peak GPU memory. I am particularly curious about the additional GPU memory consumption introduced by the parameter-level masks during training.

Response: We thank the reviewer for requesting a direct memory measurement. We profiled the original Infinity-2B training implementation on a single RTX A6000 at a resolution of 1024 × 1024 and a batch size of 1. For the eighth concept, the peak GPU memory allocated by PyTorch is 28.72 GiB, compared with 26.21 GiB for the control without masks or overlap regularization. The masks therefore introduce an additional 2.51 GiB, corresponding to a 9.57% increase. The GPU-resident Boolean masks occupy 1.50 GiB in total, while the accumulated historical union mask occupies 0.50 GiB on the CPU.

AuthorsOpenReview ↗

Friendly Reminder

Dear Reviewer ubyR Thank you again for your review and your detailed questions. As the discussion period ends on November 18 (AoE), we would like to hear whether our responses have addressed your concerns. If any issue remains or any new questions have come up, please let us know, and we will respond promptly.

Reviewer_Lh5TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes C2P-VAR for continual and multi-concept personalization in VAR models. It uses selective parameter updates and regularization to reduce forgetting, and combines separate generation branches to control where different concepts appear. The experiments show improved subject fidelity and multi-concept generation compared with the baselines.

Critical strengths

1.The paper addresses a useful setting where new concepts are learned over time and later combined in the same image. The motivation is clear and reflects practical needs in personalized generation.

2.The experiments cover both object and style concepts, with additional tests using different task orders and random seeds. These checks provide useful evidence about the stability of the method.

Critical weaknesses

1.Continual personalization has already been extensively studied in diffusion models. The authors should explain more clearly what challenges are specific to VAR and what benefits this architecture offers for personalization, supported by theoretical analysis and experimental evidence.

2.This also raises concerns about the baseline coverage. Although the authors adapt several existing methods to Infinity, comparisons with relevant approaches remain missing, including CIFC/CIDM [1], SCNS [2], for continual personalization, as well as PureCC [3] and LoRA-Composer [4] for model preservation and concept composition. Where feasible, adapting these methods to the same VAR backbone would help assess the contribution of the proposed design. At minimum, the authors should compare against their official implementations under matched data and evaluation settings, while reporting differences in backbones and computational costs. These experiments would clarify whether the proposed framework offers practical advantages over existing solutions.

  • [1] Jiahua Dong et al. How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? NeurIPS, 2024.
  • [2] Zijie Peng et al. SCNS: Continual Personalization of Diffusion Models via Submodular Concept Neuron Selection. ICML, 2026.
  • [3] Zhichao Liao et al. PureCC: Pure Learning for Text-to-Image Concept Customization. CVPR, 2026.
  • [4] Yang Yang et al. LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models. IEEE Transactions on Image Processing, 2025.

3.Tables 1 and 8–10 report final performance, making it hard to tell whether the gains come from better initial learning or less forgetting. The authors should track each concept’s performance as new concepts are learned to directly evaluate forgetting.

Questions for authors

Please address the concerns above with additional qualitative and quantitative evidence in the final version.

7 comments
AuthorsOpenReview ↗

Official Comment by Authors

W1: The authors should explain more clearly what challenges are specific to VAR and what benefits this architecture offers for personalization, supported by theoretical analysis and experimental evidence.

Response: We thank the reviewer for this important comment. Although continual personalization has been extensively studied for diffusion models, VAR exhibits distinct generation and forgetting mechanisms. Unlike diffusion models, which iteratively refine an image through repeated denoising, VAR predicts multi-scale token maps autoregressively from coarse to fine. This architecture provides substantially faster inference and an explicit hierarchical representation that naturally supports scale-specific personalization and spatial control.

However, the architecture also introduces unique challenges. Each predicted scale conditions all subsequent scales, and the early coarse scales largely determine the global structure and subject identity. Interference introduced at an early scale while learning a new subject can therefore propagate through the entire generation trajectory. It is difficult to correct this interference at later scales because they primarily refine local details. Sequentially updating these parameters may overwrite previously encoded identities and amplify catastrophic forgetting through the coarse-to-fine dependency. Continual personalization for VAR must therefore preserve historical knowledge in these shared, high-impact parameters and early-scale representations. At the same time, VAR's fast next-scale prediction and explicit multi-scale hierarchy provide important advantages for efficient personalized generation, allowing concept learning and composition to be controlled at both the parameter and scale levels.

AuthorsOpenReview ↗

Official Comment by Authors

W2: Although the authors adapt several existing methods to Infinity, comparisons with relevant approaches remain missing, including CIFC/CIDM [1] and SCNS [2] for continual personalization, as well as PureCC [3] and LoRAComposer [4] for model preservation and concept composition.

Response: We thank the reviewer for this valuable suggestion. We added experiments with the requested baselines under the same experimental settings and evaluation criteria using an eight-concept continual-learning sequence.

For CIDM, we transferred its hierarchical concept tokens, inter-task orthogonality constraint, and shared-subspace constraint. We trained a pair of LoRA adapters for each concept and used the EWA module at inference to dynamically fuse LoRAs from previous tasks according to the prompt for single-concept generation. Because CIDM requires layer-wise EWA weight computation and LoRA merging, it introduces an additional 0.128 seconds for EWA computation and 3.351 seconds for layer-wise LoRA merging for generating single image. The CPU must also load the LoRAs for weight merging, resulting in an additional peak memory consumption of 8.02 GiB.

For SCNS, we transferred its modules to Infinity, including task-gradient correlation, retention scoring based on historical gradients and a Fisher proxy, Facility Location coverage, zero-shot protection, and cost-aware greedy selection. Only the selected columns and the input embedding of the current concept are updated, while all other parameters remain frozen.

For PureCC, we adapted its two-stage learning procedure to Infinity, including concept extractor training, the concept increment provided by the frozen extractor, and the adaptive projection constraint. Each task starts from the model produced by the previous task and undergoes the two-stage training procedure, after which the student LoRA is merged. The hyperparameters of all baselines follow their original papers.

We compared C2PVAR-S with CIDM, SCNS, and PureCC. The results are shown below.

MethodDINOCLIP-ICLIP-T
CIDM66.9882.4130.31
SCNS57.1777.7432.08
PureCC54.6477.6330.32
Ours68.1782.7131.14

Under the eight-concept continual-learning setting, our method achieves the best subject fidelity and CLIP-I. In contrast to CIDM's multi-LoRA merging strategy, our method introduces no additional memory or time overhead for adapter merging.

We also adapted LoRA-Composer to Infinity using independently trained concept LoRAs. We retained its region-aware cross-attention injection, in which queries are shared across concepts while each region uses its local text condition together with the corresponding LoRA key, value, and output projections. We also retained Gaussian-weighted concept enhancement, axis-wise coverage constraints, self-attention isolation across foreground regions, and the out-of-region attention penalty. RMS-normalized gradient updates with an upper bound on relative displacement were used to control the update magnitude.

We compared C2PVAR-M with LoRA-Composer. The results are shown below.

MethodDINOCLIP-ICLIP-T
LoRA-Composer62.2582.3234.82
Ours68.3783.3535.24

Our method achieves the best performance for multi-concept composition across all three metrics.

W3: The authors should track each concept's performance as new concepts are learned to directly evaluate forgetting.

Response: We thank the reviewer for this constructive suggestion. We use backward transfer (BWT) to quantify forgetting: BWT=1T∑i=1T(AT,i−Ai,i).\mathrm{BWT}=\frac{1}{T}\sum_{i=1}^{T}\left(A_{T,i}-A_{i,i}\right). Although BWT effectively quantifies catastrophic forgetting, most LoRA-based baselines train an independent LoRA module for each task and merge the modules only after training. Standard BWT is therefore not directly applicable to those methods. Because our method performs genuine continual learning on a single unified backbone, we report the formal BWT metric for the applicable baseline methods.

The results are shown below.

MethodDINOBWT
ARBooth51.68-22.09
LoRA50.60-19.32
LwF59.05-18.71
MAS57.21-8.07
CNS45.31-0.19
SCNS57.17-4.51
Ours68.17-0.54

The results show that our method achieves the strongest overall learning performance in terms of DINO while effectively mitigating forgetting.

Reviewer_Lh5TOpenReview ↗

Response to the Rebuttal

Thank you for your detailed responses and the effort devoted to the additional experiments and clarifications. Please incorporate all the additional results presented in your responses, together with the relevant experimental details and complete, accurate citations, into the main text of the revised manuscript, as these additions do not yet appear to be fully included. The published manuscript will be the primary reference for readers and should provide a self-contained account of the work. I will reread the revised paper and reassess the submission as a whole. I will also carefully consider the other reviewers’ comments to identify any aspects I may have overlooked and support a balanced assessment. Thank you again for your time and effort.

Reviewer_55LpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies personalization in visual autoregressive models in two settings, learning new concepts sequentially and composing several learned concepts in one image. The proposed C2P-VAR has two parts. C2PVAR-S handles continual learning by updating only a small set of gradient-selected cross-attention parameters and regularizing those that overlap with earlier concepts, so all concepts live in a single model. C2PVAR-M is a training-free inference procedure that runs a global branch and one branch per concept and merges their features and logits within user-provided bounding boxes. Experiments on an Infinity-based benchmark show higher subject fidelity than diffusion-based baselines ported to VAR, with slightly lower text alignment for single concepts.

Critical strengths
  • This work tackles both continual and multi concept personalization in VAR models, with a benchmark that includes semantically similar concepts.
  • C2PVAR-M is training-free and gives explicit spatial control, which is a relatively hard task in personalization.
Critical weaknesses
  • The parameter selection step contributes little. In the ablation, selection without regularization is barely above ARBooth and nearly all of the gain comes from the masked L2 penalty. With historical masks accumulating by OR and overlap regularized rather than avoided, the method is close to a sparse variant of MAS. The insensitivity to the regularization coefficient and the lack of any mask overlap analysis support this reading.
  • The single weight continual setting is not well motivated. Adapter storage and merge cost for the baselines are negligible, and modular approaches avoid forgetting by construction. The zero overhead claim also applies only to C2PVAR-S, since C2PVAR-M runs one branch per subject and its inference cost is not reported.
  • C2PVAR-M is evaluated almost only on two subject layouts with non overlapping boxes. There is no analysis of scaling with the number of subjects, no handling of overlapping boxes, and no ablation on the blending factor that is the only mechanism for integrating subjects with the background.
  • All experiments use a single base model. Results on at least one other VAR model would help show that the findings are not specific to this backbone.
Questions for authors
  • What is the motivation behind combining continual and multi subject personalization in one framework? In particular, why is sequential training into a single set of weights preferable to keeping a separate adapter per subject, which avoids forgetting by construction?
  • What is the inference cost of C2PVAR-M, and how does it scale with the number of subjects in a single image? Table 14 only reports the cost of merging weights, which does not apply to the multi branch inference.
  • Since all tasks are subjects, I would expect the gradient selected parameters to overlap heavily across tasks. How much do the task masks actually overlap, and does performance degrade as the overlap grows? Is there any analysis of which parameters tend to be selected?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

W1: The parameter selection step contributes little. In the ablation, selection without regularization is barely above ARBooth, and nearly all of the gain comes from the masked L2 penalty. With historical masks accumulating by OR and overlap regularized rather than avoided, the method is close to a sparse variant of MAS. The insensitivity to the regularization coefficient and the lack of any mask overlap analysis support this reading.

Response: We thank the reviewer for raising this important concern. Our contribution lies in combining dynamic, task-conditioned parameter selection with the protection of historically salient parameters. These two components are complementary, as supported by the baseline comparisons, component ablations, and additional mask analyses.

In the ablation study, we further compared the settings with and without dynamic masks. The results show that using dynamic masks clearly reduces interference between similar concepts and improves the average DINO score by 2.89. Moreover, under our experimental setting, MAS, which uses regularization alone, yields substantially lower DINO and CLIP-I scores than our method. This result demonstrates that combining parameter selection with the regularization term effectively mitigates catastrophic forgetting in continual personalization.

We also analyzed mask overlap in the eight-task personalization setting. We used Jaccard similarity to measure mask overlap among all 28 concept pairs formed by the eight concepts. We provide the heatmap in the revised manuscript in Appendix A.9 and four representative concept pairs are shown below.

Concept pairJaccard similarity
dog–dog232.90%
cat–cat226.33%
duck_toy–cat22.33%
cat–teddybear26.24%

The mean Jaccard similarity across all 28 concept pairs is 26.35%, which is lower than the 37.04% parameter coverage of CNS calculated using the same procedure. This result indicates that different concepts rely on distinct but partially shared subsets of sensitive parameters. Our method can therefore identify and protect task-relevant parameters to mitigate catastrophic forgetting during continual learning.

We additionally report results for a 15-task sequence in Appendix A.5. Although mask overlap increases during long-term continual learning, our method still substantially reduces catastrophic forgetting and achieves the best overall performance. Its performance does not noticeably degrade across the 15 tasks.

W2: The single-weight continual setting is not well motivated. Adapter storage and merge cost for the baselines are negligible, and modular approaches avoid forgetting by construction. The zero-overhead claim also applies only to C2PVAR-S, since C2PVAR-M runs one branch per subject and its inference cost is not reported.

Response: Maintaining a separate adapter for each concept and loading only the corresponding LoRA at inference differs from the continual-learning setting considered in our work. Similar to methods such as C-LoRA and OA, even when a separate LoRA is stored for each concept, different merging strategies are still required to combine concept-specific LoRAs for final single-concept and multi-concept generation.

As the number of concepts increases, independent adapters also incur cumulative storage and model-management costs. They require adapter selection for single-concept queries and weight-level or feature-level composition for multi-concept generation. Automatic adapter selection may further require a learned prompt-to-adapter router. These costs are therefore not universally negligible.

By contrast, C2PVAR-S integrates all learned concepts into a single set of weights, avoiding per-concept adapter storage, external adapter selection, and LoRA fusion during single-concept generation. This single-model, fusion-free setting is also consistent with prior continual-personalization methods such as CNS.

We additionally measured the inference time and peak GPU memory of C2PVAR-M and the region-controllable sampling module used by the baseline methods with different numbers of subjects. The results are shown below.

Number of conceptsC2PVAR-M, time / peak GPU memoryRegion-controllable sampling module, time / peak GPU memory
26.15 s / 17.93 GiB2.89 s / 10.95 GiB
37.97 s / 21.44 GiB3.03 s / 10.95 GiB
49.81 s / 24.96 GiB3.14 s / 10.95 GiB

Within the evaluated range of two to four subjects, the inference time of C2PVAR-M increases with the number of subjects. Each additional subject adds approximately 1.83 seconds per image. Because C2PVAR-M performs multi-branch inference and maintains the historical state of each branch, its memory usage also increases with the number of subjects. Despite this additional cost, C2PVAR-M provides substantially better multi-concept composition quality than the standard region-controllable sampling module.

AuthorsOpenReview ↗

Official Comment by Authors

W3: C2PVAR-M is evaluated almost only on two-subject layouts with non-overlapping boxes. There is no analysis of scaling with the number of subjects, no handling of overlapping boxes, and no ablation on the blending factor, which is the only mechanism for integrating subjects with the background.

Response: We thank the reviewer for this constructive suggestion. We added 30 three-subject prompts and 5 four-subject prompts to the evaluation. The expanded multi-concept benchmark contains 75 prompts in total, and we generate 10 images for each prompt. The results are shown below.

MethodDINOCLIP-ICLIP-T
ARBooth41.0971.8332.54
LoRA39.0772.5732.35
LWF49.5176.6730.47
MAS54.6477.7734.64
C-LoRA51.7078.7333.67
OA47.7275.8132.39
CNS44.3578.0033.92
Ours64.8181.5135.66

Our method continues to achieve the best performance across all three metrics on the expanded evaluation set, demonstrating improved subject fidelity and text alignment for multi-concept personalization. The current implementation assumes non-overlapping bounding boxes, following the region-controllable sampling protocol of Mix-of-Show that is used by the baseline methods. Increasing the number of subjects leads to additional inference time and GPU memory consumption, as reported in our response to W2.

We also added an ablation study of the blending factor on the expanded evaluation set. The results are shown below.

Blending factorDINOCLIP-ICLIP-T
0.0164.4681.2235.59
0.0564.8181.5135.66
0.367.4482.6335.80
0.765.6082.0035.97

The results show that performance can be further improved by tuning the blending factor for the task. When the blending factor becomes too large, the local and global branches are not fused sufficiently, which reduces subject fidelity. Consequently, both CLIP-I and DINO fall below their optimal values.

W4: All experiments use a single base model. Results on at least one other VAR model would help show that the findings are not specific to this backbone.

Response: We thank the reviewer for this valuable suggestion. We conducted additional experiments on STAR-512. We selected one continual-learning baseline and one diffusion-based continual-personalization baseline for comparison, and evaluated both single-concept continual learning and multi-concept composition. All methods were trained on the same sequence of eight concepts using identical experimental settings and evaluation criteria.

The single-concept continual-learning results are shown below.

MethodDINOCLIP-ICLIP-T
MAS38.1470.7929.88
C-LoRA41.6572.6531.82
C2PVAR-S46.6274.2333.86

We also evaluated multi-concept composition using the same protocol as in the main manuscript. The results are shown below.

MethodDINOCLIP-ICLIP-T
MAS22.4665.8327.75
C-LoRA29.1169.2332.12
C2PVAR-M37.5472.2433.33

Our method achieves the best performance in both single-concept generation and multi-concept composition. These results demonstrate that the proposed method generalizes across different VAR backbones.

Q1: What is the motivation behind combining continual and multi-subject personalization in one framework? In particular, why is sequential training into a single set of weights preferable to keeping a separate adapter per subject, which avoids forgetting by construction?

Response: Please see our detailed response to W2 above.

Q2: What is the inference cost of C2PVAR-M, and how does it scale with the number of subjects in a single image? Table 14 only reports the cost of merging weights, which does not apply to multi-branch inference.

Response: Please see our detailed response to W2 above.

Q3: Since all tasks are subjects, I would expect the gradient-selected parameters to overlap heavily across tasks. How much do the task masks actually overlap, and does performance degrade as the overlap grows? Is there any analysis of which parameters tend to be selected?

Response: Please see our detailed response to W1 above.

AuthorsOpenReview ↗

Friendly Reminder

Dear Reviewer 55Lp, Thank you again for your review and your detailed questions. As the discussion period ends on November 18 (AoE), we would like to hear whether our responses have addressed your concerns. If any issue remains or any new questions have come up, please let us know, and we will respond promptly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal

Revisions of the Manuscript

  1. Page 2, Lines 69–75: We have clarified the challenges specific to VAR and the benefits of this architecture for personalization. (Lh5T)

  2. Page 3, Lines 160–161: We have clearly identified the specific constraints motivating continual personalization, such as the unavailability of historical data and the high cost of retraining. (ubyR)

  3. Page 6, Line 278: We have revised the selection of the blending factor. (55Lp)

  4. Tables 1, 2, and 8–12: We have added new baselines for comparison. (Lh5T)

  5. Page 15: We have provided additional implementation details for the newly introduced baselines. (Lh5T)

  6. Page 15, Line 809: We have reported the peak GPU memory consumption during training. (ubyR)

  7. Page 16, Line 818: We have specified the number of prompts used in the multi-concept evaluation. (55Lp)

  8. Page 16, Lines 860–863, and Table 6: We have added the calculation of the BWT metric to evaluate the extent of forgetting in the model. (Lh5T)

  9. Page 17, Lines 904–913, and Table 7: We have added an overhead evaluation of C2PVAR-M. (55Lp)

  10. Page 21, Lines 1106–1110: We have added an ablation study on the blending factor. (55Lp)

  11. Page 21, Lines 1111–1116: We have conducted additional experiments applying C2PVAR-M to other baseline methods. (ubyR)

  12. Pages 21–22, Table 18, and Figures 7–8: We have added a Mask Overlap and Historical Parameter Protection Analysis. (55Lp, ubyR)

  13. Page 22, Lines 1173–1187: We have added transfer experiments on other models to evaluate the generalizability of our method. (55Lp)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.