- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates continual personalization learning and multi-concept compositional generation in visual autoregressive (VAR) models, and proposes a unified framework called C2P-VAR, which consists of two modules. C2PVAR-S selects concept-relevant parameters based on gradient magnitudes and dynamically updates the parameter masks. It applies regularization only to important parameters shared between the current task and historical tasks to mitigate catastrophic forgetting. C2PVAR-M introduces global and concept-specific branches during inference and achieves multi-concept composition through spatial feature fusion and logit aggregation. The authors conduct experiments based on Infinity-2B, including continual learning of 15 concepts and multi-concept generation. The experimental results show that the proposed method outperforms existing baselines in subject fidelity while maintaining relatively reasonable text alignment performance.
Critical strengths
The research problem has practical relevance. Existing work on VAR personalization primarily focuses on single-concept training. This paper further considers continual learning and multi-concept composition, broadening the scope of research on personalized generation with VAR.
The experimental results are competitive. In the continual learning experiment involving eight concepts, C2PVAR-S achieves an average DINO score of 68.17, substantially outperforming the strongest baseline at 60.04. The experiments involving 15 concepts also show advantages, indicating that the method is effective in mitigating catastrophic forgetting.
Critical weaknesses
I have some concerns about the necessity of continual personalization in practical personalized generation scenarios. In real-world applications, users typically only need to compose a small number of concepts. For such scenarios, Joint Training can be directly applied, or independent LoRA adapters can be trained for different concepts and composed during inference. Even when new concepts are introduced later, they can be accommodated through retraining or adding independent adapters. Although continual learning avoids repeatedly accessing historical data, the paper does not clearly specify practical constraints, such as the unavailability of historical data or the high cost of retraining. It also lacks sufficient comparisons with Joint Training or Independent LoRA. Therefore, it is difficult to determine what practical advantages the proposed method offers over these simpler alternatives. Furthermore, C2PVAR-S essentially balances the learning of new and previously learned concepts through regularization within a fixed parameter space, rather than directly expanding the model's personalization capacity. As the number of concepts increases, parameter conflicts may become increasingly severe, but the paper does not sufficiently analyze the long-term scalability of the proposed method. For multi-concept composition, C2PVAR-M itself is a relatively independent inference strategy that could theoretically be applied to Joint Training or Independent LoRA. The paper does not demonstrate the necessity of combining these two modules.
The core assumption of C2PVAR-S is that different concepts rely on different subsets of important parameters, and that overlap between these parameters may lead to concept conflicts. However, the paper does not provide analyses of gradient distributions, mask overlap, or gradient similarity across different concepts. Therefore, it is unclear whether different concepts actually exhibit significant differences in parameter selection. More importantly, parameter overlap does not necessarily imply parameter conflict. Even if two concepts rely on the same parameters, their gradient directions may be consistent. The authors directly apply regularization to overlapping important parameters without verifying whether these parameters actually exhibit gradient conflicts. In addition, the paper continuously accumulates historical important parameter masks through the OR operation, but does not analyze how mask coverage and parameter overlap ratios change as the number of tasks increases. If the historical masks eventually cover most of the Cross-Attention parameters, the proposed method may gradually lose the advantages of selective regularization.
Questions for authors
Q1. Could the authors provide analyses of Mask Overlap and Gradient Cosine Similarity across different concepts to verify whether different concepts indeed rely on different parameter subsets and whether shared parameters actually exhibit gradient conflicts? In addition, how does the coverage of historical masks change as the number of tasks increases? Q2. Could the authors provide comparisons with Joint Training and Independent LoRA to demonstrate the practical advantages of continual personalization in terms of generation quality, incremental training cost, and storage overhead? Joint Training could serve as a reference upper bound when all historical data is accessible. Q3. C2PVAR-M introduces additional spatial control and multi-branch fusion mechanisms. Could the authors clarify the multi-concept generation strategies used by each baseline and provide experiments applying C2PVAR-M to other personalization methods, in order to distinguish the improvements brought by personalization training from those brought by inference-time composition strategies? Q4. The authors train Infinity-2B on a single A6000 GPU but do not report the actual Peak GPU Memory. I am particularly curious about the additional GPU memory consumption introduced by the parameter-level masks during training.
3 comments
Official Comment by Authors
Q2: Could the authors provide comparisons with Joint Training and Independent LoRA to demonstrate the practical advantages of continual personalization in terms of generation quality, incremental training cost, and storage overhead? Joint Training could serve as a reference upper bound when all historical data are accessible.
Response: We thank the reviewer for this constructive suggestion. We added comparisons with Joint Training and Independent LoRA. We report the DINO score for each concept and the average DINO score. The results are shown below.
| Method | V1 dog | V2 Ducktoy | V3 Cat | V4 teddybear | V5 Dog2 | V6 Drawing | V7 Cat2 | V8 Inkpainting | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Joint Training | 81.91 | 66.81 | 78.44 | 65.33 | 65.18 | 43.36 | 61.44 | 33.91 | 62.05 |
| Independent LoRA | 85.37 | 70.27 | 88.55 | 71.46 | 83.15 | 38.02 | 74.12 | 35.48 | 68.30 |
| Ours | 81.89 | 69.17 | 82.86 | 75.50 | 80.36 | 41.80 | 78.21 | 35.57 | 68.17 |
These results show that our method maintains or exceeds the performance of Joint Training, which serves as an upper-bound reference when all historical data are available. Joint Training is affected by similar concepts in the training set. For example, interference between V1 dog and V5 Dog2 reduces its overall performance. Independent LoRA achieves subject fidelity comparable to our method. However, it requires additional storage and repeated adapter loading and composition as user requirements evolve. Our method therefore offers a better balance between generation quality and practical deployment cost.
Q3: C2PVAR-M introduces additional spatial control and multi-branch fusion mechanisms. Could the authors clarify the multi-concept generation strategies used by each baseline and provide experiments applying C2PVAR-M to other personalization methods, in order to distinguish the improvements brought by personalization training from those brought by inference-time composition strategies?
Response: We thank the reviewer for this important suggestion. As stated in the manuscript, for multi-concept customization, we incorporate the region-controllable sampling module from Mix-of-Show into the baseline methods for a fair comparison. Bounding-box masks replace the global cross-attention outputs with the corresponding regional outputs. Outputs in overlapping regions are averaged, while uncovered positions retain the global outputs.
We also applied C2PVAR-M to other personalization methods using the same experimental settings and evaluation protocol as in the manuscript. We report regional subject fidelity measured by DINO. The results are shown below.
| Baseline | C2PVAR-M | Region-controllable sampling module |
|---|---|---|
| C-LoRA | 57.32 | 52.61 |
| OA | 61.58 | 56.52 |
| Independent LoRA | 60.59 | 56.58 |
Applying C2PVAR-M consistently improves subject fidelity for multi-concept generation across the baseline personalization methods, demonstrating that the proposed inference-time composition strategy provides benefits beyond the personalization training method itself.
Q4: The authors train Infinity-2B on a single A6000 GPU but do not report the actual peak GPU memory. I am particularly curious about the additional GPU memory consumption introduced by the parameter-level masks during training.
Response: We thank the reviewer for requesting a direct memory measurement. We profiled the original Infinity-2B training implementation on a single RTX A6000 at a resolution of 1024 × 1024 and a batch size of 1. For the eighth concept, the peak GPU memory allocated by PyTorch is 28.72 GiB, compared with 26.21 GiB for the control without masks or overlap regularization. The masks therefore introduce an additional 2.51 GiB, corresponding to a 9.57% increase. The GPU-resident Boolean masks occupy 1.50 GiB in total, while the accumulated historical union mask occupies 0.50 GiB on the CPU.
Friendly Reminder
Dear Reviewer ubyR Thank you again for your review and your detailed questions. As the discussion period ends on November 18 (AoE), we would like to hear whether our responses have addressed your concerns. If any issue remains or any new questions have come up, please let us know, and we will respond promptly.
Official Comment by Authors
Response: We thank the reviewer for raising this important point. We agree that the original manuscript did not clearly identify the specific constraints motivating continual personalization, such as the unavailability of historical data and the high cost of retraining. We will add this constraint in the final version.
Maintaining a separate adapter for each concept and loading only the corresponding LoRA at inference differs from the continual-learning setting considered in our work. Similar to methods such as C-LoRA and OA, even when a separate LoRA is stored for each concept, different merging strategies are still required to combine concept-specific LoRAs for final single-concept and multi-concept generation.
As the number of concepts increases, independent adapters incur cumulative storage and model-management costs. They require adapter selection for single-concept queries and weight-level or feature-level composition for multi-concept generation. Automatic adapter selection may further require a learned prompt-to-adapter router. These costs are therefore not universally negligible.
By contrast, C2PVAR-S integrates all learned concepts into a single set of weights, avoiding per-concept adapter storage, external adapter selection, and LoRA fusion during single-concept generation. This single-model, fusion-free setting is also consistent with previous continual-personalization methods such as CNS.
Response: We thank the reviewer for this valuable suggestion. We analyzed mask overlap in the eight-task personalization setting. We used Jaccard similarity to measure mask overlap among all 28 concept pairs formed by the eight concepts. We provide the heatmap in the revised manuscript in Appendix A.9 and four representative concept pairs are shown below.
The mean Jaccard similarity across all 28 concept pairs is 26.35%, which is lower than the 37.04% parameter coverage of CNS calculated using the same procedure. This result indicates that different concepts rely on distinct but partially shared subsets of sensitive parameters. Our method can therefore identify and protect task-relevant parameters to mitigate catastrophic forgetting during continual learning.
We also analyzed the gradient cosine similarity of overlapping parameters across different concept pairs. The mean gradient cosine similarity over the intersecting parameters is 0.2774, while an average of 40.27% of the shared parameters have nonzero gradients with opposite signs for the two concepts. These results support our hypothesis that partial gradient conflicts exist between different concepts.
Although some shared parameters have positive cosine similarity, aligned updates after removing the regularization term do not necessarily preserve previously learned concepts. Near an optimum for an earlier task, the gradient of that task may be small, while displacement along a sensitive curvature direction can still increase its loss. Our ablation study further shows that removing the regularization term substantially reduces subject fidelity and causes catastrophic forgetting, confirming the effectiveness of the proposed regularization.
We additionally report results for a 15-task sequence in Appendix A.5. Although overlap increases during long-term continual learning, our method still substantially reduces catastrophic forgetting and achieves the best overall performance. Its performance does not noticeably degrade across the 15 tasks.
We further analyzed the coverage of the accumulated historical mask. The results are shown below.
Across the evaluated sequence of eight concepts, the accumulated historical mask does not saturate and does not degenerate into an all-encompassing protection region.