- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a method of resolving the conflict between personalization and generalization in T2I diffusion models. Personalized image generation model suffer from overfitting, where unique concepts overwrite models' general prior knowledge. And exsiting method overcome this issue by distinct prior preserving mechanism. On the other hands, this paper addresses this problem as a unified problem by redefining personal concepts as "Rare Concepts" and pretrained semantics as "Frequent Concepts". The proposed method, SAP, prevents the meaning vector of the newly learned rare concept from drifting too far outside the model's knowledge space (anchor). This method successfully controls a balance between a subject's unique details and semantic consistency. The experimental results show that SAP consistently outperforms existing personalized techniques across various diffusion models.
Critical strengths
- Framing the personalization problem as a "Rare Concept Learning" problem is highly reasonable. This approach offers a new perspective and is supported by a clear mathematical foundation.
- Despite its mathematically rigorous foundation, the proposed method is simple and effective. It enables simultaneous learning personal concept while preserving prior knowledge without requiring separate training stages or any additional procedures.
- The paper is well-written and clearly organized, making it easy to follow.
Critical weaknesses
- W1. The experimental comparisons with recent works, particularly those published in 2026, are somewhat limited.
- W2. The personalization results appear to exhibit limited diversity, with generated subjects often showing similar poses and viewing angles (e.g., in Fig 3, rows 1–3 appear similar to ref 2, and row 5 is similar to ref 4)
- W3. The experiments only include object-level personalization. It is unclear whether the proposed method is limited to object-based subjects or can also handle more abstract concepts, such as artistic styles or poses.
- (Suggestion) It would be helpful to include the results of other methods listed in Table 1 in Figure 7 for a more comprehensive comparison.
Questions for authors
- Q1. Does the proposed method use only [V*] instead of [V*] dog?
- Q2. How many iterations are required for training, and what is the training time?
- Q3. Is there any analysis for the reason of the prompt mixing in row 2 of Fig 3?
4 comments
Rebuttal [2/2]
4. Clarifications on Conditioning, Training, and Prompt Mixing (Q1–Q3)
Q1. We use “[V*] [class]” such as “[V*] dog” (lines 296–298).
Q2. We train SAP for 500 steps on SD1.5 and SDXL and 700 steps on SD3. Training on SD1.5 takes an average of 323 seconds per subject on an NVIDIA RTX 3090 GPU. We added these training details to Appendix F of the revised manuscript.
Q3. The prompt mixing in the second row of Figure 3 is consistent with attribute-binding errors reported in text-to-image diffusion models [5, 6], where attributes are associated with unintended entities. This issue is not specific to personalization. While the proposed objective uses pretrained class semantics to guide subject learning, ensuring correct attribute binding involves an additional challenge in compositional generation. We expanded this discussion in Section 4.2 of the revised manuscript.
References
[1] Kim et al. Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift. ICLR 2026.
[2] Huang et al. ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance. ICLR 2025.
[3] Qiao et al. FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation. CVPR 2024.
[4] Park et al. Steering Guidance for Personalized Text-to-Image Diffusion Models. ICCV 2025.
[5] Feng et al. Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. ICLR 2023.
[6] Rassin et al. Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment. NeurIPS 2023.
[7] Pang et al. AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation. NeurIPS 2024.
Thank you for the response. The authors' responses have addressed most of my concerns.However, I have two remaining comments.
- The newly added baseline methods appear to be missing from Figure 9. Including them would make the comparison more comprehensive.
- Following Reviewer M3zh's concern regarding novelty, I would appreciate further clarification of how proposed method differs from ClassDiffusion. While the authors have explained the methodological differences, it remains unclear why the proposed approach is fundamentally more advantageous beyond these differences in implementation. A clearer explanation of the underlying benefits of SAP would help strengthen its contribution.
We appreciate Reviewer Dqd3 pointing out the remaining issues and are glad that our responses have addressed most of the concerns. We address these points below.
1. Updated Figure 9
In the revised manuscript, we updated Figure 9 to include the newly added baselines. We also expanded the discussion in Appendix J to explain the observed trade-offs and clarify the baseline comparisons presented in Section 4.2. This suggestion helped us provide a more comprehensive evaluation and a clearer presentation of SAP.
2. Comparison with ClassDiffusion
Regarding the comparison with ClassDiffusion, SAP frames personalization as learning the denoising response of an underrepresented subject relative to that of its well-represented semantic counterpart. Motivated by Theorem 1, we define a denoising target in which the class-conditioned prediction of the frozen pretrained model and the subject reconstruction signal jointly guide personalization. Proposition 1 shows that fitting this target yields the same optimal parameters as minimizing the reconstruction and anchoring objective in Eq. (6).
In contrast, ClassDiffusion uses a semantic preservation loss and a reconstruction loss to align the text representations of personalized and class prompts and to train the denoiser, respectively. The semantic preservation loss constrains text representations but does not directly anchor the personalized denoising response to the pretrained class prediction. In SAP, pretrained class knowledge serves as a reference for learning subject-specific denoising behavior through the shared target. Our method thus offers a distinct perspective by interpreting personalization as extending pretrained class knowledge to capture subject-specific attributes, rather than only balancing prior preservation and subject learning.
Rebuttal [1/2]
We appreciate Reviewer Dqd3’s careful review and recognition of the connection between rare concept learning and personalization.
The reviewer’s main concerns are:
We address these concerns and respond to the reviewer’s suggestion and questions below.
1. Comparisons with Recent Methods (W1)
We expanded the comparisons to include PNP [1], ClassDiffusion [2], and AttnDreamBooth [7], alongside SuDe [3] and Steering Guidance [4]. The evaluation uses CLIP-I and DINO for subject fidelity and CLIP-T for text alignment, consistent with the manuscript.
SAP shares the top overall rank with AttnDreamBooth across the three metrics. SAP obtains a higher CLIP-T score than AttnDreamBooth, although its CLIP-I and DINO scores are lower. Compared with ClassDiffusion and PNP, SAP obtains higher CLIP-I and DINO scores but lower CLIP-T scores. SAP nevertheless ranks above both methods when the three metrics are considered together. These comparisons support the effectiveness of semantic anchoring in achieving a competitive balance between subject fidelity and text alignment.
Rank orders methods by their average rank across CLIP-I, CLIP-T, and DINO.
2. Generation Diversity and Personalization beyond Object Subjects (W2, W3)
We acknowledge that some examples in Figure 3 resemble the reference images in pose and viewpoint. Following the evaluation protocol, we use a fixed set of hyperparameters across subjects for each method without subject-specific tuning. This shared configuration may limit flexibility for some subjects as the settings that balance subject fidelity and text alignment can differ across subjects. The second row of Figure 10 and the examples in Figure 11 show variations in pose and viewpoint beyond the reference images. Subject-specific tuning may improve this flexibility. The top overall ranks in Tables 1 and 2, together with the highest overall preference in the user study (Appendix I, Table 6), support the balance between subject fidelity and text alignment achieved by SAP.
Our evaluation follows the object-level personalization setting used in prior work, allowing comparisons with established baselines. We agree that extending personalization to artistic styles and poses is an interesting direction for future work. The current formulation uses a general class as the semantic anchor for an individual subject. Applying this objective to abstract concepts requires defining an appropriate semantic anchor and evaluating how it guides learning of the target concept. We added this discussion and future direction to Appendix A of the revised manuscript.
3. Visualization of Quantitative Results (S1)
Following the reviewer’s suggestion, we visualize the baseline results alongside the semantic anchoring weight ablation presented in Figure 7 of the manuscript. The SAP curve lies toward the upper-right region of both plots, indicating a competitive trade-off between subject fidelity and text alignment. Most baselines lie near the curve or below and to its left. Although some baselines obtain higher scores on an individual metric, these gains are accompanied by lower scores on the other metric. This comparison illustrates how semantic anchoring adjusts the balance between subject fidelity and text alignment while maintaining competitive performance. We added this figure to Appendix J of the revised manuscript.