acceptodds
Under review as a conference paper at ICLR 2027

Anchor to Expand: Semantic Anchoring for Personalized Text-to-Image Diffusion Models

Abstract

Personalizing text-to-image diffusion models extends pretrained models to represent novel user-specific concepts from only a few reference images. However, learning a new concept while building on the prior knowledge of the pretrained model remains a key challenge. When personalization focuses on learning the target concept, the model tends to overfit the reference examples and degrade its general capability. In contrast, emphasizing prior preservation can hinder capturing distinctive personalized attributes. In this paper, we address this challenge by viewing a personalized concept as an underrepresented concept whose semantic counterpart is well represented in the pretrained model. Rather than treating the learning of a new concept and prior preservation as separate objectives, we reformulate them as a single anchored learning problem. We therefore introduce Semantic Anchoring Personalization (SAP), which keeps concept learning grounded in the pretrained semantic structure while capturing subject-specific attributes. The proposed objective offers a simple yet effective formulation that can be applied across different model backbones without architectural modifications or auxiliary networks. Extensive experiments across various settings demonstrate that SAP achieves a better balance between subject fidelity and text-image alignment than baseline methods. Further ablation studies validate the contribution of semantic anchoring to personalization.

Primary area:
generative models
Keywords:
Text to Image Generation, Diffusion Model, Personalization, Semantic Anchoring

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Dqd3OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method of resolving the conflict between personalization and generalization in T2I diffusion models. Personalized image generation model suffer from overfitting, where unique concepts overwrite models' general prior knowledge. And exsiting method overcome this issue by distinct prior preserving mechanism. On the other hands, this paper addresses this problem as a unified problem by redefining personal concepts as "Rare Concepts" and pretrained semantics as "Frequent Concepts". The proposed method, SAP, prevents the meaning vector of the newly learned rare concept from drifting too far outside the model's knowledge space (anchor). This method successfully controls a balance between a subject's unique details and semantic consistency. The experimental results show that SAP consistently outperforms existing personalized techniques across various diffusion models.

Critical strengths
  • Framing the personalization problem as a "Rare Concept Learning" problem is highly reasonable. This approach offers a new perspective and is supported by a clear mathematical foundation.
  • Despite its mathematically rigorous foundation, the proposed method is simple and effective. It enables simultaneous learning personal concept while preserving prior knowledge without requiring separate training stages or any additional procedures.
  • The paper is well-written and clearly organized, making it easy to follow.
Critical weaknesses
  • W1. The experimental comparisons with recent works, particularly those published in 2026, are somewhat limited.
  • W2. The personalization results appear to exhibit limited diversity, with generated subjects often showing similar poses and viewing angles (e.g., in Fig 3, rows 1–3 appear similar to ref 2, and row 5 is similar to ref 4)
  • W3. The experiments only include object-level personalization. It is unclear whether the proposed method is limited to object-based subjects or can also handle more abstract concepts, such as artistic styles or poses.
  • (Suggestion) It would be helpful to include the results of other methods listed in Table 1 in Figure 7 for a more comprehensive comparison.
Questions for authors
  • Q1. Does the proposed method use only [V*] instead of [V*] dog?
  • Q2. How many iterations are required for training, and what is the training time?
  • Q3. Is there any analysis for the reason of the prompt mixing in row 2 of Fig 3?
4 comments
AuthorsOpenReview ↗

Rebuttal [1/2]

We appreciate Reviewer Dqd3’s careful review and recognition of the connection between rare concept learning and personalization.

The reviewer’s main concerns are:

  • Comparisons with recent personalization methods.
  • Further analysis and discussion of the proposed method.

We address these concerns and respond to the reviewer’s suggestion and questions below.


1. Comparisons with Recent Methods (W1)

We expanded the comparisons to include PNP [1], ClassDiffusion [2], and AttnDreamBooth [7], alongside SuDe [3] and Steering Guidance [4]. The evaluation uses CLIP-I and DINO for subject fidelity and CLIP-T for text alignment, consistent with the manuscript.

SAP shares the top overall rank with AttnDreamBooth across the three metrics. SAP obtains a higher CLIP-T score than AttnDreamBooth, although its CLIP-I and DINO scores are lower. Compared with ClassDiffusion and PNP, SAP obtains higher CLIP-I and DINO scores but lower CLIP-T scores. SAP nevertheless ranks above both methods when the three metrics are considered together. These comparisons support the effectiveness of semantic anchoring in achieving a competitive balance between subject fidelity and text alignment.

MethodCLIP-I ↑CLIP-T ↑DINO ↑Rank ↓
SAP (Ours)0.78330.30640.60511
AttnDreamBooth (NeurIPS 2024)0.81730.29820.67831
ClassDiffusion (ICLR 2025)0.74200.32070.53153
PNP (ICLR 2026)0.77360.31070.57644
SuDe (CVPR 2024)0.78000.30470.59534
DreamBooth-LoRA (CVPR 2023)0.73140.31950.48667
Steering Guidance (ICCV 2025)0.73810.31330.50646

Rank orders methods by their average rank across CLIP-I, CLIP-T, and DINO.


2. Generation Diversity and Personalization beyond Object Subjects (W2, W3)

We acknowledge that some examples in Figure 3 resemble the reference images in pose and viewpoint. Following the evaluation protocol, we use a fixed set of hyperparameters across subjects for each method without subject-specific tuning. This shared configuration may limit flexibility for some subjects as the settings that balance subject fidelity and text alignment can differ across subjects. The second row of Figure 10 and the examples in Figure 11 show variations in pose and viewpoint beyond the reference images. Subject-specific tuning may improve this flexibility. The top overall ranks in Tables 1 and 2, together with the highest overall preference in the user study (Appendix I, Table 6), support the balance between subject fidelity and text alignment achieved by SAP.

Our evaluation follows the object-level personalization setting used in prior work, allowing comparisons with established baselines. We agree that extending personalization to artistic styles and poses is an interesting direction for future work. The current formulation uses a general class as the semantic anchor for an individual subject. Applying this objective to abstract concepts requires defining an appropriate semantic anchor and evaluating how it guides learning of the target concept. We added this discussion and future direction to Appendix A of the revised manuscript.


3. Visualization of Quantitative Results (S1)

Following the reviewer’s suggestion, we visualize the baseline results alongside the semantic anchoring weight ablation presented in Figure 7 of the manuscript. The SAP curve lies toward the upper-right region of both plots, indicating a competitive trade-off between subject fidelity and text alignment. Most baselines lie near the curve or below and to its left. Although some baselines obtain higher scores on an individual metric, these gains are accompanied by lower scores on the other metric. This comparison illustrates how semantic anchoring adjusts the balance between subject fidelity and text alignment while maintaining competitive performance. We added this figure to Appendix J of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal [2/2]


4. Clarifications on Conditioning, Training, and Prompt Mixing (Q1–Q3)

Q1. We use “[V*] [class]” such as “[V*] dog” (lines 296–298).

Q2. We train SAP for 500 steps on SD1.5 and SDXL and 700 steps on SD3. Training on SD1.5 takes an average of 323 seconds per subject on an NVIDIA RTX 3090 GPU. We added these training details to Appendix F of the revised manuscript.

Q3. The prompt mixing in the second row of Figure 3 is consistent with attribute-binding errors reported in text-to-image diffusion models [5, 6], where attributes are associated with unintended entities. This issue is not specific to personalization. While the proposed objective uses pretrained class semantics to guide subject learning, ensuring correct attribute binding involves an additional challenge in compositional generation. We expanded this discussion in Section 4.2 of the revised manuscript.


References

[1] Kim et al. Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift. ICLR 2026.

[2] Huang et al. ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance. ICLR 2025.

[3] Qiao et al. FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation. CVPR 2024.

[4] Park et al. Steering Guidance for Personalized Text-to-Image Diffusion Models. ICCV 2025.

[5] Feng et al. Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. ICLR 2023.

[6] Rassin et al. Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment. NeurIPS 2023.

[7] Pang et al. AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation. NeurIPS 2024.

Thank you for the response. The authors' responses have addressed most of my concerns.However, I have two remaining comments.

  • The newly added baseline methods appear to be missing from Figure 9. Including them would make the comparison more comprehensive.
  • Following Reviewer M3zh's concern regarding novelty, I would appreciate further clarification of how proposed method differs from ClassDiffusion. While the authors have explained the methodological differences, it remains unclear why the proposed approach is fundamentally more advantageous beyond these differences in implementation. A clearer explanation of the underlying benefits of SAP would help strengthen its contribution.

We appreciate Reviewer Dqd3 pointing out the remaining issues and are glad that our responses have addressed most of the concerns. We address these points below.

1. Updated Figure 9

In the revised manuscript, we updated Figure 9 to include the newly added baselines. We also expanded the discussion in Appendix J to explain the observed trade-offs and clarify the baseline comparisons presented in Section 4.2. This suggestion helped us provide a more comprehensive evaluation and a clearer presentation of SAP.

2. Comparison with ClassDiffusion

Regarding the comparison with ClassDiffusion, SAP frames personalization as learning the denoising response of an underrepresented subject relative to that of its well-represented semantic counterpart. Motivated by Theorem 1, we define a denoising target in which the class-conditioned prediction of the frozen pretrained model and the subject reconstruction signal jointly guide personalization. Proposition 1 shows that fitting this target yields the same optimal parameters as minimizing the reconstruction and anchoring objective in Eq. (6).

In contrast, ClassDiffusion uses a semantic preservation loss and a reconstruction loss to align the text representations of personalized and class prompts and to train the denoiser, respectively. The semantic preservation loss constrains text representations but does not directly anchor the personalized denoising response to the pretrained class prediction. In SAP, pretrained class knowledge serves as a reference for learning subject-specific denoising behavior through the shared target. Our method thus offers a distinct perspective by interpreting personalization as extending pretrained class knowledge to capture subject-specific attributes, rather than only balancing prior preservation and subject learning.

Reviewer_M3zhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper solves the problem of personalization while preserving the model’s language alignment. Beyond the DreamBooth fine-tuning idea, the paper proposes a new loss that keeps the fine-tuned model’s prediction for a new concept close to the frozen pretrained model’s prediction for the same class of concept, given the same noisy latent.

Critical strengths

Clear structure and easy to follow; figures support the hypothesis well.

The hypothesis is supported by mathematical proof.

The problem has a clear motivation.

The ablation study is reasonable, along with extensive experiments.

Critical weaknesses

Limited novelty: The core idea is adding a new L2L_2 loss term to pull the generated concept toward the class prediction of the frozen model. This concept is not entirely new, as baseline methods like ClassDiffusion and DCO also try to constrain and lock the class of the generated concept.

The paper claims its contribution is personalization while preserving textual alignment, but provides no comparison of the base models before and after fine-tuning. There is also no evaluation comparing textual alignment, image quality, or output diversity.

Missing baseline comparisons: ClassDiffusion (ICLR 2025), AttnDreamBooth (NeurIPS 2024), DisenBooth (ICLR 2024), etc.Training details are missing: LoRA rank, steps, etc.

Questions for authors

Could you provide the detailed training setup?

Could you clarify the key contributions—what makes the proposed loss better/cleverer than other baselines?

How does the method compare with recent baselines, and does it outperform them?

4 comments
AuthorsOpenReview ↗

Rebuttal [1/2]

We appreciate Reviewer M3zh’s constructive feedback on the formulation and evaluation of SAP.

The reviewer’s main concerns are:

  • How SAP differs from existing personalization objectives.
  • Whether the experiments support its claimed benefits.

We explain these distinctions, clarify the scope of our evaluation, and present further baseline comparisons and training details below.


1. Key Contributions and Differences from Existing Methods (W and Q2)

Our contribution lies in reformulating subject learning through the relation between underrepresented and well-represented concepts. Proposition 1 shows that training with an interpolated target yields the same optimal parameters as an objective combining subject reconstruction with regularization toward the pretrained class prediction. This derivation makes explicit how pretrained class knowledge contributes to the target for learning subject-specific attributes. The anchor ablation in Table 3 supports this design choice by showing a better balance between subject fidelity and text alignment with the corresponding class than with the alternative anchors considered.

We acknowledge that prior work has used class semantics or frozen pretrained models for personalization. These methods differ in how they use pretrained knowledge to guide subject learning. ClassDiffusion [1] aligns the text representations of personalized and class prompts. DCO [2] compares the denoising errors of the fine-tuned and pretrained models under the same conditioning. SAP anchors the subject-conditioned prediction of the personalized model to the class-conditioned prediction of the frozen model on the same noisy latent, incorporating the relation between the subject and its general class into the learning objective. Its focus is on learning subject-specific attributes while retaining the ability to follow text prompts, rather than minimizing parameter changes to preserve the pretrained distribution [5]. We clarified these distinctions and expanded the discussion in Section 2 of the revised manuscript to include AttnDreamBooth [3] and DisenBooth [4] and explain their differences from SAP.

Regarding the evaluation of text alignment, subject fidelity, and the balance between them, Tables 1 and 2 demonstrate the competitive performance of SAP against baseline methods and across diffusion backbones. The user study in Appendix I (Table 6) further evaluates identity preservation, prompt alignment, visual quality, and overall preference. The second row of Figure 10 and the results in Figure 11 also show variations in pose and viewpoint.


2. Further Baseline Comparisons and Training Details (Q1 and Q3)

We expanded the baseline comparisons to include ClassDiffusion [1], AttnDreamBooth [3], and PNP [5], alongside Steering Guidance [6] and SuDe [7]. We evaluate subject fidelity using CLIP-I and DINO and text alignment using CLIP-T, following the evaluation protocol described in the manuscript.

The additional comparisons show that SAP shares the top overall rank with AttnDreamBooth across CLIP-I, CLIP-T, and DINO. SAP achieves higher text alignment than AttnDreamBooth and higher subject fidelity than ClassDiffusion and PNP. Although ClassDiffusion and PNP obtain higher CLIP-T scores, SAP ranks above both methods in the three-metric evaluation. We note that SAP uses a single training stage and requires 323 seconds per subject, compared with approximately 6,006 seconds for the three-stage training procedure of AttnDreamBooth in our experiments. These results support the effectiveness of the proposed objective in achieving competitive personalization performance with efficient training.

MethodCLIP-I ↑CLIP-T ↑DINO ↑Rank ↓
SAP (Ours)0.78330.30640.60511
AttnDreamBooth (NeurIPS 2024)0.81730.29820.67831
ClassDiffusion (ICLR 2025)0.74200.32070.53153
PNP (ICLR 2026)0.77360.31070.57644
SuDe (CVPR 2024)0.78000.30470.59534
DreamBooth-LoRA (CVPR 2023)0.73140.31950.48667
Steering Guidance (ICCV 2025)0.73810.31330.50646

Rank orders methods by their average rank across CLIP-I, CLIP-T, and DINO.

We specified in Appendix F of the revised manuscript that SAP is trained for 500 steps on SD1.5 and SDXL and 700 steps on SD3, with LoRA rank 4 for all three backbones following the default configurations of the corresponding official training scripts. Appendix F also provides the implementation links and other training settings.

AuthorsOpenReview ↗

Rebuttal [2/2]


References

[1] Huang et al. ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance. ICLR 2025.

[2] Lee et al. Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models. NeurIPS 2024.

[3] Pang et al. AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation. NeurIPS 2024.

[4] Chen et al. DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation. ICLR 2024.

[5] Kim et al. Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift. ICLR 2026.

[6] Park et al. Steering Guidance for Personalized Text-to-Image Diffusion Models. ICCV 2025.

[7] Qiao et al. FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation. CVPR 2024.

Thank you for the response. The authors' responses have addressed most of my concerns, including the additional baselines and the training details. However, I have two remaining comments.

  • In the new comparison table, SAP obtains lower CLIP-T than DreamBooth-LoRA, ClassDiffusion, and PNP, while the motivation of the paper is to preserve text alignment during personalization. I would appreciate it if the authors could explain how these results should be interpreted, and in what sense SAP achieves a better balance than these methods.
  • Regarding novelty, the authors have explained where each method applies the constraint (text representations in ClassDiffusion, denoising errors in DCO, and the denoising prediction in SAP). However, it remains unclear to me why anchoring the denoising prediction is more beneficial than these alternatives. A short intuitive explanation of this benefit would help strengthen the contribution.

We appreciate Reviewer M3zh’s follow-up feedback and are glad that our responses have addressed most of the concerns. We address the remaining points below.

1. Text Alignment and Subject Fidelity

The proposed objective uses pretrained knowledge of a well-represented class as an anchor for learning an underrepresented subject. This formulation allows the relative emphasis on subject fidelity and text alignment to be adjusted through the anchoring weight. Our claim of a better balance refers to the joint evaluation of subject fidelity and text alignment. We agree that SAP obtains lower CLIP-T scores than DreamBooth-LoRA, ClassDiffusion, and PNP in the baseline comparison. SAP obtains higher CLIP-I and DINO scores than these methods and shares the top position in the overall ranking. Figure 7 illustrates how the anchoring weight controls the use of pretrained class knowledge during subject learning. Figure 9 compares the resulting trade-offs with the baselines and illustrates the competitive balance achieved by SAP.

2. Intuition behind Semantic Anchoring

SAP views personalization as learning an underrepresented subject in relation to its well-represented semantic counterpart. A few reference images provide limited evidence for learning how the subject should respond across noise levels and textual contexts. The benefit of semantic anchoring is to use pretrained class knowledge as a reference for this learning process.

ClassDiffusion uses a semantic preservation loss and a reconstruction loss to align text representations and train the denoiser, respectively. Class semantics constrain the text representations, while the reconstruction target provides supervision for the denoising response. DCO uses a frozen pretrained model to weight reconstruction updates according to relative denoising errors under the same conditioning.

SAP incorporates the relation between the subject and its general class into the denoising target itself. The pretrained class prediction and the subject reconstruction signal jointly define this target. The class prediction provides a semantic reference for the denoising response to learn, while the reconstruction signal allows adaptation to subject-specific attributes. Theorem 1 motivates this use of a well-represented counterpart, and Proposition 1 connects the resulting target to the reconstruction and anchoring objective. This formulation makes the role of pretrained class knowledge explicit and allows us to interpret personalization as extending that knowledge to capture a new subject.

We will incorporate these clarifications into the revised manuscript. The reviewer’s feedback helped us clarify the motivation and benefits of the proposed method.

Reviewer_eXB3OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Semantic Anchoring Personalization (SAP), a method designed to balance subject fidelity and text alignment in few-shot personalized text-to-image generation. The authors view a personalized subject as an underrepresented concept that can benefit from the semantic knowledge of its corresponding general class in a pretrained diffusion model. Based on this perspective, SAP introduces a semantic anchoring objective that regularizes the personalized model's noise predictions using the class-conditioned predictions of a frozen pretrained model

Critical strengths
  1. Few-shot personalized image generation faces a fundamental trade-off between preserving subject identity and maintaining text-guided editing capabilities. The paper addresses a practically relevant problem.
  2. The proposed method is simple and has an interpretable training objective. It can be integrated into existing personalization frameworks without architectural modifications.
  3. The experiments cover not only SD1.5 but also SDXL and SD3, demonstrating the effectiveness of the proposed method across multiple diffusion backbones.
Critical weaknesses
  1. Missing comparisons with important related methods. Preserve and Personalize (PNP) [1] and ClassDiffusion [2] address closely related problems. Although ClassDiffusion is discussed in the paper, neither method is included in the experimental comparisons. Furthermore, several follow-up methods to DreamBooth, such as AttnDreamBooth [3] and DreamBooth++ [4], have demonstrated substantial improvements in subject fidelity and text alignment over the original DreamBooth.
  2. Limited relevance to recent generative models. The experiments are conducted on SD1.5, SDXL, and SD3. However, recent image generation models, such as FLUX.2 and Qwen-Image-Edit, have developed strong reference-conditioned generation capabilities that can preserve subject identity without subject-specific fine-tuning. Although fine-tuning-based personalization remains a meaningful research direction, the paper does not sufficiently demonstrate the advantages of SAP in the context of modern generative models.
  3. CLIP-T has limitations in evaluating fine-grained image attributes and complex spatial relationships. ClassDiffusion [2] specifically discusses the limitations of CLIP-T for personalized generation and introduces BLIP2-T as an alternative evaluation metric. The authors should consider including additional metrics, such as BLIP2-T, to provide a more comprehensive evaluation of text alignment and compositional generation capabilities.

[1] Kim et al. Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift. ICLR 2026.

[2] Huang et al. ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance. ICLR 2025.

[3] Pang et al. AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation. NeurIPS 2024.

[4] Fan et al. DreamBooth++: Boosting Subject-Driven Generation via Region-Level References Packing. ACM MM 2024.

Questions for authors
  1. Appendix F states that the denominator (σt)(\sigma_t) was removed from the SuDe objective due to training instability. However, SuDe has an official open-source implementation. Why did the authors choose to reimplement SuDe rather than use its official implementation?

SuDe: https://github.com/modelscope/facechain/blob/main/more_apps/Facechain-SuDe/README.md?utm_source=chatgpt.com

6 comments
AuthorsOpenReview ↗

Rebuttal [1/2]

We appreciate Reviewer eXB3’s careful review and constructive suggestions, as well as the recognition of the importance of balancing subject fidelity and text alignment, the simplicity of our objective, and its applicability across multiple backbones.

The reviewer’s main concerns are:

  • The research motivation and broader applicability of fine-tuning-based personalization (W2).
  • The need for further baseline comparisons and evaluation with fine-grained metrics (W1, W3).

We address each concern below.


1. Research Motivation and Broader Applicability (W2)

We agree that recent reference-conditioned models provide strong capabilities for subject-preserving generation. However, their success does not imply that personalization is resolved across models and domains. For example, MMPB [1] and PALM-Bench [2] report limitations in how vision-language and audio-language models handle user-specific concepts and personal context. PersonaBooth [3] studies the adaptation of pretrained diffusion models for personalized motion generation. These studies motivate research on how pretrained knowledge can support learning user-specific concepts, including in settings where specialized reference-conditioning mechanisms are unavailable.

In this context, we study personalization using a pretrained text-to-image model and 4~6 subject images, without requiring a pretrained reference-conditioning module. SAP formulates semantic anchoring as a training objective that does not rely on backbone-specific architectural components or domain-specific conditioning mechanisms. We evaluate SAP in text-to-image personalization, where established benchmarks and baselines provide a basis for assessing subject fidelity and text alignment. Our comparisons with existing baselines on SD1.5, SDXL, and SD3 support its effectiveness across different generative architectures. Extending and evaluating this objective in other domains remains a direction for future work. We added this discussion to Appendix K, “Further Discussion on Personalized Diffusion Models,” in the revised manuscript.


2. Further Baseline Comparisons and Evaluation with Fine-Grained Metrics (W1, W3)

Following the reviewer’s suggestions, we compare SAP with PNP [4], ClassDiffusion [5], and AttnDreamBooth [6], alongside SuDe and Steering Guidance. We evaluate subject fidelity using CLIP-I and DINO and complement CLIP-T with BLIP2-T for a more fine-grained evaluation of text alignment.

As shown in the table below, SAP shares the top overall rank with AttnDreamBooth across the original three metrics. The shared rank reflects different trade-offs. SAP achieves higher scores on both text-alignment metrics, while AttnDreamBooth obtains higher subject-fidelity scores. We note that SAP requires 323 seconds of training per subject in a single stage, compared with approximately 6,006 seconds for the three-stage training procedure of AttnDreamBooth in our experiments. Compared with ClassDiffusion and PNP, SAP achieves higher scores on both subject-fidelity metrics but lower text-alignment scores. SAP ranks above ClassDiffusion and PNP under the original three-metric evaluation. When BLIP2-T is included, SAP ranks second behind ClassDiffusion and ahead of AttnDreamBooth and PNP. These comparisons show that SAP offers a competitive balance between subject fidelity and text alignment through a simple objective and efficient training.

MethodCLIP-I ↑CLIP-T ↑DINO ↑BLIP2-T ↑Rank (3) ↓Rank (4) ↓
SAP (Ours)0.78330.30640.60510.484512
AttnDreamBooth (NeurIPS 2024)0.81730.29820.67830.471313
ClassDiffusion (ICLR 2025)0.74200.32070.53150.503331
PNP (ICLR 2026)0.77360.31070.57640.492343
SuDe (CVPR 2024)0.78000.30470.59530.475346
DreamBooth-LoRA (CVPR 2023)0.73140.31950.48660.505175
Steering Guidance (ICCV 2025)0.73810.31330.50640.497166

Rank (3) ranks methods by their average rank across CLIP-I, CLIP-T, and DINO. Rank (4) includes BLIP2-T as the fourth metric.

We also expanded the related work discussion in Section 2 of the revised manuscript to include DreamBooth++ [7].

AuthorsOpenReview ↗

Rebuttal [2/2]


3. SuDe Implementation (Q1)

We were aware of SuDe’s official open-source implementation when designing our experiments. It uses the CompVis/LDM training codebase, whereas our experiments use a Diffusers-based SD1.5 pipeline. To compare the methods under a common experimental setup, we reimplemented SuDe in this pipeline. Training with the posterior-variance denominator was unstable in our implementation. Removing it stabilized training but changed the original objective. The reported SuDe results therefore correspond to this modified implementation. For clarification, we explicitly noted in Section 4.1 and Appendix F of the revised manuscript that the reported SuDe results use this modified implementation.


Reference

[1] Kim et al. MMPB: It’s Time for Multi-Modal Personalization. NeurIPS 2025.

[2] Wang et al. PALM-Bench: A Comprehensive Benchmark for Personalized Audio-Language Models. arXiv:2601.03531, 2026.

[3] Kim et al. PersonaBooth: Personalized Text-to-Motion Generation. CVPR 2025.

[4] Kim et al. Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift. ICLR 2026.

[5] Huang et al. ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance. ICLR 2025.

[6] Pang et al. AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation. NeurIPS 2024.

[7] Fan et al. DreamBooth++: Boosting Subject-Driven Generation via Region-Level References Packing. ACM MM 2024.

Thank you for your response. I acknowledge that the additional comparisons and evaluation metrics have substantially strengthened the paper.

My remaining concerns are primarily as follows:

  1. SAP appears to achieve a competitive trade-off between subject fidelity and text alignment, but other methods may also achieve similar trade-offs through hyperparameter tuning. The current results do not sufficiently convince me that SAP offers a clear advantage over existing approaches.

  2. Given that existing methods have already investigated preserving pretrained knowledge during personalization, I would like to better understand what new insights SAP provides beyond introducing another regularization objective. Alternatively, is its main contribution simply a more efficient personalization approach?

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We appreciate all reviewers’ careful evaluations and constructive suggestions. The reviewers highlighted the following strengths:

  • The clear motivation (M3zh) and the perspective connecting rare concept learning with personalization (Dqd3).
  • The simple and interpretable objective (eXB3) and the absence of separate training stages or additional procedures (Dqd3).
  • The mathematical foundation supporting the formulation (M3zh and Dqd3).
  • The evaluation across multiple backbones (eXB3) and the extensive experiments and ablation studies (M3zh).

We conducted additional experiments and expanded the discussions to address the reviewers’ concerns. Detailed responses are provided in the individual threads below. The corresponding changes are included in the revised manuscript and highlighted in blue.

We would be happy to discuss any remaining concerns. Please let us know if any points require further clarification.

Dear Reviewers,

The authors of Submission 31990 have responded to all reviews. Could you please read the rebuttal and reply before the discussion period ends on November 18?

Thank you all for your time,

Area Chair

AuthorsOpenReview ↗

Summary of the Discussion

We thank the reviewers and the Area Chair for their time and feedback. All three reviewers took part in the follow-up discussion. Reviewer eXB3 wrote that the additional comparisons and evaluation metrics "have substantially strengthened the paper" and subsequently raised their rating. Reviewers M3zh and Dqd3 both indicated that our responses had addressed most of their concerns.

All three reviewers asked for comparisons with recent personalization methods. We added ClassDiffusion, PNP, and AttnDreamBooth to Table 1 and reported an additional evaluation using BLIP2-T in our rebuttal. SAP achieves higher subject fidelity than ClassDiffusion and PNP, while both methods score higher on text alignment. Against AttnDreamBooth, SAP achieves higher text alignment, while AttnDreamBooth achieves higher subject fidelity. The BLIP2-T evaluation shows the same pattern as CLIP-T in these comparisons. At the suggestion of Reviewer Dqd3, we added these baselines to Figure 9 to illustrate the competitive balance SAP achieves between subject fidelity and text alignment.

The follow-up discussion helped us clarify the central contribution of SAP. We formulate personalization as learning an underrepresented subject in relation to its well-represented semantic counterpart. In SAP, the class-conditioned prediction of the frozen pretrained model and the subject reconstruction signal jointly define the denoising target. Proposition 1 connects this target to the reconstruction and anchoring objective. This formulation allows us to interpret personalization as extending pretrained class knowledge to capture subject-specific attributes.

We appreciate the reviewers’ questions and suggestions, which helped us strengthen the evaluation and clarify how learning a new concept and preserving pretrained knowledge can be formulated as a single learning problem.

Best regards,
Authors of Submission 31990

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.