acceptodds
Withdrawn from ICLR 2027

ReGain: Fixing Re-contextualization of Synthetically Generated Subjects

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image diffusion models are personalized to a subject by DreamBooth fine-tuning on a handful of its images. Increasingly, these images come from a diffusion model rather than a camera. We show that fine-tuning on such synthetic images degrades subject fidelity, producing oversaturated color and excess high-frequency detail. To isolate the cause, we fine-tune two models from the same base model with the same DreamBooth recipe, one on real photos of a subject and one on synthetic images of that subject generated by the first. We trace the degradation to classifier-free guidance (CFG). For the model personalized on synthetic images, the angle between the conditional and unconditional noise predictions, and with it the norm of their difference, is much larger than for the model personalized on real photos. This inflation grows toward high frequencies and also appears at other prompts semantically close to the subject, such as its class noun, but not at unrelated ones. We propose ReGain, a training-free correction applied at sampling time that measures how much each frequency band of the guidance is inflated relative to the base model and scales that band down accordingly. ReGain needs no real photos. On Stable Diffusion v1.5, ReGain closes 51–64% of the subject-fidelity gap to the model personalized on real photos, as measured by DINO, DINOv2 and CLIP-I. It also improves subject fidelity on SDXL and SD 3.5 and preserves text alignment on all three backbones.

Primary area:
generative models
Keywords:
diffusion models, personalization, classifier-free guidance, synthetic training data, text-to-image generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_62mcOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper finds that diffusion models personalized on synthetic subject images exhibit inflated classifier-free guidance, especially at high frequencies, alongside reduced subject fidelity. It introduces ReGain, a training-free sampling correction that calibrates guidance by frequency and timestep against the original base model and applies it within a subject mask, improving subject fidelity across SD1.5, SDXL, and SD3.5 while largely preserving text alignment.

Critical strengths
  • Clear presentation. The paper is well written and organized, with a coherent progression from empirical observations to the proposed correction.

  • Practical relevance. The problem is concrete, and the ability to improve already personalized checkpoints without additional training makes the proposed intervention potentially useful.

  • Meaningful experimental evidence. The controlled comparisons provide useful evidence within the evaluated settings. The paper includes quantitative assessments of subject and prompt fidelity, together with qualitative examples illustrating the effects of the correction.

  • Results across multiple architectures. The method consistently improves the reported subject-fidelity metrics across several backbones, including U-Net-based SD1.5 and SDXL and transformer-based SD3.5.

Critical weaknesses

Overall, this paper is close to borderline, which requires more detailed discussions with the literature and more solid experiments. I would change my ratings if the following concerns are addressed.

  1. The ablations do not isolate the contribution of frequency-dependent calibration. In Table 3, lowering global CFG to 3 matches ReGain’s reported DINO score of 0.647, while ReGain achieves higher CLIP-T: 0.290 versus 0.281. Similarly, removing ReGain’s mask produces slightly higher DINO but lower CLIP-T. These results suggest that subject localization plays an important role in preserving text alignment, leaving the additional benefit of frequency-dependent calibration unclear. The key missing comparisons are corrections using the same subject mask with either a single scalar gain or a timestep-dependent gain without frequency decomposition. Table 4 does not resolve this issue: it measures how closely the compressed schedule matches its calibration target, rather than whether the additional frequency/time structure improves final image quality over these simpler masked corrections.
  2. The generality of the findings beyond the evaluated Stable Diffusion models remains unclear. Although the experiments cover both U-Net and transformer architectures, all evaluated backbones belong to the Stable Diffusion family. It remains uncertain whether the observed guidance inflation and the benefits of ReGain persist in newer model families. A focused evaluation on a newer compatible backbone would strengthen this claim. For example, FLUX.2 [klein] Base is explicitly designed for fine-tuning and is released without step or guidance distillation, making it a relevant candidate. The authors should also clarify which properties of the backbone and guidance formulation are required for ReGain to apply.
  3. The differentiation from ReDiFine understates the overlap in the underlying observations. ReDiFine already analyzes the norm of the conditional–unconditional score difference, reports its growth across synthetic retraining iterations, and connects this growth to high-frequency degradation. ReGain cites this work and identifies meaningful differences, including its focus on a single round of subject personalization and its sampling-time correction. However, the distinction between prior work on the guidance scale and the present work on the guidance term does not fully reflect ReDiFine’s analysis. The paper should acknowledge this closer overlap and explain more precisely what its personalization setting, geometric and frequency-resolved characterization, and base-referenced calibration add beyond those earlier findings.
Questions for authors

In which settings does fine-tuning offer a meaningful advantage over direct reference-conditioned generation? Recent foundation models support generation and editing from multiple reference images; for example, Qwen-Image-2.1 supports up to 10 reference images and is designed to preserve the identities of people and products. Could the authors clarify the circumstances in which DreamBooth or LoRA fine-tuning remains preferable—for example, in terms of fine-grained subject fidelity, consistency across prompts, controllability, or the cost of repeated generation? A clearer discussion, supported by representative evidence where available, would help establish the practical motivation for the studied setting and ReGain’s intended use cases.

1 comment

We thank the reviewer for the constructive review and for noting the clear presentation and practical relevance. We address each weakness below.

W1. Isolating frequency-dependent calibration [also R eD56 W3]

As suggested, we ran the masked correction without frequency decomposition, using the same subject mask as ReGain. The masked baseline performs strongly, confirming the role of the subject mask; however, the frequency-dependent gain of ReGain still improves over it.

Table: Masked corrections with and without frequency decomposition (SD1.5, DreamBooth, w=7.5w=7.5)

MethodCLIP-IDINODINOv2CLIP-T
MsynM_{\text{syn}}, plain CFG0.7880.6110.6110.289
MsynM_{\text{syn}}, mask + single scalar gain (w=5.0w=5.0 inside the mask)0.7960.6290.6270.289
MsynM_{\text{syn}}, ReGain (mask + per-band, per-step gain)0.8020.6470.6430.290

We have added this row to the ablation in Table 3 of the revised manuscript.

W2. Generality beyond the SD family [also R eD56 W4]

We note that our backbones already cover both U-Net (SD1.5, SDXL) and DiT (SD3.5, rectified flow) families, which are among the most widely used in the community: about 3,600 of the text-to-image LoRA adapters uploaded to the Hugging Face Hub in the last six months are built on them (see our reply to Reviewer eD56, W1). Experiments on FLUX.2 [klein] Base over the complete subject set and seeds are in progress, and we will add them to any final version. Since SD3.5 is already a flow-based DiT, we expect the findings to carry over, though the size of the gain varies across backbones (Table 1).

Regarding the required properties: our study is empirical, and ReGain applies wherever this inflation of the guidance term appears: the model must be sampled with CFG from separate conditional and unconditional predictions, and the base checkpoint it was fine-tuned from must be available.

W3. Overlap with ReDiFine

We agree that the overlap in the underlying observation is closer than L111-112 conveys. However, our work differs in four respects.

Setting. ReDiFine studies a six-iteration chain of LoRA fine-tuning on captioned datasets, with the training set regenerated at every iteration (their Sec. 3.1). We study one round of DreamBooth personalization with a subject identifier, and show that the inflation follows the subject's semantic neighborhood (L177-181).

Cause. ReDiFine reports the norm along each model's own sampling trajectory. We decompose it and show that the growth comes from the angle between the conditional and unconditional predictions, not their magnitudes (L157-161), and that it persists when both models see the same inputs (L182-189).

Frequency structure. ReDiFine's frequency analysis is on the final generated images (their Sec. 4.3, Fig. 37). We instead resolve the guidance term itself per frequency band and denoising step (L190-198).

Correction. ReDiFine modifies fine-tuning (condition drop) and decays a scalar CFG scale over steps (their Sec. 5). ReGain measures a per-band, per-step gain against the base model and corrects already-trained checkpoints without retraining (Sec. 3.3).

We regard the two sets of findings as complementary, and will revise Sec. 2 (L111-116) in any final version to state this overlap and these differences.

Q1. Fine-tuning vs. reference-conditioned generation [also R eD56 W1]

We would like to clarify that the choice between fine-tuning and reference conditioning is orthogonal to our contribution. We do not argue that one is preferable; we study what goes wrong in a setting that remains widely used, with over 12,000 text-to-image LoRA adapters uploaded to the Hugging Face Hub in the last six months alone. Fine-tuning is preferred when fine-grained subject fidelity matters, since tuning-free methods reproduce identity detail less faithfully (L103-105), and it also works with small backbones such as SD1.5, whereas reference-conditioned models such as Qwen-Image-2.1 are far larger and costlier to run for every generation. We also note that Qwen-Image-2.1 is contemporaneous with our submission.

Reviewer_eD56OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies subject personalization of text-to-image diffusion models when the personalization images are synthetically generated rather than real photographs. The paper shows that DreamBooth fine-tuning on synthetic images leads to degraded subject fidelity and artifacts such as oversaturation and excessive high-frequency detail. The paper attributes this degradation to an inflation of the classifier-free guidance (CFG) term, caused primarily by an increased angle between the conditional and unconditional noise predictions. The inflation is shown to vary across frequency bands and denoising steps and to be strongest at high frequencies. Based on this analysis, the paper proposes ReGain, a training-free sampling-time method that uses the original base model as a reference to estimate and correct the inflated guidance on a per-frequency and per-timestep basis within the subject region. Experiments across Stable Diffusion 1.5, SDXL, and SD3.5 show improved subject fidelity while largely preserving prompt alignment.

Critical strengths
  1. The guidance recalibration idea is simple, practical, and potentially applicable beyond the specific synthetic-data personalization setting considered in this work.

  2. The paper presents a thorough analysis of the observed degradation, studying CFG geometry, frequency and timestep dependence, and semantic locality, supported by quantitative metrics and ablations

Critical weaknesses
  1. The practical scope of the studied setting is somewhat unclear. The paper focuses on DreamBooth-style fine-tuning using fully synthetic subject images generated by another personalized model. However, an increasingly relevant use case for personalization is conditioning on one or a few reference images without subject-specific fine-tuning. Moreover, although the paper motivates the problem partly through edited or generated subject images, the experiments primarily consider synthetic images generated by the model​. It would be useful to better establish how representative this setup is of current personalization workflows.

  2. The attribution of the observed guidance inflation specifically to synthetic data is not fully convincing. While the controlled setup fixes the base model and fine-tuning procedure and rule out differences in CLIP diversity and generation guidance scale, the real and synthetic training sets can still differ systematically in their image statistics and distribution. Thus, it is unclear whether the observed CFG inflation is a property specific to training on synthetic images, or a consequence of a more general train-data/domain shift.

  3. The motivation and interpretation of the frequency-domain analysis are not sufficiently clear. While the paper observes stronger guidance inflation in high-frequency components of the latent guidance term and associates this with excessive fine image detail, the connection between frequency components of delta in latent space and the resulting image-space artifacts is not established very directly. Since frequency-dependent correction is central to ReGain, a clearer explanation and stronger evidence connecting latent frequency bands to the observed visual degradation would make the method considerably easier to motivate and interpret.

  4. Although the main ReGain results include SDXL and SD3.5, much of the detailed analysis and ablation supporting the proposed mechanism is conducted on Stable Diffusion v1.5. Given the architectural differences between the evaluated models, it would strengthen the paper to demonstrate that the key mechanistic findings, particularly the angle-driven and frequency-dependent CFG inflation, also hold on the newer backbones rather than showing primarily that the resulting correction transfers.

  5. The choice of the base model as the reference for “correct” guidance is practical but somewhat heuristic. Personalization is expected to modify conditional predictions relative to the base model, so it is not obvious that deviations of Msyn guidance magnitude from Mbase​ necessarily correspond to undesirable inflation. Although the empirical results support the usefulness of this reference, a stronger justification or analysis of when the base-model guidance constitutes an appropriate calibration target would strengthen the method because the base model is typically not perfect itself.

Questions for authors

Please address weaknesses

Rating justification - Paper contains some interesting ideas, particularly the guidance recalibration perspective and the detailed empirical analysis, the overall contribution is not very convincing. The practical relevance of the specific synthetic-data DreamBooth setting is not sufficiently established, and much of the detailed mechanistic analysis is conducted on Stable Diffusion v1.5, making it unclear how broadly the findings carry over to more recent models. More importantly, the interpretation of the observed guidance inflation seems insufficiently established: it remains unclear whether this is fundamentally a consequence of synthetic training data or a more general distribution-gap effect, and the motivation for treating the base model as the appropriate calibration reference is somewhat heuristic. Overall, some of the observations are promising, but the problem motivation, generality of the findings, and broader significance of the proposed method dont seem sufficiently convincing.

4 comments

We thank the reviewer for the thorough review and for calling the guidance recalibration idea "simple, practical." We address each weakness below.

W1. Practical scope of the setting [see R 62mc Q1; R gms2 W2]

We respectfully disagree that the setting is narrow: subject-specific fine-tuning remains a major personalization workflow, alongside reference conditioning.

Scale of use. The Hugging Face Hub lists over 70,000 text-to-image LoRA adapters, and over 12,000 of them were uploaded in the last six months (as of Nov 2026). About 3,600 of these recent adapters are built on the models we evaluate: SD1.5, SDXL and its fine-tuned variants, and SD3.5. Fine-tuning persists because it reproduces fine identity detail more faithfully (L103-105); see also our reply to Reviewer 62mc, Q1.

Regarding synthetic subject images: training a LoRA on generated or edited views of a subject is itself a standard recipe (Avrahami et al., 2024; L107). Our Mreal→MsynM_{\text{real}} \rightarrow M_{\text{syn}} design isolates data provenance with everything else fixed (L038-040). We add results on purely synthetic subjects in Appendix E of the revised manuscript (see our reply to Reviewer gms2, W2).

We will add this discussion of scope to Sec. 1 in any final version.

W2. Synthetic data differs in distribution from real data

We would like to clear up a possible misunderstanding: that synthetic images differ in distribution from real ones is the premise of the paper, not a confound. The synthetic training images are clean and artifact-free, so a clean output with a similar shift would be expected, yet MsynM_{\text{syn}} produces artifacts with inflated CFG, while training with the same recipe on real photos does not; we take a step toward understanding why through the frequency decomposition.

W3. Frequency analysis: latent bands to image artifacts [also R 62mc W1]

We would like to point the reviewer to Sec. 4.3 and Table 2 (L366-420), which already establish this connection in image space. Under plain CFG, images from MsynM_{\text{syn}} are 18% over-saturated, 22% over-contrasted and carry 13% excess high-band Fourier power relative to MrealM_{\text{real}} inside the subject region (L414-417), the image-space counterpart of the latent spectral excess of Fig. 2c. The same excess is visible in the power spectrum of the generated latents (Fig. 1, top right; L041-043).

Rescaling the latent frequency bands with ReGain then moves all three image statistics back toward MrealM_{\text{real}} without overshooting any of them:

MethodSaturationContrastHigh band
MrealM_{\text{real}} + CFG0.3010.2110.061
MsynM_{\text{syn}} + CFG0.3550.2580.069
MsynM_{\text{syn}} + CFG + ReGain0.3090.2340.066
Gap recovered85%51%38%

W4. Mechanistic findings on SDXL / SD3.5 [also R 62mc W2]

We repeated the analysis of Sec. 3.2 on SDXL and SD3.5, and the findings hold: the inflation is angle-driven and grows toward the high-frequency bands on both backbones. We have added these results as Appendix A.5 of the revised manuscript; we summarize them below.

Table: MsynM_{\text{syn}} / MrealM_{\text{real}} guidance statistics per backbone (dog6, mean over the sampling steps; 10 / 3 / 2 seeds for SD1.5 / SDXL / SD3.5)

BackboneAngle$\theta$ excessNorm ratio$\delta$Δ\Delta inflation, lowΔ\Delta inflation, midΔ\Delta inflation, high
SD1.5+37%1.00+34%+48%+68%
SDXL+42%1.00+35%+87%+109%
SD3.5+20%1.00−6%+7%+17%

W5. MbaseM_{\text{base}} as the calibration reference [see R gms2 W1]

We agree, and we do not claim that MbaseM_{\text{base}} corresponds one-to-one to the ideal reference. It is the best estimate available without real photos (L078-080), the Limitations already describe it as imperfectly calibrated (L471-472), and it consistently improves subject fidelity in all four settings (Table 1). Please see our reply to Reviewer gms2, W1.

Thanks for the response and the additional details.

A follow-up question regarding point 2: synthetic data generators are becoming increasingly realistic. If the domain gap between synthetic and real data were largely resolved, would this still remain a concern?

My concerns regarding points 3, 4, and 5 are resolved. However, concern 1 still remains. The problem setting feels somewhat constructed, particularly given that single-image approaches, inference-time approaches, and conditional generation approaches appear to perform considerably better. I think this is also related to point 2.

That said, I find the broader themes around frequency analysis, CFG inflation, etc. very interesting and potentially useful. It would be valuable to evaluate whether these observations hold in more general personalization settings and potentially make these phenomena a more central theme of the paper, rather than studying them primarily through this specific setting. I think this would broaden the relevance of the findings and make the research more useful to the community.

We thank the reviewer for the follow-up and are glad that points 3, 4 and 5 are resolved. We address the remaining points below.

W2. Synthetic data differs in distribution from real data

We believe the concern remains: images that look realistic to humnans can still differ in distribution from real ones. Detectors are still very good at separating images from the newest generators, including Nano Banana 2, GPT Image 1.5 and FLUX.2, at about 0.997 ROC AUC [A]. Our own synthetic training images are already clean and artifact-free, yet the model trained on them generates artifacted images. Intuitively, we believe the network amplifies the 'fingerprint' of such generative images when trained on such data, and this will remain, since a network is more prone to generate and fit such a 'fingerprint'.

W1. Practical scope of the setting

We are not aware of evidence that tuning-free approaches perform considerably better than DreamBooth LoRA, as recent work reports subject fidelity as their weak point [B, C]. Our setting also remains widely used, with over 12,000 text-to-image LoRA adapters uploaded to the Hugging Face Hub in the last six months, especially with small backbones that run locally.

Broader personalization settings

We thank the reviewer for the encouraging words on the frequency analysis and CFG inflation. We agree that these phenomena are the core of the paper: ReGain is built on the findings of Sec. 3.2, and we will make this framing more explicit in Sec. 1 in any final version.

Regarding other settings: We have added purely synthetic subjects in Appendix E of the revised manuscript (as requested by Reviewer gms2), where ReGain improves all three metrics. We agree that tuning-free and in-context settings are valuable to study. However, the findings themselves may differ in these settings: whether and where the guidance inflates, in which frequency bands and at which steps, and whether a rescaling fix applies.

We believe these settings are different enough to warrant a separate study, and we will state this in the Limitations and as future work in any final version.

Thank you for the response, will initiate a discussion with the area chair and other reviewers.

Reviewer_gms2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies DreamBooth personalization when the subject images are generated rather than real. Two models are fine-tuned from the same base with an identical recipe: Mreal on real photos and Msyn on images generated by Mreal. The authors find that Msyn's guidance term is inflated, mainly through a larger angle between the conditional and unconditional predictions, and that the inflation grows at high frequencies and varies over timesteps. They propose ReGain, a correction applied at sampling time. A one-time calibration against the base model gives a gain for each band and step, which rescales the guidance inside the subject mask. Experiments on SD1.5, SDXL and SD3.5 show partial recovery of the subject-fidelity gap to Mreal, with prompt fidelity largely preserved.

Critical strengths
  • The paper identifies and characterizes a concrete failure mode of personalization on synthetic data. The analysis is interesting.
  • The fix is practical. Calibration needs no retraining and no real photos, and works with U-Net and DiT backbones (SD1.5, SDXL, SD3.5).
Critical weaknesses
  • The guidance inflation is derived from the analysis of Msyn against Mreal, but the actual algorithm uses Mbase as the reference. There is no direct evidence that the two excesses are equivalent across frequency bands.
  • The evaluation scenario is limited. The paper argues that subject images increasingly come from generative models, but none of the realistic scenarios is evaluated, such as subjects that exist only as generations, images from external generators or editing models, or mixtures of real and synthetic data.
  • Calibration has to be repeated for every new subject, taking 12 minutes on SD1.5 alone, with no numbers reported for SDXL or SD3.5. This is clearly inefficient compared with training-free guidance baselines that add no cost, and it contradicts the claim of "no per-subject tuning", in my opinion.
Questions for authors
  • Can the authors provide direct evidence that the excess measured against Mbase matches the excess measured against Mreal in each frequency band and step?
  • Can the authors conduct qualitative and quantitative evaluations of ReGain in the realistic scenarios above: subjects that exist only as generations, images from external generators or editing models, and mixed real and synthetic data?
  • What are the calibration time and memory cost on SDXL and SD3.5? Can the calibration cost be reduced (e.g., fewer images, noise draws or steps), or can a gain schedule shared across subjects replace per-subject calibration?
1 comment

We thank the reviewer for the review and for noting that the analysis is interesting and the fix is practical. We address each weakness below.

W1/Q1. MbaseM_{\text{base}} vs. MrealM_{\text{real}} as the reference [also R eD56 W5]

We do not claim that the two excesses are equal, and indeed they are not. Our claim is only that MsynM_{\text{syn}} carries excess guidance. As stated in L078-080 and L137-138, MrealM_{\text{real}} is available only in our analysis experiments and not in practice, so MbaseM_{\text{base}}, which never saw the synthetic images (L202-204), is the best available reference. The Limitations (L471-472) already state that the base model is an imperfectly calibrated reference, so ReGain cannot fully match MrealM_{\text{real}}.

Empirically, this reference works well: it recovers 51–64% of the subject-fidelity gap on SD1.5 and improves all three subject-fidelity metrics in all four settings (Table 1).

W2/Q2. Realistic synthetic-subject scenarios

In our setting the subject exists for MsynM_{\text{syn}} only as generations: it never sees a real photo. We generate these images from the DreamBooth subjects so that the evaluation follows the standard protocol of prior work and the numbers remain comparable (Sec. 4.1, L284-285). Our focus is to isolate why this failure occurs and how to correct it; editing models and mixtures of real and synthetic data warrant a separate study.

That said, we have added results on purely synthetic subjects, i.e., subjects with no real photograph, in Appendix E of the revised manuscript. We reproduce them below for convenience.

Table: Purely synthetic subjects (SD1.5, DreamBooth, 3 subjects)

MethodCLIP-IDINOv2CLIP-T
MsynM_{\text{syn}} + CFG0.8960.8090.297
MsynM_{\text{syn}} + CFG + ReGain0.9060.8430.300

W3/Q3. Calibration cost

We would like to clarify that by "no per-subject tuning" we mean that no hyperparameter is tuned per subject. Calibration uses the same settings for all subjects and only needs forward passes of the two models.

Regarding the cost: we agree that calibration can be made faster. Its time grows linearly with the number of noise draws, and we used 10 draws per image without tuning this number. With 3 draws, calibration on SD1.5 takes under 3 minutes instead of 12 on one RTX A4000 (RTX 3070-class, 16 GB), and the measured gain changes by 0.03 on average (RMSE over the 30 subjects), which is below the tolerance of τ=0.05\tau = 0.05 that we already use to compress the schedule. For SDXL and SD3.5, which are larger models run at 1024×1024, calibration takes 10 and 15 minutes with 3 draws on the same GPU.

Regarding a shared schedule: the measured schedules differ across subjects (Fig. 4 and Fig. 10), so we keep calibration per subject.

Comments on the paper

1 comment

Dear reviewers,

This is a kind reminder that the discussion with the authors ends on 18th Nov. Please take a moment to leave a comment to acknowledge the authors’ responses and engage in the discussion if you have further questions, particularly on points that may affect your assessment.

Thank you!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.