- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper finds that diffusion models personalized on synthetic subject images exhibit inflated classifier-free guidance, especially at high frequencies, alongside reduced subject fidelity. It introduces ReGain, a training-free sampling correction that calibrates guidance by frequency and timestep against the original base model and applies it within a subject mask, improving subject fidelity across SD1.5, SDXL, and SD3.5 while largely preserving text alignment.
Critical strengths
-
Clear presentation. The paper is well written and organized, with a coherent progression from empirical observations to the proposed correction.
-
Practical relevance. The problem is concrete, and the ability to improve already personalized checkpoints without additional training makes the proposed intervention potentially useful.
-
Meaningful experimental evidence. The controlled comparisons provide useful evidence within the evaluated settings. The paper includes quantitative assessments of subject and prompt fidelity, together with qualitative examples illustrating the effects of the correction.
-
Results across multiple architectures. The method consistently improves the reported subject-fidelity metrics across several backbones, including U-Net-based SD1.5 and SDXL and transformer-based SD3.5.
Critical weaknesses
Overall, this paper is close to borderline, which requires more detailed discussions with the literature and more solid experiments. I would change my ratings if the following concerns are addressed.
- The ablations do not isolate the contribution of frequency-dependent calibration. In Table 3, lowering global CFG to 3 matches ReGain’s reported DINO score of 0.647, while ReGain achieves higher CLIP-T: 0.290 versus 0.281. Similarly, removing ReGain’s mask produces slightly higher DINO but lower CLIP-T. These results suggest that subject localization plays an important role in preserving text alignment, leaving the additional benefit of frequency-dependent calibration unclear. The key missing comparisons are corrections using the same subject mask with either a single scalar gain or a timestep-dependent gain without frequency decomposition. Table 4 does not resolve this issue: it measures how closely the compressed schedule matches its calibration target, rather than whether the additional frequency/time structure improves final image quality over these simpler masked corrections.
- The generality of the findings beyond the evaluated Stable Diffusion models remains unclear. Although the experiments cover both U-Net and transformer architectures, all evaluated backbones belong to the Stable Diffusion family. It remains uncertain whether the observed guidance inflation and the benefits of ReGain persist in newer model families. A focused evaluation on a newer compatible backbone would strengthen this claim. For example, FLUX.2 [klein] Base is explicitly designed for fine-tuning and is released without step or guidance distillation, making it a relevant candidate. The authors should also clarify which properties of the backbone and guidance formulation are required for ReGain to apply.
- The differentiation from ReDiFine understates the overlap in the underlying observations. ReDiFine already analyzes the norm of the conditional–unconditional score difference, reports its growth across synthetic retraining iterations, and connects this growth to high-frequency degradation. ReGain cites this work and identifies meaningful differences, including its focus on a single round of subject personalization and its sampling-time correction. However, the distinction between prior work on the guidance scale and the present work on the guidance term does not fully reflect ReDiFine’s analysis. The paper should acknowledge this closer overlap and explain more precisely what its personalization setting, geometric and frequency-resolved characterization, and base-referenced calibration add beyond those earlier findings.
Questions for authors
In which settings does fine-tuning offer a meaningful advantage over direct reference-conditioned generation? Recent foundation models support generation and editing from multiple reference images; for example, Qwen-Image-2.1 supports up to 10 reference images and is designed to preserve the identities of people and products. Could the authors clarify the circumstances in which DreamBooth or LoRA fine-tuning remains preferable—for example, in terms of fine-grained subject fidelity, consistency across prompts, controllability, or the cost of repeated generation? A clearer discussion, supported by representative evidence where available, would help establish the practical motivation for the studied setting and ReGain’s intended use cases.
We thank the reviewer for the constructive review and for noting the clear presentation and practical relevance. We address each weakness below.
W1. Isolating frequency-dependent calibration [also R eD56 W3]
As suggested, we ran the masked correction without frequency decomposition, using the same subject mask as ReGain. The masked baseline performs strongly, confirming the role of the subject mask; however, the frequency-dependent gain of ReGain still improves over it.
Table: Masked corrections with and without frequency decomposition (SD1.5, DreamBooth, w=7.5)
We have added this row to the ablation in Table 3 of the revised manuscript.
W2. Generality beyond the SD family [also R eD56 W4]
We note that our backbones already cover both U-Net (SD1.5, SDXL) and DiT (SD3.5, rectified flow) families, which are among the most widely used in the community: about 3,600 of the text-to-image LoRA adapters uploaded to the Hugging Face Hub in the last six months are built on them (see our reply to Reviewer eD56, W1). Experiments on FLUX.2 [klein] Base over the complete subject set and seeds are in progress, and we will add them to any final version. Since SD3.5 is already a flow-based DiT, we expect the findings to carry over, though the size of the gain varies across backbones (Table 1).
Regarding the required properties: our study is empirical, and ReGain applies wherever this inflation of the guidance term appears: the model must be sampled with CFG from separate conditional and unconditional predictions, and the base checkpoint it was fine-tuned from must be available.
W3. Overlap with ReDiFine
We agree that the overlap in the underlying observation is closer than L111-112 conveys. However, our work differs in four respects.
Setting. ReDiFine studies a six-iteration chain of LoRA fine-tuning on captioned datasets, with the training set regenerated at every iteration (their Sec. 3.1). We study one round of DreamBooth personalization with a subject identifier, and show that the inflation follows the subject's semantic neighborhood (L177-181).
Cause. ReDiFine reports the norm along each model's own sampling trajectory. We decompose it and show that the growth comes from the angle between the conditional and unconditional predictions, not their magnitudes (L157-161), and that it persists when both models see the same inputs (L182-189).
Frequency structure. ReDiFine's frequency analysis is on the final generated images (their Sec. 4.3, Fig. 37). We instead resolve the guidance term itself per frequency band and denoising step (L190-198).
Correction. ReDiFine modifies fine-tuning (condition drop) and decays a scalar CFG scale over steps (their Sec. 5). ReGain measures a per-band, per-step gain against the base model and corrects already-trained checkpoints without retraining (Sec. 3.3).
We regard the two sets of findings as complementary, and will revise Sec. 2 (L111-116) in any final version to state this overlap and these differences.
Q1. Fine-tuning vs. reference-conditioned generation [also R eD56 W1]
We would like to clarify that the choice between fine-tuning and reference conditioning is orthogonal to our contribution. We do not argue that one is preferable; we study what goes wrong in a setting that remains widely used, with over 12,000 text-to-image LoRA adapters uploaded to the Hugging Face Hub in the last six months alone. Fine-tuning is preferred when fine-grained subject fidelity matters, since tuning-free methods reproduce identity detail less faithfully (L103-105), and it also works with small backbones such as SD1.5, whereas reference-conditioned models such as Qwen-Image-2.1 are far larger and costlier to run for every generation. We also note that Qwen-Image-2.1 is contemporaneous with our submission.