- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduced a training free method which redistributes cross-attention with Gaussian smoothing and then averaging the content-token contributions and padding contributions. The authors did a comprehensive mathematical analysis and experiments to validate the efficacy of this method (better Pareto frontier in the CLIP-SSCD, ImageReward-SSCD tradeoffs in both SD v1.4 and SDv2.0) under certain assumptions.
Critical strengths
This paper is well written and the idea is clearly presented. Comprehensive experiments are done to illustrate that this idea can alleviate the memorization issue in text to image generation and have better Pareto frontier compared with existing works. The code is clean and experiments are well shown to support the idea. Proposition 1 provides a theoretical guarantee for reinforcement with localized dilution. The theory in appendix A.3 and A.4 provide a rigorous proof of complete row preservation and localized dilution and complementary roles of redistribution and gain.
Critical weaknesses
1.The assumption seems to be too strong so I am worried about the application of it in other cases. 2.The benchmark only has 500 memorized prompts. Can this method generalize to other datasets and have the same advantages over other methods.
Minor issue: 1.I would suggest to highlight/bold the values in the tables where the proposed method beats other methods, otherwise it is hard to get information from the table at the first sight. 2.It is better to write the characteristic function 1_{i\in T} in appendix A.1.
Questions for authors
1.“ We apply smoothing during an early denoising window to establish the redistributed conditioning pattern, then let Bk become the identity for the remaining steps. We use the first 5 of 50 steps for SD v1.4 and the first 10 for SD v2.0”--- is there any reasons why s/50 for SD v1.4 and 10/50 for SD v2.0?
2 what is l in equation (5) and why the assumption that |s_j-s_i|\leq l\j-i\ can be assumed? 3.Why the transfer to DeepFloydIF seems to have worse CLIP AND ImageReward compared to the base.
4 comments
Response [2/2]
Q1. Smoothing duration in SD1.4 and SD2.0
The smoothing width controls the extent of attention redistribution, while the duration controls how long it is applied. Our early-window design is consistent with prior evidence that conditioning during early denoising can steer trajectories toward memorized images [1]. We restrict smoothing to a short initial window to redistribute conditioning early while limiting continued mixing of token-specific information during later denoising. We use the first five of 50 steps for SD1.4 and the first ten for SD2.0 as model-specific settings. Content reinforcement and padding attenuation continue afterward.
The table below reports SD2.0 results at with smoothing applied during the first 5, 10, or 15 of the 50 denoising steps.
| Initial smoothing steps | Mean SSCD ↓ | CLIP ↑ | ImageReward ↑ |
|---|---|---|---|
| 5/50 | 0.2255 | 0.2596 | −0.5279 |
| 10/50 | 0.2090 | 0.2579 | −0.5482 |
| 15/50 | 0.1991 | 0.2575 | −0.5546 |
Q2. The meaning of and the local regularity assumption
For a fixed unit direction , let denote the projection of token ’s value vector onto that direction. The constant bounds how rapidly these scalar projections vary over the kernel neighborhood: . It is a local discrete Lipschitz bound on the projected values. The assumption concerns a shared projected feature; it does not require neighboring tokens to have identical meanings or identical full value vectors.
Together with a positive projection on the localized component, this condition controls the feature variation introduced by smoothing. The local variation must be small enough for a gain to compensate for it while still diluting localized attention mass: gives the admissible interval . The actual-value measurements reported under W1 quantify how often this sufficient local condition holds in the model.
Q3. Quality trade-offs in DeepFloyd IF
Our objective is to improve the suppression–quality trade-off. As Figure 2 shows, our method achieves a favorable trade-off relative to competing mitigation methods on SD1.4, although CLIP and ImageReward can decrease relative to the unmitigated Base model. The IF results exhibit a similar pattern: template matches decrease from 56/144 to 29/144, accompanied by reductions in CLIP and ImageReward. Here, we directly transferred the SD-selected configuration without IF-specific tuning. These results demonstrate that the suppression effect extends to T5-conditioned, pixel-space generation, while the associated quality cost can vary with the model and intervention setting.
References
[1] Jain et al. Classifier-Free Guidance Inside the Attraction Basin May Cause Memorization. CVPR, 2025.
[2] Somepalli et al. Understanding and Mitigating Copying in Diffusion Models. NeurIPS, 2023.
Thank you. That answered my question.
Response 2
We are glad to hear that our response answered your question. Thank you for taking the time to engage with our work and for your thoughtful feedback.
Response [1/2]
Thank you for recognizing the empirical trade-offs and theoretical analysis. We clarify the assumptions, their practical applicability, and the model-specific settings below.
W1. Strength and scope of the theoretical assumptions
The assumption concerns local coherence of a projected feature, without requiring neighboring tokens to have identical full value vectors. It identifies a sufficient regime in which reinforcement preserves that feature while redistribution dilutes localized attention mass.
We tested the proposition on actual SD1.4 value vectors at σ=1 using all 500 prompts during the first five denoising steps, across cross-attention layers and heads on a fixed query grid. Each attention row describes one image query’s attention over text tokens. For each row, we selected the content token with the largest attention weight and used the mean content-token value to define the feature direction.
The local analysis applies when the selected token has a positive projection along this direction and its entire smoothing neighborhood stays within content tokens, away from boundaries. We checked the sufficient conditions at the applied content gain, λw=2.
These measurements show that the proposition’s local regime occurs on actual model values during sampling: redistribution and gain can preserve the selected token’s projected contribution while reducing the fraction of its attention mass returned to its original position.
W2. Generalization beyond the 500-prompt benchmark
Beyond the 500-prompt SD1.4 benchmark, our evaluations show favorable comparative trade-offs on SD2.0 and SD3.0, as well as suppression transfer to DeepFloyd IF.
On the SD2.0 benchmark of 219 memorized prompts, our method also exhibits a favorable empirical Pareto trade-off. The IF evaluation further covers 48 template-verbatim prompts from 11 reference groups, extending transfer to T5 conditioning and pixel-space generation.
We further evaluated the template-memorization prompts from the SD2.0 benchmark on SD3.0, which uses a diffusion-transformer backbone. We compared our method with Jain et al. [1] and random token addition (RTA) [2] under a common evaluation protocol across multiple parameter settings. The table reports all four evaluated operating points for each mitigation method.
All four of our evaluated configurations lie on the empirical Pareto frontier in both the SSCD–CLIP and SSCD–ImageReward comparisons.
Minor comments
Thank you for the suggestion. Lower SSCD can come at the cost of CLIP and ImageReward, so highlighting individual metric optima could obscure the suppression–quality trade-offs represented by the different settings. We therefore leave these values unbolded; Figure 2 makes the comparison explicit by showing our method’s favorable trade-off relative to the baselines.
We have explicitly defined the indicator 1{i∈T} in Appendix A.1: it equals 1 when i∈T and 0 otherwise.