- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PatchRecover, a memorization detection metric for text-to-image diffusion models. PatchRecover perturbes one local patch at a time and regenerates edited images with diffusion model. It measures how strongly each edited image returns toward the no-edit reference, then aggregates patch scores into an detection score. The method supports both prompt-conditioned and prompt-free settings. The authors report strong detection results on benchmarks built with Stable Diffusion v1.4 and v2-base, and analyze denoising trajectories to investigate local recovery.
Critical strengths
The paper provides a quite straightforward while effective way to flag memorization, compared to more sophisticated metrics proposed by previous works depending on attention, probability-landscape sharpness or score anisotropy.
Critical weaknesses
- One major weaknesses of this paper is its limited novelty, both on the research question it considers and method it proposes. As far as I know, memorization detection has been extensively studied, with existing methods already achieving near-saturated performance on established benchmarks, which are also used by this paper. future research would benefit more from investigating novel memorization scenarios or model architectures than from introducing another detection method for well-studied settings.
- The contribution of theorectical explanation. The theorectical results presented in Section 3.2 regarding local contraction region is intriguing, but, lacks clear connection to the proposed metric. The effectiveness of the metric is primarily supported by empirical observations, leaving the theorectical results isolated. For example, how are and determined and how do they guide the design of the metric?
Questions for authors
- I would suggest the authors justify the unique challenges in memorization detection and the novelty of this work.
- The authors should clarify the contribution and necessity of theoretical analysis on the methodolody.
4 comments
W2 / Q2. The theoretical results in Sec. 3.2 lack a clear connection to the metric. How are and determined, and how do they guide the design of the metric? Please clarify the contribution and necessity of the analysis.
How ρ and ε are determined. Both parameters are introduced only for theoretical analysis and are not used to define or compute the detection metric. Specifically, defines the recovery region and is estimated from recorded trajectories, while represents the residual recovery error for each patch. Their purpose is to explain the local recovery behavior underlying PatchRecover, rather than directly determine its metric.
How the analysis guides the design. The analysis explains why memorized patches are more likely to recover their original content and motivates four design choices, supported by ablation studies below:
- Regeneration strength. Sufficient denoising is needed for recovery, but excessive noise makes recovery harder. Memorized patches enter the recovery region with about 16 steps left, non-memorized ones with about 8 (Table 23). Too much noise, however, starts the trajectory far from the region. Accordingly, AUC on SD1 is 0.74 at strength 0.1, 0.999 at 0.5 and 0.956 at 0.9 (Table 10).
- Top-k aggregation. Since memorization may occur only in certain patches, we focus on patches with the strongest recovery signals. Using all patches reduces SD1 AUC from 0.9990 to 0.9592 (Table 9).
- Recovery margin. The margin measures recovery toward the original content relative to the edited content. It achieves higher TPR@1%FPR (0.64) than using either distance alone (0.41 and 0.05), while reconstruction without editing performs poorly (AUC 0.59, Table 22).
- Meaningful patch edits. Recovery requires a noticeable change to the original content. Gray, Neighbor, Flip, and Blur edits work well, while JPEG compression, which largely preserves the content, performs poorly (Table 8).
Overall, the analysis explains and motivates our metric design, supported by the ablation results. We clarify this in the revised Sec. 3.2.
We hope these clarifications adequately address your concerns and welcome any further questions.
Dear Reviewer,
We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.
Thank you
~ AC
Thank you again for your review, and thanks to the AC for the reminder. In our response, we clarified the challenge this work targets: existing detectors are near-saturated when they use the prompt and its generation seed, but they fall close to chance when they receive only the image. We also explained how the recovery analysis motivates the design choices of the metric.
We would appreciate it if you could let us know whether this addresses your concerns. If so, we hope you might consider updating your assessment. With the discussion period closing soon, we are happy to clarify anything further in the time left.
We thank the reviewer for recognizing our method as simple yet effective compared with more complex prior metrics, and for finding the recovery-region analysis intriguing. We address each concern below.
We agree that existing methods achieve near-saturated performance on standard benchmarks (Table 2, AUC > 0.98). However, these benchmarks overlook a more practical challenge: detecting memorization in individual images rather than their generation prompts. Our novelty lies in addressing this overlooked image-level setting. We identify this gap, reveal the limitations of existing methods in this setting, and propose an effective image-level detection approach based on local counterfactual recovery.
We have clarified the novelty of our work and contributions in the revised Introduction. We hope they address the concern about novelty.