acceptodds
Under review as a conference paper at ICLR 2027

The Null Is the Hard Part: Exact Tests for Memorization in Generative Models

Abstract

Memorization audits of generative models read similarity scores against thresholds, with no null distribution, and the conclusions they support can be wrong. By MemBench's rule, the benchmark's mitigations roughly halve Stable Diffusion's memorization; audited with false-discovery control, two thirds of the certified images are no longer detected under random prompt perturbations, five sixths under attention rescaling, and all of them under embedding optimization. The field's data-copying test, read against its own null, flags ten of twenty-four generators that reproduce nothing. We argue that for memorization the null is the hard part, and supply two. For a whole model, training and held-out images are exchangeable given its samples, and relabelling them is a permutation test, exact for any statistic when the held-out images are a random split; under it, a nearest-neighbour preference still fires on seven of those twenty-four, and a count restricted to the near-duplicate scale on none (McNemar p=0.016). For single images, the natural nulls fail twice, measurably: ranking an image among random images yields 596 false discoveries among 2,365 controls, and resampling independent generations makes the null three times too narrow. Calibrated against matched controls, the audit certifies 36 of 61 MemBench images at 5% false-discovery rate, held-out controls are certified in 0.01% of calibration splits, and on this benchmark two generations per image recover that count. A calibrated maximum, which reads occasional rather than typical copying, certifies 46. As the scale-restricted statistic we recommend the small-scale mass of the Intersection Euler Characteristic Profile, which also counts distinct images copied and tests whether two models copy the same ones.

Primary area:
generative models
Keywords:
memorization, diffusion models, data copying, permutation test, false discovery rate, conformal calibration, topological data analysis, Euler characteristic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6KtuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this work, the authors criticize existing tests for memorization in generative image models by proposing two new null hypotheses: one an exchangeability argument for swapping labels of train and held-out evaluation images at the model level, and one at the image-level based on the intersection of clouds around the train and held-out sets with the generated images.

Critical strengths

The authors take a principled approach to designing good null hypotheses for memorization tests in generative models. They introduce tests both for models and individual images, which are two complementary problems in the area of memorization detection in generative models.

Critical weaknesses

The writing and clarity is quite poor, to the point of hindering my ability to understand and appreciate the author's work. This is often vague feedback so I'll try to make concrete points. My personal perspective is that, if your goal is to get practitioners to adopt your rigorous approach to memorization testing, then you should opt for clarity over technical complexity.

  1. The work is mainly written in passive voice to the point where I can't tell what the authors did and what prior work did. The experiments section briefly goes into active voice which is a better example of good clarity in writing.
  2. The authors use a lot of pronouns instead of nouns which sometimes make me confused on what they are referencing, especially combined with the passive voice.
  3. You introduce results and methods without justifying or explaining them, instead of walking through prior methods/results and what you do differently. Walk the reader through your thought process!
  4. Maybe related to this, you mention (baseline) methods and tests in passing without explicitly introducing them. This is especially confusing in intro where we don't even know what your approach is yet.

For example on Line 343: "For memorized image xi with its ni trigger-prompt generations (median ni = 21), the statistic is the median SSCD similarity (Pizzi et al., 2022) of the generations to xi. Embedding space is the right place for it, since it sees layout-level copies, and the wrong place for the topological statistic, since SSCD distances concentrate."

  • You just introduced SSCD similarity in-line, and then talk about embedding spaces with no context. At least say "SSCD similarity, a metric based on similarity in model embedding space"

Examples of confusing phrases:

  • "Where topology enters is narrower, and we say so."
  • "At the operating scale its specificity matches the count’s."
  • "Read against its normal reference, the data-copying statistic flags 10 of 24 generators that reproduce nothing;"
    • What is the normal reference?
    • What is the data copying statistic?
    • Isn't reproducing nothing (in a memorization benchmark) good? So the data copying statistic is flagging 10/24 generators, should it be flagging all of them or none of them? I have no clue what your contribution is here
  • "the nearest-neighbour fraction rejects on 7, the data-copying statistic on 5, and a count restricted to the near-duplicate scale on none."
    • You introduce these methods without any context or explanation. Which is yours? which are baselines? do we want to reject on all or none?
  • "The median of 21 values resampled from the pool has spread (standard deviation) 0.016"
    • Why are you resampling 21 values from the pool, or what procedure is resulting in 21 values? Earlier in the paragraph you mentioned 47 discoveries not 21.

To show an example of more clear, active voice, consider contribution 1:

  • "The exact null for memorization, and what the existing one gets wrong." ->
    • We point out challenges with existing tests for memorization and propose an exact null for memorization.
  • "Ten copies among 500 samples, injected into a model that copies nothing, are caught by the scale-restricted statistics in every replicate; the nearest-neighbour fraction needs fifty." ->
    • Using the scale-restricted statistics we are able to find 10 data copies inserted in a set of 500 unmemorized generations across multiple trials, whereas the nearest-neighbor fraction needs 50 (needs 50 what?)

Line 212: Where is Prop 3?

Notation: you define you test statistic as S(G,T,H)S(G,T,H) but then in the image-level null section write S(Gi,xi)S(G_i, x_i)

I believe that the method underlying this work is solid, but it is difficult for me to evaluate due to the presentation. I am willing to increase my score but I fear that the presentation issues would require changes on a scale beyond the normal revisions of a discussion cycle.

Questions for authors
  • If TsmallT_{small} is looking at r0r_0 balls around points, how is it different than the naive threshold tests you were criticizing earlier? Isn't the radius an arbitrary hyperparameter again here?
  • On your CIFAR-10 trained models, what is your ground truth for memorization? How do you know your statistic is finding the true memorized generations?
  • Why does statistical power change with the number of replicated samples? I understand it being dependent on the number of held-out samples but aren't you testing each replicated sample independently, or are you testing the entire set jointly?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 6Ktu

Thank you for the concrete examples; they are the most useful part of the reviews for us, and the revision will follow them (active voice throughout, every baseline introduced before it is used, no results before their justification). To your questions:

How is T_small at r₀ different from the threshold tests we criticise? The threshold rules we criticise compare a similarity to a fixed cutoff with no null, so the cutoff sets the error rate. Under the swap null, r₀ sets what is measured (how close a sample must be to count as a copy), not the error rate: the permutation test is exact at any r₀, and the choice is reported. The choice does matter for what is detected, and we will show the sensitivity (four models rerun at 0.6× to 2× the default r₀; at 2× one marginal model rejects).

Ground truth on CIFAR-10. Pixel distance: a sample within a quarter of the training-image spacing of a training image is a copy, and a generator with at least a fifth of its samples copies is a reproducer. Reviewer 2VGb points out that this makes the count's specificity partly circular, and we agree; the non-circular evidence is on Stable Diffusion, where the ground truth is MemBench's reverse-image-search verification.

Why does power depend on the number of copies? The model-level test is one statistic for the whole sample of 500 generations, tested jointly against the permutation null; it is not a per-sample test. More copies among the 500 move that one statistic further from its null, which is why power rises with the number injected. The image-level audit is the per-image test, and there the question is how many generations per image are needed (two, on this benchmark).

Line 212 / Prop. 3 / notation. Proposition 3 is in Appendix A and will move to the main text; S(G, T, H) is the model-level statistic and S(G_i, x_i) the image-level one, and the revision will use different letters.

Reviewer_6CMoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a test to check the null hypothesis of memorisation. They propose a model-level permutation test, An image-level test and a topological statistic.

Critical strengths
  • I like the premise that finding some semantic leakage don’t necessarily means there’s memorisation. Especially given that we usually prompt with the same text
  • Have a statistical test could standardize the study of memorization
Critical weaknesses
  • The null hypothesis is actually stronger than no memorisation. A model could depend of it’s training samples without depending on it. Diffusion models have been shown to generalise while at the same time being heavily tied to the training data.
  • Image level test is not correctly matched. The generated samples are matched to prompt that trigger memorisation but the reference are random images on LAION.
  • I’m not sure I understand the value of the topological statistic. A KNN should work just as well. The authors should account for this simple baseline in their experiments.
  • Overall I find the paper very hard to parse. The authors should make a significant effort on writing and presentation. This paper doesn’t meet the standards in my opinion mainly because of the poor LLM writing
  • Stable diffusion 1.4, 1.5, 2.0 are notoriously known to overfit mainly due to the lack of deduplication of their data. I think the authors should include results with more up to date SoTA diffusion models.
Questions for authors

I need clarifications from the authors on the null hypothesis. Can you the methodological issues i point out for the image level test and the topological statistic

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6CMo

Thank you. On the three points:

The null is stronger than "no memorization". Yes, deliberately. The model-level null is that training and held-out images are exchangeable given the samples, i.e. the samples carry no information about which images were training. A model can violate it by depending on its training set without reproducing anything, which is what the nearest-neighbour fraction and the data-copying statistic detect on the 1,000-to-5,000-image models; that is why we distinguish a rejection (dependence) from reproduction, and read reproduction from a statistic restricted to the near-duplicate scale. The revision will state this distinction in the introduction rather than in Section 3.

The image-level matching. Reviewer 2VGb raises the same issue precisely, and we agree: trigger-prompt candidates and natural-caption controls are not exchangeable by construction. Our reply there gives the counts and the negative control we will run (prompts from MemBench's own search pipeline for non-memorized images).

A kNN baseline. It is in the paper: the nearest-neighbour fraction T_NN is the kNN baseline, and the thresholded count at the near-duplicate scale is the simplest possible one. The count matches the topological statistic's specificity and power, and the revision will recommend the count and present the profile as the instrument, not the recommendation.

Models. We audited Stable Diffusion v1.4, v1.5, RealisticVision 5.1 and Stable Diffusion 2. The audit needs only samples, a candidate list and a control null, so it applies to any model; we chose these because they are the ones with verified memorized-image benchmarks. We will say in the limitations that a deduplicated model is where the audit's specificity matters most and that we have not run one.

Writing. Agreed, and Reviewer 6Ktu gives us a concrete list. The revision will be rewritten in active voice with every method introduced before it is used.

Reviewer_2VGbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes two tests for memorization in generative models. A model-level permutation test relabels which real images count as "training" and "held-out," and is exact for any statistic when the held-out set is a random split. An image-level audit ranks each candidate image's score against matched control images, giving conformal p-values with Benjamini–Hochberg or Storey FDR control. The recommended statistic is the small-scale mass of the Intersection Euler Characteristic Profile (IECP). On 29 CIFAR-10 generators and on Stable Diffusion with MemBench, the paper reports that Meehan et al.'s data-copying test flags 10 of 24 non-copying generators, that the audit certifies 36 of 61 MemBench images at 5% FDR, and that mitigations remove far more certifications than MemBench's fixed rule suggests.

Critical strengths

S1. A simple, general model-level test. It works for any statistic, needs no retraining or data marking, and is exact under a random split.

S2. Overfitting vs. reproduction. The distinction is useful, and the public SecMI/Diffence DDPM (App. K) shows a model vulnerable to membership inference that reproduces no training image.

S3. Two measured failure modes. Semantic leakage (596 false discoveries among 2,365 controls) and false concentration (null spread 0.016 vs. 0.050) are concrete lessons for anyone building such audits.

Critical weaknesses

W1. The image-level null is not exchangeable by construction. The FDR guarantee requires a non-memorized candidate's statistic to be exchangeable with the controls' ("produced as G_i was"). It is not. Each candidate's n_i generations come from n_i distinct MemBench trigger prompts, found by an MCMC search that maximizes a model-based memorization score and kept when their outputs clustered on one image (Hong et al., Sec. 3.4). Each control's 12 generations are seeds of one natural LAION caption. Prompts selected on the model's output plausibly carry more semantic leakage than natural captions (the paper's own "failure one"), and medians over related prompts and over seeds of one prompt have different spreads ("failure two"). Both effects are anti-conservative, and candidates with n_i < 12 cannot be count-matched at all. The held-out-control check (0.01% of splits) cannot detect this, as held-out controls are exchangeable with calibration controls by construction. Notably, the 25 non-certified candidates sit above the controls as a group (Mann–Whitney p = 0.014, App. C), which a null that is too low would also produce. The model-level test on SD has the same issue, since G is generated from T's own prompts. An informative negative control would audit non-memorized images whose prompts come from the same search pipeline.

W2. The headline mitigation result mixes two changes. MemBench's rule reads an uncalibrated maximum; the audit reads a calibrated median. Sec. 4 shows that a calibrated maximum certifies 46 of 61 images (vs. 36), yet Fig. 4 and Tab. 5 use only the median, while 21–181 generations still exceed the copy threshold under each mitigation. Hence, "two thirds to all of the certified images are no longer detected" may reflect the choice of statistic rather than calibration. The calibrated maximum under each mitigation is needed.

W3. The topological statistic is not what drives the results. A thresholded count at the same scale matches its specificity (0 of 24) and its power. The IECP, by contrast, requires a 5-dimensional PCA projection and at most 2,000 points (O(n^⌈d/2⌉)), cannot operate in SSCD space where SD memorization is detected (App. H), and its leak count fails on SD (App. M). Hypothesis (i) of Proposition 3 also fails for near-duplicate training images, a main driver of memorization (Carlini et al., 2023a), and fails more often in the projection than in pixel space. The recommendation should be narrowed, or a case shown where the two statistics disagree.

W4. The CIFAR-10 evaluation is close to circular. "Reproduction" is defined as a sample within r0 of a training image, which the thresholded count measures almost by definition; hence, 0 of 24 vs. 7 of 24 for the nearest-neighbour fraction largely restates the definition (the paper itself calls the latter correct rejections). This ground truth also misses SD's memorization, where only 39 of 1,826 generations (2.1%) are pixel-level copies. In addition, r0, d0 = 5 and the statistic were chosen and evaluated on the same 29 generators, McNemar treats seeds of one configuration as independent, and the power experiment uses five replicates per cell.

W5. The critique of Meehan et al. targets the wrong statistic. The normal-reference comparison is reported only for the global Z_U, while Meehan et al. recommend the cell-wise C_T. Moreover, "10 of 24" counts generators that meet their definition (Δ_T < 1/2) but not the paper's. The paper concedes the test "is valid for that null," so the abstract should describe a disagreement between definitions, not a failure of the test.

W6. The paper is very hard to read. The abstract contains about twenty numbers and terms used before definition; nonstandard terms ("five-dimensional shadow," "readout protocol," "leaning") recur; T denotes both the training set and every statistic; the same quantity receives several values (Fig. 1: 132 vs. 7, then 131 vs. 15; App. N: 108 vs. 116); much of the evidence sits in 14 appendices; and neither test is given as an algorithm. The main message is also unclear: the title stresses the null, the abstract recommends the IECP, and Sec. 3 finds a plain count equally specific.

W7. Figure 2 is too small. At this size every pair looks identical, so exact, near, and layout-level copies cannot be distinguished, which is precisely the distinction Sec. 4 relies on. I suggest enlarging the images, printing a similarity value under each pair, stating how examples were selected, and adding a contrast row from a non-memorizing model. The left panel of Fig. 1 and Fig. 5 are similarly hard to read.

W8. Table 1 reports runs inconsistently. Second seeds appear only at 200 and 1,000 images and the 300k-step run only at 1,000, while Tab. 3 lists further seeds at 1,000, 2,000 and 5,000 that Tab. 1 omits (with training clouds capped at 1,000). No reason is given; if these runs were added after borderline results, this should be stated. All sizes should have at least three seeds, reported in one table. The "of 2,000" cap and the negative leak counts (−6, −11) also need explanation.

W9. Minor. The abstract's "two thirds," "five sixths" and "all" should name the mitigations they refer to, and several references have encoding errors (e.g., Kynkäänniemi, Mücke, Tramèr).

Questions for authors

Q1. (W1) What does the audit give on non-memorized images whose prompts come from MemBench's search pipeline? How many candidates have n_i < 12, and how many of those are certified?

Q2. (W1) Do any of the audit's prompt–seed pairs coincide with those MemBench used to find or verify images?

Q3. (W2) How many images does the calibrated maximum certify under each mitigation?

Q4. (W3) Is there any experiment in which T_small and the thresholded count disagree at r0? How sensitive are the results to d0?

Q5. (W5) What does C_T give against its normal reference on the 24 non-reproducing generators?

Q6. (W8) Were the additional seeds and the 300k-step run planned before the first results?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2VGb (part 1 of 2)

Thank you for a careful reading. We answer the questions first, with new numbers where we have them, then say what we will change.

Q1 / W1 (exchangeability of the image-level null). We agree that the guarantee rests on an exchangeability the design does not enforce: candidates are generated from search-selected trigger prompts, controls from seeds of one natural caption, and both differences you name (more semantic leakage from selected prompts; different spread for medians over related prompts than over seeds of one prompt) would push candidate statistics up relative to the null. The budget-matched check in the paper (12 generations each, 35.4 ± 1.3 certified) addresses only the count, and the paper says so; it does not address prompt selection. The held-out-control check cannot detect this either, for the reason you give. Counts: 13 of the 61 candidates have fewer than 12 generations (n_i from 1 to 11; the median over all 61 is 21, the maximum 138), and 4 of those 13 are certified (n_i = 2, 6, 10 and 11). The negative control you propose is the right experiment: run MemBench's prompt search on Stable Diffusion v1.4 for source images that the search does not verify as memorized, generate their selected prompts exactly as we did the candidates', and audit them against the caption null. If those controls are certified above the nominal rate, the null is too low, and the certified count must be read against that rate. We cannot complete it inside the discussion window. The revision will state the image-level guarantee as conditional on candidate–control exchangeability, give this control as its test, and report its outcome.

Q2. No. Our generations use MemBench's prompt text with a seed that is a hash of that text (crc32(prompt)), so none of our prompt–seed pairs is one MemBench used to find or verify an image.

Q3 / W2 (calibrated maximum under each mitigation). You are right that the headline mixes the choice of statistic with calibration. Calibrated maximum at a matched budget of 12 generations per image and per control, Benjamini–Hochberg at q = 0.05, 200 subsamples (mean ± sd certified of 61), with the calibrated median at the same budget beside it:

armcalibrated mediancalibrated maximum
none35.6 ± 1.346.2 ± 1.2
random tokens (RTA)8.9 ± 2.128.3 ± 1.8
random numbers (RNA)9.7 ± 1.033.6 ± 2.0
Wen, target 1.60.032.2 ± 2.1
Wen, target 10.07.9 ± 2.7
Ren, C₁ = 1.17.9 ± 3.026.6 ± 1.9

(The full-generation calibrated medians reproduce Table 5: 36, 11, 10, 0, 0, 6.) So the mitigations remove typical copying (the median) from most or all certified images, but occasional copying (the maximum) survives in well over half of them under the two prompt perturbations, the weaker Wen target and Ren's rescaling, and in a sixth under the stronger Wen target. The abstract's "two thirds / five sixths / all" is a statement about the median only. The revision will report both statistics in Figure 4 and Table 5, name the mitigations in the abstract, and state the conclusion as: mitigation suppresses typical reproduction far more than occasional reproduction, and the benchmark's uncalibrated maximum reads the latter.

(continued in part 2)

AuthorsOpenReview ↗

Response to Reviewer 2VGb (part 2 of 2)

(part 2 of 2)

Q4 / W3 (does the topology drive anything). At r₀ on the 29 generators the thresholded count and T_small agree on 28: both detect the four reproducers, and the one disagreement is a T_small rejection (p = 0.02) of the 5,000-image model under the random split, which copies nothing and which the count does not reject. In the power experiment both reach full power at 10 injected copies; they differ only at 5 or fewer, within five-replicate noise. On r₀: four models were rerun at 0.6×, 1.4× and 2× the default; at 2× the 300k-step run rejects (p = 0.005 against 0.055) and the second 1,000-image seed moves from 0.68 to 0.105, so r₀ is not innocuous and the revision will show this. d₀ was not varied; we will add that. We accept the conclusion: the recommendation will be narrowed to the thresholded count at the near-duplicate scale as the statistic to use under the swap null, with the profile presented as the instrument that produced the leak count and the shared-copy test, and its limits (5-d projection, point cap, embedding spaces) stated where it is introduced rather than in appendices.

W4 (circularity of the CIFAR evaluation). For the count's specificity, yes: a generator with no sample within r₀ of a training image cannot fire the count, and that row of Table 2 is a consistency check, not evidence. What the CIFAR sweep does show that is not circular is where the other statistics fire on those generators and why. The non-circular evidence for the audit is on the Stable Diffusion side, in SSCD space, where the ground truth is MemBench's reverse-image-search verification. We will say this, drop the McNemar comparison that treats seeds of one configuration as independent (or compute it per configuration), and raise the power experiment above five replicates.

Q5 / W5 (C_T). On the 24 non-reproducers under the random split, C_T read against a standard normal flags 6 (Z_U flags 10); under the swap null C_T flags 4 (Z_U 5). Meehan et al. give no reference distribution for C_T, and its null variance depends on the cell weights (with equal weights over five cells it is 1/5, under which 11 are flagged), which is one reason we reported Z_U. You are right about the framing: these generators meet Meehan et al.'s definition (Δ_T < ½) and do not reproduce, so the abstract should say that the data-copying test answers a different question, not that it fails. We will.

Q6 / W8 (were the extra runs planned). The first sweep was designed as training-set size × seed × training steps and included the second seeds at 200 and 1,000 images and the 300k-step run at 1,000. The further seeds at 1,000, 2,000 and 5,000 images were trained a week later to bring the number of non-reproducing diffusion models to eleven for the specificity table, after the first sweep's results were known and not in response to any single borderline result. The revision will report every run in one table with at least three seeds per size. The "of 2,000" cap is the size of the random subset of the training set that the Alpha complex can take; the leak count is an excess over a held-out background count and is negative when the background exceeds it by chance. Both will be explained where the table is.

W6, W7, W9. We will cut the abstract to one claim and a handful of numbers, define every term before use, reserve T for the training set, report one leak count per model (the 132 / 7 against 131 / 15 and 108 against 116 pairs are the same quantity read on two scale grids; one will go), give both tests as algorithms in the main text, enlarge Figure 2 with similarity values, the selection rule and a non-memorizing contrast row, redraw Figures 1 and 5, and fix the bibliography encoding.

I thank the authors for their response! I would've appreciated if they kept the rebuttal within the character limit and a single response.

(Q2), (Q3 / W2), (Q4 / W3), (Q5 / W5), (Q6 / W8): Acknowledged and sounds good.

Q1 / W1: I don't see this in Section 4 or Appendix C of the revised PDF.

W4: I don't see this in Section 3 of the PDF.

W6, W7, W9: I don't see this in the abstract or Figures 1, 2 and 5 of the PDF.

I'm open to increasing my score if this gets resolved in the revision.

AuthorsOpenReview ↗

Revision uploaded: where each item now lives

Thank you for checking. The revised PDF is now uploaded; we had answered before revising, and we apologise for the two-part reply. Where each item now lives:

Q1/W1, the negative control (Sec. 4 "The negative control", p. 8–9; protocol in App. C, p. 16). We ran the experiment you proposed. MemBench's own search (its augmentation chain: Gibbs sampling over BERT proposals, scored by the Wen et al. magnitude, at the benchmark's default settings) was run from the captions of 61 further control images outside both nulls; the 21 highest-scoring variants per image (the candidates' median n_i) were generated exactly as the candidates were and audited against the same 300-caption null. Result: 0 of 61 certified at q = 0.05 (3.05 expected), 0.0 ± 0.0 at the matched budget of 12 over 200 subsamples. The negatives' median own-source similarity is 0.031 against the null's 0.064, below it, because the search moves a prompt away from its caption. The search raises the selection score from 0.84 (captions) to 1.42 (selected variants) against 5.8 for the trigger prompts, so the negatives are matched on the selection pipeline, not on the score; the gap in score is the memorization itself. One control image turned out to be genuinely memorized (8 of 21 search-selected generations above 0.5 against its own source, p at the floor 1/301), which the first BH threshold 0.05/61 cannot certify alone; we report it. The image-level guarantee is now stated as conditional on candidate–control exchangeability (Sec. 2, "The image-level null"), with this control as its test. The counts you asked for are in Sec. 4 "Data": n_i from 1 to 138, median 21, 13 images below 12.

W4 (Sec. 3 "Specificity", p. 7). The count's row of Table 2 is now described as a consistency check rather than evidence, with the non-circular evidence placed on the Stable Diffusion side; the McNemar p-values are removed (seeds of one configuration are not independent generators); the r_0 sensitivity is reported (six models at 0.6×, 1.4×, 2×: the 300k-step run goes from p = 0.055 to 0.01 and 0.005, the 5,000-image model from 0.17 to 0.05 and 0.015, the second 1,000-image seed from 0.68 to 0.52 and 0.105); d_0 not varied and five power replicates are stated as limitations (Sec. 3 and Sec. 6).

W2/Q3 (Sec. 5, Fig. 4, Table 5 in App. I). The calibrated maximum at budget 12 is reported under every mitigation (46, 28, 34, 33, 8, 27 against the calibrated median's 36, 9, 10, 0, 0, 8), in the figure as a third series and in the table as two columns, and the conclusion is restated as typical versus occasional reproduction.

W3/Q4 (Sec. 1, Sec. 3). The recommendation is the thresholded count; the profile is presented as the instrument with its limits (5-d projection, 2,000-point cap, no use in embedding spaces) stated where it is introduced; the one disagreement with the count (the 5,000-image model at p = 0.02) is in Sec. 3.

W5/Q5 (Sec. 3). C_T against a standard normal flags 6 of the 24; the abstract and Sec. 3 now say the data-copying test answers a different question (its definition is met) rather than that it fails.

W6, W7, W9. Abstract cut to one claim with the mitigations named; T reserved for the training set (statistics are S_NN, S_cnt, S_small, S_mass); one leak count per model (Fig. 1 caption, App. O); both tests given as Algorithms 1–2 (p. 4); Proposition 3 moved to the main text; SSCD introduced before use; the "21" explained as the median n_i. Fig. 2 redrawn with the statistic under every pair, the selection rule in the caption, and contrast rows from the 50,000-image DDPM and from control captions (App. N, Fig. 6); Fig. 1's left panel enlarged; Fig. 5 (App. G) redrawn with legends. The bibliography renders correctly in the PDF; the encoding error was in OpenReview's text extraction, and we have checked the .bib entries.

W8/Q6 (Table 1). Every DDPM run is in one table, with the cap "of 2,000" and the negative leak counts explained in the caption and the run planning stated in Sec. 3.

The full active-voice rewrite that Reviewer 6Ktu asked for goes beyond what we could do in the window; the sections above are rewritten, the rest is unchanged in substance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.