- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper examines whether memorization audits reliably measure training-data memorization and leakage in code-completion models. The authors inject canaries into a fine-tuning corpus and introduce matched non-members, a retrained reference model, and an over-memorized checkpoint as controls. They identify three failure modes: high reference-model likelihood ratios do not reliably distinguish members from non-members; a completion-based sequential test may detect fine-tuning rather than membership; and confidence bounds for zero extraction results vary substantially with the statistical unit. The paper also explains why pointwise likelihood ratios cannot support its originally intended information-theoretic leakage certificate and proposes a minimum set of audit controls.
Critical strengths
The problem is practically relevant. The paper demonstrates why a high audit score alone is insufficient evidence of training-data memorization, helping prevent unsupported interpretations of privacy risks.
The matched non-member controls are informative. Non-members follow the same generation procedure as members, and the authors additionally retrain a reference model without the canaries. The resulting AUC of 0.4855 directly challenges the interpretation of high exposure as evidence of membership.
The evaluation considers both detection capability and false positives. The over-memorized checkpoint confirms that the extraction pipeline can detect leakage, while the high non-member detection rates show why reporting member detection alone can be misleading.
Critical weaknesses
The generalizability of the main findings is limited. The central membership comparison relies on one model scale, one primary training configuration, and a single-insertion regime. As acknowledged by the authors, this regime may produce weak memorization, making it difficult to separate an insensitive statistic from an insufficiently strong memorization signal
The sequential test’s failure is closely tied to its construction. The test scores target-generated completions using the likelihood ratio between the target and reference models, which can naturally respond to distributional differences between those models. This establishes a limitation of the specific implementation, but its implications for membership inference more broadly require clearer qualification.
The assumptions underlying the confidence intervals need clarification. Table 3 reports “exact” Clopper–Pearson bounds for repeated queries involving the same secrets, without sufficiently explaining the source of randomness, dependencies between queries, or differences in success probabilities. Changing the denominator illustrates different estimands, but does not by itself establish the claimed coverage of each interval.
Positive controls do not cover all audit instruments. The over-memorized checkpoint validates extraction, as explicitly stated in Table 6. However, the main text does not establish whether exposure and the sequential membership test can distinguish members from non-members under known strong memorization, leaving their detection capabilities incompletely characterized.
The contribution beyond established evaluation principles needs sharper articulation. Non-member controls, false-positive reporting, and explicit statistical units are valuable requirements. However, the paper should better distinguish broadly applicable new findings from empirical demonstrations of these principles and corrections to the authors’ own certification framework.
Questions for authors
Do existing results across training checkpoints or insertion frequencies help distinguish weak memorization from an audit statistic’s inability to identify memorization? If such results are unavailable, could the authors explicitly narrow the corresponding conclusions?
What hypotheses does the sequential test actually distinguish: membership, or whether a completion originates from the target rather than the reference model? Is this test adapted from an established method or constructed specifically for this paper?
What random process defines the probabilities and coverage guarantees in Table 3? How are dependencies and heterogeneous success probabilities across repeated queries to the same secret handled when applying Clopper–Pearson intervals?
Have exposure AUC and the sequential test’s TPR/FPR already been evaluated on the over-memorized checkpoint? If not, should the positive-control claim be explicitly restricted to extraction?
What is the paper’s main generalizable contribution beyond existing practices of non-member calibration and false-positive reporting? Please distinguish conclusions specific to the evaluated statistics from those intended to apply more broadly.
5 comments
Reply to Reviewer 53pe (Part 2)
W3 and Question 3. Assumptions behind the Clopper–Pearson intervals
Section 6.4 now states the sampling model explicitly.
- Random process. The target, the prompt suite, and greedy decoding are fixed. The random element is the draw of canaries from the generator.
- Per-canary interval. "At least one prompt recovers this secret" is one Bernoulli trial per canary. The interval has nominal coverage for the probability that a generator-drawn secret is extractable, under independence across canaries. We note that canaries trained into one model and sharing templates may be positively dependent, which would make even this interval optimistic.
- Per-request figures. They treat the 7 or 35 prompts on one secret as independent trials with a common success probability. Neither assumption holds. We no longer call them valid intervals for any secret-level quantity; they are what a query-level audit would report.
We did not run a coverage simulation, and we say so.
A further caveat from the new experiments (Table 16). Canaries share prompt contexts: 1,000 literal secrets share 16. A prompt therefore cannot identify one secret, and the per-canary rate is capped far below what the model retains. A model from which sampling recovers 615 of 1,000 inserted secrets returns a canary's own secret for 0.18% of prompts. The zero-hit result on the primary target should be read with this cap in mind, and the revision says so.
We would appreciate knowing whether this statement of assumptions is sufficient for W3.
W4 and Question 4. Exposure and the sequential test on the over-memorized checkpoint
Both were evaluated (Appendix E.3, Table 17), and the result changed our description of that checkpoint.
- Exposure AUC is 0.569.
- The sequential test flags 100% of members and 100% of non-members.
- We traced the training configuration. Each canary was inserted once. The low training loss came from 700 clean documents seen through overlapping windows, not from the canaries.
The checkpoint was not over-memorized in the sense we claimed, and we have withdrawn the label. The positive controls are now the sweep for exposure (above) and the set-level measurement for extraction (615 of 1,000 secrets recovered, no never-inserted secret in 3,200 samples; Table 16).
An error this exposed. The checkpoint's extraction hits are all vulnerability canaries, at the same rate for members (10.25%) and same-template non-members (9.89%), and zero for template-disjoint non-members. The cause is in our panel: 142 of 283 non-member vulnerability secrets are identical to inserted ones, so that type cannot test membership. The disjointness check in the submitted version was wrong for that type. The other three types share no secret, and the central result is unaffected (Appendix E.4). We report the correction openly.
We would be grateful for the reviewer's confirmation that W4 is addressed.
W5 and Question 5. The generalizable contribution
The principles are not ours, and Appendix A.6 now says so. We reviewed the relevant protocols and could not identify a published canary protocol that prescribes a member-only reference ratio.
Specific to the statistics we evaluated: the sequential-test result and the defense evaluation.
Intended to apply to any reference-ratio canary audit:
- The template term. On the primary 1.5B model, never-inserted strings that share the canary templates gain 49.5 nats over a matched reference, as members do; template-disjoint strings gain 0.6 (Table 15).
- Non-members must match the members' templates. On the same model the AUC is 0.4855 against same-template non-members and 0.962 against template-disjoint ones.
- A floor at low repetition below which a correctly controlled audit reports nothing.
- The gap between set-level and per-canary extraction when canaries share contexts.
We would welcome the reviewer's view on whether this separation is now clear.
What we were not able to do
- A repetition sweep at the 1.5B scale (about 45 hours per run plus a reference).
- The set-level extraction measurement on the primary target.
- A coverage simulation for the intervals of W3.
Each is stated in Appendix F.
We are grateful for the reviewer's time and for a review that made the paper more precise. We respectfully ask the reviewer to consider whether the repetition sweep, its replications, and the narrowed conclusions change the assessment of the paper, and if so, whether the rating could be revisited.
Thank you again for your careful and constructive review. Since the discussion closes on 18 November, we would like to briefly summarize how each of your concerns has been addressed in the revised manuscript.
W1 & Q1 (weak memorization versus audit sensitivity). We conducted a controlled repetition sweep with matched references (Table 4). Exposure AUC increases from near chance at R = 1–4 to 1.000 for literal and paraphrase canaries at R = 64, demonstrating that the statistic detects sufficiently strong memorization. Additional training and non-member seeds, together with a fixed-canary-share design, confirm the low-repetition floor and increasing detectability (Tables 13–14). We have narrowed the primary null claim accordingly.
W2 & Q2 (sequential-test interpretation). We clarified that the test is our own construction and distinguishes target-generated from reference-generated text, not membership. Its TPR and FPR remain nearly identical across repetition levels, including R = 128 (99.5% versus 97.0%), where exposure separates literal canaries perfectly. We no longer generalize this failure to established membership-inference methods (Section 6.3; Table 11).
W3 & Q3 (confidence-interval assumptions). Section 6.4 now explicitly defines the sampling model and distinguishes per-canary Bernoulli intervals from invalid independence assumptions in per-request intervals. We acknowledge potential dependence across canaries and no longer interpret the per-request intervals as valid secret-level uncertainty estimates. We also clarify the shared-context limitation of per-canary extraction (Table 16).
W4 & Q4 (7B checkpoint and positive controls). We evaluated both instruments on the checkpoint (Table 17), traced its training configuration, and withdrew the unsupported “over-memorized” label. We also identified and disclosed overlapping vulnerability secrets in the original non-member panel, corrected the interpretation, and established stronger positive controls through the repetition sweep and set-level extraction results (615/1,000 inserted secrets recovered; Table 16).
W5 & Q5 (generalizable contribution). We now distinguish findings specific to our audit instruments from broader lessons about reference-ratio canary evaluation. The primary 1.5B experiment shows that same-template non-members gain 49.5 nats, versus 0.6 for held-out templates, with AUC changing from 0.4855 to 0.962 (Table 15). Together with the repetition sweep and the set-level versus per-canary extraction gap, these results support more precise recommendations about control matching and audit interpretation.
We have also explicitly documented the experiments that could not be completed within the discussion period, including the 1.5B repetition sweep and confidence-interval coverage simulation (Appendix F).
We would be grateful if you could revisit our responses and updated manuscript before the discussion closes and let us know whether these revisions address your concerns. If the new evidence changes your assessment, we would sincerely appreciate your consideration of a stronger recommendation and increased confidence.
Thank you again for your valuable feedback and guidance.
Reply to Reviewer 53pe (Part 1)
We thank the reviewer for a clear and fair review. Each weakness led to a new experiment or a correction. During the discussion period we fine-tuned twelve additional models (a six-level repetition sweep with a matched reference, two second-seed replications, and a second design with two seeds and its own reference) and re-evaluated the primary 1.5B target and the 7B checkpoint with new control panels. Section and table numbers below refer to the revised PDF, in which new text is blue. We answer each weakness together with its matching question.
W1 and Question 1. Weak memorization, or a statistic that cannot detect memorization?
Weak memorization. We ran the experiment that separates the two.
Repetition sweep (Section 6.5, Table 4). Qwen2.5-Coder-0.5B, 50K clean documents, 2,566 canaries each inserted R times, and a matched no-canary reference. Exposure AUC against same-template non-members:
The statistic is at chance at 1 to 4 insertions and separates members perfectly at 64. It is sensitive when memorization is strong.
Beyond one run and one regime. A second training seed at R = 16 and R = 64 changes no AUC by more than 0.018, and a second non-member seed by at most 0.010 (Table 13). Because the canary share of the corpus grows with R in the sweep, we also trained a second design with all repetition levels in one model, canaries at 8% of documents, and two seeds (Table 14): AUC is 0.52 to 0.58 at R ≤ 4 and rises monotonically to 0.87 to 0.96 at R = 64.
Narrowed conclusions, as the reviewer asked. The abstract and Section 6.2 now state that the primary null is a controlled null in a single-insertion regime. We claim a chance-level floor at low repetition and a monotone dose-response. We do not claim a transferable number of insertions for full separation, because that level differs between the two designs (Appendix F).
We would be grateful if the reviewer could confirm whether this resolves W1.
W2 and Question 2. What does the sequential test distinguish?
The reviewer is right. The test was constructed for this paper; it is not adapted from an established method. It distinguishes "this text was generated by the target" from "this text was generated by the reference", not membership. Section 6.3 now states both facts, and the abstract and introduction no longer present the result as a failure of likelihood-ratio membership inference in general.
The sweep confirms the diagnosis. The test's true- and false-positive rates differ by at most 3 points at every repetition level and under every defense, including R = 128 where exposure separates literal canaries perfectly (99.5% against 97.0%). Its detection rate on template-disjoint non-members is 90.5% to 100% (Table 11).
We hope this qualification is the one the reviewer had in mind.