acceptodds
Under review as a conference paper at ICLR 2027

Fooled by Fine-Tuning: Why Memorization Audits of Code-Completion Models Need Controls

Abstract

Memorization audits of code models commonly inject canaries into training data and measure how strongly the resulting model prefers them, often without evaluating comparable non-members. We show that this practice can produce convincing but misleading conclusions. Through a controlled audit of fine-tuned code models with more than canaries, multiple attacker families, checkpoints, and controls, we identify three recurring failure modes. First, large reference-model exposure scores do not necessarily distinguish injected canaries from never-injected controls, even with a matched-corpus reference. Second, a completion-based likelihood-ratio membership test can respond similarly to members and non-members, effectively detecting fine-tuning rather than membership. Third, extraction guarantees can be substantially overstated when confidence bounds are computed for a different estimand than the security claim being made. Positive-control experiments confirm that these findings are not simply failures of the auditing pipeline. We further revisit the information-theoretic certificate motivating our audit. We correct a vacuous additive bound formulation, but show that the pointwise likelihood ratios computed by reference-ratio audits do not directly estimate the divergence required by such a certificate; accordingly, we make no certification claim. Our results motivate a stricter methodology for memorization auditing based on non-member controls, matched references, per-instrument positive controls, and estimand-aware reporting.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
memorization auditing, code language models, membership inference, training data extraction, canary auditing, privacy leakage, fine-tuning, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_53peOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines whether memorization audits reliably measure training-data memorization and leakage in code-completion models. The authors inject canaries into a fine-tuning corpus and introduce matched non-members, a retrained reference model, and an over-memorized checkpoint as controls. They identify three failure modes: high reference-model likelihood ratios do not reliably distinguish members from non-members; a completion-based sequential test may detect fine-tuning rather than membership; and confidence bounds for zero extraction results vary substantially with the statistical unit. The paper also explains why pointwise likelihood ratios cannot support its originally intended information-theoretic leakage certificate and proposes a minimum set of audit controls.

Critical strengths

The problem is practically relevant. The paper demonstrates why a high audit score alone is insufficient evidence of training-data memorization, helping prevent unsupported interpretations of privacy risks.

The matched non-member controls are informative. Non-members follow the same generation procedure as members, and the authors additionally retrain a reference model without the canaries. The resulting AUC of 0.4855 directly challenges the interpretation of high exposure as evidence of membership.

The evaluation considers both detection capability and false positives. The over-memorized checkpoint confirms that the extraction pipeline can detect leakage, while the high non-member detection rates show why reporting member detection alone can be misleading.

Critical weaknesses

The generalizability of the main findings is limited. The central membership comparison relies on one model scale, one primary training configuration, and a single-insertion regime. As acknowledged by the authors, this regime may produce weak memorization, making it difficult to separate an insensitive statistic from an insufficiently strong memorization signal

The sequential test’s failure is closely tied to its construction. The test scores target-generated completions using the likelihood ratio between the target and reference models, which can naturally respond to distributional differences between those models. This establishes a limitation of the specific implementation, but its implications for membership inference more broadly require clearer qualification.

The assumptions underlying the confidence intervals need clarification. Table 3 reports “exact” Clopper–Pearson bounds for repeated queries involving the same secrets, without sufficiently explaining the source of randomness, dependencies between queries, or differences in success probabilities. Changing the denominator illustrates different estimands, but does not by itself establish the claimed coverage of each interval.

Positive controls do not cover all audit instruments. The over-memorized checkpoint validates extraction, as explicitly stated in Table 6. However, the main text does not establish whether exposure and the sequential membership test can distinguish members from non-members under known strong memorization, leaving their detection capabilities incompletely characterized.

The contribution beyond established evaluation principles needs sharper articulation. Non-member controls, false-positive reporting, and explicit statistical units are valuable requirements. However, the paper should better distinguish broadly applicable new findings from empirical demonstrations of these principles and corrections to the authors’ own certification framework.

Questions for authors

Do existing results across training checkpoints or insertion frequencies help distinguish weak memorization from an audit statistic’s inability to identify memorization? If such results are unavailable, could the authors explicitly narrow the corresponding conclusions?

What hypotheses does the sequential test actually distinguish: membership, or whether a completion originates from the target rather than the reference model? Is this test adapted from an established method or constructed specifically for this paper?

What random process defines the probabilities and coverage guarantees in Table 3? How are dependencies and heterogeneous success probabilities across repeated queries to the same secret handled when applying Clopper–Pearson intervals?

Have exposure AUC and the sequential test’s TPR/FPR already been evaluated on the over-memorized checkpoint? If not, should the positive-control claim be explicitly restricted to extraction?

What is the paper’s main generalizable contribution beyond existing practices of non-member calibration and false-positive reporting? Please distinguish conclusions specific to the evaluated statistics from those intended to apply more broadly.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer 53pe (Part 1)

We thank the reviewer for a clear and fair review. Each weakness led to a new experiment or a correction. During the discussion period we fine-tuned twelve additional models (a six-level repetition sweep with a matched reference, two second-seed replications, and a second design with two seeds and its own reference) and re-evaluated the primary 1.5B target and the 7B checkpoint with new control panels. Section and table numbers below refer to the revised PDF, in which new text is blue. We answer each weakness together with its matching question.


W1 and Question 1. Weak memorization, or a statistic that cannot detect memorization?

Weak memorization. We ran the experiment that separates the two.

Repetition sweep (Section 6.5, Table 4). Qwen2.5-Coder-0.5B, 50K clean documents, 2,566 canaries each inserted R times, and a matched no-canary reference. Exposure AUC against same-template non-members:

RLiteralParaphraseSemantic
10.5180.4320.526
40.5320.4780.551
160.6390.6370.605
641.0001.0000.915
1281.0001.0000.997

The statistic is at chance at 1 to 4 insertions and separates members perfectly at 64. It is sensitive when memorization is strong.

Beyond one run and one regime. A second training seed at R = 16 and R = 64 changes no AUC by more than 0.018, and a second non-member seed by at most 0.010 (Table 13). Because the canary share of the corpus grows with R in the sweep, we also trained a second design with all repetition levels in one model, canaries at 8% of documents, and two seeds (Table 14): AUC is 0.52 to 0.58 at R ≤ 4 and rises monotonically to 0.87 to 0.96 at R = 64.

Narrowed conclusions, as the reviewer asked. The abstract and Section 6.2 now state that the primary null is a controlled null in a single-insertion regime. We claim a chance-level floor at low repetition and a monotone dose-response. We do not claim a transferable number of insertions for full separation, because that level differs between the two designs (Appendix F).

We would be grateful if the reviewer could confirm whether this resolves W1.


W2 and Question 2. What does the sequential test distinguish?

The reviewer is right. The test was constructed for this paper; it is not adapted from an established method. It distinguishes "this text was generated by the target" from "this text was generated by the reference", not membership. Section 6.3 now states both facts, and the abstract and introduction no longer present the result as a failure of likelihood-ratio membership inference in general.

The sweep confirms the diagnosis. The test's true- and false-positive rates differ by at most 3 points at every repetition level and under every defense, including R = 128 where exposure separates literal canaries perfectly (99.5% against 97.0%). Its detection rate on template-disjoint non-members is 90.5% to 100% (Table 11).

We hope this qualification is the one the reviewer had in mind.

AuthorsOpenReview ↗

Reply to Reviewer 53pe (Part 2)

W3 and Question 3. Assumptions behind the Clopper–Pearson intervals

Section 6.4 now states the sampling model explicitly.

  • Random process. The target, the prompt suite, and greedy decoding are fixed. The random element is the draw of canaries from the generator.
  • Per-canary interval. "At least one prompt recovers this secret" is one Bernoulli trial per canary. The interval has nominal coverage for the probability that a generator-drawn secret is extractable, under independence across canaries. We note that canaries trained into one model and sharing templates may be positively dependent, which would make even this interval optimistic.
  • Per-request figures. They treat the 7 or 35 prompts on one secret as independent trials with a common success probability. Neither assumption holds. We no longer call them valid intervals for any secret-level quantity; they are what a query-level audit would report.

We did not run a coverage simulation, and we say so.

A further caveat from the new experiments (Table 16). Canaries share prompt contexts: 1,000 literal secrets share 16. A prompt therefore cannot identify one secret, and the per-canary rate is capped far below what the model retains. A model from which sampling recovers 615 of 1,000 inserted secrets returns a canary's own secret for 0.18% of prompts. The zero-hit result on the primary target should be read with this cap in mind, and the revision says so.

We would appreciate knowing whether this statement of assumptions is sufficient for W3.


W4 and Question 4. Exposure and the sequential test on the over-memorized checkpoint

Both were evaluated (Appendix E.3, Table 17), and the result changed our description of that checkpoint.

  • Exposure AUC is 0.569.
  • The sequential test flags 100% of members and 100% of non-members.
  • We traced the training configuration. Each canary was inserted once. The low training loss came from 700 clean documents seen through overlapping windows, not from the canaries.

The checkpoint was not over-memorized in the sense we claimed, and we have withdrawn the label. The positive controls are now the sweep for exposure (above) and the set-level measurement for extraction (615 of 1,000 secrets recovered, no never-inserted secret in 3,200 samples; Table 16).

An error this exposed. The checkpoint's extraction hits are all vulnerability canaries, at the same rate for members (10.25%) and same-template non-members (9.89%), and zero for template-disjoint non-members. The cause is in our panel: 142 of 283 non-member vulnerability secrets are identical to inserted ones, so that type cannot test membership. The disjointness check in the submitted version was wrong for that type. The other three types share no secret, and the central result is unaffected (Appendix E.4). We report the correction openly.

We would be grateful for the reviewer's confirmation that W4 is addressed.


W5 and Question 5. The generalizable contribution

The principles are not ours, and Appendix A.6 now says so. We reviewed the relevant protocols and could not identify a published canary protocol that prescribes a member-only reference ratio.

Specific to the statistics we evaluated: the sequential-test result and the defense evaluation.

Intended to apply to any reference-ratio canary audit:

  1. The template term. On the primary 1.5B model, never-inserted strings that share the canary templates gain 49.5 nats over a matched reference, as members do; template-disjoint strings gain 0.6 (Table 15).
  2. Non-members must match the members' templates. On the same model the AUC is 0.4855 against same-template non-members and 0.962 against template-disjoint ones.
  3. A floor at low repetition below which a correctly controlled audit reports nothing.
  4. The gap between set-level and per-canary extraction when canaries share contexts.

We would welcome the reviewer's view on whether this separation is now clear.

What we were not able to do

  • A repetition sweep at the 1.5B scale (about 45 hours per run plus a reference).
  • The set-level extraction measurement on the primary target.
  • A coverage simulation for the intervals of W3.

Each is stated in Appendix F.

We are grateful for the reviewer's time and for a review that made the paper more precise. We respectfully ask the reviewer to consider whether the repetition sweep, its replications, and the narrowed conclusions change the assessment of the paper, and if so, whether the rating could be revisited.

Thank you again for your careful and constructive review. Since the discussion closes on 18 November, we would like to briefly summarize how each of your concerns has been addressed in the revised manuscript.

W1 & Q1 (weak memorization versus audit sensitivity). We conducted a controlled repetition sweep with matched references (Table 4). Exposure AUC increases from near chance at R = 1–4 to 1.000 for literal and paraphrase canaries at R = 64, demonstrating that the statistic detects sufficiently strong memorization. Additional training and non-member seeds, together with a fixed-canary-share design, confirm the low-repetition floor and increasing detectability (Tables 13–14). We have narrowed the primary null claim accordingly.

W2 & Q2 (sequential-test interpretation). We clarified that the test is our own construction and distinguishes target-generated from reference-generated text, not membership. Its TPR and FPR remain nearly identical across repetition levels, including R = 128 (99.5% versus 97.0%), where exposure separates literal canaries perfectly. We no longer generalize this failure to established membership-inference methods (Section 6.3; Table 11).

W3 & Q3 (confidence-interval assumptions). Section 6.4 now explicitly defines the sampling model and distinguishes per-canary Bernoulli intervals from invalid independence assumptions in per-request intervals. We acknowledge potential dependence across canaries and no longer interpret the per-request intervals as valid secret-level uncertainty estimates. We also clarify the shared-context limitation of per-canary extraction (Table 16).

W4 & Q4 (7B checkpoint and positive controls). We evaluated both instruments on the checkpoint (Table 17), traced its training configuration, and withdrew the unsupported “over-memorized” label. We also identified and disclosed overlapping vulnerability secrets in the original non-member panel, corrected the interpretation, and established stronger positive controls through the repetition sweep and set-level extraction results (615/1,000 inserted secrets recovered; Table 16).

W5 & Q5 (generalizable contribution). We now distinguish findings specific to our audit instruments from broader lessons about reference-ratio canary evaluation. The primary 1.5B experiment shows that same-template non-members gain 49.5 nats, versus 0.6 for held-out templates, with AUC changing from 0.4855 to 0.962 (Table 15). Together with the repetition sweep and the set-level versus per-canary extraction gap, these results support more precise recommendations about control matching and audit interpretation.

We have also explicitly documented the experiments that could not be completed within the discussion period, including the 1.5B repetition sweep and confidence-interval coverage simulation (Appendix F).

We would be grateful if you could revisit our responses and updated manuscript before the discussion closes and let us know whether these revisions address your concerns. If the new evidence changes your assessment, we would sincerely appreciate your consideration of a stronger recommendation and increased confidence.

Thank you again for your valuable feedback and guidance.

Reviewer_1VKnOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines whether canary-based memorization audits of fine-tuned coding models support the security interpretations attached to their measurements. Using Qwen2.5-Coder-1.5B fine-tuned with 11,566 synthetic canaries, the authors evaluate reference-model exposure, membership inference, and extraction using matched non-members and auxiliary control experiments. The central finding is that large exposure scores coexist with near-chance member/non-member separation, even when the reference model is fine-tuned on the same background corpus without the canary documents. The paper also demonstrates that a completion-based membership test can flag members and non-members at similar rates, and that extraction confidence changes substantially depending on whether the statistical unit is a query or a secret. Its theoretical analysis identifies why pointwise likelihood ratios on externally generated canaries cannot directly support the proposed information-theoretic certificate, which the authors consequently withdraw. Overall, the work provides a useful empirical challenge to interpreting likelihood increases as instance-specific memorization, while format-level generalization remains a promising but unverified explanation for its main observation.

Critical strengths
  1. Table 4 provides a compelling counterexample to interpreting large reference-model exposure scores as evidence of strong instance-specific memorization: members and matched non-members have nearly identical mean scores, with an AUC of 0.4855. This goes beyond reporting an unsuccessful attack, because the seemingly strong memorization signal persists even when it lacks corresponding membership discrimination. The format-level generalization hypothesis offers a valuable perspective on this discrepancy by distinguishing learning the structure of canary continuations from retaining particular secret values.

  2. The matched-reference experiment makes the central observation more informative than a comparison against an unfine-tuned base model alone. By retraining the reference on the same background corpus without canary documents and checking panel disjointness and injection fidelity, the authors reduce the plausibility of ordinary domain adaptation or accidental data overlap as sufficient explanations for their findings. These controls strengthen the empirical case that improving reference matching does not automatically make raw exposure scores informative about membership.

  3. The paper illustrates how appropriate controls can materially change the interpretation of apparently convincing security results. The completion-based test's 99% member detection rate is accompanied by a 100% false-positive rate, while changing the statistical unit from requests to secrets increases the reported upper endpoint of the two-sided 95% interval for zero hits from 0.105% to 3.622% (Tables 3 and 5). These concrete examples make the recommended evaluation practices useful to practitioners rather than merely restating that controls and statistical definitions matter.

  4. The theoretical analysis clarifies the distinction between an empirical memorization score and a formal leakage guarantee by identifying the mismatch between pointwise likelihood ratios and the output-distribution KL divergence required by the proposed certificate. Explicitly separating this unresolved estimation problem from the corrected information-theoretic expressions prevents an unsupported certification claim and explains what additional justification a valid guarantee would require.

Critical weaknesses
  1. The paper proposes two plausible explanations for the limited membership discrimination observed in Table 4: weak instance-specific memorization due to limited canary exposure and format-level generalization that increases exposure scores for both members and non-members. While these hypotheses provide interesting insights into the observed results, the current experiments do not sufficiently investigate either explanation. Additional experiments varying canary repetition frequency and the degree of template sharing would help validate these hypotheses and better explain the underlying phenomenon.

  2. The central member/non-member analysis is conducted using a single target/reference model pair under one fine-tuning and canary-injection setting. Although this controlled setup supports the observed failure of membership discrimination, it remains unclear whether the findings generalize across different training configurations or stronger memorization regimes. Additional evaluations across independently trained models and different fine-tuning settings would strengthen the generalizability of the conclusions.

Questions for authors
  1. The matched reference excludes all canary documents, so it controls for background-domain fine-tuning but does not match the target's exposure to canary formats. Could the authors train a reference on the same templates with disjoint secret values, while matching training steps and token budgets, to test whether format learning explains the similarly large scores for members and non-members?

  2. The main experiment uses a single-insertion regime, and the existing positive control establishes extraction capability rather than the sensitivity of exposure or membership inference. Could the authors repeat the experiment across several canary repetition levels, holding the architecture and background corpus fixed, and report exposure separation, membership true-positive/false-positive rates, and extraction together?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer 1VKn (Part 1)

We thank the reviewer for an encouraging and precise review. The two questions asked for exactly the experiments the paper was missing, and both are now central to it. During the discussion period we fine-tuned twelve additional models and re-evaluated the primary 1.5B target with a new control panel. Section and table numbers below refer to the revised PDF, in which new text is blue.


Q1 and W1. Does format learning explain the similar scores of members and non-members?

Yes. We now show this directly on the primary 1.5B model.

What we could not do. The reviewer suggests a reference trained on the same templates with disjoint secrets. Training it requires about 173 GPU-hours at this scale, which we could not complete in the discussion period. We say so in Section 7 and Appendix F.

What we did instead. We built a second non-member panel from held-out templates: new contexts and secret formats for literal canaries, new wrappers for paraphrase canaries, new layouts for semantic canaries. We scored it with the existing target and the existing matched reference (Appendix E.3, Table 15). This tests the same hypothesis from the evaluation side: if format learning is the cause, never-inserted strings should score high when they share the templates and low when they do not.

Non-member panelMean score, members (nats)Mean score, non-members (nats)AUC
Same templates as members49.549.50.4855
Held-out templates49.50.60.962

By type, the template-disjoint AUC is 0.992 (literal), 0.946 (paraphrase), and 1.000 (semantic). The same-template run reproduces the submitted AUC of 0.4855, which confirms that the checkpoints and code are unchanged.

Reading. Never-inserted strings gain as much as members when they share the templates and almost nothing when they do not. The large scores of both groups are template learning. The same pattern appears in the 0.5B sweep: 72 to 105 nats against 2 to 4 at 1 to 16 insertions (Table 4), and it is stable under a second non-member seed (Table 13).

A consequence we had not anticipated. If an audit's non-members do not share the members' templates, it reports AUC 0.96 for a model with no per-canary signal. A non-member panel is therefore not sufficient; it must match the templates. This is now one of the paper's main recommendations.

We would be grateful if the reviewer could confirm whether this evaluation-side test answers Q1, given that the same-template reference was not trained.


Q2 and W2. Several repetition levels, with exposure, TPR/FPR, and extraction reported together

Done, with architecture and background corpus held fixed (Section 6.5, Table 4; Appendix E.3). Qwen2.5-Coder-0.5B, 50K clean documents, 2,566 canaries each inserted R times, and a matched no-canary reference.

RExposure AUC, literalparaphrasesemanticSequential test TPR / FPR (%)Inserted secrets recovered (of 1,000)
10.5180.4320.526100.0 / 99.0not run
40.5320.4780.55199.5 / 100.0not run
160.6390.6370.605100.0 / 100.0not run
641.0001.0000.91599.5 / 100.0162
1281.0001.0000.99799.5 / 97.0615
  • Exposure is at chance for 1 to 4 insertions and separates members perfectly at 64. The instrument works when memorization is strong, so the null on the primary target is a controlled null.
  • The sequential test never separates members from non-members, at any level.
  • Extraction succeeds at the level of the inserted set: 615 of 1,000 secrets at R = 128, and none of 3,200 samples contains a never-inserted secret. A prompt returns its own canary's secret only 0.18% of the time, because the 1,000 secrets share 16 prompt contexts (Table 16). This explains the near-zero extraction rates of the submitted version: they were per-canary rates, which shared contexts cap far below what the model retains.

We would appreciate the reviewer's confirmation that this table provides what Q2 asked for.

AuthorsOpenReview ↗

Reply to Reviewer 1VKn (Part 2)

W2. Generalization across independently trained models and other settings

We added three checks (Appendix E.3).

  1. Second training seed. We retrained the R = 16 and R = 64 models with a different seed. No AUC changes by more than 0.018 (Table 13).
  2. Second non-member seed. Rescoring R = 1, 64, 128 with a new non-member panel changes no AUC by more than 0.010 (Table 13).
  3. A second design. In the sweep the canary share of the corpus grows with R. We therefore trained one model that contains all repetition levels at once, with 200K clean documents and canaries at 8% of documents, under two seeds (Table 14). AUC is 0.52 to 0.58 at R ≤ 4 and rises monotonically to 0.87 to 0.96 for literal and paraphrase canaries at R = 64.

The chance-level floor at low repetition holds in both designs and across seeds. The level needed for full separation differs between designs (1.00 against 0.87 to 0.96 at R = 64), so the revision claims a floor and a monotone dose-response, not a transferable threshold.

We hope this addresses the concern about a single training configuration, and would welcome the reviewer's view.

We are grateful for the reviewer's support and for questions that improved the paper. If the reviewer finds that the template-disjoint control and the repetition sweep answer Q1 and Q2, we respectfully ask the reviewer to consider whether the revised paper merits a higher rating, and whether the added evidence allows greater confidence in the assessment.

Thank you again for your encouraging and constructive review. Since the discussion closes on 18 November, we would like to briefly summarize how your concerns have been addressed in the revised manuscript.

W1 & Q1 (format learning and non-member controls). We added a template-disjoint non-member panel on the primary 1.5B model (Table 15). Same-template non-members gain 49.5 nats, identical to members, whereas held-out-template non-members gain only 0.6 nats. The corresponding AUC changes from 0.4855 to 0.962. This directly supports the format-learning explanation and demonstrates why non-member controls must be template-matched. We acknowledge that the suggested same-template reference model could not be trained within the discussion period.

W2 & Q2 (repetition sweep and positive controls). We trained models across multiple repetition levels while holding the architecture and background corpus fixed (Table 4). Exposure AUC increases from approximately 0.5 at low repetition to 1.000 at R = 64 for literal and paraphrase canaries, while the sequential test continues to fail. At R = 128, sampling recovers 615 of 1,000 inserted secrets, demonstrating substantial set-level memorization despite low per-canary extraction rates (Table 16).

W2 (robustness and generalization). Additional training and non-member seeds change AUC by at most 0.018 and 0.010, respectively (Table 13). A second fixed-canary-share design independently confirms the low-repetition floor and increasing detectability with repetition (Table 14). We have narrowed our claims accordingly and no longer suggest a universal repetition threshold.

These additions directly address the experimental controls and generalization concerns you raised, and we believe they substantially strengthen the revised paper.

We would be grateful if you could revisit our responses and updated manuscript before the discussion closes. If you find that the new evidence resolves your concerns, we would sincerely appreciate your consideration of whether the revised contribution warrants a stronger recommendation and increased confidence.

Thank you again for your valuable feedback and support.

Reviewer_yZEXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether canary-based audits reliably support claims about memorization in fine-tuned code-completion models. Using Qwen2.5-Coder-1.5B with 11,566 injected canaries, the authors find that large reference-model likelihood ratios do not distinguish members from matched non-members, and that their completion-based sequential test responds to fine-tuning rather than membership. They also illustrate how extraction confidence intervals change with the statistical unit. The paper withdraws its original certification claims and advocates stronger evaluation controls. The case study is useful, but its methodological novelty and the validation of its central measurement instruments remain insufficiently established.

Critical strengths
  1. The matched-reference experiment provides a meaningful control for interpreting likelihood-ratio scores. The authors train a reference model on the same corpus with the canary documents removed, rather than relying exclusively on the original base model. Member/non-member separation nevertheless remains near chance. This is consequential because it demonstrates that replacing an unmatched reference does not automatically make the raw score informative about instance-level membership in the evaluated setting.

  2. The paper makes the consequences of choosing the statistical unit concrete. For the zero-hit extraction results, the reported two-sided 95% upper endpoint changes from approximately 0.105% over 3,500 requests to 3.622% over 100 canaries. This clearly illustrates why a confidence statement about requests cannot be substituted for one about secrets. Although the underlying statistical principle is established, the example is useful for interpreting security evaluations.

Critical weaknesses
  1. The paper does not sufficiently establish the research gap or connect its demonstrated failures to representative existing auditing methods. The central framing suggests that memorization audits commonly lack the controls needed to distinguish memorization from other effects. However, The Secret Sharer already defines exposure through comparison with same-format alternative candidates; LiRA explicitly distinguishes models trained with and without the target example and emphasizes false-positive control; and Duan et al. already investigate misleading MIA performance under distribution shift and ambiguous member/non-member boundaries. The completion-based SPRT particularly illustrates this concern. Section 6.4 scores target-generated text using the target/reference output likelihood ratio. This directly compares output distributions of two models, rather than constructing the member-versus-non-member hypotheses underlying standard membership inference. Its failure therefore establishes a problem with this particular construction, but does not by itself demonstrate a previously unrecognized failure of established likelihood-ratio MIAs. The paper needs to identify concrete existing audit protocols affected by its findings and explain what remains unresolved after their prescribed controls are applied. Without this connection, the contribution is primarily a correction of the authors’ original audit, with uncertain broader methodological significance.
  2. The central measurement instruments are not positively validated, leaving the interpretation and generality of the null results unresolved. The positive control demonstrates successful extraction from an over-memorized 7B checkpoint, but Table 6 and Appendix F explicitly acknowledge that exposure separation and the membership test were not positively validated. Successful extraction does not establish that these other instruments can detect instance-level memorization when it is present. Moreover, the principal member/non-member evidence comes from one target/reference training pair, one insertion regime, and a panel with substantial template reuse. The near-chance AUC is therefore compatible with weak instance-level memorization, insufficient instrument sensitivity, or shared-format generalization obscuring a smaller membership effect. These possibilities have materially different implications, but the experiments do not distinguish them. A controlled repetition sweep with fixed architecture and corpus, together with instrument-specific positive controls, would substantially strengthen the contribution. The current results support a useful local counterexample, but provide limited evidence about when the diagnosed failure occurs or how generally it affects reference-ratio auditing.
Questions for authors
  1. Relationship to existing audit protocols. Which published methods use the member-only reference-ratio interpretation or completion-based SPRT evaluated here? Can you identify a concrete conclusion from prior work that changes under your controls when the original method is implemented according to its prescribed protocol?

  2. Instrument sensitivity and positive controls. Do you have exposure and membership-separation results for the over-memorized checkpoint, or another setting with independently established instance-level memorization? Under a fixed model and corpus, does increasing canary repetition produce detectable separation?

  3. Meaning of the DP comparison. In Table 12, does ϵ\epsilon describe the complete DP training procedure or a mechanism executed separately for each query? If the comparison concerns repeated inference from a fixed DP-trained model, why is the bound composed over BB queries rather than analyzed through post-processing? How would the interpretation change under that analysis?

  4. Independence of the defense evaluation. Appendix F acknowledges overlap between classifier training and evaluation data, threshold selection on an overlapping utility subset, and prompt-level bootstrap resampling. What is the extent of this overlap, and are results available using disjoint canaries, a frozen threshold, and canary-level resampling? This would determine how much independent evidentiary weight the defense results should receive.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer yZEX (Part 1)

We thank the reviewer for a careful and constructive review. Every weakness and question led to a new experiment or to a correction in the paper. During the discussion period we fine-tuned twelve additional models (a six-level repetition sweep with a matched reference, two second-seed replications, and a second design with two seeds and its own reference) and re-evaluated the primary 1.5B target and the 7B checkpoint with new control panels. Section and table numbers below refer to the revised PDF, in which new text is blue.


W1 and Q1. Research gap and relation to existing audit protocols

The reviewer is right on both points, and we have changed the paper accordingly.

The sequential test (Section 6.3). The test is our own construction. The revision states this in the abstract, the introduction, and Section 6.3. It also states what the test's two hypotheses actually are: "this text was generated by the target" versus "this text was generated by the reference". We no longer present its failure as a failure of likelihood-ratio membership inference. We did not run multi-shadow-model LiRA and say so; the per-example calibrated statistic in our audit is exposure evaluated against non-members, a single-reference ratio.

Published protocols (Appendix A.6, Table 6). We reviewed the protocols the reviewer names and the main extraction studies of code models. We could not identify a published canary protocol that prescribes a target-versus-base ratio on inserted canaries alone. The Secret Sharer ranks each canary against same-format candidates, and LiRA calibrates on shadow models. The member-only reading was a shortcut of our own audit. We therefore do not claim that a specific published conclusion changes under our controls, and the revision no longer suggests it.

What the paper now claims. Three measurements that we did not find in prior work:

  1. The size of the template term, on the primary 1.5B target (Table 15). Never-inserted strings that share the canary templates gain 49.5 nats over a matched no-canary reference, exactly as members do. Never-inserted strings from held-out templates gain 0.6 nats.
  2. A condition on the non-member control (Table 15). With same-template non-members the AUC is 0.4855. With template-disjoint non-members it is 0.962, for a model with no per-canary signal. Randomly inserted canaries are the recommended remedy for member/non-member shift; our result shows the remedy only works when the never-inserted candidates share the inserted canaries' templates.
  3. A gap between set-level and per-canary extraction (Table 16). From a model at 128 insertions, sampling recovers 615 of 1,000 inserted secrets, and none of 3,200 samples contains a never-inserted secret. Yet a prompt returns its own canary's secret only 0.18% of the time, and 90 of 100 prompts built from never-inserted canaries elicit an inserted secret, because canaries share prompt contexts. This bears on lookup-based extraction studies, which report member-only rates.

We would be grateful if the reviewer could confirm whether this narrower statement of the contribution addresses W1 and Q1.

AuthorsOpenReview ↗

Reply to Reviewer yZEX (Part 2)

W2 and Q2. Positive controls and a controlled repetition sweep

We ran the experiment the reviewer describes: fixed architecture and corpus, varying repetition, with instrument-specific positive controls.

Repetition sweep (Section 6.5, Table 4; Appendix E.3). Qwen2.5-Coder-0.5B, 50K clean documents, 2,566 canaries each inserted R times, and a matched no-canary reference.

RExposure AUC, literalparaphrasesemanticSequential test TPR / FPR (%)
10.5180.4320.526100.0 / 99.0
40.5320.4780.55199.5 / 100.0
160.6390.6370.605100.0 / 100.0
641.0001.0000.91599.5 / 100.0
1281.0001.0000.99799.5 / 97.0

Exposure detects instance-level memorization once it is strong, so the null on the primary target is a controlled null. The sequential test fails at every level.

Is this one run per level? No longer. A second training seed at R = 16 and R = 64 changes no AUC by more than 0.018, and a second non-member seed by at most 0.010 (Table 13).

Is the floor confounded with the canary share of the corpus? In the sweep the share grows with R (Table 12). We therefore trained a second design that places all repetition levels in one model, with 200K clean documents and canaries at 8% of documents, under two training seeds (Table 14). AUC is 0.52 to 0.58 at R ≤ 4 and rises monotonically to 0.87 to 0.96 for literal and paraphrase canaries at R = 64. The chance-level floor at low repetition holds in both designs. The level needed for full separation does not, and the revision says so rather than quoting one threshold.

The over-memorized 7B checkpoint (Appendix E.3, Table 17). Exposure AUC is 0.569. The sequential test flags 100% of members and 100% of non-members. We traced the training configuration: each canary was inserted once, and the low training loss came from 700 clean documents seen through overlapping windows. We have withdrawn the label "over-memorized".

An error this exposed. The checkpoint's extraction hits (10.25% for members, 9.89% for non-members) are all vulnerability canaries, with none on template-disjoint non-members. The cause is in our panel: 142 of 283 non-member vulnerability secrets are identical to inserted ones, so that type cannot test membership. The disjointness check in the submitted version was wrong for that type. The literal, paraphrase, and semantic types share no secret, and the central result is unaffected (Appendix E.4). We report this correction openly.

We would appreciate the reviewer's confirmation that these controls address W2 and Q2.


Q3. Meaning of the DP comparison

The reviewer is right. For a DP-trained model, ε describes the training procedure, and every query to the fixed model is post-processing; the guarantee does not compose over B queries. Our table compared mismatched objects. It has been removed, and Appendix E.7 explains why. We trained no DP model and the revision makes no DP comparison.

We hope this resolves Q3.

AuthorsOpenReview ↗

Reply to Reviewer yZEX (Part 3)

Q4. Independence of the defense evaluation

The reviewer's doubt was justified.

Extent of the overlap. The run metadata of the original evaluation were not retained, so we cannot report the overlap from logs. The code indicates it was complete: all 32 evaluation canaries were among the classifier's training positives, and the threshold was calibrated on the same 164 HumanEval problems used to measure utility.

Re-evaluation as requested (Appendix H.1, Table 23). Canary-disjoint splits (460 to train the classifier, 240 to choose the threshold, 432 to evaluate), a frozen threshold, and canary-level resampling, on three sweep models.

RWorkloadNo defenseFilterLearned refusalFilter minus learned (pp) [95% CI]
16W42.60%2.60%2.53%0.066 [0.016, 0.140]
64W43.22%3.21%3.19%0.017 [0.000, 0.053]
128W44.00%3.95%3.97%−0.017 [−0.052, 0.000]

Out of sample, the learned refusal removes at most 0.07 percentage points relative to the filter. The earlier results carry no independent evidentiary weight. The defense is no longer listed as a contribution, and the original tables are labelled as in-sample.

We would be grateful to know whether this answers Q4.

We are grateful for the reviewer's time. Three of the reviewer's points led us to correct our own paper, and we believe the revision is more accurate for it. We respectfully ask the reviewer to consider whether the added controls and the narrowed claims change the assessment of the paper, and if so, whether the rating could be revisited.

Thank you again for your thoughtful and constructive review. Since the discussion closes on 18 November, we would like to briefly summarize how we have addressed each of your concerns in the revised manuscript.

W1 & Q1 (research gap and audit protocols). We clarified that the sequential test is our own construction, not a published membership-inference protocol, and narrowed our claims accordingly. New controls show that never-inserted strings sharing canary templates gain 49.5 nats, comparable to members, while held-out templates gain only 0.6 nats. Same-template non-members yield AUC 0.4855, versus 0.962 for template-disjoint controls (Table 15). We also distinguish set-level extraction from per-canary recovery (Table 16).

W2 & Q2 (positive controls and repetition). We conducted a controlled repetition sweep with matched references and additional seeds. Exposure AUC rises from near chance at low repetition to 1.000 at R = 64 for literal and paraphrase canaries, while the sequential test remains ineffective (Table 4). A second fixed-canary-share design confirms the low-repetition floor (Table 14). We also corrected the characterization of the 7B checkpoint and disclosed the non-member vulnerability-canary overlap.

Q3 (DP comparison). We agree that our comparison conflated training-level differential privacy guarantees with repeated queries to a fixed model. We removed the misleading comparison and clarified the distinction in Appendix E.7.

Q4 (defense evaluation independence). We identified overlap in the original evaluation and conducted a new canary-disjoint assessment with a frozen threshold (Table 23). The learned refusal provides at most a 0.07 percentage-point reduction relative to the filter. We have withdrawn the defense contribution and explicitly labeled the original results as in-sample.

These revisions include substantial new experiments, corrections, and narrower claims directly motivated by your feedback.

We would be grateful if you could review our updated responses and manuscript and let us know whether your concerns have been resolved. If the new evidence changes your assessment, we would sincerely appreciate your reconsideration of the rating before the discussion closes.

Thank you again for your time and valuable guidance.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of the PDF revision for the three reviewers, AC, SAC, and Program Chairs

We thank the reviewers. All three asked for positive controls for exposure and the sequential test, and for a repetition sweep. We ran both, plus a template-disjoint control on the primary model. New text in the revised PDF is blue.

New results

  1. Template learning on the primary 1.5B target (Section 6.5; Appendix E.3, Table 15). Never-inserted strings that share the canary templates gain 49.5 nats over the matched reference, exactly as members do. Template-disjoint strings gain 0.6. AUC is 0.4855 against same-template non-members and 0.962 against template-disjoint ones.
  2. Repetition sweep (Table 4): 0.5B, R = 1 to 256, matched reference. Exposure AUC for literal canaries is 0.52, 0.53, 0.64, 1.00, 1.00 at R = 1, 4, 16, 64, 128. A second training seed moves no AUC by more than 0.018, a second non-member seed by at most 0.010 (Table 13).
  3. A second design with all repetition levels in one model (canaries 8% of documents, two seeds; Table 14). AUC is 0.52 to 0.58 at R ≤ 4 and rises monotonically to 0.87 to 0.96 at R = 64.
  4. Extraction control (Table 16). Sampling recovers 615 of 1,000 inserted secrets at R = 128, with none of 3,200 samples containing a never-inserted secret. A canary's own secret is returned for 0.18% of prompts, because canaries share 16 prompt contexts.
  5. Out-of-sample defense evaluation (Appendix H.1) and a comparison with published protocols (Appendix A.6).

Corrections to the submitted version. Four statements did not survive the new controls.

  1. The 7B checkpoint was not "over-memorized": each canary was inserted once and exposure AUC is 0.569. Its extraction hits are all vulnerability strings that the non-member panel shares (Table 17).
  2. The panel-disjointness check was wrong for vulnerability canaries: 142 of 283 non-member secrets equal inserted ones. Literal, paraphrase, and semantic canaries share none, so the central result is unaffected.
  3. The learned refusal does not beat the content filter out of sample (at most 0.07 percentage points). It is no longer a contribution.
  4. The DP comparison table is removed; Reviewer yZEX is right that queries to a DP-trained model are post-processing.

Reframed. The sequential test is presented as our own construction (Section 6.3). The sampling model behind the Clopper–Pearson intervals is stated (Section 6.4).

Unchanged. The primary experiment and all of its numbers; we re-ran it and reproduced AUC 0.4855.

Not done. A reference trained on the same templates with disjoint secrets; a repetition sweep at 1.5B scale; multi-shadow-model LiRA; the set-level extraction measurement on the primary target. Each is stated in Appendix F.

Scale of the additional work. Twelve additional fine-tuned models (six sweep targets and a matched reference, two second-seed replications, two mixed-repetition targets and their reference), plus re-evaluation of the primary 1.5B target and the 7B checkpoint with new control panels.

We would be grateful if the reviewers could tell us whether these additions resolve their concerns, and whether any further experiment would affect their assessment. We will run whatever is feasible before the discussion period closes and will say clearly what is not.

AuthorsOpenReview ↗

Official Announcement by The Authors

Dear Chairs and Reviewers,

Thank you for your reviewing service. The authors have provided summaries, rebuttals, and clarifications on November 14. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your current recommendations.

Thank you.

AuthorsOpenReview ↗

Note to the Area Chair

Experimental concerns: resolved, by the reviewers' statements.

  • Reviewer yZEX: the repetition sweep "addresses my concern about the sensitivity of the exposure measure", and the same-template versus template-disjoint comparison is "informative".
  • Reviewer 1VKn: "I have no more concerns."
  • Reviewer 53pe: "extensive supplementary experiments" with "notable transparency"; no technical objection to any of them.

No reviewer has identified an unsupported claim or a technical flaw in the revised paper.

The two remaining objections.

  1. Reviewer yZEX: which conclusions drawn with established protocols would change? We have answered with a concrete case (follow-up comment, Table 16). Under the standard discoverable-extraction protocol (prompt with the training prefix, count exact reproduction), a model yields "1 of 100 secrets memorized" while it in fact reproduces 615 of 1,000, and 90 of 100 prompts for never-inserted examples eliciting a training secret. For rank-based exposure and LiRA we state plainly that no conclusion changes; the paper measures the cost of departing from them and the condition under which the canary remedy holds.

  2. Reviewer 53pe: the revision exposes problems in the original draft, narrows the scope, and so needs a new round of review. This is a procedural objection, not a technical one. The narrowing was requested by the same review (its Questions 1 and 4 and Weaknesses 2 and 5). The corrections affect one of four canary types, an auxiliary checkpoint, and two appendix items. The primary experiment was re-run and reproduces AUC 0.4855, and the three central findings are unchanged. Two of the four corrections were found by the authors with the controls the reviewers requested, and were reported unprompted.

What we ask. We ask that the paper be judged on its revised content: twelve additional trained models, a template-disjoint control on the primary model, replication across seeds and designs, and claims narrowed to what the evidence supports. We believe a submission should not be rated lower for correcting itself during review than it would have been had it arrived in its corrected form.

We are grateful for the Area Chair's time and for the reviewers' engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.