acceptodds
Under review as a conference paper at ICLR 2027

Extractable Memorization From First Principles

Abstract

Recent work on extractable memorization in language models suffers from two contrasting validity problems. Some studies overstate extraction, for example, by using sequences too short to distinguish memorization from predictability. Others imply that extraction is unreliable evidence because models can reproduce real-world text they weren't explicitly trained on. Both overlook what makes a valid extraction claim: the model must generate a training sequence with high enough probability to indicate memorization. To determine what's high enough, one has to perform a matched comparison, measuring generation probabilities of training and comparable non-training sequences. Since non-training sequences can't have been memorized, their probabilities are a baseline for predictability; exceeding this baseline is evidence of memorization. We formalize matched comparisons with (1) a conformal test calibrated to a chosen false-positive rate when sequences are sampled from populations, and (2) a single-document census that calibrates against a matched non-training document. Matched comparisons enable rigorous, calibrated memorization claims and clarify where prior setups have validity issues. On Wikipedia, OLMo 2 32B reproduces non-training 10-token suffixes roughly 24% as often as training ones: that share reflects false positives, not memorization. For Llama 3.1 70B on books, calibrated census thresholds reach as low as , supporting memorization claims for sequences no feasible sampling budget would extract. We therefore refine "extractable memorization" to require both a valid memorization claim and near-certain generation within a realistic budget.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
memorization, extraction, measurement validity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KhdGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that a model reproducing a training sequence is not, by itself, evidence of memorization: one has to compare against how often the model reproduces matched text it was not trained on. It frames any extraction method as a fixed procedure (model, decoding policy, prompt rule, score, verification rule) and proposes setting the verification threshold from the scores of matched non-member "controls".

Critical strengths
  • The paper addresses an important problem.
  • The paper is solid in math.
Critical weaknesses
  1. The significance of the paper is very unclear.

    • As far as I understands, the paper aims to re-define ``extractable memorization'': but the focus of paper totally deviates from `extractable memorization'', where preivous work use this term to determine whether a (training data) sentence is extractable by some prefix or not. What the paper actually did is bascially membership inference attacks (MIA), which ****have been studied for almost 10 years.
    • In this context, the authors did not differentiate their membership inference atttack method with baseline methods, and therefore the evaluation of the proposed method is very weak. As far as I understand, formulation of MIA using statistical tools has been very popular, e.g. [1] [2].
    • There is almost no related work. Authors are strongly to encouraged to include a dedicated ``related work'' section and distingush your paper with others.

    For the reasons above, the paper falls under a category that neither training data extraction nor MIA. The implication to the memorization community is embrassingly unclear.

  2. Method soundness

  • For consus experiments, the census threshold seems to be by a manual, one-sided decontamination step.
  • The Wikipedia experiment splits members and non-members by date (2021–23 vs 2024), the same design as WikiMIA in [1]. However, Duan et al. 2024 and Das et al. (“Blind Baselines Beat Membership Inference Attacks”) showed this is inappropriate, since date splits are confounded by distribution shift, so the “matched” controls may not actually be matched.
  1. This paper is very hard to follow.
  • The paper uses terms which don't follow standard in memorization literature. For example, census'', population'' are almost never seen in memorization works. Population has been explored in a previos work [3] by Maini et al, and Maini et al has described as a ``dataset infernce attack".
  • the meaning of many words are not clear, e.g. “predictability” in introduction section,
  • Unnesscary bold text across section 2, where the uahthors boldize many terms in LLM decoding steps, which are elementary (and do not exhibit high relavance to the focus on this paper) for reviewers.
  • As a follow up of concern 1, the limitation of preivous work is not clear to me. please provide addtional contexnt.

I belive the paper needs a thorough polish to meet the standard of ICLR.

[1] Shi, Weijia, et al. "Detecting pretraining data from large language models." International Conference on Learning Representations. 2024.

[2] Zhang, Jingyang, et al. "Min-k%++: Improved baseline for pre-training data detection from large language models." International Conference on Learning Representations. 2025.

[3] Maini, Pratyush, et al. "LLM Dataset Inference: Did you train on my dataset?." Advances in Neural Information Processing Systems 37 (2024): 124069-124092.

Questions for authors

My main concern is concern 1 above. Please try to answer:

  • How do this paper distinguish from MIA? and can you provide any comparsion between your method and state-of-the-art MIA / DIA conceptually? And experimental results of these comparsions are neccessary.

For concern 2, please answer my two subconcerns directly. For concern 3, please justify the reasons of adopting non-standard terms used in major memorization works.

Reviewer_1CDvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a statistical framework for measuring memorisation and extraction in LLMs with a certified false-positive-rate (FPR) guarantee. The key idea is to subtract a matched non-member baseline: how much more often this happens for members than for distributionally-matched non-members, constructed using pre-cutoff data as members and post-cutoff data as controls.

Contributions:

  1. A population-level conformal test (Prop. 1) that certifies a per-candidate FPR of αN=K/(N+1)≤α\alpha_N = K/(N+1) \le \alpha via a rank/exchangeability argument, yielding a calibrated memosization rate M^=G^−αN\hat{M} = \hat{G} - \alpha_N, with one control pool reusable across many candidates.
  2. A book-level "census" that audits copyrighted works using same-author, post-cutoff books as controls and a per-book threshold τ(book)\tau(\text{book}) (Eq. 30).
  3. A distinction between memorised sequences (clearing the calibrated threshold) and extractable ones (also generable within a sampling budget).

Validation A Wikipedia population experiment (N=5,000N=5{,}000 members vs. 5,000 post-cutoff controls) on Llama 3.1 models, with 50-token prefixes and 10- or 50-token suffixes, scored by q(s∣p)q(s\mid p) under top-k=40k=40; plus a census over a small set of author-matched books (four pairs collected, three reported), with manual/LLM-assisted decontamination of the control tail.

Critical strengths
  • The conformal certified-FPR test (Prop. 1) is a methodological improvement over the field's usual raw extraction rates; subtracting a matched non-member baseline is the right idea, and it rests on a minimal assumption (exchangeability).
  • Separating memorised from extractable is an operationally meaningful distinction that the field often blurs, and the paper formalises it clearly.
Critical weaknesses
  • The paper's concern regarding "10-token suffixes roughly 24%" and a primary experimental condition is resting on the 10-token suffix. However, 50-token suffix has been a standard since Carlini et al. (2023) (https://openreview.net/forum?id=TatRHT_1cK), and recent studies use far longer suffixes (up to 2k–8k tokens). The authors' own Fig. 3 shows the false-positive floor is already near-zero at 50 tokens. The short-suffix concern the paper foregrounds is therefore one that standard practice has already avoided.
  • Membership is treated as binary, therefore overlooks frequency, which a large body of the work identifies as the dominant driver for memorisation. This could be the most natural explanation for the census thresholds τ(book)\tau(\text{book}) varying by ~6 orders of magnitude.
Questions for authors
  • Can the authors report how memorisation rates and the per-book thresholds τ(book)\tau(\text{book}) relate to training-data frequency?
Reviewer_1raKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits when verbatim or near-verbatim generation provides evidence of memorization in language models. Its central proposal is to evaluate candidate sequences against matched non-member controls using the same extraction procedure, thereby accounting for generation attributable to general predictability or information supplied by the prompt. For population experiments, the authors instantiate this comparison as a conformal hypothesis test with a finite-sample, marginal false-positive guarantee under exchangeability. For individual documents, they propose a census procedure that sets a threshold from the maximum score in a decontaminated control document.

Experiments use OLMo 2 models on pre and post-cutoff Wikipedia passages and Llama 3.1 models on four same-author book pairs. The results illustrate how suffix length, control contamination, and decoding policy affect the conclusions drawn from generation scores. The paper also distinguishes evidence of memorization from practical reproducibility by incorporating a query budget and reporting sampling and deterministic-recovery results. The main contribution is a methodological framework for interpreting extraction evidence.

Critical strengths
  • The paper offers a useful way to think about evidence of memorization. I appreciate the emphasis on applying the same extraction procedure to training examples and comparable non-training examples. This helps clarify why successfully generating a sequence may not, by itself, provide strong evidence of memorization, especially when the sequence is short or the prompt reveals part of the answer. Turning this comparison into an explicit calibration procedure gives researchers a practical way to make their evaluation criteria more transparent.
  • The experiments illustrate why these evaluation choices matter in practice. The analyses of suffix length, decoding policy, model size, and control contamination provide useful context for interpreting extraction results. I found the decontamination analysis particularly informative: removing a small number of contaminated controls substantially changes the reported memorization rate. The distinction between evidence of memorization and recovery within a realistic budget is also helpful, and I appreciate that the appendix considers deterministic recovery in addition to random sampling.
Critical weaknesses
  1. Some interpretations of the reported rates need further justification. My main concern is the step from identifying unusually high scores relative to non-members to estimating how much training data has been memorized. Proposition 1 provides a guarantee for non-members under the stated assumptions, but it is less clear how that guarantee applies to training examples that have not been memorized. In particular, I would find the interpretation of G(τ)−FPR(τ)G(\tau)-\mathrm{FPR}(\tau) as a lower bound on latent memorization more convincing with an explicit assumption or derivation connecting these quantities. This also affects the statement that approximately 24% of the apparent 10-token extractions are false positives. The reported ratio shows that the control generation rate is about one quarter of the member generation rate, but this does not directly establish that one quarter of the generated member sequences were not memorized. The discussion in Section 4 and Appendix C.5 is more qualified, and bringing the headline wording into closer agreement with that discussion would strengthen the paper.

  2. The quality of the matched controls remains an important source of uncertainty. The authors clearly recognize that constructing suitable controls is difficult, and the decontamination analyses are a valuable step. However, a passage from a post-cutoff document may still contain short suffixes that appeared in training. Since Equation 2 defines membership at the suffix level, I would like to see clearer evidence that the controls supporting the 10-token result satisfy that definition. For the book experiments, matching by author and genre is a reasonable starting point. Still, exceeding the maximum score in one control book does not establish how unusual that score would be across other comparable non-training texts. The threshold sensitivity analyses are helpful, but some uncertainty about control selection remains. I would therefore be more comfortable with the reported percentages being described as coverage above the chosen control threshold, unless additional evidence supports their interpretation as memorized coverage.

Questions for authors
  1. Could you clarify the assumptions under which the difference between member and non-member generation rates provides a lower bound on latent memorization? In the same context, how should readers interpret the reported 24%: as a ratio of generation rates, or as a claim about the fraction of member outputs that were not memorized?

  2. Were the 10-token control suffixes checked for occurrence elsewhere in the training corpus, particularly the 91 controls reproduced in Figure 3? It would be helpful to know whether suffix-level verification changes the comparison.

  3. For greedy decoding, Appendix D.3 computes the “certified FPR” as (m+1)/(N+1), using the observed number of reproduced controls. Could you explain how this relates to Proposition 1, where the target level is fixed before observing the calibration scores?

  4. Have you examined how a threshold derived from one control book behaves on another comparable non-training document, or how member coverage changes when a different control is used? Such evidence would help assess the robustness of the census interpretation. If this has not been evaluated, could the coverage claims be qualified more explicitly as relative to the selected control?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.