acceptodds
Under review as a conference paper at ICLR 2027

Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models

Abstract

When a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data. Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not. This paper removes the guessing. Two model families, OLMo-2 and Pythia, publish their pretraining corpora, and a public index over those corpora returns the exact number of times any sentence appeared in each. Those counts make three questions answerable directly. How many copies does it take before a model's loss carries a trace of exposure? Can that trace be separated from the ordinary fact that some sentences are easier than others? And how much of the reported success of membership tests comes from the way their control sentences were built? The answers form a pincer, closing from two sides. At the duplication levels ordinary text actually has, five models from 1B to 13B parameters carry at most a faint trace of their own exposure. We measure that trace with a design that reads the same sentence through two models, which cancels fluency and quality by construction, and it comes to a rank correlation near −0.08. Where the trace does become strong, above roughly a thousand copies, the two corpora agree on which sentences those are, because they are the famous ones, so exposure can no longer be told apart from fame. Two further measurements show how apparent membership signal gets manufactured. A common way to build a non-member is to change one word of a member. The model does prefer the original, but the gap is the same whether the original appeared once or a hundred times, so what the model is rewarding is the author's word choice, not memory. Above a thousand copies the gap grows with model size on the twelve sentences we can test there, at the same boundary where the pincer closes. And swapping the controls for sentences that differ from the members in register moves a detector from 0.83 to 0.94 AUC. We release the sentence banks, counts, and code.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
membership inference, training data detection, language models, data duplication, pretraining corpora, evaluation confounds

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JhrHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies membership inference in LLMs, in particular observed that loss is not an discriminative features on low-frequency training instances, a well-established findings in the membership inference literature, and find the loss is discriminative when the frequency is high (i.e., likely when models already verbatim memorize the sequences), however, at which point the members is confounded by near-duplicate edits, again, another established finding in the memorization literature.

Critical strengths

None

Critical weaknesses

Novelty: Research question and observations have been well explored in the literature:

  • There is a rich literature on membership inference and rigorous methodologies to measure counterfactual memorization, and it appears that this work is on the journey of rediscovering some of these well known findings in the community. The research question around when is membership inference possible and the core observations regarding repetitions, near-duplications are well known in the literature.
  • While the core claim "Loss-based evidence of membership becomes strong only when a sentence has been duplicated on the order of a thousand times or more." is plausible, it is also trivial in some sense, at this repetition range, there is no need for membership inference as the model likely can verbatim output the members.
  • In general, there appears to be insufficient engagement with literature. This perhaps can be explained by a surprisingly short list of reference.

Presentation: Excessive use of vague and anthropomorphizing languages that does not meet the standard of scientific writing:

  • For a paper that studies membership inference, there is no mention of membership inference terminologies in the abstract, and in fact exact two mentions of "membership inference" in the main text, including one the related work. instead, the paper is filled with vague and anthropomorphizing phrases. Just use the opening sentences of the abstract as an example: "When a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data. Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not." What does "cheap to predict" mean here? And how does a probability model "finds" something cheap to predict? "Almost every published test" which published test? "of that inference" Which inference?
Questions for authors

Apart from addressing lack of novelty discussed above, it would be great if the authors can clarify a few claims made in the paper.

  • "so what the model is rewarding is the author’s word choice, not memory": What does memory mean here? Why reconstructing a particular author's word choice is not memory?
  • "The unit is a sentence, not a paragraph": Why sentence? Why not shorter n-grams? Why not more complicated syntactic structures.
  • "Membership evidence from loss is either weak or confounded, and the transition between the two regimes is not a place where a careful test can stand": What does it mean "not a place where a careful test can stand"?
Reviewer_t3xUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is about membership inference: given a sentence, can we infer whether it occurred in a language model’s (pre)training data? The experiments show that if we use the sentence’s loss under the model as a way to detect membership, then accuracy isn’t much better than chance (a finding reported in previous work), except for sentences with extremely high frequency in the training data. The paper provides evidence against a simple dose-response theory of memorization: seeing a sentence more frequently does not strongly predict its loss. It also provides evidence that minimally edited sentences are not necessarily equally natural controls and suggests that naturalness may be a confound in arguments about memorization.

The paper argues, like others, that variation in training-set duplication counts is a poor predictor of model loss at ordinary exposure levels. The authors repeatedly interpret this as evidence against membership inference using probability calculations. I think this doesn’t follow. The marginal effect of additional exposures may be negligible even when the effect of the first exposure is substantial. Their member vs. non-member comparisons do not resolve this issue, because they confound exposure with differences between sentences.

Critical strengths

The best idea in the paper is to try to control for as much as possible in pairing a member sentence with a non-member. This is done by drawing the non members from the same books as the members, hence controlling for author, time, length, etc., and then changing one word to a near-synonym. (The paper doesn’t explain how the register, level, and style are categorized. No complete examples are given.)

The paper extensively documents the cautions that the reader should apply in understanding the method and the results; this is at times tedious and repetitive, but overall it’s good practice. I especially appreciated the discussion of what kinds of future experiments would show that the paper’s findings are wrong.

Critical weaknesses

The introduction sets things up by talking about “two things missing from the conversation” about membership inference, claims that the paper provides them, but never says why these added ingredients (membership as a graded value rather than a binary, and what we might call — though the paper doesn’t — natural experiments where non-members are selected using an alternative corpus) make a difference to the current thinking on membership inference. The paper’s framing suggests fundamental flaws in past methods for doing membership inference, but rather than landing in a conclusion like “these methods are flawed” or “the findings from these methods are flawed” the paper instead introduces a new method and discusses its findings as if none of what came before matters. The paper never clearly states what its conclusions teach us beyond what was established in prior work.

The loss-based membership test seems to be a straw man. Why not evaluate Shi’s Min-K% Prob and Mattern’s full neighborhood attack on the new, verified-membership dataset? That would directly test whether the reported confounds invalidate the prior methods, rather than only challenging some of their motivating intuitions.

Line 70 — it sounds like the proposed test rests on the idea of comparing probabilities from the two different models. But if they differ in vocabularies/tokenization, these values are not directly comparable quantities.

Line 162 — “tests that harder” — this is another straw man. The sentence is constructed to make it sound like the choice was between using 6 novels or 6 novels plus a handful of other books. In fact the main design choice the authors have made is to sample a tiny number of sentences from novels to test their claims, and this is never defended.

The writing is disjointed and organized in a confusing way. Rather than explaining the existing findings, defining necessary terminology, stating a hypothesis, and then developing experiments to test the hypothesis, the paper jumps around among a bunch of methods choices with no clear flow (sections 2 and 3) and then a series of measurements and discussion (sections 4 through 8).

Section 5 is well-intentioned and correctly points out which finding shouldn’t be trusted. It hinges mainly on reporting rank correlations. A better and more conventional approach would have been to fit a linear mixed-effects regression predicting differences in model loss from differences in log exposure counts, accounting for book-level effects and distinguishing within-book from between-book relationships. This would yield interpretable effect sizes and confidence intervals, allow a direct comparison across model sizes, and reveal whether the relationship is consistent across books rather than driven by a few of them.

In Section 5, the authors square their Spearman correlation of approximately −0.08 and conclude that exposure accounts for less than 1% of the variance in loss differences. That is not generally a valid interpretation of squared Spearman correlation. It measures association between ranks, not the proportion of variance explained in the original loss measurements.

The paper is full of terminology that’s not standard or explained. It reads like project-internal jargon between the researcher and their favorite chatbot. It’s often metaphorical and colorful in a distracting and inconsistent way that makes it impossible to confidently understand the intended meaning. Some examples (not exhaustive):

Line 23 — what is “ordinary text”? Training data for LMs is often deduplicated.

Line 69 — These corpora were not assembled independently. Olmo was more than a year later, and inspired heavily by the Pythia effort. The corpora have overlapping sources (wikipedia, arXiv, common crawl, books).

Line 73 — “loss based membership” is confusing terminology. Membership is the construct, loss is one way to estimate it.

The paper uses “membership” and “exposure” interchangeably; this just adds confusion.

Line 127 —- what is an index that is “down”?

Line 130 — what is a “bank”?

Figure 1 — what does “moves” mean? Why use causal language here? It’s misleading.

Line 140 — “the two corpora barely agree” — this gives a dramatic, bold impression while saying nothing. It’s based on an analysis using 747 sentences, a tiny drop in the bucket no matter how we count.

Line 143 — “member sentences that carry both counts” — what does “carry” mean?

Line 183 — “The twelve famous-line edits were written one at a time rather than drawn from a table” — what does “drawn from a table” mean?

Line 261 — what does “cancel” mean?

Line 266 — what does “taste” mean?

Line 347 — “The flatness has an edge, and it falls where the rest of this paper says it should” — I don’t understand the metaphor here

Line 477 — “jaws”?

Questions for authors

My main concern is that the paper's evidence concerns how loss varies with duplication count, whereas membership inference is fundamentally about distinguishing zero exposures from one or more. Could you explain how your findings bridge this gap and what they establish about membership inference beyond the conclusions of prior work?

Reviewer_5nhgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyses the link between the perplexity of a sentence for a LLM, and the number of occurrences of this sentence (its exposure) in its training set. In particular, the goal is to distinguish the effect this exposure from the effect of other linguistic properties or différences introduced by the construction of non members.

In the experiments, Infinigram is used to count the exact number of occurrences of each sentence in 2 corpora, the one of OLMo-2 and the one of pythia for which training and test sets are documented. The paper compares for certain given sentences, the difference in loss between 2 models trained on each copus, with respect to the difference in exposure. An analysis per book helps reducing the influence of general preferences of the models for certain texts.

The results show a weak association between the differences in exposure and the differences in loss across models for ordinary sentences (with a few repetitions). The models generally prefer the original sentences to their variants, but this advantage does not clearly increase with the number of copies in this group. Sentences that are highly frequently repeated are better distinguished from their variants using loss. Finally, detection performance varies depending on the non members used, which shows the importance of their construction.

In conclusion, this paper provides a complementary analysis of the interpretation of MIA for LLMs. Its main value lies in the combination of measured exposure counts and a comparison of the same sentences across models, rather than in uncovering the role of duplication or evaluation biases.

Critical strengths
  • the question of the correlation between the exposure of a sentence and its loss is important in MIA evaluation. Exact counts of occurences enable a more detailled analysis than the sole distinction between members and non members.

  • The comparison of the same sentences between 2 models, with counts of occurences in their respective corpus, is an interesting methodology. It avoids constructing non members for the central analysis, which could introduces bias, and takes into account differences between books using by adjusting for the average exposure and loss differences between the models within each book.

  • the presentaiton is clear and the paper recognises several limits.

Critical weaknesses
  • Kandpal et al., which is mentionned in the related work section, already studies detection performance in function of the deplication level (beyond the sole case of unique sequences). The bias in the construction of MIA benchmarks are also documented, eg, by Duan et al. and Das et al (also cited in the paper). The most distinctive contribution seems to be the comparison between models, and its Added values should be made modre explicit.

  • This comparison for the same sentences between models does not fully isolate the effect of number of occurrences in the training set. The models also differ by their training and tokenization. The counts of full sentences do not capture additional exposure the n-grams constituent of the sentences in the corpus, which may explain part of the differences between models. The paper mentions certain limitations, but their impact on the interpretation of the correlations is unclear.

  • The better separation observed between on the 12 famous sentences from their edited variants, versus ordinary/less frequent sentences, seems not really enough to establish a threshold around 1000 occurrences. Experiment with a larger set of frequent sentences would help, and would also show if detection improves progressively with the number of occurrences.

Questions for authors
  • Could you explain explicitely, beyond the duplication analyses by Kandpal et al., the benefit of comparing the same sentences across models using the counts of occurrences measured in each respective corpus ?

  • Regarding Figure 4, did you look if the loss between an original sentence and the variant of that sentence depends on the frequency of ngram fragments around the replaced word ?

  • Did you look at membership scores other than the average per token loss of the sentence, such as Min-K% or scores adjusted using a reference model or neighboring sentences ? or should the conclusions be restricted to the score studied ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.