acceptodds
Under review as a conference paper at ICLR 2027

EXPOSURE EMBEDDINGS FOR CORPUS AUDITING AND DOWNSTREAM EVALUATION

Abstract

Author-conditioned title likelihood offers a behavioral signal for corpus auditing, but its interpretation depends on document identity and the construction of comparison papers. We study this signal using Comma v0.1 and the arXiv component of Common Pile. Starting from metadata-matched paper pairs, an independent identifier audit retains 1,222 pairs whose positive paper is present and negative paper is absent from that component. On these pairs, Comma’s title score achieves AUC 0.6523 with matched distractors and 0.6139 with lexically similar distractors, compared with 0.5316 and 0.5154 for GPT-Neo-2.7B. These associations concern component presence, not verified membership in the complete training mixture. Replacing the correct author list with a within-year, one-to-one derangement reduces Comma’s AUC by 0.0369 and 0.0337 under the two distractor conditions; the paired intervals exclude zero, while GPT-Neo changes by less than 0.0006. Comma’s exploratory within-year AUCs range from 0.642 to 0.663 in 2020–2023, compared with 0.455–0.569 for GPT-Neo. In a separate citation-prediction cohort, holding out query papers gives SPECTER2 AUC 0.90756 and fusion AUC 0.90770; the paired 90% interval for the difference is [−0.00095, 0.00119]. The results distinguish behavioral discrimination of an audited corpus component from incremental utility in one downstream task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
exposure embeddings, corpus auditing, author-conditioned probing, pretraining data detection, scholarly document embeddings, citation link prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YNUYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a log-likelihood-based method for detecting the association between paper titles and content, which can be used to detect whether an LLM has seen the corresponding data during training. This study is not about membership inference, nor does it aim to design source attribution. Finally, in the experimental section, this paper honestly reports both positive and negative results.

Critical strengths

Through empirical study, this paper honestly reports both positive and negative experimental results.

Critical weaknesses
  1. There are too many traces of AI editing; I strongly doubt human involvement in this paper.
  2. This method is similar to [1]; please refer to it and learn more about the LLM-based MIA background.

References:

[1] Detecting Pretraining Data from Large Language Models, ICLR 2024

Questions for authors

See Weakness.

Reviewer_UoiJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a test of whether a language model's title predictions reflect which scientific papers were in its training corpus. The model is prompted with "Which paper has authors [author list]? Answer:", and four candidate titles, the true one and three 'distractors'. The score is the probability of the true title. They used Comma v0.1, an open language model released with Common Pile (Kandpal et al., 2025), a public corpus of public-domain and openly licensed text. As the corpus is public, papers in its arXiv component can be compared with matched papers that are absent from it. Comma separates the two groups better than GPT-Neo, with AUCs of about 0.65 against 0.53. Adding the correct authors help a little and wrong authors hurt the prediction. Adding the score to SPECTER2 does not help citation prediction.

Critical strengths
  1. The label audit was the most useful part for me. Over a hundred of the papers that title matching marked as absent are in the component when checked by identifier. I would have expected title screening to be more reliable, and a lot of training-data auditing relies on string matching. The authors fix the labels with metadata and drop the affected pairs whole, which keeps the matching.

  2. Using a model whose training corpus is public means the labels come from the corpus and do not have to be guessed. The pairs are also matched on year, field and citations, which reduces the usual differences between papers that are in the corpus and papers that are not.

Critical weaknesses

I would like to know what the four-title construction adds over the likelihood of the title alone. Removing the authors only lowers the AUC slightly, which suggests most of the signal comes from the title. I could not find a comparison with Comma's log-likelihood of the true title without distractors. A similar AUC there would make the method close to a standard likelihood test on a short string, and the contribution would then be mostly the setting and the labels. Without this comparison it is hard to judge what the method offers, and that is the main reason for my score.

  1. I am also unsure what the model "knows" when it picks the right title. It may have seen this exact paper, or other papers by the same authors. It may have seen the title cited in other papers' reference lists. Another possibility is that openly licensed papers differ from their matches in ways any model picks up. GPT-Neo being above chance suggests this last explanation plays some role, and the authors themselves say GPT-Neo should not be treated as a chance-level null. The Figure 2 caption, which says correct author-title associations drive the effect "rather than author tokens alone", goes further than §3.5 and §6. I do not expect all of this to be settled, but I would like to know which explanation the authors favor.
Questions for authors
  1. How does Comma's log-likelihood of the true title (summed or per token, no distractors) compare with the four-title score on the same pairs? This is the question most likely to change my score.

  2. Which of the explanations above do you think accounts for most of the effect? You scanned the whole component, so could you check whether positives whose title also appears in other papers' references score higher? That would help tell apart learning a paper from its own text and learning it through citations.

  3. What did you expect the two-number vector (score and margin) to capture about citations? I do not see why two papers with similar title scores should cite each other, so I may be missing the idea behind the downstream experiment.

  4. Which Comma checkpoint was used (1T or 2T), and what is the exact prompt without authors?

Reviewer_JkRTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores "behavioral corpus auditing", exploring whether a language model gives higher likelihood to a specific paper title when the list of authors, and the multiple factors that might make that true, getting at whether author-conditioned title likelihood can distinguish papers that are present versus absent in a known corpus component.

Critical strengths

The paper is careful about controls, e.g. harder distractors, shuffled authors, and matched negatives.

Critical weaknesses

...but I am having trouble seeing the significance of the contribution. Practically speaking, the results are not strong: corpus-presence discrimination is fairly modest, and the behavioral features add essentially no value in the downstream citation-prediction experiment. The paper presents a careful empirical study, but I am not convinced that the proposed approach is sufficiently novel or broadly useful methodology.

Questions for authors

Why focus specifically on author-conditioned title likelihood? What makes the author/title relation preferable to other possible metadata relationships for corpus auditing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.