acceptodds
Under review as a conference paper at ICLR 2027

SpanAudit: Benchmark Contamination Detection as Span Retrieval

Abstract

Benchmark contamination can cause evaluation scores to reflect prior exposure rather than the capabilities that a benchmark is intended to measure. Corpus-side detection can identify benchmark-specific information in training data before model training, but practical methods commonly rely on long exact matches, predefined retrieval units, or both. Rewritten benchmark content may no longer contain a sufficiently long contiguous -gram, while the relevant passage may occupy only part of a long document or cross a segmentation boundary. To address these coupled challenges, we introduce **SpanAudit** a query-conditioned method for approximate retrieval over variable-length source spans. **SpanAudit** keeps each complete benchmark item as a query, uses banded MinHash–LSH to identify spans with likely token-set overlap, and shares sketch computation across overlapping intervals instead of enumerating the quadratic span space. It then merges and localizes the retrieved candidates to produce inspectable evidence for downstream contamination adjudication. Across controlled experiments with progressively stronger paraphrases and increasingly long surrounding context, **SpanAudit** achieves the highest or tied-highest planted-pair recall in every reported setting, while retaining favorable latency among high-recall retrievers. Audits of three previously decontaminated instruction corpora also recover confirmed evidence absent from the other methods, including repeated answer and reasoning exposure.

Primary area:
datasets and benchmarks
Keywords:
benchmark contamination, corpus decontamination, data curation, variable-length span matching, set similarity search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_PrtMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies corpus-side detection of benchmark contamination, where benchmark items or their answers appear in a training corpus. The authors argue that existing methods rely on long exact matches or fixed retrieval units, and so can miss rewritten content or passages that sit inside long documents. They propose SpanAudit, which uses each benchmark item as a query and searches corpus spans of any length using character 4-gram MinHash with banded LSH, while sharing computation across overlapping spans to avoid enumerating all of them. Retrieved candidates are merged and localized into evidence spans, which are then judged by LLMs. In controlled experiments with planted rewrites and increasingly long documents, SpanAudit reaches the highest or tied-highest recall in every setting, with competitive latency. Audits of three previously decontaminated instruction corpora also find remaining contamination, much of it including answers or reasoning.

Critical strengths

1. A solution to a known problem. It is well known that exact n-gram matching breaks under small rewrites, and that fixed units such as paragraphs or windows can miss passages that are shorter, longer, or split across boundaries. The paper addresses both issues at once by searching spans of any length, and does so without the cost of enumerating all possible spans.

2. Good results on long documents. SpanAudit keeps recall near 99.8% as the surrounding context grows from 512 to 4,096 tokens per side, while the embedding-based baseline drops from 97.89% to 72.53% (Tab. 2). It also runs several times faster than both Infini-gram and LLM-Decontaminator in this setting, which matters for corpus-scale audits.

3. Transparency and inspectable outputs. The authors release code, data, prompts, and the retrieved evidence, so the results can be checked and reproduced.

Critical weaknesses

1. The paper measures retrieval, not contamination, and does not use ground truth where it matters. This is my strongest concern. The paper itself states that SpanAudit "returns localized evidence rather than final contamination labels"; however, its results are framed as contamination detection. In the controlled experiments, the planted positions are known, but recall is measured only at document level: "the planted pair is recovered if any retained evidence belongs to its carrier document." Localization is a central claim, yet it is never evaluated against the planted spans, and no precision is reported. In the real-corpus audits, "confirmed" means two LLM judges agreed, with no human validation, and the second judge "does not recover first-stage false negatives." Exclusive evidence is defined relative to the union of the five methods, so contamination missed by all of them is invisible. Without ground truth, it remains unclear whether the retrieved evidence reflects actual contamination or merely lexical overlap.

2. The controlled benchmark is built around the proposed method. The rewrites are constructed to hit Jaccard targets "under token-set Jaccard computed after the same normalization and character 4-gram tokenization used by SPANAUDIT," with the hardest level at [0.4, 0.6). Since SpanAudit uses the same measure with a threshold of τ = 0.4, every planted rewrite is guaranteed to lie at or above its threshold. This setup favors SpanAudit and may explain why it achieves the highest recall in every setting.

3. The rewrites may not reflect realistic paraphrasing. Sec. 5.1 states that hard examples "extensively rephrase and restructure the item." Yet the only example shown (Tab. 8) builds the moderate and hard versions by inserting noise tokens, such as "Write a(ignore) very angry(ignore) letter," and "Remove every parenthesized boundary marker before reading." This is closer to an obfuscation attack than to a paraphrase. It would help to know how many rewrites follow this pattern, and how the results change on natural paraphrases.

4. Some claims are not supported by their citations. The introduction states that corpus-side decontamination "removes, downweights, or flags suspected documents before training or evaluation," citing [1] and [2]. However, [1] attempts to remove overlapping training text (only partly, due to a bug) and then flags benchmark examples as clean or contaminated, while [2] only analyzes the contamination level of evaluation samples after training. Neither paper downweights or flags training documents. The later claim of policies "ranging from recording or downweighting suspected exposure to excluding it" is given without any citation.


References

[1] Language Models are Few-Shot Learners (Brown et al., NeurIPS 2020)

[2] Llama 2: Open Foundation and Fine-Tuned Chat Models (Touvron et al., arXiv 2023)

Questions for authors
  1. How does benchmark contamination appear in post-training corpora? The introduction defines it as a case "in which benchmark-specific information appears in a pretraining or post-training corpus before evaluation," but post-training data is usually curated from selected tasks rather than scraped from the web.

  2. All planted rewrites have a Jaccard similarity of at least 0.4, which matches SpanAudit's threshold of τ = 0.4. How does SpanAudit perform on rewrites with Jaccard below 0.4, for example in a [0.2, 0.4) range, compared with the other methods?

Reviewer_GwkBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SpanAudit, a method for detecting benchmark contamination in LLM training corpora. The approach is based on localization of n-grams overlapping with benchmark data and effective search for spans with high Jaccard similarity to benchmark questions based on matching hash signatures. The method was compared against 4 baselines in a synthetic evaluation and on auditing real-world training datasets.

Critical strengths
  • Effectiveness and novelty. Authors aimed to propose a method for efficient n-gram overlap computation and localization of high-overlap spans in text. The proposed approach is reliable and, to my best knowledge, it is substantially different from other known methods. The method has a few parameters to tune, but seems to be simple enough to be practical.
  • Evaluation setup (in general). The overall approach to evaluating the method both in controlled synthetic scenarios and cross-comparison on real corpora is a good way to evaluate the quality. The missing details are discussed below.
Critical weaknesses
  • No ablations. The components of SpanAudit are extensively explained, but evaluation lacks any ablations to verify the importance and influence of them. In particular:
    • Which parameters are critical to the final performance? How should I choose d,b,rd, b, r, and other parameters? What are the tradeoffs? In general, I understand that they control the balance between time and precision, but I have no clue what is the sensitivity of the pipeline and how to choose them. In particular, which values were used for your experiments? How they were chosen? How easy/hard is to find good-enough set? How the method behaves depending on the number of hashes d? Do I need ~3 or ~30 of them?
    • One of the key algorithmic contributions of the method is localizing the interesting spans from T^2 possible intervals. How important is that to the final performance? What if we replace that component with something simplistic, e.g. sliding window? Sliding window is also computationally efficient and commonly used, so how much do you benefit from using span sketches instead?
  • Details of the experimental validation. I acknowledge the effort taken to validate the method, as both methods (synthetic datasets and real-world corpora decontamination) are creative and relevant. However, there are a few details here I'd like to note:
    • The (only) metric used in the first evaluation is recall@20. That seems like quite a lot. Essentially, to remove nearly all benchmark samples, we'd also remove 19x benign samples. That may be acceptable, but it would be nice to show also different values, starting from as low as 1. What about precision? Does it mean that precision is 5%? Precision-recall curves would help here.
    • How do the synthetic rewrites look like? I see only a single example of each strength in the appendix. Is that a representative sample? It doesn't seem to match the description in Section 5.1. Please show me random samples from those datasets. In particular, SpanAudit is inherently exact-match oriented and it should be blind to strong paraphrases, translations, and other forms of word-damaging augmentations. That itself is not a major problem, as efficient exact-matching methods are also valuable, at the very least as a complement to heavier semantic methods (see e.g. decontamination in "Nemotron 3.5 Super and Lightning" technical report), but that needs to be made clear. The more that SpanAudit works on rather short n-grams, so it has the potential to hold even against moderately strong paraphrases if it correctly handles spurious matches. Hence, I'd like to know to what extent it outperforms standard n-gram methods and by how much it trails behind semantic methods on moderate-to-strong paraphrases. If that evidence is already in the paper, it is not justified enough.
    • Also, it would be nice to see what are the unique strengths of SpanAudit (what kind of contamination it can capture that other methods miss) and conversely, what it cannot capture.
    • The quality of the synthetic contamination evaluation strongly depends on the relation between the corpus and benchmark used (true, it's symmetric to all methods). However, to cut out that distractor, you should split a benchmark (e.g. MMLU) in two parts and treat one as a corpus, one as a benchmark. Then, it becomes clear that high scores mean the method is able to separate question-derived content from same-distribution distractors. Given that you use recall@20, I'd expect that such an experiment would be too hard for any method tested. But at least a good goal to pursue.
    • I acknowledge the real-corpus audit experiments. However, compared to synthetic study, the gap between SpanAudit and LLM-Decontaminator is much more narrow, both in terms of precision and running time. Why is it the case? Clearly, LLM-Decontaminator is sensitive to semantic similarity while SpanAudit not, hence high precision. But I don't understand why the running times of both methods are now similar? In particular, what are the parameters used for this experiment? Note that this is NOT a weakness itself. However, I believe it should be honestly acknowledged in Results (line 432+), since now it's rather hidden.
    • (minor) In the real-corpus audit, how does that "original cleaning suite" looks like? Is that simply a set of benchmarks?
  • Little to no related works discussion. Other works are briefly mentioned in a few paragraphs inline. The overview should cover different methods, including: exact n-gram and substring matching, fuzzy/soft matching, semantic matching, LLM judge, different types of indexes/databases used, etc. See e.g. "Are LLM Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods" (https://aclanthology.org/2026.gem-main.50.pdf) for examples.

Overall, while I support the idea, the experimental validation needs to be improved to confirm the effectiveness of the method. Currently, I'm missing answers to: are all the novel components needed and how important are they to the final performance, as well as what are the key advantages from the data perspective.

Questions for authors
  1. Please explain the parameters' values and their choice, as explained in Weaknesses.
  2. How does your evidence localization compare to sliding window (e.g. size 512 stride 256 or something similar)?
  3. Please show me the samples used for the synthetic evaluation. How does your method work against semantic rephrases? Where does it land on the scale between easy nothing-changed leaks and extremely hard no-word-no-style-shared semantic paraphrases?
  4. Please answer the questions about real-corpus audits detailed in Weaknesses.
Reviewer_uFgxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a corpus-side benchmark contamination detection method that treats each benchmark item as a query and searches the training corpus for text spans of arbitrary length and position that overlap heavily with it, using hashing-based approach designed to keep this search efficient. The matched region is then narrowed down into a localized piece of evidence that is passed to a downstream judge to decide whether it constitutes contamination. The evaluation is based on synthetic settings with rewritten benchmark items and on real, previously decontaminated corpora, and the method is reported to outperform existing contamination detection tools on recall while finding evidence that those tools miss.

Critical strengths
  1. This work addresses a practically important problem and has a clear motivation to move beyond exact matching and fixed retrieval units in decontamination pipelines.

  2. The central idea is simple and easy to scale in lengthy documents

  3. Using a localized evidence span rather than a document-level flag is a sensible approach that practically fits how audits are carried out.

Critical weaknesses
  1. It is unclear how the proposed method differs from standard lexical passage or near-duplicate retrieval, and no standard IR baselines are included. Stripped of the contamination framing, the retrieval step is a fixed set of queries, a large corpus, and a search for passages with high lexical overlap, which is a well-studied problem in IR and near-duplicate detection. The paper only compares against contamination-specific tools, so it is hard to tell whether the reported gains come from the proposed span mechanism or from using a good lexical passage retriever where the baselines use exact matching or document-level embeddings. The part of contamination detection that genuinely differs from IR, which is about deciding what counts as contamination, seems to depend entirely on an LLM judge, which is not a contribution of this work.

  2. A method that returns many loosely related candidates can achieve high recall while being much less useful in practice, since every candidate must be put into another assessment cycle. The controlled experiments only report recall, and the real-corpus audits do not report precision. Some measure of precision or false-positive rate is needed to support the claim of practical usefulness.

  3. The proposed method relies on several components and hyperparameters, but there is no systematic ablation study measuring their individual contributions. It also seems to omit a basic baseline that runs the same pipeline with fixed-length windows of a few sizes.

  4. Several key comparisons rest on small samples and the differences are marginal. Confidence intervals, significance tests, or repeated runs are needed, as a large part of this study is LLM-generation based.

Questions for authors
  1. The rewrite intensity levels in Appendix F.1 are defined by Jaccard intervals computed with the same tokenization that the proposed method uses, so the difficulty of the synthetic benchmark is measured by the proposed method's own similarity function. Table 8 also suggests that a rewrite can satisfy its target interval mainly by inserting extra tokens. How was it validated that the rewrites are actually paraphrased to the intended degree, independently of this meausre, and are the Table 1 gains robust to that validation?
Reviewer_9uU4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates corpus-side benchmark contamination screening as query-conditioned, variable-length span retrieval. SpanAudit combines character-4-gram MinHash–LSH, shared interval-sketch computation, and candidate merging/localization to return inspectable evidence. It achieves the highest or tied-highest Recall@20 in controlled rewrite and long-context experiments and contributes additional evidence in three previously decontaminated instruction corpora. The system is practically relevant, but evaluation–method coupling, missing close baselines, and the absence of independently calibrated real-audit labels limit the conclusions.

Critical strengths
  1. Separating retrieval from adjudication and returning localized evidence with context makes the proposed workflow inspectable and avoids equating lexical similarity with a final contamination judgment.
  2. Query-side indexing and shared interval computation are appealing engineering choices. The long-context experiment preserves planted pairs and distractors across lengths, providing a useful controlled stress test.
  3. The evaluation spans several task types and real corpora, reports retrieval resources as well as recall, and explicitly acknowledges lexical-overlap dependence, probabilistic routing, and complementarity with semantic retrieval.
Critical weaknesses

1.The synthetic evaluation is selected using the method's own similarity representation. Appendix F.1 generates and accepts rewrites using the same normalization, character 4-grams, and Jaccard metric as SpanAudit. Hard rewrites are constrained to [0.4,0.6), while the retrieval threshold is 0.4. This is not an independent test of robustness to arbitrary strong semantic paraphrases. Moreover, the moderate/hard examples in Table 8 mainly insert repeated ignore/nonsemantic markers, breaking contiguous matches while preserving much of the original wording. Please quantify the prevalence of such constructions and evaluate independently generated, human-validated rewrites without Jaccard-based acceptance, including low-overlap cases. 2.The algorithmic contribution and the benefit of variable-length retrieval are insufficiently isolated. Appendix D.5 explicitly adopts the GlobalList results of Zhang et al. (2026), including interval counting, traversal, and composite-hash maintenance. The main text should clearly distinguish inherited machinery from new contributions and compare a query–document adaptation of LSHAlign. Multi-scale overlapping-window/paragraph MinHash, lexical retrieval, and chunked embedding or hybrid baselines are also needed. Degradation of a whole-document embedding baseline does not by itself establish the necessity of this interval algorithm.

Questions for authors

see in weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.