acceptodds
Under review as a conference paper at ICLR 2027

From Detection to Intervention: Counterfactual Decontamination for Language-Model Evaluation

Abstract

Benchmark decontamination is commonly treated as an overlap-detection problem, even though evaluation signal can enter training data through paraphrases, answer-bearing context, and reusable solution templates. We introduce Counterfactual Decontamination, an intervention-first framework that represents benchmark-to-training links as typed edges and maps them to auditable data actions. The framework combines lexical, semantic, answer, and structural evidence, then constructs matched contaminated, exact-pruned, graph-pruned, reweighted, and oracle-pruned training arms. This design separates three questions that are often conflated: whether a candidate edge can be retrieved, whether it constitutes actionable leakage, and whether removing it changes the evaluation conclusion. In controlled real-text mathematics contamination, exact matching recovers copies but misses non-exact channels, whereas the typed graph recovers paraphrase, answer, and template exposures while retaining hard benign neighbors. Across contamination levels and random seeds, exact pruning reduces but does not eliminate the target-to-clean training-effect gap; graph pruning closes 99.1% of the gap and approaches the oracle intervention under matched evaluation. Natural-pool blind validation and matched clean controls further test whether high-severity edges remain actionable beyond controlled injection. The resulting protocol replaces overlap counts with a falsifiable criterion: a decontamination policy is supported only when its detected edges survive blinded validation and its intervention preserves the model comparison the benchmark is meant to measure.

Primary area:
datasets and benchmarks
Keywords:
benchmark contamination, decontamination, counterfactual intervention, exposure graphs, language-model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZHetOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that removing exact duplicates is insufficient for benchmark decontamination, and proposes a graph based decontamination policy that covers a wider range of decontamination edge types (exact, paraphrase, answer leakage, template, benign). They perform intervention experiments by retraining with different pruning policies, and they measure how the NLL gap closes against baselines. Authors also perform human review to verify the precision on the natural pool.

Critical strengths

The experiment control design seems to be thorough. The authors also report negative results such as 32B gate. However, the experimental design's strength is contingent on proper description of implementation details that are not sufficiently provided in the paper (see weakness section).

Critical weaknesses

The premise of this paper relies on the understanding that exact duplicate removal is insufficient for decontamination, this is reasonable but already widely established in the field (Yang et al, 2023; Jiang et al, 2024). This paper also does not cite highly relevant works with overlapping research problems (Yang et al, 2023; Jiang et al, 2024). For instance, Jiang et al, 2024 used counterfactual training to measure contamination effects, which overlaps the framework in this paper. Authors need to cite these papers and establish the novelty of their contribution over prior work.

Many terms are named but not defined. Examples of missing details include edge-type features (how is it computed?), thresholds (numbers?), and severity rules. A major example of this is the cutoff value for graph-pruning, I do not see this cutoff declared in the paper. Also, some important methods are not explained, just named. For example, what exactly is “graph-structured” contamination in table 7? I can only find one mention of this word in the text (line 143), with no further explanation. Another similar case is “retrieval-leak”, there are many cases like these that the authors need to fix to make the paper clear and reproducible.

Even though the authors disclose the use of LLMs for writing assistance, the style of writing in this paper sounds highly “AI-like”, making it overly verbose and difficult to read. The overly complex sentence structure and overuse of noun-stacking are signs of AI-writing that make the paper difficult to read. Throughout the paper, the authors overuse semi-colons in their writing. The worst offending paragraph is the last paragraph in the introduction section (line 54-60). This paragraph is one long sentence that presents a list of 4 items separated by semi-colons, and each individual item is itself a comma separated list. Even though it is grammatically correct, this kind of long, list-of-lists sentence is difficult to understand for a human reader. The authors should take time to revise the writing style in this paper to communicate their ideas in a more clear and straightforward fashion. As it stands, it reads like an AI-generated paper with insufficient human auditing.

Citations: Yang, Shuo, et al. "Rethinking benchmark and contamination for language models with rephrased samples." arXiv preprint arXiv:2311.04850 (2023). Jiang, Minhao, et al. "Investigating data contamination for pre-training language models." arXiv preprint arXiv:2401.06059 (2024)

Questions for authors

Table 1 says Typed Graph recovers 100% of controlled edges that were injected. So why does the neural graph arm consistently underperform the oracle (not seed noise because it’s consistent across seeds in supp.table 4), can you report edge recall and precision for neural-tier contamination? It is also unclear to me where I can see the deletion count for the neural arm?

Reviewer_wDcXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses benchmark contamination beyond literal overlap, including paraphrases, answer-bearing contexts, and solution templates, while noting that semantic similarity alone can falsely flag legitimate data. The authors propose Counterfactual Decontamination, which represents test–training relationships as a typed, severity-labeled bipartite graph and maps evidence to actions such as deletion or downweighting. Each resulting dataset is used for LoRA fine-tuning from the same initialization. Effectiveness is assessed through the NLL gap between targeted and matched clean questions and the fraction of that gap closed.

Critical strengths

1. An important problem and fresh perspective. The paper explores an important problem and provides a fresh perspective by framing decontamination as a verifiable data intervention, assessed by whether data removal changes evaluation outcomes.

2. Comprehensive controls. The paper includes many ablation controls: size-matched random pruning, benign-edge placebos, TracIn, influence functions, and an oracle. The experiments cover several model scales, tasks, contamination channels, and two model families.

3. Transparent statistical reporting. The paper provides paired intervals, Wilson intervals, and per-seed results.

Critical weaknesses

1. The core metric is not clearly defined, and its numerical magnitude is hard to interpret. The paper does not state whether completion NLL is averaged over tokens or summed over the sequence.

2. Experimental details are severely missing, making the work impossible to reproduce. For the neural experiments, the size and source of the training data pool, how contamination is injected, and the LoRA rank, learning rate, and number of steps are all not reported.

3. The contamination is injected by the authors themselves through LoRA fine-tuning. The base model may have already been exposed to GSM8K, MATH, and other data during pretraining, and this real contamination falls outside the scope of what the method tests. The paper therefore verifies whether “injected contamination can be removed,” rather than real pretraining contamination. I suggest authors check the paper "Hubble: a Model Suite to Advance the Study of LLM Memorization". They onpened the model weights and training data, which means authors could check their method's effectiveness directly through comparing with training data.

4. Insufficient description of the method. The severity thresholds and the decision rules for each edge type are not given concrete definition or algorithm, so it's hard to know how the typed graph is actually constructed.

5.Missing citations and comparisons with related methods. The paper has only small amount of references in total and omits a large body of core work in the data contamination field.

Questions for authors
  1. How exactly is completion NLL computed (averaged over tokens or summed over the sequence, and is it computed over the answer or the full solution)?

  2. What exactly are the decision rules for each edge type and the severity thresholds in the typed graph? How does the graph method perform in recall on contamination not constructed by the authors (e.g., paraphrases generated by other methods or by other people)?

  3. What advantages does this method have over existing paraphrase contamination detection methods? Could the authors add a comparison?

Reviewer_vCFFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Counterfactual Decontamination, an intervention-based framework for identifying and mitigating benchmark contamination in language model evaluation. Rather than relying solely on textual overlap or semantic similarity, the proposed method constructs a typed exposure graph to distinguish exact duplicates, paraphrases, answer leakage, and structural similarities, and uses the resulting evidence to guide training-data removal. Experiments across multiple model scales, benchmarks, and contamination types demonstrate that graph-based pruning substantially reduces contamination-induced performance gaps, outperforming exact matching, retrieval-based pruning, and data attribution baselines while approaching oracle-level decontamination.

Critical strengths
  1. This paper evaluates whether removing suspected contaminated training examples actually changes benchmark performance. This provides a more meaningful way to assess the impact of contamination than retrieval accuracy or overlap statistics alone.

  2. The proposed method is interpretable.

  3. Writing is good and easy to follow.

Critical weaknesses
  1. In the controlled experiment, BM25 already achieves 100% Recall@10, suggesting that the main improvement may come from the subsequent actionability classification rather than retrieval itself. Similar retrieval-and-verification pipelines have already been adopted in large-scale data engineering. For example, OpenMathInstruct-2 [1] employs embedding-based retrieval followed by LLM-based verification to identify contaminated examples. This raises an important question about the practical advantages of the proposed typed graph over existing approaches. I would encourage the authors to compare their method against this simpler baseline, i.e., BM25 (or embedding retrieval) combined with a lightweight LLM judge (or JEV). More importantly, the comparison should explicitly consider the cost–performance trade-off and scalability. Real-world data engineering often involves millions or billions of training examples, making expensive verification impractical at scale. If a simple retrieval-plus-lightweight-verification pipeline can achieve comparable decontamination effectiveness at substantially lower cost, it may offer greater practical value. Without such comparisons, it remains unclear whether the proposed method provides sufficient benefits to justify its additional complexity.

  2. I suggest using numbered subsections (e.g., via the latex \subsection{} command) to improve readability.

[1] Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. ICLR 2025.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.