acceptodds
Under review as a conference paper at ICLR 2027

The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation

Abstract

Large language models (LLMs) have demonstrated impressive reasoning abilities across a wide range of tasks, but data contamination undermines the objective evaluation of these capabilities. This problem is further exacerbated by malicious model publishers who use evasive, or indirect, contamination strategies, such as paraphrasing benchmark data to evade existing detection methods and artificially boost leaderboard performance. Current approaches struggle to reliably detect such stealthy contamination. In this work, we uncover a critical phenomenon: a model's generated reasoning steps actively mask its underlying memorization. Inspired by this, we propose the Zero-CoT Probe (ZCP), a novel black-box detection method that deliberately truncates the entire Chain-of-Thought (CoT) process to expose latent shortcut mappings. To further isolate memorization from the model's intrinsic problem-solving capabilities, ZCP compares the model's zero-CoT performance on the original benchmark against an isomorphically perturbed reference dataset. Furthermore, we introduce Contamination Confidence, a metric that quantifies both the likelihood and severity of contamination, moving beyond simple binary classifications. Extensive experiments on both previously identified contaminated models and specially fine-tuned contaminated models demonstrate that ZCP robustly detects both direct and evasive data contamination.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, data contamination, reasoning, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iRCZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the Zero-CoT Probe (ZCP), a contamination detection method that truncates the Chain-of-Thought process to expose latent shortcut mappings. The proposed method can detect data contamination without requiring access to the model's token probabilities.

Critical strengths

The authors aim to develop a dataset-level contamination detection method, which is important for ensuring reliable evaluation of LLMs.

Critical weaknesses
  1. Lack of baseline comparisons. The manuscript does not compare the proposed method against existing detection methods, making it unclear whether it achieves superior performance.

  2. Limited novelty. Existing work has explored the use of performance differences between the original and reference datasets to detect contamination [1]. Additionally, introducing a reference dataset does not necessarily isolate memorization from the model's intrinsic ability. The authors do not provide experimental evidence to support this claim.

  3. Unconvincing empirical evaluation. The experimental results do not convincingly demonstrate the effectiveness of ZCP. In particular, the authors should compare Zero-CoT and Full-CoT under the same controlled setting. Additionally, it is unclear whether the performance gap stems from ZCP or differences between the target and reference datasets. The experimental results in Table 18 show that the measured metrics are highly sensitive to the dataset.

Reference: [1] ConStat: Performance-Based Contamination Detection in Large Language Models, Neurips 2024

Questions for authors
  1. In Figure 2, what is the experimental setup? How are the contaminated questions and clear questions subsets established?

  2. In line 219, what does 'where response prefixes cannot be explicitly pre-filled' mean? From my perspective, the instruction can be placed directly into the prompt.

  3. What motivates the use of a Bayesian evidence framework and an upper bound on the Bayes factor?

  4. In Figure 2, why are solid lines used in the legend, while dashed lines are used in the plot?

6 comments
AuthorsOpenReview ↗

Response to Reviewer iRCZ 1/3

Thank you for the detailed review and valuable advice. We have revised the paper (changes in dark green); section, table, figure, and line numbers below refer to this version.

W1: Lack of baseline comparisons with existing detection methods

Sec. 3.2 already evaluates existing methods on the same contaminated models: the scores of DPCC (Table 1) and data reconstruction (Table 2) drop sharply from the original to the paraphrased data (e.g., for Qwen-Math on MATH, DPCC 0.730 → 0.191, ROUGE-L 0.621 → 0.267).

Existing methods produce different outputs (a thresholded fraction of low-RMIA examples for DPCC, ROUGE-L and accuracy for reconstruction, per-example scores for membership inference), and ZCP's paired test needs reference data paired one-to-one with the original data. We therefore do not rank the methods by one score, but test whether each method's own signal, which is stronger in the direct (original) setting, still raises an alarm in the evasive (paraphrased) setting.

We add Min-K% (Shi et al., 2023), a widely used membership inference method (Appendix I.4). Following Shi et al., we score the question of each example with k = 20%, treat the original (or paraphrased) examples as positives and their unseen reference examples as negatives, and report AUROC and the true positive rate (TPR) at 5% false positive rate.

Table R11. Min-K%: AUROC / TPR at 5% FPR, with the reference examples as negatives (random guessing: 0.5 / 5%).

Model · dataOriginalParaphrased
Qwen-Math · GSM8K0.862 / 38.2%0.162 / 0.0%
Qwen-Math · MATH0.651 / 13.0%0.091 / 0.0%
DeepSeek-Math · GSM8K0.596 / 13.0%0.178 / 0.2%
DeepSeek-Math · MATH0.545 / 5.6%0.213 / 0.1%

On the original data, Min-K% ranks the contaminated examples above their references (AUROC 0.545–0.862, TPR up to 38.2%). On the paraphrased data, the signal disappears: TPR is at most 0.2%, and AUROC falls well below 0.5. Like DPCC and data reconstruction, Min-K% relies on the exact tokens of the training data, which paraphrasing removes.

W2: Limited novelty, as existing work (ConStat) uses performance differences on reference data; a reference dataset may not isolate memorization from intrinsic ability

We agree that comparing performance on original and reference data is an established idea, and now cite and discuss related work such as ConStat (Dekoninck et al., 2024) in Sec. 2. However, ZCP differs from it in two ways:

  • Signal. ConStat compares performance under the standard evaluation, but reasoning masks memorization (Sec. 3.3, Figure 2): with CoT, a contaminated model also solves the reference problems, so the gap disappears, as Table R12 in W3 shows for all paraphrased settings (Full-CoT Ccont≤0.616\mathcal C_{cont} \le 0.616). ZCP therefore compares zero-CoT performance.
  • Reference data. ConStat uses rephrased samples, synthetic samples from the same distribution, or another benchmark for the same task; since their difficulty can differ from the original benchmark's, it corrects for this with a set of reference models. ZCP builds each reference example by rewriting the text and changing the numbers while keeping the logical structure and the solution steps (Sec. 3.5, Table 3), so the model cannot have seen it. The one-to-one pairing allows a paired test on the target model alone, without reference models, which gives Ccont\mathcal C_{cont}.

Our results show that the reference data isolate memorization from intrinsic ability:

  • (a) Uncontaminated benchmarks show no gap: Ccont\mathcal C_{cont} of AccAcc is 0.500 on GSM1K for both Qwen-Math and DeepSeek-Math (Table 5) and 0.551 / 0.567 on Dataset U for the FT models (Table 8).
  • (b) The reference data are as difficult as the original data. In the human evaluation (Appendix C.1, Table 9), raters judged the difficulty equal for 93/100 GSM8K and 94/100 MATH examples.
  • (c) Six additional reference datasets lead to the same conclusion (Appendix C.2, Figure 4): for Qwen-Math and DeepSeek-Math on GSM8K and MATH, Ccont>0.997\mathcal C_{cont} > 0.997 in all 96 original-vs-reference tests (4 settings × 4 metrics × 6 references), and the AccAcc gap is 10.8–28.7 points between the original data and each reference, but at most 6.6 points between any two references.
  • (d) On the same examples and reference data, the gap appears only after training (Appendix I.3, Table 21): when Qwen3-8B is trained with SFT on six paraphrased versions of the 241 business questions in Dataset C, its zero-CoT AccAcc on the original / reference data goes from 16.6% / 15.8% before training (Ccont\mathcal C_{cont} = 0.500) to 28.2% / 15.4% after 400 steps (Ccont\mathcal C_{cont} = 0.999).

References

AuthorsOpenReview ↗

Response to Reviewer iRCZ 2/3

W3: Compare Zero-CoT and Full-CoT under the same controlled setting; is the performance gap caused by ZCP or by differences between the target and reference datasets; Table 20 (Table 18 in the original submission) shows that the metrics are highly sensitive to the dataset

In the original submission, Appendix I.1 (Table 17) compares the two settings only for Qwen-Math on GSM8K. We now evaluate all existing-model settings of Table 4 and the OLMo models of Table 6 with both Full-CoT and zero-CoT, on the same original examples and the same reference data (Table R12 in the rebuttal and Table 19 in the revised paper).

Table R12. AccAcc on original / reference (SS / SrefS_{ref}, %) and Ccont\mathcal C_{cont} with Full-CoT and zero-CoT. All rows are means over 3 seeds.

ModelDataFull-CoTZero-CoT
Qwen-MathGSM8K original95.9 / 96.2, 0.50045.7 / 33.2, 1.000
Qwen-MathGSM8K paraphrased94.3 / 96.3, 0.50042.3 / 33.2, 0.996
Qwen-MathMATH original78.6 / 76.4, 0.67549.3 / 31.3, 1.000
Qwen-MathMATH paraphrased74.5 / 76.3, 0.50043.1 / 31.3, 1.000
DeepSeek-MathGSM8K original96.6 / 93.7, 0.94229.7 / 21.7, 0.993
DeepSeek-MathGSM8K paraphrased90.2 / 93.9, 0.50027.3 / 21.7, 0.917
DeepSeek-MathMATH original59.4 / 49.5, 1.00037.3 / 20.4, 1.000
DeepSeek-MathMATH paraphrased52.4 / 50.0, 0.61630.0 / 20.4, 1.000
OLMo-2-InstructMATH33.8 / 32.9, 0.51713.0 / 10.0, 0.929
OLMo-3-InstructMATH76.0 / 75.0, 0.62922.1 / 17.0, 0.990

Full-CoT finds strong evidence (Ccont≥0.942\mathcal C_{cont} \ge 0.942) only on the original GSM8K and MATH data of DeepSeek-Math, and little or no evidence on any paraphrased (evasive) setting (Ccont≤0.616\mathcal C_{cont} \le 0.616). On GSM8K, Full-CoT accuracies on the original and reference data are close. With zero-CoT, Ccont≥0.917\mathcal C_{cont} \ge 0.917 in all 10 settings, including all paraphrased ones. This is the masking effect of Sec. 3.3: the model's reasoning also solves the reference problems and hides the memorization.

Table R12 also shows that the gap comes from ZCP rather than from differences between the original and reference datasets. Full-CoT and zero-CoT are evaluated on exactly the same original and reference data, so any difference between the two datasets affects both settings in the same way. Yet with Full-CoT, the accuracies on the original and reference data are close in most settings (e.g., 94.3% vs. 96.3% for Qwen-Math on paraphrased GSM8K), and strong evidence appears only on the original data of DeepSeek-Math; with zero-CoT, the same data give Ccont≥0.917\mathcal C_{cont} \ge 0.917 in all 10 settings (e.g., 42.3% vs. 33.2%). Zero-CoT does not create a gap by itself either: on the uncontaminated GSM1K and Dataset U, Ccont\mathcal C_{cont} of zero-CoT AccAcc is 0.500–0.567 (W2 (a)).

Table 20 varies the number of evaluated examples N. As an evidence score derived from a p-value, Ccont\mathcal C_{cont} is expected to depend on N: with few examples, even a real gap provides limited evidence. For the same contaminated model and a similar gap, Ccont\mathcal C_{cont} of AccAcc rises from 0.578 at N = 50 to 1.000 at N = 1000. The logit-based metrics are much less sensitive to N: Ccont\mathcal C_{cont} of Pfirst\mathcal P_{first} is at least 0.946 from N = 50, and that of Pall\mathcal P_{all} is at least 0.997 from N = 100. As recommended in Appendix I.2 (l.1314–1328), Pfirst\mathcal P_{first} and Pall\mathcal P_{all} should be preferred whenever logits are available; AccAcc and ConCon are a compromise for cases where logits or ground-truth answers are unavailable, and they require a sufficiently large N.

Q1: What is the experimental setup of Figure 2, and how are the contaminated and clean question subsets established?

We thank the reviewer for raising this question. Figure 2 uses Qwen-Math and DeepSeek-Math. The contaminated questions are the 700 MATH training examples of Table 4, on which both models were trained (Sec. 4.1), and the clean questions are their counterparts in the reference dataset, which the models have not seen. For each question, we keep the first X% (by characters) of the model's Full-CoT solution and append The final answer is: \[ \boxed{ for the model to complete the answer; X = 0% is zero-CoT, and X = 100% is the full solution. Δ is the accuracy on the contaminated questions minus that on the clean ones. We have added this description to Appendix I.1 (l.1261–1266).

AuthorsOpenReview ↗

Response to Reviewer iRCZ 3/3

Q2: What does "where response prefixes cannot be explicitly pre-filled" (l.224–225; l.219 in the original submission) mean? The instruction can be placed directly into the prompt.

For open-weight models, we write the beginning of the model's response ourselves (The final answer is: \[ \boxed{, l.222–224), so the model must answer immediately without reasoning. The APIs of closed-source models such as GPT do not allow us to edit the model's response directly. We can only edit the user prompt, so for them we place the instruction like "Please ONLY put your final answer within \boxed{} directly without any other content before or after it (e.g., reasoning or explanation)" in the prompt instead (l.224–228). Since an instruction does not guarantee that reasoning is skipped, we measure how often the models comply: GPT-4o and GPT-5.1 output the boxed answer directly in 92.8–100% of cases on MATH500 and GSM8K (Table 24, Appendix J). We have clarified this in Sec. 3.3 (l.224–230).

Q3: What motivates the use of a Bayesian evidence framework and an upper bound on the Bayes factor?

We thank the reviewer for raising this question. First, a binary clean/contaminated label cannot capture how the evidence of contamination varies with the exposure frequency and the proportion of leaked data (l.310–312). We therefore want a continuous measure of evidence rather than an accept/reject decision at a certain threshold. A p-value does not consider how likely the gap is under contamination, and by itself it does not provide a good measure of evidence for a hypothesis (Wasserstein & Lazar, 2016). The Bayes factor compares how likely the observed data are under contamination and under no contamination, and a p-value can be converted into a bound on it (Sellke et al., 2001; Bayarri et al., 2016).

Second, an exact Bayes factor requires a model of the data under contamination, which depends on the unknown amount of exposure and varies across models and benchmarks. The bound BF10=−1/(e⋅pln⁡p)BF_{10} = -1/(e \cdot p \ln p) only requires a valid p-value. With a neutral prior of 0.5, it gives Ccont∈[0.5,1)\mathcal C_{cont} \in [0.5, 1), where 0.5 means no evidence. Because it is based on an upper bound, Ccont\mathcal C_{cont} is an evidence score rather than a calibrated probability, which we now state in Sec. 3.6 (l.341–343).

Q4: In Figure 2, why are solid lines used in the legend, while dashed lines are used in the plot?

Thank you for pointing this out. The DeepSeek-Math curve is dashed, and its legend entry uses the same dashed style, but the legend line is too short for the dashes to be visible. We have lengthened the legend lines in Figure 2 so that the line styles are clearly visible.

References

Reviewer_8SW6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a Zero-CoT probe (ZCP) for detecting data contamination within LLMs, particularly evasion contamination that bypasses existing detection methods. The authors found that the CoT inference process can mask memorization effects within the model. To clarify this, the authors induced the model to generate the correct answer directly without an inference step, a method called Zero-CoT. Next, ZCP detects memorization by leveraging performance differences between reference datasets that share the same problem and logical structure but differ in numerical values and context. Furthermore, the authors introduce the concept of 'contamination confidence' to quantify the degree of data contamination and validate the effectiveness of their proposed method through experiments with both the original model and the model that generates data contamination.

Critical strengths
  • By using a prompting method called Zero-CoT and a simple approach involving comparisons with reference data, the paper demonstrates the possibility of detecting indirect data pollution without accessing the model's training data or internal parameters.
  • The paper is well written overall, and the motivation for the methods and the experimental design are clearly explained, making it easy to understand.
  • The experiments are comprehensive, validating the proposed method not only on existing models but also on models in which data perturbations were induced.
Critical weaknesses
  • In Section 3.1, paraphrased data with the same correct answer is defined as contaminated data. However, it is unclear how to interpret cases in which the training data are derived from the original benchmark but the numerical values differ from the correct ones. In addition, even when a model is trained on problems of a similar type that are unrelated to the original benchmark, a similar performance gap may occur between the original and reference problems.
  • Zero-CoT is a prompting method that guides models to respond without an explicit reasoning process. It may be natural for performance to degrade more on new problems than on familiar ones.
  • The assumption that reference data present the same level of difficulty for the model is a strong one. Previous studies, such as GSM-Plus [1], have also observed that small numerical substitutions or perturbations can affect inference performance. Therefore, these effects may be mixed with signals of actual data memorization.
  • The claim that contamination confidence indicates the actual severity of contamination should be carefully considered. In Table 18 of Appendix I.2, although the data contamination settings and accuracy differ, Contamination Confidence increases from 0.578 to 1.000 as the number of evaluation samples increases.

[1] Li et al., "GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers," ACL 2024.

Questions for authors
  • How do ZCP's detection results correlate with the overestimation of actual leaderboard performance? The authors present the overestimation of leaderboard performance caused by data contamination as the main motivation. However, if numerical variations of problems derived from benchmarks are learned and generalized to reference problems, ZCP may fail to detect contamination even if performance improves. Also, Full CoT shows high accuracy on both the original and reference problems, raising the question of whether actual performance can be considered overestimated.
  • Does ZCP work on more difficult tasks, such as AIME or GPQA, or on tasks where the answer is not fixed as a single number or string, such as code generation?
  • Would it be more appropriate to include the OLMo experiment in the main experiments? OLMo's results provide important validation for models whose training data are relatively transparent.
12 comments
AuthorsOpenReview ↗

Response to Reviewer 8SW6 1/3

Thank you for the detailed review and valuable advice. We have uploaded a revised version of the paper that incorporates your suggestions, with the modified text highlighted in dark green; the section, table, figure, and line numbers below follow this revised version.

W1: Does training on data derived from the benchmark but with different numerical values count as contamination? Can training on problems of a similar type that are unrelated to the benchmark also cause a gap?

We follow the common definition of contamination as "any leakage of information that provides a signal for the correct label for at least one example in the test set" (Palavalli et al., 2024). Training on paraphrased benchmark data with the same answers is therefore widely regarded as contamination (Dekoninck et al., 2024; Yang et al., 2023), and this is how Sec. 3.1 defines evasive contamination (l.130–137). Singh et al. (2024) similarly assess contamination by whether the model benefits from the leaked examples. Under this definition, training on numerical variants of the benchmark leads to three cases:

  • If the model only memorizes the answers of the variants, these answers differ from those of the original questions, so the model gains no advantage on the benchmark. This is not contamination, and ZCP does not flag it.
  • If the model learns from many variants to solve the problem for new numbers, it has learned the solution rather than memorized the answer. Such training is also costly, as it requires generating and training on many variants of each question. This is benign data augmentation (Appendix H, l.1216–1239): the model performs similarly on the original and reference data, and ZCP correctly finds no evidence of contamination.
  • If the model is trained on only a few variants (too few to learn the solution) mixed with paraphrases that keep the original answers, it still memorizes the benchmark answers. Appendix H tests this case: Qwen3-8B trained on three paraphrased and three perturbed (paraphrase + renumber) variants is detected with Ccont\mathcal C_{cont} = 0.987 / 0.862 / 0.954 / >0.998 on AccAcc / ConCon / Pfirst\mathcal P_{first} / Pall\mathcal P_{all} (Table 18).

Training on problems of a similar type does not cause a gap in our experiments. Qwen-Math and DeepSeek-Math are trained on GSM8K, but show no gap on GSM1K (Zhang et al., 2024), whose problems are of the same type (Table 5). FT Qwen-Math and FT Qwen3 are trained on paraphrased Dataset C, but show no gap on Dataset U, the held-out half of the same benchmarks (Table 8). We further add OpenR1-Qwen-7B, whose training data contain AIME 1983–2023 problems, and evaluate it on AIME 2025/26 problems released after its training (setting in Q2).

Table R7. Zero-CoT AccAcc on original / reference (SS / SrefS_{ref}, %) and Ccont\mathcal C_{cont} on uncontaminated benchmarks of the same type as the training data.

ModelTrained onUncontaminated benchmarkSS / SrefS_{ref}Ccont\mathcal C_{cont}
Qwen-MathGSM8KGSM1K23.3 / 22.20.500
DeepSeek-MathGSM8KGSM1K16.0 / 18.30.500
FT Qwen-MathDataset C (paraphrased)Dataset U13.8 / 12.30.551
FT Qwen3Dataset C (paraphrased)Dataset U15.3 / 14.40.567
OpenR1-Qwen-7BAIME 1983–2023AIME 2025/261.7 / 1.70.500

Across all metrics, Ccont\mathcal C_{cont} stays between 0.500 and 0.638 on these benchmarks. Training on problems of the same type alone does not create a gap between the original and reference data.

W2: Under zero-CoT, it may be natural for performance to degrade more on new problems than on familiar ones

This depends on what "familiar" means. If the model has seen the problems during training, this is exactly the contamination that ZCP is designed to detect. If the model is only familiar with the problem type but has not seen the problems, its performance does not degrade on the new reference problems: although the models in Table R7 are trained on GSM8K, Dataset C, and older AIME problems, their zero-CoT performance on the references of GSM1K, Dataset U, and AIME 2025/26 is similar to that on the originals.

References

AuthorsOpenReview ↗

Response to Reviewer 8SW6 2/3

W3: The assumption that the reference data present the same level of difficulty is strong, and small numerical perturbations can affect performance (GSM-Plus)

We agree that, for an individual question, a numerical perturbation can make the problem easier or harder, as shown by GSM-Plus (Li et al., 2024) and GSM-Symbolic (Mirzadeh et al., 2024). However, ZCP is a dataset-level test that compares performance over the whole benchmark, and at this level such effects are small:

  • (a) on uncontaminated benchmarks, zero-CoT performance does not degrade on the reference data (Table R7);
  • (b) the contamination conclusion is stable across six generated reference datasets, and the performance gaps among these reference datasets are very small.

For (b), we used the pipeline of Appendix B to generate six reference variants for every original example of GSM8K and MATH used in Table 4. We compared the originals with each variant, and each variant with every other variant (30 ordered pairs). Ccont>0.997\mathcal C_{cont} > 0.997 in all 96 original-vs-variant tests (4 settings × 4 metrics × 6 variants). In every setting, the gap between the original data and any variant is larger than the gap between any two variants, on all four metrics. Averaging the six variants as the reference further reduces the influence of any single variant and gives Ccont>0.998\mathcal C_{cont} > 0.998 in all settings; we recommend this practice in Appendix C.2.

Table R8. Performance gap (S−SrefS - S_{ref}) between the original examples and each of the six variants (range), and the largest gap between two variants. AccAcc in points, Pall\mathcal P_{all} × 100.

Model · dataAccAcc: original vs. variantsAccAcc: max variant vs. variantPall\mathcal P_{all}: original vs. variantsPall\mathcal P_{all}: max variant vs. variant
DeepSeek-Math · MATH22.9–23.91.017.9–20.32.3
DeepSeek-Math · GSM8K10.8–17.46.612.2–19.67.4
Qwen-Math · MATH22.6–28.76.116.0–17.92.0
Qwen-Math · GSM8K17.6–24.26.613.6–20.16.5

W4: Contamination Confidence should not be claimed to indicate the severity of contamination, and it increases with the number of evaluation samples N (Table 20; Table 18 in the original submission)

Thank you for pointing this out. We agree that our claim was not precise. Ccont\mathcal C_{cont} is obtained from the one-sided p-value and an upper bound on the Bayes factor (Sec. 3.6, Appendix A), so it measures the strength of evidence for contamination, not its severity. We have corrected the abstract (l.026), the contributions (l.067), the conclusion (l.509), l.142, the Table 4 caption, and l.1320 accordingly.

As an evidence score, Ccont\mathcal C_{cont} is expected to depend on N. With few examples, even a real gap provides limited evidence, and Ccont\mathcal C_{cont} becomes high only when the gap is consistent over enough examples. This is what Table 20 shows: for the same contaminated model and a similar gap, Ccont\mathcal C_{cont} of AccAcc rises from 0.578 at N = 50 to 1.000 at N = 1000.

The logit-based metrics are much less sensitive to N: Ccont\mathcal C_{cont} of Pfirst\mathcal P_{first} is at least 0.946 from N = 50, and that of Pall\mathcal P_{all} is at least 0.997 from N = 100 (Table 20). As recommended in Appendix I.2 (l.1314–1328), Pfirst\mathcal P_{first} and Pall\mathcal P_{all} should be preferred whenever logits are available. AccAcc and ConCon are a compromise for cases where logits or ground-truth answers are unavailable, and they require a sufficiently large N.

We further fix N = 200 and vary the proportion of contaminated examples for Qwen-Math (Appendix I.3, Figure 6) by mixing k GSM8K examples (contaminated) with 200 − k GSM1K examples (uncontaminated). Since Ccont\mathcal C_{cont} depends on which examples are drawn, for each proportion we draw 500 random benchmarks and report the median Ccont\mathcal C_{cont}.

Table R9. Qwen-Math, N = 200: median Ccont\mathcal C_{cont} of AccAcc over 500 random benchmarks vs. the proportion of contaminated examples.

Contaminated proportion0%10%20%30%40%50%60%70%80%90%100%
Median Ccont\mathcal C_{cont}0.5000.5130.5640.6260.7000.8020.8540.9010.9580.9730.985

Ccont\mathcal C_{cont} increases with the contaminated proportion: it stays close to 0.5 at 10–20%, rises steadily from 30% to 80%, and flattens above 80% (0.958–0.985). So at a fixed N, more contamination yields stronger evidence, but Ccont\mathcal C_{cont} remains an evidence score rather than a measure of severity.

References

AuthorsOpenReview ↗

Response to Reviewer 8SW6 3/3

Q1: How do ZCP's detection results correlate with the overestimation of leaderboard performance?

ZCP is designed to detect data contamination, i.e., the memorization of benchmark data. Overestimation of leaderboard performance is one possible consequence of contamination, not the quantity that ZCP measures.

If the model learns from numerical variants and generalizes to the reference problems, it has learned the solution rather than memorized the answers (W1; Appendix H, l.1216–1239). Its benchmark performance then reflects genuine capability rather than an overestimation, and ZCP correctly does not flag it.

When the model's reasoning is strong enough for the benchmark, as with Full-CoT on GSM8K (95.9% vs. 96.2%, Table 19), contamination may not show in the score of this benchmark, so the overestimation on this benchmark can be small. However, the contamination is still present: the model partly relies on memorized answers, so the benchmark may overestimate its reasoning ability, which can be exposed on harder tasks or in other settings.

Q2: Does ZCP work on more difficult tasks, such as AIME or GPQA, or on code generation?

Thank you for raising this question. We add experiments on AIME (Appendix F.2, Table 15). OpenR1-Qwen-7B (Hugging Face, 2025) is Qwen2.5-Math-7B-Instruct fine-tuned on OpenR1-Math-220k, which contains AIME 1983–2023 problems. We evaluate it on 60 AIME 1983–2023 problems from its training data (contaminated) and on all 60 AIME 2025/26 problems, released after the model (uncontaminated), with reference data from the pipeline of Appendix B. ConCon is omitted because the model's Full-CoT outputs exceed our length limit.

Table R10. OpenR1-Qwen-7B on AIME: zero-CoT SS / SrefS_{ref} and Ccont\mathcal C_{cont} (AccAcc in %).

MetricAIME 1983–2023 (contaminated)AIME 2025/26 (uncontaminated)
AccAcc28.3 / 0.0, 1.0001.7 / 1.7, 0.500
Pfirst\mathcal P_{first}0.472 / 0.196, >0.9980.122 / 0.139, 0.500
Pall\mathcal P_{all}0.398 / 0.094, >0.9980.081 / 0.089, 0.500

On these competition-level problems, ZCP finds strong evidence of contamination on all three metrics, and no evidence on the problems released after training.

For code generation, the answer is not a single number or string, so our current reference construction, which perturbs the numerical values of a question with a single final answer, does not apply directly. We have not yet found a suitable way to build reference data for code, and leave it to future work.

Q3: Would it be more appropriate to include the OLMo experiment in the main experiments?

We thank the reviewer for this suggestion. We have moved the OLMo results (Table 13 in Appendix F of the original submission) into Sec. 4.1 (Table 6).

References

Reviewer_ZUbBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Zero-CoT Probe (ZCP), a black-box method for detecting direct and evasive benchmark contamination in large language models. The paper's hypothesis is that generated chain-of-thought reasoning can conceal memorization. That is, a contaminated model may produce a plausible reasoning trace even when its answer is driven primarily by a memorized question–answer association. ZCP therefore instructs the model to produce only the final answer and compares its performance on the original benchmark with its performance on an automatically generated reference set in which the context and numerical values are modified while the nominal reasoning structure is preserved. The paper also introduced Contamination Confidence, a p-value-derived score intended to quantify the likelihood and severity of contamination.

Critical strengths
  • The method does not require access to model parameters, gradients, or training corpora. In principle, it can therefore be used to audit proprietary models through their generated outputs, provided that instruction compliance can be verified.

  • The paper considers evasive rather than only verbatim contamination. This is an important threat model because semantic transformations, paraphrasing, and translation can preserve benchmark information while defeating surface-form overlap detectors.

Critical weaknesses
  • The measured performance gap is not specific to contamination. A model may perform better on the original benchmark than on perturbed examples because of numerical sensitivity, template familiarity, differences in arithmetic difficulty, answer-token priors, or artifacts introduced by the reference-set generator. Prior work such as GSM-Symbolic has already shown that changing numerical values can substantially affect model accuracy without establishing that the corresponding difference is caused by item-level contamination.

  • The proposed Contamination Confidence score is statistically misinterpreted. It is derived from a p-value and an upper bound on a Bayes factor. It is therefore neither a calibrated posterior probability of contamination nor a measure of contamination severity. The score may approach one for a very small effect if the evaluation set is sufficiently large, while clipping it at 0.5 prevents it from representing evidence in favor of the null hypothesis.

Questions for authors
  • Existing evidence shows that models can lose accuracy when only names or numerical values are changed. What experiment demonstrates that the observed zero-CoT gap is caused by prior exposure to the original examples rather than by arithmetic difficulty, answer-token priors, canonical template familiarity, or generator artifacts? Specifically, how does ZCP distinguish contamination from generic perturbation sensitivity?

  • How sensitive is ZCP to the construction of the reference dataset? If five or ten independent reference variants are generated for each original example, how much does Contamination Confidence vary? Does the contamination conclusion remain stable across generators, prompts, random seeds, numerical ranges, and contextual rewrites?

  • Since the Contamination Confidence score is based on an upper bound derived from a p-value, what calibration evidence supports statements such as “definitive contamination”? Can the authors show empirically that model-dataset pairs assigned scores of approximately 0.9 are contaminated approximately 90% of the time?

  • What happens when only 0.1%, 1%, 5%, or 10% of benchmark examples are included in training, each with a single exposure? Does ZCP detect pretraining-only contamination without solutions or repeated paraphrases?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZUbB 1/3

Thank you for the detailed review and valuable advice. We have revised the paper to address your comments, and all changes are marked in dark green. All section, table, figure, and line numbers below refer to the revised paper.

W1, Q1 & Q2: The zero-CoT gap may be caused by generic perturbation sensitivity, not specific to contamination, and may depend on how the reference dataset is constructed

We thank the reviewer for raising this important question. We agree that, for an individual example, a perturbation can make the problem easier or harder. However, ZCP is a dataset-level test that compares performance over the whole benchmark, and at this level such effects are small. The zero-CoT gap is mainly caused by training on the data, and the contamination conclusion is stable across reference variants. The evidence is as follows:

  • (a) for uncontaminated benchmarks, the performance gap between the original and reference data is not significant;
  • (b) on the same examples, the gap appears only after the model is trained on them;
  • (c) six reference variants lead to the same conclusion, and the performance gaps among the reference variants are very small.

(a) Contaminated vs. uncontaminated benchmarks on the same model. The paper already contains two such uncontaminated settings: GSM1K (Zhang et al., 2024) for Qwen-Math and DeepSeek-Math (Table 5), and Dataset U for the FT models (Table 8). The datasets are clean on the corresponding models. We further add OpenR1-Qwen-7B (Hugging Face, 2025), which is contaminated on AIME 1983–2023 problems and clean on AIME 2025/26 (60 problems each; Appendix F.2). All uncontaminated originals are also human-written and their references come from the same pipeline as the contaminated data. Numerical sensitivity, arithmetic difficulty, answer-token priors, template familiarity, and generator artifacts are therefore present on both sides.

Table R1. Zero-CoT AccAcc on original / reference (SS / SrefS_{ref}, %) and Ccont\mathcal{C}_{cont} on contaminated vs. uncontaminated data.

ModelContaminatedUncontaminated
Qwen-MathGSM8K: 45.7 / 33.2, 1.000GSM1K: 23.3 / 22.2, 0.500
DeepSeek-MathGSM8K: 29.7 / 21.7, 0.993GSM1K: 16.0 / 18.3, 0.500
FT Qwen-MathDataset C: 24.3 / 16.9, 1.000Dataset U: 13.8 / 12.3, 0.551
FT Qwen3Dataset C: 25.0 / 15.1, 1.000Dataset U: 15.3 / 14.4, 0.567
OpenR1-Qwen-7BAIME 1983-2023: 28.3 / 0.0, 1.000AIME 2025/26: 1.7 / 1.7, 0.500

Due to space limits, the table only shows AccAcc, and the full results are in Tables 4, 5, 8, and 15. Across all four metrics, Ccont\mathcal{C}_{cont} is 0.500–0.638 on the uncontaminated benchmarks and 0.964–1.000 on the contaminated ones. (ConCon is excluded for OpenR1, whose Full-CoT outputs are too long.) The results show that the gap only appears on the contaminated data and does not appear on clean data.

(b) Same examples and reference dataset; only exposure changes. We train Qwen3-8B with SFT for 400 steps on six paraphrased versions of the business questions in the multi-domain Dataset C (Appendix I.3, Table 21). The checkpoints are all evaluated on the same original examples and the same reference dataset. Numerical sensitivity, arithmetic difficulty, answer-token priors, template familiarity, and generator artifacts are therefore held fixed.

The results show that the performance gap emerges and grows as training exposure increases.

Table R2. Zero-CoT AccAcc (SS / SrefS_{ref}, %) and Ccont\mathcal{C}_{cont} of Qwen3-8B at different SFT steps, with the same examples and reference dataset throughout.

SFT steps0100200300400
SS / SrefS_{ref}16.6 / 15.815.8 / 14.122.0 / 17.424.5 / 16.628.2 / 15.4
Ccont\mathcal{C}_{cont}0.5000.5080.6780.9510.999

References

AuthorsOpenReview ↗

Response to Reviewer ZUbB 2/3

(c) Six reference variants. We used the pipeline of Appendix B to generate six reference variants for every original example of GSM8K and MATH used in Table 4. We conducted originals vs. each variant, and each variant vs. every other variant comparisons (30 ordered pairs). The results are shown in Table R3. We observed Ccont>0.997\mathcal C_{cont} > 0.997 in all 96 original-vs-variant tests (4 settings × 4 metrics × 6 variants), the same conclusion as with the paper's reference dataset (Table 4). In every setting, the original–variant gap is larger than every variant–variant gap. This also holds for ConCon and Pfirst\mathcal P_{first}. So the gap mainly comes from contamination, rather than other factors.

We can also average different reference variants to further reduce the influence of any single variant. With the average of the six variants as the reference, Ccont>0.998\mathcal C_{cont} > 0.998 in all settings. We recommend this practice and show these comparisons in Appendix C.2 and Figure 4.

Table R3. Performance gap (S−SrefS - S_{ref}) between the original examples and each of the six variants (range), and the largest performance gap between two variants. AccAcc in points, Pall\mathcal P_{all} × 100.

Model · dataAccAcc: original vs. variantsAccAcc: max variant vs. variantPall\mathcal P_{all}: original vs. variantsPall\mathcal P_{all}: max variant vs. variant
DeepSeek-Math · MATH22.9–23.91.017.9–20.32.3
DeepSeek-Math · GSM8K10.8–17.46.612.2–19.67.4
Qwen-Math · MATH22.6–28.76.116.0–17.92.0
Qwen-Math · GSM8K17.6–24.26.613.6–20.16.5

W2: Contamination Confidence is neither a calibrated posterior probability of contamination nor a measure of contamination severity

We thank the reviewer for this precise point and agree with it. Ccont\mathcal C_{cont} is obtained from the one-sided p-value and an upper bound on the Bayes factor (Sec. 3.6, Appendix A). It only measures the strength of evidence for contamination, not its probability or its severity. Sec. 3.6 already calls Ccont\mathcal C_{cont} a "bound-based contamination evidence score" (l.313), but the abstract (l.026), the contributions (l.067), and the conclusion (l.509) described it as measuring severity. We have corrected all of them.

As the reviewer notes, Ccont\mathcal C_{cont} approaches one for a fixed gap once N is large. This is reasonable, because observing the same gap on a larger dataset is stronger evidence. We also add an experiment (Appendix I.3, Figure 6) that fixes N = 200 and varies the proportion of contaminated examples by mixing k GSM8K examples (contaminated) with 200 − k GSM1K examples (uncontaminated). Since Ccont\mathcal C_{cont} depends on which examples are drawn, for each proportion we draw 500 random benchmarks and report the median Ccont\mathcal C_{cont}.

Table R4. Qwen-Math, N = 200: median Ccont\mathcal C_{cont} of AccAcc over 500 random benchmarks vs. the proportion of contaminated examples.

Contaminated proportion0%10%20%30%40%50%60%70%80%90%100%
Median Ccont\mathcal C_{cont}0.5000.5130.5640.6260.7000.8020.8540.9010.9580.9730.985

Ccont\mathcal C_{cont} increases with the contaminated proportion, but not linearly: it stays close to 0.5 at 10–20%, rises steadily from 30% to 80%, and flattens above 80% (0.958–0.985). So at a fixed N, more contamination yields stronger evidence. Finally, because the Bayes-factor bound is never below 1 (Eq. 1), Ccont\mathcal C_{cont} is bounded below by 0.5, which means "no statistical evidence" rather than evidence for the null hypothesis. Light contamination can produce a small gap and thus a score near 0.5 (Table R4), and we now state both points explicitly in Sec. 3.6 (l.341–345) and Appendix I.3 (l.1359–1360).

AuthorsOpenReview ↗

Response to Reviewer ZUbB 3/3

Q3: What is the evidence of "definitive contamination"? Are model-dataset pairs assigned scores of approximately 0.9 contaminated approximately 90% of the time?

We agree that "definitive contamination" is not justified and have removed it from the paper, because Ccont\mathcal C_{cont} is an evidence score rather than a probability or a measure of severity. To test the calibration directly, we build 16,800 model-dataset pairs with known contamination status from our existing per-example results. Each dataset is a random subset of 30, 50, or 100 examples drawn from one contaminated pool (the GSM8K and MATH training splits for Qwen-Math and DeepSeek-Math; the AIME problems in the training data of OpenR1-Qwen-7B) or one uncontaminated pool (GSM1K; all AIME 2025/26 problems), and half of the pairs (8,400 pairs) are contaminated. Table R5 (Table 22 in Appendix I.3) reports the share of contaminated pairs in each Ccont\mathcal C_{cont} range.

Table R5. Share of contaminated pairs in each Ccont\mathcal C_{cont} range.

Ccont\mathcal C_{cont}AccAccConConPfirst\mathcal P_{first}Pall\mathcal P_{all}
= 0.512.0%15.4%14.8%3.4%
(0.5, 0.6]54.5%56.4%30.1%17.3%
(0.6, 0.7]80.5%84.3%53.1%41.4%
(0.7, 0.8]94.3%94.8%67.2%68.8%
(0.8, 0.9]97.6%98.0%84.3%85.1%
(0.9, 0.95]99.5%99.7%93.1%96.2%
(0.95, 0.99]100%100%98.2%98.9%
(0.99, 1]100%100%100%100%

We can see Ccont\mathcal C_{cont} is not a probability, but it is positively related to one: the share of contaminated pairs increases monotonically with Ccont\mathcal C_{cont} for all four metrics. Above 0.6, AccAcc and ConCon are conservative, whereas Pfirst\mathcal P_{first} and Pall\mathcal P_{all} overstate between 0.5 and 0.8. Uncontaminated pairs reach Ccont≥0.9\mathcal C_{cont} \ge 0.9 in at most 0.83% of cases. Along the SFT trajectory (Table R2), Ccont\mathcal C_{cont} likewise increases with exposure.

Q4: What happens when only 10% of benchmark examples are included in training, each with a single exposure? Does ZCP detect pretraining-only contamination?

Table R4 is the test-side analogue of this question. With 10% contaminated examples, the median Ccont\mathcal C_{cont} at N = 200 is only 0.51 (AccAcc), and clear evidence needs roughly 50%. At 0.1–1%, a 200-example benchmark contains only 0–2 contaminated examples, which is too few for dataset-level detection methods. Therefore, if the proportion of contaminated data is very low, Ccont\mathcal C_{cont} will also be around 0.5.

For pretraining-only contamination, we use OLMo-2-1124-7B (OLMo Team et al., 2025), an open-data model whose stage-2 pretraining data contain the 500 GSM8K training examples. We evaluate its released checkpoints before and after exposure on these examples and the corresponding reference dataset. Table R6 (Table 14 in Appendix F.1) shows that before exposure there is no evidence (Ccont\mathcal C_{cont} around 0.5), while after exposure ZCP finds evidence on all four metrics. The results demonstrate the effectiveness of ZCP in detecting pretraining-only contamination.

Table R6. OLMo-2-1124-7B (base) on 500 GSM8K examples seen during stage-2 pretraining: zero-CoT AccAcc (SS / SrefS_{ref}, %) and Ccont\mathcal C_{cont}.

CheckpointZero-CoT AccAccCcont\mathcal C_{cont} (AccAcc / ConCon / Pfirst\mathcal P_{first} / Pall\mathcal P_{all})
End of stage 1 (before exposure)2.8 / 2.60.500 / 0.500 / 0.529 / 0.508
Stage 2, after 26B tokens training13.8 / 8.20.991 / 0.988 / 0.977 / 0.973

References

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.