- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes three behavioral contamination detectors for multiple-choice benchmarks, each with an explicit null hypothesis. DOPR rotates option order and uses Latin-square residuals to remove letter and position preferences; DWPC compares the model's support for the correct answer under original versus paraphrased wording; and CBA hides the options, asks the model to answer from the question stem alone, and builds a null distribution by permutation. Evidence from the three is combined with e-values and controlled with e-BH and Holm. The authors compare against 19 published methods on two controlled testbeds (WikiCutoffQA and SyntheticSourcesQA) with Qwen3.5-4B, and audit 35 models on MMLU, MMLU-Pro, GPQA-D, and HLE.
Critical strengths
- The core idea is clear and valuable. A clean model's own preferences (letter preference, option-content preference, sensitivity to paraphrasing) can easily masquerade as contamination signals. The paper explicitly separates these preferences from contamination and states an explicit null hypothesis for each detector, which is more rigorous than many detection methods that output only a score.
- The statistical framework is solid and the results are reproducible. The e-value combination and the e-BH and Holm procedures are clearly derived. I recomputed the e-values and the e-BH and Holm decisions in the main tables from the paper's formulas, and they all match; I found no numerical errors.
- The controlled experiments are well designed and convincing. Under SFT exposure, DOPR and DWPC-QA reach mean AUROCs of about 0.85 and 0.95, clearly above the published methods (at most about 0.59–0.69), and the ablations show that the debiasing steps work.
- The two main detectors are complementary, and this is analyzed well. When option order is shuffled, DOPR fails while DWPC-QA stays near 1.0; when questions are paraphrased, the opposite happens. The explanation is clear and useful for practitioners.
- The audit is large and the figures are of high quality. Auditing 35 models on four benchmarks is substantial work; all figures are vector graphics, and I verified the worked examples in the illustrations (Figures 2 and 3). The paper is also generally careful to state that rejecting the null is not proof of contamination.
Critical weaknesses
- My main concern is that the null is never calibrated on real, human-written benchmarks. All 28 open models are flagged on MMLU, and even the clean Qwen has a DWPC mean that is significantly different from 0 on the testbed (CI [−0.063, −0.014]). At n≈10⁴, a small misspecification of the null is enough to produce extreme p-values, so without a large clean control it is hard to tell "widespread contamination" from "a biased null". Adding a human-written multiple-choice set of comparable size, released after the models' cutoffs, as a negative control, plus a known contamination case as a positive control, would make the audit conclusions much more credible.
- The source of the paraphrases for the real benchmarks is not described, yet 7 of the 9 non-MMLU e-BH rejections rely on DWPC-QA. Please state which model generated the paraphrases, how many there are per item, and how meaning preservation was checked, and report whether the conclusions are stable under a different paraphraser.
- The controlled exposure is narrow, and the "leading performance" claim should be stated more cautiously. The experiments use a single 4B model, 20 epochs of fine-tuning, over 95% accuracy on exposed items, and "ten selected" checkpoints. In the CLM setting, which is closer to pre-training contamination, LOSS and Min-K% reach 1.000 while DWPC-QA reaches 0.91/0.88; under SFT, CBA is also below LOSS. I suggest adding dose–response experiments (e.g., a single epoch, or mixing into a larger corpus), reporting multiple seeds and AUROC confidence intervals, and adjusting the abstract accordingly.
- The annotation pipeline is circular. GPT-5.6 Sol generates the testbeds and annotates the item artifacts used for filtering, while also being one of the audited models, and it is flagged on MMLU. A manual spot check of the annotations, or a comparison with a second annotator model, would help.
Minor: the appendix figures for "contaminated" models (e.g., Figures 6a and 10a) use all 512 questions, mixing the 256 exposed and 256 control items, which differs from the setting of Table 1. The x-axis of Figure 1(a) is truncated at 20, which hides differences in magnitude between models. Typo on p.2: "themeselves".
Questions for authors
- How were the paraphrases for the real benchmarks generated and verified? Do the conclusions hold with a different paraphraser?
- How do the detectors behave on a large, human-written negative control set released after the models' cutoffs?
- Does the CBA conclusion hold when the permutation is stratified by question type and by the N and Vfail annotations?
- Do the 512 questions in the appendix figures mix exposed and control items? What is the effect size on the exposed items alone?
Overall, I find this a clear and statistically rigorous paper that makes a substantive contribution to the important problem of contamination detection, and I currently lean toward acceptance. If the authors can address questions 1 and 2 in the rebuttal, I would be more confident and would consider raising my score.
2 comments
Reply part 2
"GPT-5.6 Sol generates the testbeds and annotates the item artifacts used for filtering, while also being one of the audited models, and it is flagged on MMLU...comparison with a second annotator model, would help."
That’s a fair point. We used three to five GPT Sol 5.6 judgments per annotation to construct the original filters, which we also applied to GPT itself. To address this concern, we repeated the annotation protocol with Claude Opus 5.5. The final filtering decisions agree on approximately 99.8% of questions. For example, there are only 31 disagreements among the 14,042 MMLU questions.
The following preliminary results compare GPT Sol 5.6’s results on MMLU under the different filters:
| Filtering annotations | p value |
|---|---|
| Original Sol | 2.22e−66 |
| Opus only | 2.15e−64 |
| Exclude if either annotator excludes | 3.47e−64 |
GPT Sol 5.6 remains flagged using Opus’s annotations alone, or when excluding every question excluded by either annotator.
We also checked the other audited models and found that all flagging decisions remain unchanged: flagged models remain flagged, and unflagged models remain unflagged.
We would also like to clarify that GPT Sol 5.6 was not an audited model on the testbeds it generated, so its role in their generation does not create the same circularity.
"the appendix figures for "contaminated" models (e.g., Figures 6a and 10a) use all 512 questions, mixing the 256 exposed and 256 control items"
Thank you for this suggestion. We agree that showing only the 256 exposed questions is more informative and makes the signals from DOPR and DWPC-QA more pronounced. Restricting the analysis to these questions gives:
- DOPR (Figure 6a): mean 1.175 logits, 95% CI [0.934, 1.415].
- DWPC-QA (Figure 10a): mean 1.411 logits, 95% CI [1.167, 1.655].
We will update these histograms and their captions accordingly in the revision.
"The x-axis of Figure 1(a) is truncated at 20"
We truncated the x-axis to save space, but included arrows with the exact e-values. In the revision, we will add a version showing the full x-axis range to the supplementary material to make the scale clear.
Thank you also for pointing out the typo. We will correct it.
“Does the CBA conclusion hold when the permutation is stratified by question type and by the N and Vfail annotations?”
If we understand you correctly, you are asking to compare the original CBA answer-key shuffle with the subject conditioned shuffle (Appendix A.3.3), retaining the original filtration that excludes reconstructable mappings (M) and deducible correct answers and labels (GT∩L). For completeness, following your question, we compare both procedures with and without additional stratification by N×VFail. The results below are for MMLU.
| Shuffle procedure | Original e-BH flags retained | Original Holm flags retained | Changes to original flags |
|---|---|---|---|
| Original answer-key shuffle | 28/28 | 30/30 | None |
| Answer-key shuffle within N×VFail | 28/28 | 30/30 | None |
| Subject conditioned answer-key shuffle | 27/28 | 29/30 | DeepSeek-V4-Flash loses e-BH. Gemma-3-27B-IT loses Holm. |
| Subject conditioned answer-key shuffle within N×VFail | 27/28 | 29/30 | DeepSeek-V4-Flash loses e-BH. Gemma-3-27B-IT loses Holm. |
DeepSeek-V4-Flash loses its e-BH flag under subject conditioned shuffling both with and without N×VFail stratification, while retaining Holm. Its e value decreases from 41.32 originally to 8.16 with subject conditioning alone and 8.91 with the additional stratification. Thus, the flag change already occurs under subject conditioning alone.
Similarly, Gemma-3-27B-IT loses its Holm flag under subject conditioning both with and without N×VFail stratification. Its raw p value used by Holm changes from 0.002776 originally to 0.053490 with subject conditioning alone and 0.054992 with the additional stratification.
We will include the results of all these additional experiments and analyses in the revision. We hope we have addressed your remaining concerns, particularly questions 1 and 2, and that you now feel more confident and would consider raising your score.
Reply part 1
Thank you for your detailed and constructive review.
You make a valid point. Adding human written controls would strengthen the audit beyond our existing synthetic controls and examples of unflagged models on real benchmarks (Figures 5 and 9).
Following your review, we ran preliminary tests on 553 published MCQs from three collections. We used questions published after 2 March 2026, the date of the model version we tested. We screened for recent events and reports, excluding clear historical fact questions while preserving the original questions, options and answer keys.
We used each full dataset as a negative control. Neither method flags any of the three datasets.
For positive controls, we split each of the ABC and RealTimeQA collections into exposure and control halves, then trained separate models on each dataset only on the exposure half. Means, CIs and p values below concern exposed questions. AUROC compares exposed questions with untouched controls.
These new results, in accordance with your suggestion, further support our null assumptions on human written MCQs: neither method flags the clean sets, and both detect the known contamination.
For the real benchmarks, we've used the same protocol and model (GPT Sol 5.6) as we did for the testbeds. We generated one paraphrase per item and checked meaning preservation using independent annotator instances. We will clarify this in the revision.
That’s a great question. Following your question, we used Claude Opus 5.5 to generate alternative paraphrases and repeated DWPC-QA for these seven models to test whether the results remain consistent. Recomputing the non-MMLU aggregates using the final revised paraphrases preserves all e-BH and Holm rejections. Interestingly, the evidence becomes stronger for all seven models, reflected in higher aggregate e-values and lower model p values supplied to Holm.
We agree and will tone down the claim in the revision. We would still like to emphasize the performance demonstrated in our SFT experiments on our testbeds, for example by revising the abstract to “demonstrate leading detection performance under SFT exposure on our testbeds.” Would this narrower wording address your concern?
We would also like to point out that, in our response to reviewer nTE5, we report preliminary results on additional LLMs beyond Qwen3.5-4B, supporting the same conclusions on our testbeds.
Thank you also for your other suggestions, which we will work to incorporate in the revision.
[1] Kasai et al. RealTime QA: What's the Answer Right Now? NeurIPS 2023