acceptodds
Under review as a conference paper at ICLR 2027

Benchmark Deja Vu: Debiasing Signals for Multiple-Choice Contamination

Abstract

Benchmark contamination compromises the reliability of evaluations used by frontier labs to measure progress and by users to compare language models. Because ordinary model behavior can mimic contamination signals, reliable detection remains challenging. We introduce three contamination detectors with explicit null hypotheses that adjust for specified response preferences. We apply these new detectors to 35 contemporary open and closed models across four widely used multiple-choice benchmarks: MMLU, MMLU-Pro, GPQA Diamond, and the multiple-choice subset of Humanity’s Last Exam (HLE). This audit reveals that most models produce at least one statistically significant signal on MMLU, whereas signals on other benchmarks are more selective, with family members sometimes sharing signals on the same benchmark. We aggregate detector-specific e-values to rank models by the strength of evidence against the global null across detectors and benchmarks. In addition, we construct two new testbed datasets, WikiCutoffQA and SyntheticSourcesQA, whose questions are screened so that model accuracy before exposure is consistent with random guessing. With these datasets, we can induce controlled exposure by training models on the original questions (or on versions with shuffled choices or paraphrased wording). Using these testbeds, we compare our detectors with 19 published contamination detection methods and demonstrate leading detection performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, MCQ Contamination Detection, MCQ Benchmark Contamination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pj7AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes three behavioral contamination detectors for multiple-choice benchmarks, each with an explicit null hypothesis. DOPR rotates option order and uses Latin-square residuals to remove letter and position preferences; DWPC compares the model's support for the correct answer under original versus paraphrased wording; and CBA hides the options, asks the model to answer from the question stem alone, and builds a null distribution by permutation. Evidence from the three is combined with e-values and controlled with e-BH and Holm. The authors compare against 19 published methods on two controlled testbeds (WikiCutoffQA and SyntheticSourcesQA) with Qwen3.5-4B, and audit 35 models on MMLU, MMLU-Pro, GPQA-D, and HLE.

Critical strengths
  1. The core idea is clear and valuable. A clean model's own preferences (letter preference, option-content preference, sensitivity to paraphrasing) can easily masquerade as contamination signals. The paper explicitly separates these preferences from contamination and states an explicit null hypothesis for each detector, which is more rigorous than many detection methods that output only a score.
  2. The statistical framework is solid and the results are reproducible. The e-value combination and the e-BH and Holm procedures are clearly derived. I recomputed the e-values and the e-BH and Holm decisions in the main tables from the paper's formulas, and they all match; I found no numerical errors.
  3. The controlled experiments are well designed and convincing. Under SFT exposure, DOPR and DWPC-QA reach mean AUROCs of about 0.85 and 0.95, clearly above the published methods (at most about 0.59–0.69), and the ablations show that the debiasing steps work.
  4. The two main detectors are complementary, and this is analyzed well. When option order is shuffled, DOPR fails while DWPC-QA stays near 1.0; when questions are paraphrased, the opposite happens. The explanation is clear and useful for practitioners.
  5. The audit is large and the figures are of high quality. Auditing 35 models on four benchmarks is substantial work; all figures are vector graphics, and I verified the worked examples in the illustrations (Figures 2 and 3). The paper is also generally careful to state that rejecting the null is not proof of contamination.
Critical weaknesses
  1. My main concern is that the null is never calibrated on real, human-written benchmarks. All 28 open models are flagged on MMLU, and even the clean Qwen has a DWPC mean that is significantly different from 0 on the testbed (CI [−0.063, −0.014]). At n≈10⁴, a small misspecification of the null is enough to produce extreme p-values, so without a large clean control it is hard to tell "widespread contamination" from "a biased null". Adding a human-written multiple-choice set of comparable size, released after the models' cutoffs, as a negative control, plus a known contamination case as a positive control, would make the audit conclusions much more credible.
  2. The source of the paraphrases for the real benchmarks is not described, yet 7 of the 9 non-MMLU e-BH rejections rely on DWPC-QA. Please state which model generated the paraphrases, how many there are per item, and how meaning preservation was checked, and report whether the conclusions are stable under a different paraphraser.
  3. The controlled exposure is narrow, and the "leading performance" claim should be stated more cautiously. The experiments use a single 4B model, 20 epochs of fine-tuning, over 95% accuracy on exposed items, and "ten selected" checkpoints. In the CLM setting, which is closer to pre-training contamination, LOSS and Min-K% reach 1.000 while DWPC-QA reaches 0.91/0.88; under SFT, CBA is also below LOSS. I suggest adding dose–response experiments (e.g., a single epoch, or mixing into a larger corpus), reporting multiple seeds and AUROC confidence intervals, and adjusting the abstract accordingly.
  4. The annotation pipeline is circular. GPT-5.6 Sol generates the testbeds and annotates the item artifacts used for filtering, while also being one of the audited models, and it is flagged on MMLU. A manual spot check of the annotations, or a comparison with a second annotator model, would help.

Minor: the appendix figures for "contaminated" models (e.g., Figures 6a and 10a) use all 512 questions, mixing the 256 exposed and 256 control items, which differs from the setting of Table 1. The x-axis of Figure 1(a) is truncated at 20, which hides differences in magnitude between models. Typo on p.2: "themeselves".

Questions for authors
  1. How were the paraphrases for the real benchmarks generated and verified? Do the conclusions hold with a different paraphraser?
  2. How do the detectors behave on a large, human-written negative control set released after the models' cutoffs?
  3. Does the CBA conclusion hold when the permutation is stratified by question type and by the N and Vfail annotations?
  4. Do the 512 questions in the appendix figures mix exposed and control items? What is the effect size on the exposed items alone?

Overall, I find this a clear and statistically rigorous paper that makes a substantive contribution to the important problem of contamination detection, and I currently lean toward acceptance. If the authors can address questions 1 and 2 in the rebuttal, I would be more confident and would consider raising my score.

2 comments
AuthorsOpenReview ↗

Reply part 1

Thank you for your detailed and constructive review.

"...Adding a human-written multiple-choice set of comparable size, released after the models' cutoffs, as a negative control, plus a known contamination case as a positive control, would make the audit conclusions much more credible."

You make a valid point. Adding human written controls would strengthen the audit beyond our existing synthetic controls and examples of unflagged models on real benchmarks (Figures 5 and 9).

Following your review, we ran preliminary tests on 553 published MCQs from three collections. We used questions published after 2 March 2026, the date of the model version we tested. We screened for recent events and reports, excluding clear historical fact questions while preserving the original questions, options and answer keys.

  • RealTimeQA (423 questions): CNN and The Week news quizzes covering current world events, March to November 2026 [1].
  • ABC (100): similar news MCQs from the Australian Broadcasting Corporation’s weekly quizzes, April to November 2026.
  • The Point (30): questions about recent Australian and international affairs from the Australia Institute’s weekend quizzes, June to October 2026.

We used each full dataset as a negative control. Neither method flags any of the three datasets.

DatasetDWPC-QA μ [95% CI]DOPR μ [95% CI]
ABC (100)0.050 [−0.048, 0.149]−0.007 [−0.074, 0.059]
RealTimeQA (423)0.0108 [−0.0253, 0.0468]0.0182 [−0.0095, 0.0460]
The Point (30)0.090 [−0.145, 0.325]−0.037 [−0.145, 0.070]

For positive controls, we split each of the ABC and RealTimeQA collections into exposure and control halves, then trained separate models on each dataset only on the exposure half. Means, CIs and p values below concern exposed questions. AUROC compares exposed questions with untouched controls.

DatasetDWPC-QA μ [95% CI]DWPC-QA AUROC (p value)DOPR μ [95% CI]DOPR AUROC (p value)
ABC5.384 [3.975, 6.793].978 (2.97e−10)1.395 [0.656, 2.134].875 (.000204)
RealTimeQA0.0811 [0.0579, 0.1043].866 (3.27e−11)0.0741 [0.0530, 0.0952].855 (2.58e−11)

These new results, in accordance with your suggestion, further support our null assumptions on human written MCQs: neither method flags the clean sets, and both detect the known contamination.

The source of the paraphrases for the real benchmarks is not described

For the real benchmarks, we've used the same protocol and model (GPT Sol 5.6) as we did for the testbeds. We generated one paraphrase per item and checked meaning preservation using independent annotator instances. We will clarify this in the revision.

“7 of the 9 non-MMLU e-BH rejections rely on DWPC-QA...Do the conclusions hold with a different paraphraser?”

That’s a great question. Following your question, we used Claude Opus 5.5 to generate alternative paraphrases and repeated DWPC-QA for these seven models to test whether the results remain consistent. Recomputing the non-MMLU aggregates using the final revised paraphrases preserves all e-BH and Holm rejections. Interestingly, the evidence becomes stronger for all seven models, reflected in higher aggregate e-values and lower model p values supplied to Holm.

ModelGPT e-valueOpus e-valueGPT p valueOpus p value
Qwen3-1.7B3.69e418.86e821.66e−454.86e−87
Qwen3-8B4.05e242.05e491.94e−282.74e−53
Qwen3-14B1.87e52.68e317.72e−92.61e−35
Qwen3-30B-A3B9.91e211.00e948.32e−264.02e−98
Qwen3.6-35B-A3B4.92e1661.61e2725.96e−1711.36e−276
MiniMax-M2.11.82e1001.43e1432.13e−1042.24e−147
GLM-5.21.65e101.28e146.91e−147.81e−18

“the ‘leading performance’ claim should be stated more cautiously...”

We agree and will tone down the claim in the revision. We would still like to emphasize the performance demonstrated in our SFT experiments on our testbeds, for example by revising the abstract to “demonstrate leading detection performance under SFT exposure on our testbeds.” Would this narrower wording address your concern?

We would also like to point out that, in our response to reviewer nTE5, we report preliminary results on additional LLMs beyond Qwen3.5-4B, supporting the same conclusions on our testbeds.

Thank you also for your other suggestions, which we will work to incorporate in the revision.

[1] Kasai et al. RealTime QA: What's the Answer Right Now? NeurIPS 2023

AuthorsOpenReview ↗

Reply part 2

"GPT-5.6 Sol generates the testbeds and annotates the item artifacts used for filtering, while also being one of the audited models, and it is flagged on MMLU...comparison with a second annotator model, would help."

That’s a fair point. We used three to five GPT Sol 5.6 judgments per annotation to construct the original filters, which we also applied to GPT itself. To address this concern, we repeated the annotation protocol with Claude Opus 5.5. The final filtering decisions agree on approximately 99.8% of questions. For example, there are only 31 disagreements among the 14,042 MMLU questions.

The following preliminary results compare GPT Sol 5.6’s results on MMLU under the different filters:

Filtering annotationsp value
Original Sol2.22e−66
Opus only2.15e−64
Exclude if either annotator excludes3.47e−64

GPT Sol 5.6 remains flagged using Opus’s annotations alone, or when excluding every question excluded by either annotator.

We also checked the other audited models and found that all flagging decisions remain unchanged: flagged models remain flagged, and unflagged models remain unflagged.

We would also like to clarify that GPT Sol 5.6 was not an audited model on the testbeds it generated, so its role in their generation does not create the same circularity.

"the appendix figures for "contaminated" models (e.g., Figures 6a and 10a) use all 512 questions, mixing the 256 exposed and 256 control items"

Thank you for this suggestion. We agree that showing only the 256 exposed questions is more informative and makes the signals from DOPR and DWPC-QA more pronounced. Restricting the analysis to these questions gives:

  • DOPR (Figure 6a): mean 1.175 logits, 95% CI [0.934, 1.415].
  • DWPC-QA (Figure 10a): mean 1.411 logits, 95% CI [1.167, 1.655].

We will update these histograms and their captions accordingly in the revision.

"The x-axis of Figure 1(a) is truncated at 20"

We truncated the x-axis to save space, but included arrows with the exact e-values. In the revision, we will add a version showing the full x-axis range to the supplementary material to make the scale clear.

Thank you also for pointing out the typo. We will correct it.

“Does the CBA conclusion hold when the permutation is stratified by question type and by the N and Vfail annotations?”

If we understand you correctly, you are asking to compare the original CBA answer-key shuffle with the subject conditioned shuffle (Appendix A.3.3), retaining the original filtration that excludes reconstructable mappings (M) and deducible correct answers and labels (GT∩L). For completeness, following your question, we compare both procedures with and without additional stratification by N×VFail. The results below are for MMLU.

Shuffle procedureOriginal e-BH flags retainedOriginal Holm flags retainedChanges to original flags
Original answer-key shuffle28/2830/30None
Answer-key shuffle within N×VFail28/2830/30None
Subject conditioned answer-key shuffle27/2829/30DeepSeek-V4-Flash loses e-BH. Gemma-3-27B-IT loses Holm.
Subject conditioned answer-key shuffle within N×VFail27/2829/30DeepSeek-V4-Flash loses e-BH. Gemma-3-27B-IT loses Holm.

DeepSeek-V4-Flash loses its e-BH flag under subject conditioned shuffling both with and without N×VFail stratification, while retaining Holm. Its e value decreases from 41.32 originally to 8.16 with subject conditioning alone and 8.91 with the additional stratification. Thus, the flag change already occurs under subject conditioning alone.

Similarly, Gemma-3-27B-IT loses its Holm flag under subject conditioning both with and without N×VFail stratification. Its raw p value used by Holm changes from 0.002776 originally to 0.053490 with subject conditioning alone and 0.054992 with the additional stratification.

We will include the results of all these additional experiments and analyses in the revision. We hope we have addressed your remaining concerns, particularly questions 1 and 2, and that you now feel more confident and would consider raising your score.

Reviewer_nTE5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes three contamination detectors with explicit null tests and two controlled MCQ testbeds. It compares against 19 prior methods and also audits 35 released models on MMLU MMLU-Pro GPQA and HLE.

Critical strengths

Very thorough evaluation. The controlled testbeds are especially useful since exposure is known and the new methods are much stronger than prior baselines under SFT.

Critical weaknesses

The controlled exposure experiments only use Qwen3.5-4B so not clear how much the results transfer to other model families or training recipes. Also the strongest methods need logits so the closed model audit mostly relies on CBA which looks quite weaker.

Questions for authors

Did you test controlled exposure on any model other than Qwen3.5-4B? I think this would affect how confident I am that the detector gains are general.

1 comment

Thank you for your constructive feedback.

The controlled exposure experiments only use Qwen3.5-4B so not clear how much the results transfer to other model families or training recipes

You make a good point. We agree that experiments on more model families would help show transferability. Following your feedback, we started running controlled exposure experiments with Nemotron 3 Nano 30B A3B and Gemma 4 31B instruct. Their training predates the knowledge in our testbeds. Below we give preliminary results in a table similar in structure to Table 1 in the paper.

ModelTestbedExposure formDOPRDWPC-QACBA-shufflee-value aggregate
Gemma 4WCQAComplete item0.94530.94530.62890.9766
Gemma 4WCQAQuestion paraphrase1.00000.72660.56250.9961
Gemma 4SSQAComplete item0.99540.92940.61110.9890
Gemma 4SSQAQuestion paraphrase0.99760.91980.49440.9937
Nemotron 3 NanoWCQAComplete item0.78520.98830.66210.9023
Nemotron 3 NanoWCQAQuestion paraphrase0.95310.92970.77340.9883
Nemotron 3 NanoSSQAComplete item0.98360.95750.56030.9819
Nemotron 3 NanoSSQAQuestion paraphrase0.99930.97750.53860.9985

These new results solidify the results in the paper (Table 1). Moreover, following your comment, we also evaluated all literature methods from Tables 11 and 12. Our new results here show again that our methods are superior on the testbeds with SFT. Among the literature methods, the best scores remained moderate and below DOPR and DWPC-QA in the corresponding configurations. Among the literature methods, the highest AUROC across the eight configurations (with Gemma and Nemotron) was 0.7305, achieved by “Neighbourhood Comparison” [1] on Gemma 4 with WCQA under Complete-Item Exposure. However, its mean AUROC across the eight configurations was 0.583. Across all eight configurations, the highest average AUROC among the literature methods was 0.605, achieved by Min-K% [2].

We will of course include the full results in the revision.

...CBA which looks quite weaker

We agree that CBA is weaker overall than DOPR and DWPC-QA. For this reason, we are working on a simple enhancement- instead of making a single generation per question, we sample N responses, with the answer choices still hidden. This helps distinguish lucky correct guesses from consistently choosing the correct label. Preliminary results already show higher AUROC than the baseline single generation variant on the testbeds, reaching 0.789 under Complete item exposure.

On the real benchmarks (MMLU, MMLU Pro, GPQA D, and HLE), interestingly, replacing CBA with this enhanced variant leaves the existing Holm and e-BH flags of the combined analysis unchanged. On individual benchmarks, the enhanced CBA identifies additional signals. In HLE, MiniMax M2.5 and MiniMax M2.1 now have CBA p values below 0.05 on HLE, at 0.000315 and 0.04535, respectively. This strengthens the already existing signal from this model family on the HLE dataset. On MMLU, the enhanced CBA also identifies Qwen 3 4B and DeepSeek V4 Pro, which were already identified by DOPR or DWPC QA.

[1] Mattern et al. Membership Inference Attacks against Language Models via Neighbourhood Comparison. ACL 2023

[2] Shi et al. Detecting Pretraining Data from Large Language Models. ICLR 2024

Reviewer_E59mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose three contamination detectors with null hypotheses that adjust for specified response preferences in current benchmarks. The three detectors are compared with 19 published contamination detection methods and claim to demonstrate leading detection performance.

Critical strengths
  • Broad selection of models across open- and closed-weights and general capability evaluations.
  • Four forms of benchmark item exposure are outlined which remain tractable and focus on answer-order.
Critical weaknesses
  • Writing is incoherent/non-linear at times (line 217, for example), perhaps worth having further human review.
  • Narrative of the overall paper does not lead me towards discussions of applicability or novelty.
  • Figures need to be clearer, perhaps worth having further human review to ensure captions provide a clear takeaway. I do not see a single figure with a clear takeaway.
  • Related work section is missing
  • Discussions section is missing

If the authors could please respond to these points in the rebuttal, more than happy to reconsider my score.

Questions for authors

Could you please clarify the novelty and applicability of your work?

2 comments
AuthorsOpenReview ↗

Reply part 1

Thank you for your review. Since you mentioned human review, we wanted to emphasize that we worked out the content and structure of the paper ourselves. We used LLMs to help with the writing and polishing, as we explain in the AI use statement. But humans were involved, responsible and reviewing everything throughout.

Writing is incoherent/non-linear at times

We wrote the paper with the following logical flow: first the introduction and problem setup, then our detection methods, then the testbeds and results to evaluate their effectiveness. Finally, we present signals on widely used benchmarks in recent LLMs. We tried to build every part on top of the previous one.

If you could give us further concrete examples for incoherent parts, we would happily revise the text. We would also appreciate a clarification if you meant something else specifically.

line 217, for example

Could you please clarify this example? Line 217 presents the DOPR formula, and the paragraph explains the intuition behind it. The score asks whether the correct answer gets extra support in its original arrangement after accounting for ordinary label and answer preferences. If you mean the part with zi⋆−μi,σ(0)z_i^\star - \mu_{i,\sigma^{(0)}}, we didn't explicitly show how the first two terms equal this expression:

= z_i^\star - \mu_{i,\sigma^{(0)}}.$$ > Could you please clarify the novelty and applicability of your work? Our work makes several contributions to contamination detection. We list them below, along with their novelty compared with previous work: (a) Three detection methods with explicit null hypotheses, designed to separate exposure signals from label and answer preferences. DOPR and DWPC-QA outperform all 19 published baselines overall in our SFT experiments (Table 1 and Appendix E.2). Some of these baselines detect CLM exposure well but perform poorly under SFT in the same experiments. (b) Two testbeds using recent or fictional knowledge, screened for chance accuracy before exposure. We evaluate original, shuffled and paraphrased exposure, with control questions and detector paraphrases kept out of training. This addresses a problem with existing testbeds: public benchmarks may already be familiar to the model, while data collected after one model's cutoff may enter the training of later models. (c) An updated study of exposure signals in 35 recent open and closed LLMs on widely used benchmarks, including HLE, GPQA Diamond and MMLU Pro. (d) We identify question properties that violate detector assumptions and annotate 26,783 benchmark MCQs so they can be filtered separately for each method. We apply these filters to our own results. Appendix G.4 also shows how natural answer order alone can cause published methods to flag unexposed questions. Some previous work acknowledged this issue but left it as a limitation. Hidayat et al. [1], for example, discuss false positives from common answer patterns and leave handling them to future work. Here, we provide filters and apply them to our benchmark audits. This work allows practitioners to (i) look for exposure signals using our methods, and (ii) develop and evaluate their own detectors on our testbeds and apply them to real benchmarks using our filters. We also believe that clear null hypotheses and assumptions should receive more attention in the field. This is not novel on its own, but our emphasis on it is meant to encourage a move toward methods whose results are easier to interpret than heuristic scores alone. These contributions are already summarized throughout the introduction, especially in its last paragraph. In light of your comment, we will revise the paper to make the novelty and applicability clearer. [1] Hidayat et al. Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation. Eval4NLP 2025
AuthorsOpenReview ↗

Reply part 2

Figures need to be clearer...

We do understand your concern. Could you please point out specific figures or parts that are unclear so we can improve them? We do acknowledge that Figure 2 is too text heavy and packed, and intend to rework it in the revision. Figure 3 may also benefit from moving the uniform guessing null, which we kept for pedagogical reasons, to the appendix.

As for takeaways, most figures in the main body are intended only to explain the intuition behind the algorithms and filtration. For example, Figure 4 shows how question structure can produce false contamination signals, and why different methods need different filters.

As for figures with clear takeaways in general, we do think Figure 1 and Tables 1 and 2 convey them both visually and in their captions. Figure 1, which we intentionally placed near the start of the paper, highlights models with strong contamination signals using their names, red shading and threshold lines to communicate statistical significance. Its main takeaway is that signals are widespread on MMLU and more selective on the other benchmarks, often recurring within the same model families, as the repeated family names suggest. Table 2 gives more detailed results by model and benchmark, while Table 1 shows the success of our methods across the testbeds. But please direct us to any points you find confusing if you feel there are any in these so we can improve them.

Related work section is missing

We intentionally placed related work throughout the paper in the sections where they were most relevant because we felt it would make the paper easier to read. For example, we discuss previous testbeds at the beginning of Section 4. Appendix E, where we report results for detection methods from the literature, starts with a detailed summary and categorization of these methods.

However, if you feel this organization makes the related work harder to follow, we are open to consider adding a dedicated related work section that brings these discussions together. If you do agree with our structure however, we could also add a complete related works section in the appendix (for space considerations) and refer to it from the main paper.

Discussions section is missing

Section 6, titled "Conclusion", is our "Discussions" section. It includes a discussion of the paper and of future work. We agree a more extensive discussion is in order, and intend to extend it in the final revision.

We hope we've answered your points, and that you will reconsider your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.