- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies why benchmark exposure through policy distillation may evade conventional data-contamination detection. Existing detectors commonly rely on abnormal confidence, low entropy, or overfitting signatures produced by hard-label training, whereas policy distillation asks the student to match a teacher's soft distribution. Consequently, exposure to evaluation examples need not produce the same detectable signature. The paper presents an idealized theoretical analysis, constructs the Distillation-MIA benchmark using MATH, MulDimIF, XSum, and AIME2026, and compares matched SFT, online-distillation, and offline-distillation settings. Eight conventional detectors perform close to chance in most distillation settings. The paper then proposes teacher-student KL-based detectors, PD-Div and PD-Div-Gain, which improve detection when the teacher and, for PD-Div-Gain, the pre-distillation student checkpoint are available.
Critical strengths
-
The paper identifies an important and insufficiently studied evaluation-security problem: a model can be exposed to benchmark data through policy distillation without developing the confidence signatures assumed by conventional contamination detectors.
-
The theory, controlled benchmark, and proposed detectors form a coherent investigation. In particular, matched comparisons between SFT and online/offline distillation help attribute the detection failure to the training objective rather than to unrelated dataset or model differences.
-
The experiments cover mathematical reasoning, instruction following, and summarization, and report both AUROC and low-FPR metrics. PD-Div-Gain improves average AUROC over the strongest conventional detector by approximately 13.1 and 12.4 points in the online and offline settings, respectively.
Critical weaknesses
- PD-Div requires access to the teacher's full output distribution, while PD-Div-Gain additionally requires the pre-distillation student checkpoint. These objects are rarely available to third-party auditors, making the methods privileged-access diagnostics rather than generally deployable detectors.
2.The theoretical results rely on idealized independent-logit dynamics, path-length conditions, and Lipschitz detectors. They do not model cross-example coupling induced by shared neural-network parameters and do not establish general undetectability.
3.The empirical study mainly uses one Qwen2.5 teacher-student family.
Questions for authors
1.In the realistic setting without the pre-distillation student checkpoint, what are the complete AUROC and low-FPR gains of PD-Div over the strongest conventional baseline on each dataset? If most of the improvement comes from PD-Div-Gain, please clarify how strongly the main detection claim depends on checkpoint access.
2.Since the Qwen3 audit has no reliable membership labels, what non-contamination explanations could account for the observed divergence signals, such as teacher capability, task difficulty, or model-origin mismatch? Please state precisely what this case study does and does not establish.
Thank you for recognizing the importance of this problem. We address the three weaknesses and two questions below:
Weakness 1. Practical access and the detection signal
Our methods really requires the teacher's output distribution or the pre-distillation student and these references are difficult to obtain in public-model audits. However, our main contribution is identifying and formalizing the detection problem introduced by policy distillation, not simply proposing another detector. Our methods show a possible way to recover a signal; their access requirements also illustrate why this problem remains difficult in practice.
Weakness 2. Theoretical assumptions and shared parameters
You are right. However, these are sufficient conditions in theorem 1: our goal is to establish that detection can fail under policy distillation, not to require every training scheme or sample to satisfy them. Our experiments use more realistic shared-parameter training, where conventional detectors still struggle to distinguish known members from nonmembers. This indicates that the contamination problem is universal existance in both theory and real.
Weakness 3. Model coverage
Given the time and computational constraints, we added a smaller Llama 3B teacher / 1B student pair, keeping the remaining settings the same as in the Qwen experiments(Table 11-16).
PD-Div-Gain achieves mean AUROCs of 67.82%/67.87% online/offline, compared with 56.95%/59.75% for the strongest conventional baseline averages. This supports the benefit of teacher-aware calibration beyond the Qwen family. We also repeated the experiments with three training seeds and computed bootstrap confidence intervals(Table 5-26). All results are included in Appendix D.
Question 1. Performance without the initial checkpoint
As described in Section 5.3 and Table 3, PD-Div uses only the teacher and trained student; PD-Div-Gain additionally calibrates against their pre-existing agreement using the initial student. Table 4 reports both methods. The best baseline's mean AUC is 53.1%/52.1% (online/offline) while PD-Div's mean AUC are 57.96%/56.87% and 68.39%/66.00% for PD-Div-Gain. Thus, checkpoint calibration substantially strengthens detection.
To test whether this improvement comes merely from checkpoint access, we also added pre/post-checkpoint calibration to all eight conventional baselines under identical settings(Table 17-20).
Their best four-dataset mean AUC are only 56.09%/56.95%. This supports a benefit from teacher-aware scoring beyond checkpoint access alone.
Question 2. Interpretation of the Qwen3 audit
You are right that currently our analysis can only concern distribution-level differences rather than definitive membership judgments for individual problems. This because there are only 30 problems per year and no verified positive membership labels, the current audit does not provide enough evidence to conclude that AIME 2025 was contaminated.
We therefore supplemented high PD-Div-Gain samples for qualitative clues. Some high-Gain AIME 2025 responses appeal to problem-specific "known results" after incomplete derivations, yet give incorrect answers. These patterns motivate further investigation of possible exposure to solution narratives. We have included these exploratory examples and temporal-control comparisons in Appendix D.6. We hope this work can attract more attention on this issue and encourages larger auditing datasets with verified exposure labels and documented training histories.