- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FIRE, which detects test set contamination by analyzing the loss trajectories obtained from fine-tuning a suspect model on the target benchmark. The authors argue that a model previously exposed to the benchmark may show different learning dynamics when trained on it again, compared to a model seeing it for the first time. To detect contamination, FIRE 1) fine-tunes the suspect model on the benchmark and records its loss trajectory, 2) zero-normalizes the trajectory to remove the initial loss level, and 3) uses dynamic time warping (DTW) with k-nearest neighbors (kNN) to classify whether the model is contaminated based on the trajectory shape. The authors evaluate FIRE by deliberately introducing benchmark data during training and comparing clean and contaminated models. Experiments across six models and 44 benchmarks show 77–92% detection accuracy when contamination increases benchmark performance by at least 5 percentage points. The authors also evaluate cross-model transfer, partial contamination, and the effects of subsequent DPO training. They further argue that FIRE is more difficult to evade than static contamination detectors, as an adversary would need additional fine-tuning to observe the detection signal.
Critical strengths
-
Using training dynamics as a contamination signal: Unlike existing methods that mainly rely on static model outputs, FIRE actively fine-tunes the suspect model and analyzes how its loss changes. The authors also show that the signal remains after removing the initial loss level, suggesting that the learning dynamics themselves contain useful information about previous data exposure. This provides an interesting alternative approach to test set contamination detection.
-
Extensive evaluation across models and benchmarks: The authors evaluate FIRE on six models and 44 benchmarks under different poisoning ratios and training configurations. Table 2 shows that the detection signal transfers across different models with an average accuracy of 87%. The additional experiments on partial contamination and larger models further support the applicability of the proposed approach beyond a single experimental setting.
Critical weaknesses
-
Insufficient comparison with simpler trajectory-based methods: The authors compare FIRE mainly against five membership inference baselines, but do not clearly show whether DTW+kNN is necessary for detecting contamination. Figure 4 shows that clean and contaminated models have different loss decay patterns, and the early part of the trajectory already provides strong detection performance. However, it is unclear whether simpler methods, such as the initial loss reduction, average loss change, or Euclidean distance between normalized trajectories, could achieve similar results. Since the main idea is to use the loss trajectory as a contamination signal, these comparisons are important to distinguish the contribution of DTW+kNN from the basic differences in learning dynamics.
-
Limited validation under realistic contamination scenarios: The experiments simulate contamination by deliberately including benchmark data during an SFT-based training stage, and mainly evaluate cases where contamination increases benchmark performance by at least 5 percentage points. Although this provides a controlled evaluation, it may not fully reflect contamination occurring during large-scale pre-training. Moreover, Section 4.4 shows that FIRE can detect contamination even when the probing samples have little overlap with the leaked samples. This raises the question of whether FIRE specifically detects benchmark leakage or simply recognizes that a model has already been trained on similar data distributions. Without further evaluation distinguishing these cases, it is difficult to assess the reliability of FIRE in realistic auditing scenarios.
Questions for authors
-
Clarification of the contribution of DTW+kNN: Figure 4 shows that the early loss trajectory already contains a strong contamination signal. Could the authors compare FIRE against simpler trajectory-based methods, such as initial loss reduction, average loss change, or Euclidean distance between normalized trajectories? This would help clarify whether DTW provides additional benefits beyond the basic differences in learning dynamics.
-
Clarification of realistic contamination detection: Section 4.4 shows that FIRE can detect contamination even when the probing samples have little overlap with the leaked samples. Could the authors clarify how FIRE distinguishes actual benchmark contamination from legitimate training on different but distributionally similar examples? In particular, would a clean model trained on similar but non-overlapping data also be classified as contaminated? This would help assess the false-positive risk in realistic scenarios.
-
Clarification of adaptive evasion resistance: Section 4.5 argues that evading FIRE requires approximately 850 times more computation, but the paper does not experimentally evaluate an adversary directly optimizing against the detection signal. Could the authors clarify whether cheaper surrogate objectives or alternative poisoning strategies could reduce the detection accuracy without requiring a full probing run at every poisoning step? This would help assess the claimed resistance to adaptive evasion.
3 comments
Rebuttal (2/2)
W2. "The experiments simulate contamination by deliberately including benchmark data during an SFT-based training stage [...] it may not fully reflect contamination occurring during large-scale pre-training."
We inject contamination during mid-training because that is where benchmark-like data enters modern pipelines; open pipelines such as OLMo 2 add curated task data at this stage. Two results suggest the signal survives dilution and later training: FIRE detects contamination when benchmark data is only 0.5% of the training mix (§4.1), and reaches 84.4% accuracy after an additional DPO stage (§4.7). Contaminating full-scale pre-training in a controlled way would require pre-training models from scratch, which is outside the scope of this paper.
We focus on contamination that raises benchmark scores by at least 5 points because that is the contamination that misleads evaluation: Magar & Schwartz (2022) show that a model can see benchmark data without exploiting it, in which case its scores are unaffected.
We hope these results resolve the reviewer's concerns about soundness, and would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.
I thank the authors for answering my questions. I am now able to clarify what the main problem the paper wants to solve. I increased the score from 2 to 3.
We thank the reviewer for recognizing that "the learning dynamics themselves contain useful information about previous data exposure," and for the concrete suggestions. We ran the suggested simpler comparisons and investigated the realistic-contamination and evasion questions.
Thank you for the insightful suggestion! We ran all three comparisons on the same data and cross-validation splits as the main results (now in §4.2; per-model numbers in Appendix A.5).
Every method that reads the zero-normalized loss curve recovers the signal (0.79–0.86), while static-loss baselines stay at 0.57–0.61. Contamination is visible in how the model re-learns the benchmark, and this does not depend on DTW.
DTW's advantage is handling trajectories of different lengths, allowing audits with fewer samples than the reference library. Unlike Euclidean distance, it relaxes exact step-by-step alignment: a loss drop at step 10 can be matched with a similar drop at step 15. This discards some timing information and may reduce accuracy slightly, but our goal is to show that trajectory shape alone retains a strong contamination signal. With a 100-sample audit against a 1,000-sample library, DTW leads Euclidean distance on all six models (0.794 vs. 0.651).
Great question! Our experiments cover two levels of similarity: data that is distributionally similar but truly non-overlapping, and distributionally identical data drawn from the benchmark itself. The latter is sadly often a common practice to boost benchmark scores (e.g., generating synthetic data that looks exactly like the test set, vulgarly known as "benchmaxxing"), and we believe it should count as test set contamination.
Distributionally similar, non-overlapping data is the negative class in every experiment. Clean and contaminated models are matched pairs (§2): both are trained on Tulu-v2, an instruction mixture covering task types that many of our benchmarks test (natural-language inference, paraphrase detection, question answering), with the same number of samples and the same hyperparameters. The only difference is that the contaminated model has benchmark items in place of part of that data. FIRE classifies 84% of these clean models as clean (false-positive rate 16.1%; 8.5–22.0% across models).
Distributionally identical data is detected (§4.4). The model is trained on 100 items from a benchmark, and the auditor probes with a different 100 items from the same benchmark, largely disjoint from those leaked. FIRE detects contamination at 78.1%, close to the 79.4% when probing with the leaked items themselves.
The cheap strategies we tested weaken the signal only slightly. Without running the audit, an adversary can make the model memorize less or leak less data. The paper tests both:
Memorizing less: label smoothing during contaminated training lowers FIRE's accuracy from 71.2% to 68.3%, still about 7 points ahead of every baseline (§4.5).
Leaking less: FIRE detects contamination when only 100 of 1,000 benchmark items leak (78.1–79.4%, §4.4), and when benchmark data is only 0.5% of the training mix (§4.1).
Going further means optimizing against FIRE itself, which requires running the audit on the model at every poisoning step. §4.5 estimates this at about 850× the cost of the contamination itself (roughly 1,700 vs. 2 GPU-hours for a 7B model).