acceptodds
Under review as a conference paper at ICLR 2027

WildTrace: Evaluating Training Data Detection in Realistic Large-Scale LLM Post-Training

Abstract

Although post-training significantly enhances model performance, the use of massive undisclosed data raises concerns about benchmark contamination and copyright infringement. This drives growing interest in training data detection for post-trained models. Existing evaluations for training data detection often leverage benchmarks constructed post hoc, which can introduce distribution shifts. Alternatively, controlled evaluations rely on small-scale training setups that diverge from realistic post-training settings. As a result, their reported performance may not faithfully reflect the effectiveness of detection methods in real-world scenarios. To this end, we introduce , a reliable benchmark designed to evaluate detection methods in realistic, large-scale post-training settings. Built on a multi-stage pipeline, encompasses stage-specific models throughout the post-training lifecycle, spanning SFT, DPO, and RL. To perform a reliable evaluation, we construct model-specific evaluation sets across four diverse tasks, featuring distribution-matched members and non-members with verifiable membership labels. Our results reveal that even the best-performing method achieves an average AUC of only 0.529 on our benchmark, which approaches random-guessing level (0.5). Furthermore, our analysis suggests that evaluation protocols can significantly affect measured detection performance. These findings highlight the significant potential for developing effective, practical, and robust detection methods under realistic scenarios.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-training, Data Contamination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b2LuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper developed WildTrace, which was tested on three post-training stages and four types of tasks in OLMo-3 7B. It uniformly evaluated whether existing methods could identify the data used for training. Most methods performed close to random guessing under the tested conditions; for the experiments on SFT, it was also found that the detection effect declined as the data volume increased. The main contributions are the unified testing resources and systematic experimental analysis.

Critical strengths
  1. The study uniformly evaluated 12 detection methods in the SFT, DPO and RL stages, and the methods were identical for both members and non-members, resulting in relatively reliable results.

  2. It revealed that the existing methods have limited effectiveness in large-scale post-training, which is of reference value for practical applications.

Critical weaknesses
  1. The evaluation does not examine whether the same SFT samples remain detectable after DPO and RL, leaving the effect of subsequent training on earlier membership signals unclear.

  2. For DPO and RL, the reference-based experiments do not use the immediately preceding SFT and DPO checkpoints, respectively. It is therefore unclear whether reference-based methods also provide only marginal gains at these stages.

  3. All target-model experiments are conducted on OLMo-3 7B. Without validation on larger models or other model families, the representativeness and generalizability of the benchmark findings remain unclear.

Questions for authors

I would consider raising my score if the authors can better explain why existing detection methods fail under large-scale training and provide additional analyses or ablations to support this explanation.

7 comments

Thanks for your recognition and the valuable suggestions. Please find our response below.

1. Evaluating the detectability of SFT samples after DPO and RL training [W1]

Thank you for the comment. To examine how subsequent training affects earlier membership signals, we track the same SFT samples and non-members across successive post-training stages. To clearly observe how membership signals change across stages, we conduct this experiment under the small-scale setting in Section 5.1. Specifically, starting from ORACLE-SFT-10K, we further train it with DPO and then RL, each on 10K examples. The table below reports the AUC of detection on WildTrace-SFT at each checkpoint. The results show that subsequent DPO and RL training gradually weakens the detectability of SFT samples, with the average AUC decreasing from 0.800 to 0.786. These findings suggest that subsequent DPO and RL training can weaken detection signals, while the membership signals established during SFT remain largely detectable. We provide the detailed results in Appendix C.7.

ModelMathCodeIFQAAvg.
SFT0.7470.8400.8180.7930.800
SFT → DPO0.7390.8290.8120.7920.793
SFT → DPO → RL0.7340.8160.8040.7890.786

2. Results on reference-based methods [W2]

Thank you for the suggestion. We extend reference-based methods to all three post-training stages, using the preceding checkpoint of each stage as the reference model. The table below reports the average AUC across four tasks. The results show that reference-based methods still achieve limited detection performance, with an average AUC of at most 0.527 on ORACLE-DPO and 0.514 on ORACLE-RL. This confirms that reference-based methods can provide only marginal gains at the DPO and RL stages under large-scale post-training. We include the detailed results in Appendix C.7 of the revised version.

MethodORACLE-SFTORACLE-DPOORACLE-RLAvg.
Reference modelOLMo-3-1025-7BOLMo-3-7B-Instruct-SFTOLMo-3-7B-Instruct-DPO–
Perplexity0.5480.5120.5090.523
Entropy0.4820.5080.5030.498
Lowercase0.5740.5130.4990.529
Zlib0.5510.5220.5140.529
MIN-K%0.5560.5190.5020.525
MIN-K%++0.5920.5270.4850.535

3. Motivation for choosing OLMo-3 in our benchmark. [W3]

Thank you for the comment. We clarify our choice from the following three aspects.

Rationale for choosing OLMo-3. Our research aims to provide a reliable benchmark with verifiable membership labels under realistic post-training settings. This requires full training data transparency. We therefore choose OLMo-3, one of the few competitive model families that release all training data and recipes. Other mainstream LLMs from industry do not disclose their post-training data and recipes, so benchmarks constructed post-hoc on these models may introduce noisy membership labels, undermining their reliability. Moreover, OLMo-3 follows a standard architecture and a multi-stage post-training pipeline that closely mirror those of mainstream LLMs from industry, making our findings applicable to realistic post-training regimes. Although the OLMo-3 family includes a 32B model, repeating the post-training pipeline at this scale exceeds our current compute budget. We are actively seeking industry compute support to pursue this in future work.

Evaluation on another model family. To further examine the generalizability of our findings, we conduct experiments on the Apertus model family [1], a fully open model family that releases its complete training data. Due to limited computational resources, we construct the evaluation sets post hoc using the publicly released training data. Specifically, we construct balanced evaluation sets with equal numbers of members and non-members on QA and Math tasks. While post-hoc construction lacks the verifiable provenance emphasized in our manuscript, it can still serve as a sanity check here given the transparent data disclosure. The table below show that existing methods still exhibit limited detection performance on both the 8B and 70B models. These results are consistent with our main findings on OLMo-3, suggesting that our conclusions generalize to larger models and other model families.

MethodApertus-v1.5-8BApertus-v1.5-70B-SFT
MathQAAvg.GSM8KQAAvg
Perplexity0.4970.5670.5320.5080.5610.534
Entropy0.5310.4900.5110.5220.5280.525
Lowercase0.4910.4970.4940.5240.4790.501
Zlib0.4840.5480.5160.4900.5450.517
MIN-K%0.4880.5700.5290.5010.5630.532
MIN-K%++0.4680.5350.5020.4900.5490.519

4. Explanation of the limited detection performance under large-scale training [Q1]

Thank you for the question. We attribute this failure to a shift from memorization to generalization at scale, and validate this mechanism through an ablation analysis.

Large-scale training promotes generalization and weakens detection signals. Existing detection methods mainly rely on the likelihood gap between members and non-members. In small-scale settings with repeated exposure, the model overfits to individual training samples, resulting in a clear likelihood gap [2]. In contrast, realistic post-training uses large-scale and diverse data for only one or a few epochs. Each sample contributes little to parameter updates, and the model learns to generalize by capturing the underlying data distribution rather than overfitting to individual samples. Recent research also finds that when the training data exceeds the model's memorization capacity, unintended memorization decreases as the model begins to generalize [3]. Since members and non-members are drawn from the same distribution, such generalization benefits non-members as well, narrowing the discriminative gap.

Scaling up training data narrows the likelihood gap. We measure the average negative log-likelihood (NLL) of members and non-members across models trained on different data sizes. Before SFT, the base model assigns nearly identical NLL to members and non-members. After SFT on a small dataset, member NLL drops substantially while non-member NLL increases, creating a large gap that provides a strong membership signal. As the training data scales up, member NLL generally increases, whereas non-member NLL decreases substantially at larger scales as the model generalizes to unseen samples. As a result, the gap between members and non-members narrows steadily, leaving only a weak membership signal for detection.

SettingMember NLLNon-member NLLGap
Base model0.94320.95710.0138
ORACLE-SFT-100K0.20891.16510.9562
ORACLE-SFT-400K0.47131.22780.7565
ORACLE-SFT-1000K0.61821.18000.5617
ORACLE-SFT-2000K0.47070.73000.2593

[1] Apertus 1.5: from open text models to open multimodal reasoners. arXiv preprint arXiv:2610.15542, 2026.

[2] Duan et al. Do Membership Inference Attacks Work on Large Language Models? COLM 2024.

[3] Morris et al. How Much Can Language Models Memorize? ICML 2026.

Dear Reviewer,

As the discussion period is coming to an end (November 18, AoE), we would appreciate it if you could provide your comments on our rebuttal before the deadline.

Thank you for your time.

Authors

Reviewer_bsk7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WildTrace benchmarks training data detection across SFT, DPO, and RL using an Olmo-3-based post-training setup. It constructs evaluation sets with known membership labels and finds that the tested detectors generally perform poorly. The paper also examines how training scale, distribution shifts, and threshold calibration affect detection.

Critical strengths
  1. Evaluating membership inference under larger-scale post-training is a useful direction beyond small, heavily overfitted training setups.

  2. Covering SFT, DPO, and RL within a common evaluation framework could make comparisons across post-training stages easier.

Critical weaknesses
  1. The DPO evaluation omits a directly relevant baseline. PREMIA [A] specifically targets membership inference on preference data, but WildTrace does not evaluate it and scores only chosen responses. Table 6 also reports reference-based results only for SFT. The results therefore do not establish whether detection fails at scale or whether the evaluation misses stronger, preference-aware signals.

  2. It seems that the RL benchmark does not establish its claimed distribution matching. This is one major issue that might invalidate the results. Members are filtered for non-zero training advantages, but no equivalent filter is described for non-members. This selection can introduce differences in prompt difficulty despite both groups coming from the same source. Near-chance performance from the tested blind classifiers does not rule out this mismatch. I am also curious if a stronger blind classifier can capture a distributional shift better, for example, BERT or a pretrained LLM?

  3. The oracle models are not validated on task performance. The paper provides no task scores, reward improvements, or learning curves showing that its training produces useful models. The competitiveness of the original Olmo family does not validate these newly trained checkpoints. Weak detection could therefore reflect limited learning rather than a limitation of membership inference under successful post-training.


[A] Qizhang Feng, Siva Rajesh Kasa, Santhosh Kumar Kasa, Hyokun Yun, Choon Hui Teo, and Sravan Babu Bodapati. Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment. AISTATS, 2025. https://proceedings.mlr.press/v258/feng25a.html

Questions for authors

Please see weaknesses above.

5 comments

We thank the reviewer for the comments and questions. Please find our response below.

1. Additional baselines designed for DPO [W1]

Thank you for the comment. We address this concern by evaluating a DPO-specific method and extending reference-based methods to ORACLE-DPO.

  • Experimental results on PREMIA. We evaluate an additional baseline, PREMIA, on our benchmark.PREMIA leverages both chosen and rejected responses to capture membership signals [1]. We consider two reference models: the base model OLMo-3-1025-7B (PREMIA-Base) and the preceding checkpoint OLMo-3-7B-Instruct-SFT (PREMIA-SFT). The table below reports AUC scores of PREMIA on the ORACLE-DPO model. These results show that PREMIA achieves limited performance, with an average AUC of at most 0.525. We provide detailed results in Appendix C.5 of the revised manuscript.
MethodMathCodeIFQAAvg.
PREMIA-Base0.5170.4740.5370.5720.525
PREMIA-SFT0.5190.4720.5280.5710.522
  • Reference-based methods on ORACLE-DPO. We further extend reference-based methods to ORACLE-DPO by using the preceding checkpoint OLMo-3-7B-Instruct-SFT as the reference model. The table below compares the AUC on ORACLE-DPO with and without the reference model. The results show that introducing a reference model yields only marginal gains. We provide detailed results in Appendix C.6 of the revised manuscript.
MethodSettingMathCodeIFQAAvg.Δ\Delta
Perplexityw/o Ref.0.5000.4510.5270.5140.498–
w/ Ref.0.5030.4670.5310.5460.512+0.014
Entropyw/o Ref.0.5240.4470.4870.4880.487–
w/ Ref.0.5080.5210.5110.4920.508+0.021
Lowercasew/o Ref.0.4980.4380.5140.5230.493–
w/ Ref.0.4880.5130.5330.5180.513+0.020
Zlibw/o Ref.0.4660.5380.5260.4650.499–
w/ Ref.0.4950.5320.5420.5190.522+0.023
MIN-K%w/o Ref.0.4960.4590.5350.5190.502–
w/ Ref.0.5030.4960.5240.5510.519+0.017
MIN-K%++w/o Ref.0.5040.5430.4730.5440.516–
w/ Ref.0.5210.5320.4840.5700.527+0.011

In summary, these results suggest that the limited performance on DPO is not due to our evaluation missing preference-aware signals, but reflects the limitation of existing detection methods under large-scale post-training settings.

2. Clarification on distribution matching in the RL benchmark [W2]

Thank you for the comment. We clarify it from the following three aspects.

Rationale for excluding zero-advantage samples. In GRPO, prompts that elicit identically rewarded responses yield zero advantages and thus produce no gradient updates. Such samples are effectively not learned during RL training. We therefore select members only from prompts with non-zero advantages to ensure valid membership labels. Although this filtering may introduce differences in prompt difficulty between members and non-members, potentially inflating detection performance, our results show that existing methods achieve near-random performance on WildTrace-RL. This further supports our conclusion that existing methods exhibit limited detection performance under large-scale post-training settings.

Applying zero-advantage filtering to non-members. To eliminate these potential differences, we apply an equivalent filter to non-members. Specifically, we also filter out zero-advantage non-members and construct a balanced evaluation set. As shown below, all detection methods still exhibit limited detection performance on the filtered evaluation set, consistent with our original conclusion.

MethodPerplexityEntropyLowercaseZlibMIN-K%MIN-K%++
AUC0.4380.5390.5200.5390.4560.447

Evaluation with additional blind classifiers. We further evaluate two additional blind classifiers, built on BERT and the pretrained LLM Pythia, respectively. Specifically, we fine-tune each model as a text binary classifier to distinguish members from non-members. For each task, we randomly split members and non-members into training and test sets with an 80/20 ratio, and report the average AUC over three random seeds. The table below shows that these classifiers also achieve near-random AUC on WildTrace-RL, indicating little exploitable distribution shift between members and non-members.

ClassifierModelMathCodeIFQAAvg.
BERTORACLE-SFT0.4710.5410.5260.4140.488
ORACLE-DPO0.4490.4920.5320.4870.490
ORACLE-RL0.5090.4930.4910.4800.493
PythiaORACLE-SFT0.4880.4780.5170.4880.493
ORACLE-DPO0.5260.5730.5410.5210.540
ORACLE-RL0.4960.4870.5050.4890.494

3. Training dynamics and downstream performance of checkpoints [W3]

Thank you for raising this point. To verify that our post-training is effective, we examine both the training dynamics and the downstream performance of each checkpoint. The detailed training curves and downstream performance of all three stages are provided in Appendix B.8. The training curves show stable optimization across all post-training stages. Specifically, the cross-entropy loss decreases steadily during SFT. In the subsequent DPO stage, the preference loss decreases, while both preference accuracy and reward margin increase. During RL training, the reward improves as training progresses, without training collapse. Additionally, as shown in the table below, accuracy improves consistently on downstream benchmarks as post-training progresses. These results confirm that our oracle models are well-trained. The weak detection performance on our benchmark therefore reflects the limitations of existing methods, rather than insufficient training of the oracle models.

BenchmarkOracle-SFTOracle-DPOOracle-RL
MATH50060.0077.8089.00
HumanEvalPlus63.4168.9074.39
GPQA31.8238.8942.42
AIME 20243.3326.6750.00

[1] Feng Q, et al. Exposing privacy gaps: membership inference attack on preference data for LLM alignment. AISTATS, 2025.

Dear Reviewer,

As the discussion period is coming to an end (November 18, AoE), we would appreciate it if you could provide your comments on our rebuttal before the deadline.

Thank you for your time.

Authors

Reviewer_xJY3OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a benchmark, WildTrace, for detecting whether examples were used in large-scale LLM post-training, covering SFT, DPO, and RL across mathematics, coding, instruction following, and question answering. Its main contribution is controlled training with held-out non-members sampled from the same data sources, producing 6K evaluation examples. Across all evaluated methods, the highest average AUROC is only 0.529, which indicates its hardness.

Critical strengths

The design of holding out non-members before training provides stronger membership evidence than inferring labels retrospectively. Besides, the author conducts extensive experiments evaluating 3 post-training stages and 4 domains beyond narrowly controlled, small-scale fine-tuning experiments.

Critical weaknesses
  1. The comparison across stages does not establish that DPO and RL produce weaker memorization. In Sec. 4.2, lines 303–310, the author connects lower detection performance to weaker memorization signals based on the results in Tab. 1. However, from my perspective, these evaluations use different training datasets, sample formats, and objectives and they do not follow the same examples through a controlled comparison. Thus, a lower AUROC could therefore reflect how well the detector matches each training objective. I’m wondering if the author could provide more explanation on this.

  2. Addition of more baseline methods. I’m wondering if the author could report the performance of more baselines such as SPV-MIA [1] on the proposed benchmark so that we can have a better understanding about the effectiveness of the proposed benchmark.

  3. For the conclusion of “strong detection performance in small-scales does not transfer to large-scale training” in line 314, I think it requires more supporting evidence. In Fig. 2, the author shows this occurrence using Min-K%++ as the method. The conclusion would be stronger if the author can include the experiments using more methods and achieving a similar results.

[1] Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration. NeurIPS, 2024.

Questions for authors

Please refer to weaknesses.

4 comments

We thank the reviewer for the comments and questions. Please find our response below.

1. Clarification on the cross-stage comparison of detectability [W1]

Thank you for the comment. We clarify that our comparison aims to characterize the detectability of training data across post-training stages, rather than to provide an instance-level comparison of memorization. Since SFT, DPO, and RL require different training data formats due to their distinct training objectives, evaluating the same examples across stages is infeasible. We agree that membership inference does not provide a sample-level measure of memorization. We explain the differences in detectability from the perspective of the training objectives:

  • SFT directly maximizes the likelihood of target responses given their prompts via token-level supervision. The detector has access to all supervised tokens, which leaves an explicit signal for detection.
  • DPO maximizes the preference margin between chosen and rejected responses while staying close to the reference policy, without directly increasing the likelihood of chosen responses. In fact, the likelihood of chosen responses may even decrease during training [1], providing only a weak signal for detection.
  • RL optimizes the reward of responses sampled from the policy, rather than fitting fixed target responses. Since the detector can only access the prompt, which is not directly supervised during training, the available membership signal is limited.

In summary, the differences in detectability across stages likely stem from their distinct training objectives, and a lower AUC alone does not necessarily indicate weaker sample-level memorization. Accordingly, we have revised the claim in Sec. 4.2 to state that training data from DPO and RL leave weaker signals detectable by existing methods.

2. Results on additional baselines [W2]

Thanks for your suggestion. We evaluate two additional baselines, SPV-MIA [2] and CDD [3], on our benchmark. The table below reports the AUC scores on each task across the three post-trained models. The results show that SPV-MIA and CDD achieve overall average AUCs of 0.506 and 0.497, respectively, both close to random guessing. This further confirms our finding that existing methods exhibit limited effectiveness under large-scale post-training settings. We have included the detailed results in Table 1 and the implementation details in Appendix B.7 of the revised version.

MethodModelMathCodeIFQA
SPV-MIAORACLE-SFT0.4930.5540.5770.591
SPV-MIAORACLE-DPO0.5060.4470.4860.481
SPV-MIAORACLE-RL0.4540.5040.5010.483
CDDORACLE-SFT0.5010.5000.4780.497
CDDORACLE-DPO0.5060.5160.4770.515
CDDORACLE-RL0.5050.4990.4780.492

3. Evaluating additional detection methods across training scales [W3]

Thank you for the suggestion. We evaluate additional detection methods across training scales from 10K to 400K samples. The table below reports the AUC averaged over four tasks. All methods show an overall decline as the training scale increases, moving toward random guessing at large scales. For example, Entropy drops from 0.791 at 10K to 0.502 at 400K. These findings confirm that detection performance observed in small-scale settings does not transfer to large-scale post-training. We include the detailed results in Appendix C.8.

ModelPerplexityEntropyLowercaseZlibMIN-K%MIN-K%++
Oracle-SFT-10K0.7010.7910.6960.6740.6670.800
Oracle-SFT-50K0.6990.7590.7100.6700.6680.634
Oracle-SFT-100K0.6890.6950.6960.6560.6570.496
Oracle-SFT-200K0.6770.6470.6810.6420.6600.505
Oracle-SFT-400K0.5990.5020.6140.5800.5730.511
Oracle-SFT-1000K0.5730.4810.5840.5540.5510.480

[1] Ren Y, et al. Learning dynamics of LLM fine-tuning. ICLR, 2025.

[2] Fu W, et al. Membership inference attacks against fine-tuned large language models via self-prompt calibration. NeurIPS, 2024.

[3] Dong Y, et al. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. Findings of ACL, 2024.

Dear Reviewer,

As the discussion period is coming to an end (November 18, AoE), we would appreciate it if you could provide your comments on our rebuttal before the deadline.

Thank you for your time.

Authors

Thank you for the in-depth explanation! I think the motivation and results make more sense to me now. I have raised my score.

Thank you for your feedback and for raising your score. We greatly appreciate your recognition of our motivation and results.

We hope our rebuttal has addressed your concerns. If so, we would greatly appreciate it if you could reconsider your overall score.

We would be happy to discuss further if you have any additional questions.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for reviewing our manuscript. As the discussion period is drawing to a close (November 18, AoE), we would greatly appreciate it if you could share any further feedback on our rebuttal. Please let us know if you have any additional questions.

Thank you for your time.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.