- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper developed WildTrace, which was tested on three post-training stages and four types of tasks in OLMo-3 7B. It uniformly evaluated whether existing methods could identify the data used for training. Most methods performed close to random guessing under the tested conditions; for the experiments on SFT, it was also found that the detection effect declined as the data volume increased. The main contributions are the unified testing resources and systematic experimental analysis.
Critical strengths
-
The study uniformly evaluated 12 detection methods in the SFT, DPO and RL stages, and the methods were identical for both members and non-members, resulting in relatively reliable results.
-
It revealed that the existing methods have limited effectiveness in large-scale post-training, which is of reference value for practical applications.
Critical weaknesses
-
The evaluation does not examine whether the same SFT samples remain detectable after DPO and RL, leaving the effect of subsequent training on earlier membership signals unclear.
-
For DPO and RL, the reference-based experiments do not use the immediately preceding SFT and DPO checkpoints, respectively. It is therefore unclear whether reference-based methods also provide only marginal gains at these stages.
-
All target-model experiments are conducted on OLMo-3 7B. Without validation on larger models or other model families, the representativeness and generalizability of the benchmark findings remain unclear.
Questions for authors
I would consider raising my score if the authors can better explain why existing detection methods fail under large-scale training and provide additional analyses or ablations to support this explanation.
7 comments
4. Explanation of the limited detection performance under large-scale training [Q1]
Thank you for the question. We attribute this failure to a shift from memorization to generalization at scale, and validate this mechanism through an ablation analysis.
Large-scale training promotes generalization and weakens detection signals. Existing detection methods mainly rely on the likelihood gap between members and non-members. In small-scale settings with repeated exposure, the model overfits to individual training samples, resulting in a clear likelihood gap [2]. In contrast, realistic post-training uses large-scale and diverse data for only one or a few epochs. Each sample contributes little to parameter updates, and the model learns to generalize by capturing the underlying data distribution rather than overfitting to individual samples. Recent research also finds that when the training data exceeds the model's memorization capacity, unintended memorization decreases as the model begins to generalize [3]. Since members and non-members are drawn from the same distribution, such generalization benefits non-members as well, narrowing the discriminative gap.
Scaling up training data narrows the likelihood gap. We measure the average negative log-likelihood (NLL) of members and non-members across models trained on different data sizes. Before SFT, the base model assigns nearly identical NLL to members and non-members. After SFT on a small dataset, member NLL drops substantially while non-member NLL increases, creating a large gap that provides a strong membership signal. As the training data scales up, member NLL generally increases, whereas non-member NLL decreases substantially at larger scales as the model generalizes to unseen samples. As a result, the gap between members and non-members narrows steadily, leaving only a weak membership signal for detection.
| Setting | Member NLL | Non-member NLL | Gap |
|---|---|---|---|
| Base model | 0.9432 | 0.9571 | 0.0138 |
| ORACLE-SFT-100K | 0.2089 | 1.1651 | 0.9562 |
| ORACLE-SFT-400K | 0.4713 | 1.2278 | 0.7565 |
| ORACLE-SFT-1000K | 0.6182 | 1.1800 | 0.5617 |
| ORACLE-SFT-2000K | 0.4707 | 0.7300 | 0.2593 |
[1] Apertus 1.5: from open text models to open multimodal reasoners. arXiv preprint arXiv:2610.15542, 2026.
[2] Duan et al. Do Membership Inference Attacks Work on Large Language Models? COLM 2024.
[3] Morris et al. How Much Can Language Models Memorize? ICML 2026.
Dear Reviewer,
As the discussion period is coming to an end (November 18, AoE), we would appreciate it if you could provide your comments on our rebuttal before the deadline.
Thank you for your time.
Authors
Thanks for your recognition and the valuable suggestions. Please find our response below.
1. Evaluating the detectability of SFT samples after DPO and RL training [W1]
Thank you for the comment. To examine how subsequent training affects earlier membership signals, we track the same SFT samples and non-members across successive post-training stages. To clearly observe how membership signals change across stages, we conduct this experiment under the small-scale setting in Section 5.1. Specifically, starting from ORACLE-SFT-10K, we further train it with DPO and then RL, each on 10K examples. The table below reports the AUC of detection on WildTrace-SFT at each checkpoint. The results show that subsequent DPO and RL training gradually weakens the detectability of SFT samples, with the average AUC decreasing from 0.800 to 0.786. These findings suggest that subsequent DPO and RL training can weaken detection signals, while the membership signals established during SFT remain largely detectable. We provide the detailed results in Appendix C.7.
2. Results on reference-based methods [W2]
Thank you for the suggestion. We extend reference-based methods to all three post-training stages, using the preceding checkpoint of each stage as the reference model. The table below reports the average AUC across four tasks. The results show that reference-based methods still achieve limited detection performance, with an average AUC of at most 0.527 on ORACLE-DPO and 0.514 on ORACLE-RL. This confirms that reference-based methods can provide only marginal gains at the DPO and RL stages under large-scale post-training. We include the detailed results in Appendix C.7 of the revised version.
3. Motivation for choosing OLMo-3 in our benchmark. [W3]
Thank you for the comment. We clarify our choice from the following three aspects.
Rationale for choosing OLMo-3. Our research aims to provide a reliable benchmark with verifiable membership labels under realistic post-training settings. This requires full training data transparency. We therefore choose OLMo-3, one of the few competitive model families that release all training data and recipes. Other mainstream LLMs from industry do not disclose their post-training data and recipes, so benchmarks constructed post-hoc on these models may introduce noisy membership labels, undermining their reliability. Moreover, OLMo-3 follows a standard architecture and a multi-stage post-training pipeline that closely mirror those of mainstream LLMs from industry, making our findings applicable to realistic post-training regimes. Although the OLMo-3 family includes a 32B model, repeating the post-training pipeline at this scale exceeds our current compute budget. We are actively seeking industry compute support to pursue this in future work.
Evaluation on another model family. To further examine the generalizability of our findings, we conduct experiments on the Apertus model family [1], a fully open model family that releases its complete training data. Due to limited computational resources, we construct the evaluation sets post hoc using the publicly released training data. Specifically, we construct balanced evaluation sets with equal numbers of members and non-members on QA and Math tasks. While post-hoc construction lacks the verifiable provenance emphasized in our manuscript, it can still serve as a sanity check here given the transparent data disclosure. The table below show that existing methods still exhibit limited detection performance on both the 8B and 70B models. These results are consistent with our main findings on OLMo-3, suggesting that our conclusions generalize to larger models and other model families.