acceptodds
Under review as a conference paper at ICLR 2027

Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective

Abstract

Detecting pretraining data in large language models is challenging because high likelihood can reflect either training exposure or strong generalization. In the joint space of prediction loss and predictive entropy, a likelihood-only detector uses a horizontal boundary and can mistake inherently predictable non-members for members. Motivated by this, we introduce an inclined boundary that evaluates prediction loss relative to predictive entropy. Our analysis shows that entropy correction can preserve the expected membership signal while reducing its variance, thereby improving standardized member–non-member separation. We further extend the mean–variance analysis to the more general setting with a nonzero mean entropy gap. Interestingly, this entropy-adjusted score admits a Helmholtz free-energy interpretation, leading to Energy Transfer Detection (ETD), which views pretraining data detection from a macroscopic residual free-energy transfer perspective. Extensive experiments show that ETD achieves the best average detection performance, improving average AUROC by up to 3.5% and TPR@5%FPR by up to 5.1%, while remaining robust across diverse settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Pretraining Data Detection, Free Energy, Data Contamination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZyELOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Energy Transfer Detection (ETD), a method for detecting pretraining data from LLMs. The proposed method integrates prediction loss and predictive entropy by selecting the first-occurrence tokens. The authors evaluate the effectiveness of the proposed method across various datasets and models.

Critical strengths

The authors investigate predictive entropy as an additional signal for calibrating loss-based membership signals, which can benefit pretraining data detection.

Critical weaknesses
  1. The theoretical motivation lacks justification. In this paper, the theoretical analysis mainly establishes variance reduction under specific assumptions. However, lower variance does not necessarily indicate better training data detection performance. The connection between the theoretical analysis and the proposed method remains unclear.

  2. The experimental evaluation is not solid. In the main text, the experimental evaluation is primarily based on WikiMIA and StackMIAsub. Existing works show that benchmark post-hoc b construction can introduce distribution shifts, which can artificially inflate detection performance. The authors should further conduct experiments on more reliable benchmarks. For instance, the dataset used in "LLM Dataset Inference: Did You Train on My Dataset?" could be a more reliable benchmark for evaluation.

Questions for authors
  1. What unique insights or practical benefits does the free-energy interpretation provide for the detection score?

  2. In Theorem 2, how does variance reduction ensure improvements in AUC?

  3. In the main experiments, how is the hyperparameter determined?

5 comments

We sincerely appreciate your recognition of predictive entropy as a useful calibration signal and thank you for your constructive questions. Below is our point-by-point response:

Q1: lower variance does not necessarily indicate better training data detection performance. The connection between the theoretical analysis and the proposed method remains unclear. (Weakness 1 & Question 2)

A1: Thank you for this insightful comment. We agree that lower variance alone does not guarantee better detection. However, when the positive expected member–non-member gap is preserved, variance reduction yields more stable separation and a higher distribution-free AUROC lower bound, as demonstrated below.

Following Eq.(4) (Line 181), the member–non-member score gap is ΔCt=C(non-members)−C(members)\Delta\mathcal C_t=\mathcal C(\text{non-members})-\mathcal C(\text{members}), where ΔLt\Delta L_t denotes the corresponding gap under conventional loss-only scoring. Theorems 1 & 2 show that entropy correction can preserve a positive expected gap while reducing its variance, thereby improving standardized separation (Eq. (10)):

{\sqrt{\operatorname{Var}(\Delta\mathcal C_t)}}> \frac{\mathbb E[\Delta L_t]} {\sqrt{\operatorname{Var}(\Delta L_t)}}.$$ Furthermore, using Cantelli's inequality, **we establish a distribution-free lower bound on AUROC:** $$\mathrm{AUROC}_S \geq P(\Delta S_t>0) \geq \frac{\mathbb E[\Delta S_t]^2} {\operatorname{Var}(\Delta S_t)+\mathbb E[\Delta S_t]^2}= \frac{\gamma_S^2}{1+\gamma_S^2},$$ where $\Delta S_t$ denotes the score gap between a non-member and a member, and $\gamma_S=\mathbb E[\Delta S_t]/\sqrt{\operatorname{Var}(\Delta S_t)}>0$ is the standardized separation. Since $\gamma^2/(1+\gamma^2)$ is strictly increasing for $\gamma>0$, Eq.(10) implies that **ETD has a strictly higher AUROC lower bound than loss-only scoring.** As shown in Table 4, entropy correction reduces the member–non-member gap variance by 53.14% / 52.34%, while improving standardized separation by 19.18% / 16.20% (Ori./Para.), **empirically supporting this mechanism.** We provide the complete proof in Appendix L (How Lower Variance Relates to Better Detection Performance). Thank you again for this insightful and important feedback! > **Q2:** The authors should further conduct experiments on more reliable benchmarks. For instance, the dataset used in "LLM Dataset Inference: Did You Train on My Dataset?" could be a more reliable benchmark for evaluation. (Weakness 2) **A2:** Thank you for this valuable suggestion. We evaluate ETD on the distribution-matched Pile-derived dataset described in [1], covering 20 subsets and four Pythia model scales: [Table A - Average AUROC & TPR@5%FPR] | Method | AUROC ↑ | TPR@5%FPR ↑ | |---|---:|---:| | PPL | 49.83 ± 0.51 | 5.70 ± 0.43 | | Ref | 51.10 ± 0.50 | 5.91 ± 0.39 | | Lowercase | 50.84 ± 0.49 | 6.18 ± 0.39 | | Zlib | 50.40 ± 0.50 | 5.60 ± 0.38 | | Min-K% | 50.93 ± 0.50 | 5.81 ± 0.45 | | Min-K%++ | 53.44 ± 0.46 | 5.83 ± 0.38 | | AECA | 50.89 ± 0.50 | 5.72 ± 0.44 | | DC-PDD | 50.22 ± 0.49 | 5.33 ± 0.34 | | ETD | **55.03 ± 0.46*** | **7.40 ± 0.40*** | As shown, **ETD outperforms the strongest baselines by 1.59% and 1.22% in AUROC and TPR@5%FPR, with statistically significant improvements.** Full results are provided in Appendix M of the updated manuscript. Thank you for your valuable feedback! > **Q3:** What unique insights or practical benefits does the free-energy interpretation provide for the detection score? (Question 1) **A3:** Thank you for the question. The free-energy perspective provides a macroscopic interpretation of the relationship between pretraining exposure and membership detection. As discussed in Section 4.4 (Lines 261–267), in thermodynamics, when a system is brought into contact with a reservoir, it relaxes toward equilibrium by dissipating free energy. Analogously, **pretraining members have previously interacted with the model and may involve less residual free-energy transfer during inference.** Thus, lower scores suggest stronger membership evidence, consistent with Figure 1. > **Q4:** In the main experiments, how is the hyperparameter determined? (Question 3) **A4:** Thank you for pointing this out. Detailed hyperparameter settings are provided in **Appendix D.2 (Implementation Details, Line 1013)**. We will highlight this in the main text. For $\lambda$, Theorem 2 derives the theoretically optimal coefficient $\lambda_t^*=\operatorname{Cov}(\Delta L_t,\Delta H_t)/\operatorname{Var}(\Delta H_t)$. For simplicity, we fix $\lambda=0.5$ across all datasets and target models in the main experiments, without specific tuning. We hope these clarifications and additional analyses can address your concerns. Thank you again for your constructive feedback. Please feel free to reach out if you have any further concerns! Reference [1] Maini et al. LLM Dataset Inference: Did You Train on My Dataset? NeurIPS 2024.

Thank you for your detailed rebuttal. However, I am still confused about some points.

  1. A higher AUROC lower bound does not guarantee improvement in the actual AUC. For example, based on results in Table 7, the Cantelli-based lower bound increases from 0.182 to 0.241, both well below the AUC of 0.5 (random guessing). Therefore, I do not think this bound establishes a meaningful connection between variance reduction and a higher actual AUC.

  2. For W2, could you provide detailed results for each subset in your revised manuscript?

  3. I still do not understand how the free-energy interpretation helps understand training data detection. Is the connection between free energy and the proposed detection score a similarity in mathematical form?

  4. Additionally, could the authors specify how the reported 95% bootstrap confidence intervals were computed?

We sincerely thank you for carefully considering our previous responses and providing further constructive feedback. We have further strengthened the theoretical analysis, added detailed experimental results, and clarified the statistical procedures in the revised manuscript. Below are our point-by-point responses.

Re: Q1

Thank you for this important clarification. We agree that a higher AUROC lower bound does not necessarily guarantee an improvement in actual AUROC. Our previous analysis used Cantelli's inequality to establish a distribution-free guarantee without assuming any specific score-gap distribution, which explains why the resulting lower bound is conservative.

To establish a more direct connection with actual AUROC, we further derive Proposition 4 (AUROC Improvement under Approximate Gaussian Score Gaps) (Line 2164). Under a quantified approximate Gaussian assumption, Proposition 4 establishes the following bound:

\geq \Phi(\gamma_{\mathcal C})-\Phi(\gamma_{\mathcal L}) -(\epsilon_{\mathcal C}+\epsilon_{\mathcal L})$$ where $\Delta AUROC = AUROC_{ETD} - AUROC_{Loss}$ and $\gamma_S=\mu_t/\sigma_{S,t}$. Specifically, when the member–non-member score gaps approximately follow Gaussian distributions and the approximation errors are sufficiently small, preserving a positive mean gap while reducing its variance guarantees a strictly higher actual AUROC for ETD, rather than merely a higher lower bound. Based on the results in Table 7, **using the Gaussian approximation associated with Proposition 4 yields a predicted increase in average AUROC from 65.46% (loss-only) to 67.52% (ETD), an improvement of 2.06 percentage points** across 16 settings. The predicted ETD AUROC also closely matches the observed average of 67.54%. **These results empirically support the connection between improved standardized separation and higher actual AUROC.** We thank you again for this insightful comment, which has helped us strengthen the theoretical analysis. > **Re: Q2** Thank you for your suggestion. In our previous revision, we focused on the aggregated results across the 20 subsets to provide a concise comparison of overall detection performance. We agree that subset-level results are also important for assessing performance variations across different data domains. Therefore, we have now included detailed AUROC and TPR@5%FPR results for all 20 subsets in Appendix N (Figures 15–16, Lines 2538) of the revised manuscript. > **Re: Q3** Thank you for your follow-up. **The free-energy connection goes beyond algebraic similarity.** We previously explored this connection in depth but did not elaborate on it due to space constraints. We have now included the detailed analysis in the updated manuscript. As established in Proposition 5 (Pretraining as Non-equilibrium Free-Energy Minimization, Line 2220), defining the model-induced energy $\mathcal E_\theta(x)=-T\log P_\theta(x)$ yields the following non-equilibrium free-energy functional: $$\mathcal F_{\mathrm{neq}} =\mathbb E_{P_{\mathrm{data}}}[\mathcal E_\theta(x)] -T H(P_{\mathrm{data}}) =T D_{\mathrm{KL}}(P_{\mathrm{data}}\Vert P_\theta).$$ Consequently, **minimizing the maximum-likelihood pretraining objective is equivalent to minimizing this free-energy functional**: $$\arg\min_{\theta}\mathcal F_{\mathrm{neq}}(P_{\mathrm{data}},\theta)= \arg\min_{\theta}\mathcal L_{\mathrm{CE}}(\theta).$$ This establishes **a rigorous mathematical connection between maximum-likelihood pretraining and free-energy minimization**. During pretraining, the model learns to assign higher probabilities and thus lower prediction losses to training samples. From the free-energy perspective, this reduces their energy-like loss contributions. **This provides a theoretical motivation for why members tend to exhibit lower residual free-energy scores than non-members.** The complete derivation is provided in Appendix M. > **Re: Q4** We used 10,000 class-stratified bootstrap resamples within each subset, shared across methods and models. Metrics and averages were recomputed for each resample. The reported intervals are the point estimate ± 1.96 times the bootstrap standard error. We sincerely appreciate your continued engagement and constructive suggestions. We hope that these additional theoretical results, detailed experiments, and methodological clarifications can help address your remaining concerns. If anything remains unclear, we'd be happy to discuss it further.
Reviewer_YjYDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Energy Transfer Detection (ETD), a method for detecting whether a text was included in an LLM's pretraining data. The key idea is to calibrate prediction loss using predictive entropy to distinguish training exposure from strong generalization. The authors provide a mean–variance analysis showing that, under certain assumptions, entropy correction can preserve the expected member–non-member gap while reducing its variance, and interpret the resulting score from a Helmholtz free-energy perspective. Experiments on WikiMIA, StackMIAsub, and MIMIR demonstrate that ETD achieves improved detection performance over existing baselines. Ablation studies and additional analyses further examine the contributions of entropy correction, sensitivity to the calibration coefficient, and robustness across sequence lengths and training exposure levels.

Critical strengths

1. Strong empirical performance. The proposed method achieves consistent average improvements over existing baselines across multiple benchmarks and model architectures. The authors also provide comprehensive ablation studies and additional analyses to validate the effectiveness of entropy correction and its robustness under different settings. These results provide convincing empirical evidence for the practical value of the method.

2. Simplicity and practicality. The proposed method is simple and easy to implement, requiring only the target model's predictive statistics without additional reference models or training. This makes it a practical and appealing approach to pretraining data detection, particularly when full token logits are accessible.

Critical weaknesses

1. Discrepancy between the theoretical formulation and practical settings. The theoretical analysis assumes that members and non-members are sampled from the same distribution, with non-members independently drawn without explicitly enforcing non-membership. This differs from the practical membership detection setting, where the two groups should be disjoint. These discrepancies raise concerns about how well the theoretical findings generalize to practical pretraining data detection.

Questions for authors

Please check the weaknesses section.

1 comment

We appreciate your recognition of this paper's clear presentation, the proposed method's strong empirical performance, comprehensive ablation studies, and ease of implementation. We also thank you for your detailed review and constructive feedback. Below is our response:

Q1: Discrepancy between the theoretical formulation and practical settings. The theoretical analysis assumes that members and non-members are sampled from the same distribution, with non-members independently drawn without explicitly enforcing non-membership. This differs from the practical membership detection setting, where the two groups should be disjoint. These discrepancies raise concerns about how well the theoretical findings generalize to practical pretraining data detection.

A1: Thank you for this important observation. We acknowledge that independent sampling does not strictly guarantee disjointness. However, we would like to clarify that our theoretical sampling scheme follows a standard formulation in membership inference research, and our empirical results are consistent with the theoretical predictions.

  • Standard theoretical formulation. Following prior membership inference studies [1, 2], we sample the training set D∼PND\sim P^N, select a member XmX_{\mathrm m} uniformly from DD, and independently draw a fresh sample Xn∼PX_{\mathrm n}\sim P. This sampling scheme is commonly used to study membership inference under a shared data distribution. Related studies also investigate randomized training membership [3] and pretraining data detection using unseen samples from the same domain [4]. Although independent sampling does not strictly guarantee disjointness, it provides a well-defined theoretical setting for analyzing training-induced membership signals.

Under this standard sampling scheme, Theorem 1 establishes the connection between the expected membership gap and the generalization gap:

E[ΔLt]=E[R(θt)−R^D(θt)].\mathbb{E}[\Delta L_t]=\mathbb{E}[R(\theta_t)-\widehat{R}_D(\theta_t)].

Theorem 2 does not require members and non-members to follow the same distribution. Its variance-reduction result follows from

Var⁡(ΔCt)=vL,t−2λct+λ2vH,t<vL,t,\operatorname{Var}(\Delta C_t)=v_{L,t}-2\lambda c_t+\lambda^2v_{H,t}<v_{L,t},

provided that ct=Cov⁡(ΔLt,ΔHt)>0c_t=\operatorname{Cov}(\Delta L_t,\Delta H_t)>0 and 0<λ<2ct/vH,t0<\lambda<2c_t/v_{H,t} (Eq.(7)–Eq.(8)). These conditions can also hold for strictly disjoint groups. Furthermore, Appendix C extends the analysis to nonzero mean entropy gaps, characterizing when standardized separation improves.

  • Empirical validation. Our existing Table 4 already provides empirical support for this mechanism under practical benchmark membership settings. We reproduce the relevant results below:

Table A. Empirical mean–variance analysis (from Table 4 of our paper).

Settingλupper∗\lambda_{\mathrm{upper}}^*λupper\lambda_{\mathrm{upper}}Var⁡(ΔCt)Var⁡(ΔLt)\frac{\operatorname{Var}(\Delta C_t)}{\operatorname{Var}(\Delta L_t)}E[ΔCt]/Var⁡(ΔCt)E[ΔLt]/Var⁡(ΔLt)\frac{\mathbb{E}[\Delta C_t]/\sqrt{\operatorname{Var}(\Delta C_t)}}{\mathbb{E}[\Delta L_t]/\sqrt{\operatorname{Var}(\Delta L_t)}}
Ori.1.2154$^{\dagger}$2.0416$^{\dagger}$0.4686 (↓53.14%)†^{\dagger}1.1918 (↑19.18%)†^{\dagger}
Para.1.1145$^{\dagger}$1.9822$^{\dagger}$0.4766 (↓52.34%)†^{\dagger}1.1620 (↑16.20%)†^{\dagger}

↓ and ↑ indicate variance reduction and standardized-separation improvement. †^{\dagger} indicates statistical significance based on bootstrap 95% confidence intervals.

As shown in Table A, entropy correction reduces the member–non-member gap variance by over 52% while improving standardized separation by over 16% in both settings. These results are consistent with the variance-reduction mechanism predicted by Theorem 2 and the generalized analysis in Appendix C.

We will further clarify the standard sampling formulation and distinguish the assumptions and applicability of Theorems 1 and 2 in the revised manuscript.

Thank you again for this constructive suggestion. We hope these theoretical clarifications and existing empirical evidence can address your concerns. If anything remains unclear, we'd be happy to discuss it further!

References

[1] Yeom et al. Privacy risk in machine learning: Analyzing the connection to overfitting. IEEE CSF 2018.

[2] Leemann et al. Gaussian membership inference privacy. NeurIPS 2023.

[3] Tong et al. How much of my dataset did you use? Quantitative data usage inference in machine learning. ICLR 2025.

[4] Zhang et al. Fine-tuning can help detect pretraining data from large language models. ICLR 2025.

Reviewer_3B7pOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Motivated by correction through a mean–variance analysis and interpret it using Helmholtz free energy, this paper proposes Energy Transfer Detection (ETD) for identifying texts used in LLM pretraining. ETD subtracts predictive entropy from token prediction loss, then averages this score over first-occurrence token positions. The results in experiments shows improvements of up to 3.5 percentage points in average AUROC and 5.1 points in TPR@5%FPR with robustness.

Critical strengths

The analogy to free-energy is interesting and the proposed ETD requires simple and consistent settings. It requires no additional LLMs as reference model comparison and the same coefficient can generalized to all evaluated datasets and models, while achieving a comparable performance at the same time. The ablation study shows in Tab. 3 illustrates the usefulness of proposed first-occurrence aggregation.

Critical weaknesses

Weaknesses:

  1. The methodological novelty is insufficiently distinguished from existing methods. Min-K%++ already centers token log-probability using its vocabulary-wide expectation, which equals negative predictive entropy. Consequently, at λ=1\lambda=1, ETD’s token contribution is exactly the negative of Min-K%++’s numerator, except changing entropy weight and aggregation and removing standard-deviation normalization. The first-occurrence aggregation also has appeared in DC-PDD [1], which has been mentioned but not compared with. These connections suggest an incremental calibration contribution rather than an entirely new detection principle.

  2. The performance improvement is limited. Though the parameter setting of ETD is simple, its performance seems really close to existing methods. For example, as shown in Tab. 1, the average improvement of ori. and para. compared to Min-K%++ and AECA are 1.9% and 1.7%, respectively. In addition, as illustrated in Tab. 10, on 7 datasets, ETD only achieves best performance on Github and EM Mathematics, which further limits its generalizability.

  3. The robustness requires more statistical support and analysis. Tab. 1 and Tab. 2 report averages without confidence intervals for ETD’s difference from competing methods. Although bootstrap analysis elsewhere provides statistical information and controlled training, the statistical analysis for Tab.1 and Tab. 2 is import because four WikiMIA length splits contain inequal number of data samples, yet receive equal weight.

[1] Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. “Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method.” EMNLP, 2024

Questions for authors

Please refer to weaknesses.

1 comment

Thank you for your detailed review and constructive comments. We appreciate your recognition of the free-energy perspective, the simplicity of our proposed method, and the ablation studies. Our responses follow:

Q1: Min-K%++ already centers token log-probability using its vocabulary-wide expectation, which equals negative predictive entropy. Consequently, at λ=1\lambda=1, ETD’s token contribution is exactly the negative of Min-K%++’s numerator, except changing entropy weight and aggregation and removing standard-deviation normalization.

A1: Thank you for pointing out this connection. We acknowledge the algebraic similarity to Min-K%++'s numerator [1], but the methods differ:

  • Normalization-Induced Performance Degradation in Min-K%++. Min-K%++ normalizes token scores using vocabulary-level standard deviation σt\sigma_t. ETD analyzes the member–non-member score gap (Eq.(4)) without this normalization. Our new theoretical analysis shows that Min-K%++ can amplify low-probability token scores more strongly than ETD due to division by the standard deviation σt\sigma_t, leading to detection errors. Longer sequences provide more opportunities for such amplification. This mechanism helps explain Min-K%++'s performance drop at the longer sequence length of 256 (Figure 5), which is partially mitigated by removing σt\sigma_t (new Figure 13). Detailed analyses are provided in the updated Appendix K (Line 1890).

  • Interpretability. ETD's score admits a Helmholtz free-energy interpretation (Section 4.4, Figure 3).

  • Empirical performance. Table A (Q2) shows ETD improves average AUROC and TPR@5%FPR over Min-K%++ by 2.5% and 3.5%, with statistically significant gains.

Importance of long-text detection. As long-context processing becomes increasingly important for LLMs, reliable detection on longer texts is essential [2]. Longer sequences increase the risk of Min-K%++'s normalization amplification, whereas ETD exhibits robustness.

Thank you for raising this important question. We will clarify these distinctions in the revision.

Q2: The first-occurrence aggregation also has appeared in DC-PDD [3], which has been mentioned but not compared with.

A2: Yes. We cited DC-PDD but omitted a direct comparison due to its similarity to AECA, as both use external corpora. We now provide the results.

[Table A - WikiMIA & StackMIAsub (AVG.)]

MethodAUROCTPR@5%FPR
Min-K%++62.2 ± 1.912.6 ± 2.4
DC-PDD62.2 ± 2.012.6 ± 2.2
ETD64.7 ± 2.0 ∗†^{*\dagger}16.1 ± 2.6 ∗†^{*\dagger}

Superscripts * and †^{\dagger} indicate that ETD significantly outperforms Min-K%++ and DC-PDD.

As shown, ETD significantly outperforms DC-PDD (Tables 19–24, updated manuscript).

Q3: The performance improvement is limited. Though the parameter setting of ETD is simple, its performance seems really close to existing methods.

A3: We appreciate this observation. Although modest, the gains in Tables A and B are statistically significant under paired-bootstrap testing. We will clarify this in the revision.

Q4: ETD only achieves best performance on Github and EM Mathematics, which further limits its generalizability.

A4: Thank you for raising this concern. MIMIR is challenging (Line 347); even Min-K%++ is not consistently best across domains or in average AUROC (Tables 2 & 9 in the Min-K%++ paper). ETD achieves the best or second-best performance on most domains (Tables 10–11), and performs best on average with significant gains:

[Table B - MIMIR (AVG.)]

MethodAUROCTPR@5%FPR
Min-K%++57.3 ± 0.812.1 ± 1.0
DC-PDD58.7 ± 0.913.5 ± 1.0
ETD60.0 ± 0.9 ∗†^{*\dagger}16.6 ± 1.0 ∗†^{*\dagger}

Thank you for this question. We will clarify our cross-domain claim in the revision.

Q5: Tab. 1 and Tab. 2 report averages without confidence intervals for ETD’s difference from competing methods...yet receive equal weight. (Weakness 3)

A5: Thank you for this important suggestion. Tables A and B provide paired-bootstrap 95% CIs and significance. Sample-size-weighted average results on both datasets follow:

MethodAUROCTPR@5%FPR
Min-K%++62.7 ± 1.212.5 ± 1.7
DC-PDD62.4 ± 1.312.7 ± 1.6
ETD64.1 ± 1.3∗†^{*\dagger}14.7 ± 1.7∗†^{*\dagger}

ETD remains best with significant gains (Tables 25–28, updated manuscript).

We hope these analyses address your concerns. Thank you again for your constructive feedback, and we look forward to your further comments!

Reference

[1] Zhang et al. Min-k%++: Improved baseline for detecting pre-training data from large language models. ICLR 2025.

[2] Ke et al. Rethinking Pretraining Data Detection for LLMs: From Local to Global. ICML 2026.

[3] Zhang et al. Pretraining data detection for large language models: A divergence-based calibration method. EMNLP 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.