acceptodds
Under review as a conference paper at ICLR 2027

Machine Text Detectors are Membership Inference Attacks

Abstract

Although membership inference attacks (MIAs) and machine-generated text detection target different goals, their methods often exploit similar signals based on a language model’s probability distribution, and the two tasks have been studied independently. This can result in conclusions that overlook stronger methods and valuable insights from the other task. In this work, we theoretically and empirically demonstrate the transferability, i.e., how well a method originally developed for one task performs on the other, between MIAs and machine text detection. We prove that the metric achieving asymptotically optimal performance is identical for both tasks. We unify existing methods under this optimal metric and hypothesize that the accuracy with which a method approximates this metric is directly correlated with its transferability. Our large-scale empirical experiments demonstrate a strong rank correlation () in cross-task performance. Notably, we also find that a detector achieves the strongest performance among evaluated methods on both tasks, demonstrating the practical impact of transferability. To facilitate cross-task development and fair evaluation, we introduce MINT, a unified evaluation suite for MIAs and detection, implementing 15 recent methods from both tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Membership Inference Attack, Machine-generated Text Detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_T5h5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the connection between membership inference attacks (MIAs) and machine-generated text detection. Under idealized asymptotic assumptions, it argues that both tasks share the same optimal likelihood-ratio statistic comparing the target model’s distribution with the human-text population distribution. It interprets existing methods as approximations to this statistic and evaluates their cross-task transferability using MIMIR and RAID. The experiments report substantial correlation between method rankings and show that Binoculars, originally developed for text detection, performs strongly on membership inference as well. Additional analyses examine black-box detection and methods with limited transferability, such as Zlib. The paper also introduces MINT, a unified evaluation suite for methods from both tasks.

Critical strengths
  1. The paper is well written.
  2. The paper provides a diverse range of empirical results.
Critical weaknesses

My main concern is that the paper’s conclusions do not appear particularly surprising. Connecting these two research areas seems fairly straightforward, and I believe there is a substantial body of similar work, particularly from 2025–2026.

Questions for authors

Could you briefly explain what is novel about the paper’s conclusions and what actionable insights they offer for designing future machine-generated text detectors?

3 comments
AuthorsOpenReview ↗

Response to Reviewer T5h5

Thank you for your valuable feedback. We sincerely appreciate your positive assessment of the clarity of our paper and the breadth of our empirical results.

We’ll reply to each question below.

Could you briefly explain what is novel about the paper’s conclusions and what actionable insights they offer for designing future machine-generated text detectors?

We agree that similar methods have emerged in both MIAs and machine-generated text detection, making their connection appear intuitive. However, methodological similarities alone do not establish whether the two tasks, which have fundamentally different objectives (members vs. non-members and machine-generated vs. human-written text), share a theoretical foundation or whether methods developed for one task perform well on the other. As discussed in Section 4, prior studies have noted specific similarities, but a systematic understanding of their relationship has remained limited.

Our novelty lies in moving beyond these individual observations to establish a theoretical and empirical foundation for cross-task transferability. Specifically, (1) we show that both tasks share the same optimal likelihood-ratio statistic under idealized assumptions—and during the rebuttal, we further proved a non-asymptotic theorem showing that for any finite-capacity, single-epoch model, cross-task transfer between the exact optimal MIA and detection statistics is directly proportional to expected counterfactual memorization γ(x)/PQ(x)\gamma(x)/P_Q(x) (2) we provide, to our knowledge, the first systematic evaluation of methods from both communities on each task's standard benchmark under a unified implementation; and (3) we reveal substantial cross-task performance correlation and demonstrate that top detectors match leading MIA methods (and vice versa in black-box detection). These results establish that the connection is not merely a methodological similarity but also has practical significance.

Our findings also offer guidance for future detector development. First, our empirical results motivate considering MIA methods as candidates for machine text detection (for instance, Min-K% and DC-PDD outperform several dedicated detectors in black-box detection), rather than developing methods independently within each community. Second, our likelihood ratio perspective identifies the approximation of the unknown human text distribution as a common design objective, with existing methods adopting different strategies, such as reference models or perturbation-based sampling. This provides a principled basis for exploring new approximation strategies and combining existing approaches. For instance, selecting methods that approximate this distribution differently may help exploit complementary signals in ensembling. While this direction remains to be validated experimentally, it illustrates how our framework can inform future method design.

Thank you for your rebuttal. I will keep my positive score. I think this is well written with a diverse range of empirical results. Good luck!

Thank you! We appreciate it

Reviewer_vW5vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that membership inference attacks (MIAs) and zero-shot machine text detection share one optimal test statistic: the likelihood ratio Λ(x) = P_M(x)/P_Q(x) between the target model and the distribution of human text (Theorem 2.2). The MIA half of the proof assumes that the model exactly reproduces the empirical distribution of its training set. The paper then rewrites existing methods as approximations of Λ (Table 1) and hypothesises that a method transfers better across the two tasks when it approximates Λ more closely. In the experiments, 15 methods are ranked by AUROC on MIMIR for MIA, using five Pythia models, and on RAID for detection. The two rankings have Spearman ρ = 0.66, and Binoculars, a detector, has the best average AUROC on both tasks. The authors release MINT, a unified evaluation suite.

Critical strengths
  1. A worthwhile question and a reusable evaluation. To my knowledge, this is the first study to run MIA methods and detectors on each task's standard benchmark under one implementation. The appendix reports every domain and model, and the code is released. The finding that detectors match state-of-the-art MIAs on MIMIR is a useful baseline message for MIA evaluation, and it stands independently of the theory.

  2. Candid reporting. The appendix includes the domains excluded from the ranking, and §3.3 analyses Zlib as a method that does not transfer. The Remark also acknowledges that the theorem covers an idealised training regime. These choices make the claims easy to check.

Critical weaknesses
  1. The theorem covers only a degenerate regime, and the likelihood-ratio view gives no tested design guidance.

    • Theorem 2.2 assumes that P_M equals the empirical distribution of the training set. Under this assumption a sample from the model is a training example, so machine-generated text and member text are the same set by construction, and the theorem says little about models that generalise. The same assumption makes D_KL(P_Q ‖ P_M) infinite because P_M has finite support, which leaves the advantage bound vacuous. Step 3 also derives the bound for detection, and the paper does not justify applying it to MIA.
    • The paper never measures how closely a method approximates Λ, so its hypothesis about transfer is untested, and the appendix contains counterexamples. Reference is the most literal form of P_M/P_Mref, yet it falls below chance on white-box detection (21.7 on Abstracts with GPT-2, 12.6 on Recipes with LLaMA). Loss makes no attempt to approximate P_Q and reaches 97–100 on most detection cells. Min-K% and Lastde++, which the paper places outside the likelihood-ratio view, transfer well.
    • Contribution (1) claims practical guidance for developing stronger methods, but the only guidance given is to approximate P_Q better. The paper does not say which of the two approximation routes to prefer or how to choose a reference model. Table 1 restates existing methods in a common notation without predicting which will work, and no method is derived or improved from it. The headline result is also unexplained. On MIMIR, Binoculars uses Pythia-deduped as its reference model, which was trained on essentially the same corpus as the target model. Its advantage may therefore come from a well-matched reference model, and no ablation separates this from the detector's design.

    The theory is the paper's stated novelty over earlier observations that the two tasks are similar. Without it, the contribution is an empirical comparison across two benchmarks.

  2. The empirical evidence is too thin for the headline claims.

    • The rank correlation is computed over n = 15 methods, several of which are near-duplicates. An approximate 95% interval for ρ = 0.66 at this sample size is roughly 0.2 to 0.9. The ρ = 0.78 figure for the top-10 MIA methods comes from a subset chosen after seeing the results.
    • The rankings depend on very small gaps, and no confidence intervals are reported. On MIMIR every method lies between 51 and 58 AUROC. Averaging Table 2 over the five models gives approximately 54.48 for Binoculars, 54.44 for Fast-DetectGPT, and 54.28 for Min-K%++, so the claim that Binoculars is best on both tasks rests on a 0.04-point margin. On white-box RAID many methods score 99–100, which leaves the top ranks to noise.
    • The MIA ranking excludes GitHub and DM Mathematics, the two domains where detectors are clearly worse than baselines. On GitHub, for example, Fast-DetectGPT scores 57.8–72.3 and Loss scores 76.3–83.6.
    • Each task is evaluated on one benchmark. For detection, benchmarks with open-weight generators would allow further white-box tests, for example MGT-Academic (Liu et al., "On the Generalization and Adaptation Ability of Machine-Generated Text Detectors in Academic Writing", KDD 2025) and M4 (Wang et al., "M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection", EACL 2024). For MIA, a setting with stronger membership signal, such as controlled fine-tuning with known members, would show whether the ranking holds when methods can be separated.
    • All 15 methods depend on the target model's likelihood, so some positive rank correlation is expected under any explanation. The correlation alone cannot confirm that a shared optimal statistic is the cause.
Questions for authors

Clarification requests (answers could change my assessment):

  1. Is D_KL(P_Q ‖ P_M) finite under the assumptions of Theorem 2.2? How does Step 3 apply to the MIA test, and what are the asymptotic conditions?
  2. Please report bootstrap confidence intervals for per-method AUROC and for ρ, and report ρ with GitHub and DM Mathematics included. Can the MIMIR ordering of Binoculars, Fast-DetectGPT, and Min-K%++ be distinguished statistically?
  3. Can you define how closely a method approximates Λ and test the hypothesis directly, for example in a controlled setting where P_Q is known? How does the likelihood-ratio view account for Reference falling below chance and Loss being near-perfect on detection?
  4. Which design decision does the likelihood-ratio view recommend that a practitioner would not already make? Can you use it to modify an existing method and show an improvement on both tasks?
  5. How sensitive is Binoculars' MIA result to the reference model, and how does it compare to Reference when both use the same reference model?
  6. Do the rank correlation and the Binoculars result hold on additional benchmarks, such as MGT-Academic and M4 for detection and a stronger-signal setting for MIA?

Suggestion for future work (would not change my assessment): extend the analysis to finite-capacity models trained for a single epoch.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vW5v (part 1)

Thank you for the valuable feedback. We sincerely thank the reviewer for recognizing the value of connecting membership inference and machine text detection, the usefulness of our unified cross-task evaluation, and the practical significance of identifying strong transfer across the two tasks.

We’ll reply to each question below.

Is DKL(PQ∥PM)D_{\mathrm{KL}}(P_Q \parallel P_M) finite under the assumptions of Theorem 2.2?

No, not necessarily. Under our assumptions, PMP_M is the empirical distribution of a finite training dataset sampled from PQP_Q. Since PMP_M assigns zero probability to unseen texts, DKL(PQ∥PM)D_{\mathrm{KL}}(P_Q \parallel P_M) can be infinite, as the reviewer correctly points out. However, since the support of PMP_M is contained within that of PQP_Q, DKL(PM∥PQ)D_{\mathrm{KL}}(P_M \parallel P_Q) is finite. We will correct the bound by applying Pinsker's inequality using this reverse direction, which provides a valid upper bound without affecting the derivation of the shared optimal likelihood ratio statistic.

How does Step 3 apply to the MIA test?

Thank you for raising this. Yes while MGTD considers TV(PM,PQ)\mathrm{TV}(P_M, P_Q), you are right to point out that MIA considers the TV distance between distributions over trained models under H0H_0 and H1H_1 so the advantage is not naively equal. This is something we originally had missed and we appreciate you pointing this out to us.

On further analysis we managed to show that they are equal in expectation. From Theorem 2.2, the MIA optimal likelihood ratio under our assumptions is P(M∣H1)/P(M∣H0)=PM(x)/PQ(x)P(M \mid H_1)/P(M \mid H_0) = P_M(x)/P_Q(x). Plugging this into the definition of TV distance we get TVMIA(x)=12PQ(x)EM∼H0[∣PM(x)−PQ(x)∣]\mathrm{TV}_{\mathrm{MIA}}(x) = \frac{1}{2P_Q(x)}\mathbb{E}_{M \sim H_0}[\lvert P_M(x)-P_Q(x) \rvert]. Taking the expectation over x∼PQx \sim P_Q cancels PQ(x)P_Q(x) and proves that the expected optimal advantages of the two tasks are exactly equal in our asymptotic regime:

Ex∼PQ[TVMIA(x)]=∑x∈XPQ(x)(12PQ(x)EM∼H0[∣PM(x)−PQ(x)∣])=EM∼H0[TV(PM,PQ)].\mathbb{E}_{x \sim P_Q}\left[\mathrm{TV}_{\mathrm{MIA}}(x)\right] = \sum_{x \in \mathcal{X}} P_Q(x)\left(\frac{1}{2P_Q(x)}\mathbb{E}_{M \sim H_0}\left[\lvert P_M(x)-P_Q(x) \rvert\right]\right) = \mathbb{E}_{M \sim H_0}\left[\mathrm{TV}(P_M, P_Q)\right].

We’ll include this proof in the appendix of the final version.

what are the asymptotic conditions?

Our asymptotic assumption in Theorem 2.2 is that the model has sufficient capacity relative to its training dataset and is trained for sufficiently many epochs to perfectly maximize dataset likelihood, such that PM(x)=count(x,D)/NP_M(x)=\mathrm{count}(x,D)/N, where NN is the size of DD.

Extension to Non-Asymptotic Models (Addressing W1). To address your concern that the empirical-distribution assumption covers only a degenerate regime without generalization, we also proved a non-asymptotic task-transfer theorem valid for any finite-capacity, single-epoch model M←T(D)M \leftarrow \mathcal{T}(\mathcal{D}) trained on D=(X1,…,XN)∼PQN\mathcal{D}=(X_1,\dots,X_N)\sim P_Q^N. For optimal statistics ΛMGTD(x;M)=PM(x)/PQ(x)\Lambda_{\mathrm{MGTD}}(x; M)=P_M(x)/P_Q(x) and ΛMIA(x;M)=P(M∣Xi=x)/P(M)\Lambda_{\mathrm{MIA}}(x; M)=P(M \mid X_i=x)/P(M), positive expected transfer holds iff CovM(Λ,S)>0\mathrm{Cov}_M(\Lambda, S) > 0 under H0H_0. Since EM[ΛMIA]=1\mathbb{E}_M[\Lambda_{\mathrm{MIA}}]=1, expanding the covariance yields:

CovM(ΛMIA,ΛMGTD)=EM∣Xi=x[PM(x)]−EM[PM(x)]PQ(x)=γ(x)PQ(x),\mathrm{Cov}_M(\Lambda_{\mathrm{MIA}}, \Lambda_{\mathrm{MGTD}}) = \frac{\mathbb{E}_{M \mid X_i=x}[P_M(x)] - \mathbb{E}_M[P_M(x)]}{P_Q(x)} = \frac{\gamma(x)}{P_Q(x)},

where γ(x)\gamma(x) is the expected probability gain induced by training on xx—often referred to as counterfactual memorization [1, 2] (full citations in Part 3). Thus, without requiring memorization of the empirical distribution, positive transfer holds iff γ(x)>0\gamma(x) > 0. Moreover, by Cauchy–Schwarz, any practical score S(x;M)S(x; M) satisfies:

CovM(ΛMIA,S)≥γ(x)PQ(x)−VarM(ΛMIA)⋅∥ΛMGTD−S∥L2.\mathrm{Cov}_M(\Lambda_{\mathrm{MIA}}, S) \ge \frac{\gamma(x)}{P_Q(x)} - \sqrt{\mathrm{Var}_M(\Lambda_{\mathrm{MIA}})} \cdot \lVert \Lambda_{\mathrm{MGTD}} - S \rVert_{L_2}.

This means any practical detector SS inherits the optimal detector's MIA advantage up to a penalty proportional to its L2L_2 approximation error to ΛMGTD\Lambda_{\mathrm{MGTD}} (and symmetrically for MGTD), formally grounding our hypothesis that better approximations transfer better.

AuthorsOpenReview ↗

Response to Reviewer vM5v (Part 2)

(Continued from Part 1)

Please report bootstrap confidence intervals for per-method AUROC and for ρ\rho

Since sample-level scores were not retained, we computed 95% confidence intervals (CIs) using 10,000 domain-level bootstrap replicates, resampling domains while keeping all model sizes and generators within each domain together (capturing cross-domain variability):

MethodMIMIR AUROC [95% CI]RAID AUROC [95% CI]
Loss53.4 [52.1, 55.1]97.3 [95.1, 98.7]
Rank52.9 [52.3, 53.6]82.2 [71.6, 89.9]
LogRank53.4 [52.1, 55.1]97.6 [95.4, 99.0]
Entropy52.4 [51.2, 54.0]81.6 [77.5, 85.6]
Reference53.7 [52.8, 54.7]63.1 [54.3, 70.8]
Zlib53.5 [52.5, 54.8]81.2 [77.4, 86.0]
Neighborhood / DetectGPT53.9 [52.7, 55.0]81.9 [79.5, 84.5]
Min-K%53.7 [52.6, 55.1]99.0 [98.2, 99.7]
Min-K%++54.3 [53.0, 55.7]98.4 [97.4, 99.3]
ReCaLL53.6 [51.9, 55.1]82.5 [77.5, 86.7]
DC-PDD53.4 [51.9, 55.1]84.5 [81.7, 87.4]
Fast-DetectGPT54.4 [53.0, 56.0]99.4 [98.7, 99.8]
Binoculars54.4 [53.0, 55.9]99.9 [99.8, 99.9]
DetectLLM54.0 [52.6, 55.3]99.4 [98.8, 99.9]
Lastde++54.0 [52.9, 55.2]98.5 [97.1, 99.6]

For cross-task rank correlation, we obtain ρ\rho = 0.66 [0.10, 0.83], supporting a positive correlation despite uncertainty in magnitude. Excluding any single domain preserves positive correlation (ρ\rho = 0.54–0.71 on MIMIR and 0.55–0.73 on RAID). We will incorporate these results into the revised paper.

report ρ\rho with GitHub and DM Mathematics included.

As noted in the footnote of Section 3.1, we focus on textual domains to ensure a fair comparison under aligned conditions, since RAID covers textual domains. Including GitHub and DM Mathematics yields ρ\rho = -0.11 [-0.20, 0.65]. This decrease aligns with Duan et al. [3], who report that n-gram decontamination selects atypical nonmembers in high-overlap domains like GitHub, introducing distribution shifts. Consistent with this, Loss achieves 76.3 AUROC on GitHub even at 160M, while Reference performs below chance on both technical domains—favoring uncalibrated likelihood attacks independently of transferability to detection.

Can the MIMIR ordering of Binoculars, Fast-DetectGPT, and Min-K%++ be distinguished statistically?

Pairwise comparisons on MIMIR (five textual domains) using 10,000 shared domain-level bootstrap replicates show no statistically significant differences:

ComparisonDifference averaged over model sizes [95% CI]
Binoculars - Fast-DetectGPT+0.00 [-0.48, +0.52]
Binoculars - Min-K%+++0.16 [-0.51, +0.75]
Fast-DetectGPT - Min-K%+++0.16 [-0.51, +0.94]

Accordingly, we will revise "outperforms" to "performs comparably to". Importantly, this preserves our main finding that detectors achieve performance comparable to a state-of-the-art MIA method.

AuthorsOpenReview ↗

Response to Reviewer vW5v (Part 3)

(Continued from Part 2)

Can you define how closely a method approximates Λ\Lambda and test the hypothesis directly, for example in a controlled setting where PQP_Q is known?

We used GPT-2 fine-tuned on WikiText-2 (3 epochs) as PMP_M and Qwen2.5-14B as a proxy for PQP_Q, approximating ℓ^(x)=log⁡PM(x)−log⁡PQ(x)\hat{\ell}(x)=\log P_M(x)-\log P_Q(x) and defining approximation error as standardized rank-MSE on 2,000 held-out C4 documents (excluding 13-gram WikiText-2 overlaps):

Performance metricSpearman ρ\rho [95% CI]
MIA AUROC-0.36 [-0.55, -0.06]
Detection AUROC-0.57 [-0.69, -0.34]
Harmonic mean of both AUROCs-0.52 [-0.65, -0.25]

Approximation error negatively correlates with both tasks and their harmonic mean. Measuring rank-MSE to log⁡PM(x)\log P_M(x) alone showed no negative correlation (ρ\rho = +0.44 for MIA, 0.00 for detection, +0.16 for harmonic mean), confirming that likelihood ratio approximation quality—not closeness to target likelihood alone—drives cross-task performance.

How does the likelihood-ratio view account for Reference falling below chance and Loss being near-perfect on detection?

As with Zlib in Section 3.3, Loss excels on detection because machine-generated texts have lower loss under the target model than human texts. However, Reference subtracts a smaller reference model's loss. Unlike MIAs (where both classes come from PQP_Q), in detection the reference model may also assign lower loss to machine-generated texts, reversing the discriminative signal below chance.

Which design decision does the likelihood-ratio view recommend that a practitioner would not already make? Can you use it to modify an existing method and show an improvement on both tasks?

Our framework identifies approximating PQP_Q as a shared design objective across two independent communities, motivating: (1) considering MIA methods as candidates for detection, and (2) categorizing methods by how they approximate PQP_Q (external reference vs. perturbation sampling) to combine complementary categories in ensembles rather than arbitrarily. While demonstrating new detector gains remains future work, our primary contribution is establishing a theoretical and empirical foundation. We will clarify this scope in the final version.

How sensitive is Binoculars' MIA result to the reference model, and how does it compare to Reference when both use the same reference model?

In our original experiments, we followed recommended configurations (Pythia-70M for Reference; Pythia-deduped for Binoculars; both trained on the same underlying corpus). Comparing both with identical reference models on Pythia-6.9B across all MIMIR domains gives:

Reference modelReferenceBinoculars
Pythia-70M55.154.4
Pythia-70M-deduped55.554.2
Pythia-deduped (target-sized)53.556.1

Reference leads by 0.7–1.3 points with 70M models, whereas Binoculars leads by 2.6 points with the target-sized deduped model (~2-point sensitivity). We will include this ablation alongside recommended configurations.

Do the rank correlation and the Binoculars result hold on additional benchmarks, such as MGT-Academic and M4 for detection and a stronger-signal setting for MIA?

For stronger-signal MIA, we evaluated GPT-2 (EleutherAI/bergson-wikitext-2-gpt2) fine-tuned on WikiText-2 across 1–3 epochs (400 members/non-members, 400 human/generated continuations, 1,000 bootstrap resamples):

EpochVal - train lossMIA AUROC (mean)Spearman ρ\rho (MIA vs. detection)
10.0990.595 [0.568, 0.622]0.64 [0.42, 0.79]
20.1760.677 [0.655, 0.703]0.73 [0.58, 0.83]
30.2460.740 [0.718, 0.763]0.79 [0.60, 0.83]

As MIA signal strengthens (0.595 to 0.740), cross-task ρ\rho rises (0.64 to 0.73 to 0.79). Given rebuttal time limits, we leave MGT-Academic and M4 for future work.

References

[1] Feldman & Zhang. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. NeurIPS 2020.

[2] Zhang et al. Counterfactual Memorization in Neural Language Models. NeurIPS 2023.

[3] Duan et al. Do Membership Inference Attacks Work on Large Language Models? COLM 2024.

Reviewer_eFtMOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the transferability between membership inference attacks and machine-generated text detection, two tasks that both rely on a language model's probability distribution but have been studied independently. Theoretically, the authors show that under asymptotic assumptions both tasks share the same optimal test statistic, the likelihood ratio P_M(x)/P_Q(x) between the target model distribution and the true human text distribution. They interpret many existing methods from both tasks as approximation of the intractable denominator P_Q(x), either via an external reference distribution or via text sampling, and hypothesise that better approximations yield higher transferability. Empirically, they evaluate 15 methods on MIMIR (MIA) and RAID (detection), finding a Spearman rank correlation of ρ = 0.66 between the two tasks (0.78 among the top-10 methods), and that Binoculars, a detector, achieves the best average performance on both tasks. They also release MINT, a unified evaluation suite implementing 15 methods.

Critical strengths

S1. Connecting two communities that use similar likelihood-based signals but rarely compare methods is valuable. The finding that methods from one task can be competitive or even superior on the other (e.g., Binoculars on MIA, and Min-K% and DC-PDD in black-box detection) is a useful practical insight for both communities.

S2. Framing existing methods as different approximations of P_Q(x), via external references or text sampling, offers a unified way to understand methods from both fields. For example, the Neighbourhood attack and DetectGPT share the same formulation. In addition, the analysis of Zlib provides an interpretable explanation of why it performs moderately well on MIA but poorly on detection: machine-generated text is more compressible, so in detection the loss and the compression entropy shift in the same direction and cancel each other out.

S3. The study covers 15 methods on two widely used benchmarks, across multiple domains, model sizes, and both white-box and black-box detection settings, with full per-domain results reported in the appendix.

S4. The released MINT suite implements methods from both tasks in a common framework, which supports fair cross-task comparison and reproducibility.

Critical weaknesses

W1. Detection is formalised as testing whether x comes from the human distribution or from the specific model used for scoring. In practice, a text may be generated by a different LLM, by several models, or co-written by humans and models. The theorem therefore applies only to the while-box setting, where the scoring model coincides with the generator. In the black-box setting (Figure 4), text are generated by ChatGPT and GPT-4 but scored with surrogate models, so the surrogate likelihood ratio is no longer the Neyman-Pearson optimal statistic. Moreover, detecting text from any LLM involves a composite alternative hypothesis, for which a single most powerful test generally does not exist.

W2. The MIA derivation assumes infinite capacity and perfect memorisation, so that P_M(x) equals the empirical frequency of x in the training set. Under this assumption, membership is almost trivially determined (non-members essentially receive zero probability), which is far from the realistic regime: all methods achieve only 52–55 AUROC on MIMIR. The authors acknowledge in the Remark that real models are usually trained for a single pass.

W3. The paper hypothesises that transferability correlates with how well a method approximates, but never measures this approximation quality for any method. The evidence is only the rank correlation between the two tasks, which weakens the link between the theory and the empirical results.

Questions for authors
  1. How does the theoretical result extend to the black-box setting, e.g. texts are generated by ChatGPT and GPT-4 but scored with surrogate models? Is there any guarantee that the surrogate likelihood ratio remains close to optimal?

  2. How would the analysis change if the negative class included text generated by other LLMs, or if the positive class consisted of text from multiple or unknown generators?

  3. Under the perfect-memorization assumption, membership inference becomes almost trivially solvable, whereas all methods achieve only 52–55 AUROC on MIMIR. How does the theory inform this realistic regime, where models are trained for roughly one epoch? A non-asymptotic analysis would be ideal, but even an empirical check would help, for example, fine-tuning a small model for an increasing number of epochs and testing whether the cross-task rank correlation strengthens as training approaches the asymptotic regime.

  4. Could you provide evidence that a method’s approximation quality to Λ(x) correlates with its transferability? I do not expect a full-scale study during the rebuttal; even a small synthetic experiment where P_Q is known, or an analysis using a strong reference model as a proxy for P_Q (while noting its limitations), would help support the central hypothesis.

6 comments
AuthorsOpenReview ↗

Response to Reviewer eFtM (Part 1)

Thank you for the valuable feedback. We sincerely thank the reviewer for recognizing the value of connecting membership inference and machine text detection, the unified formulation of existing methods under approximate likelihood ratio tests, the comprehensive cross-task evaluation, and the practical significance of identifying strong transfer across the two tasks. (Due to OpenReview's character limit, our response is split into Part 1 and Part 2.)

We’ll reply to each question below.

How does the theoretical result extend to the black-box setting, e.g. texts are generated by ChatGPT and GPT-4 but scored with surrogate models? Is there any guarantee that the surrogate likelihood ratio remains close to optimal?

Thank you for the insightful question. For the black-box detection setting, we can define a composite alternative hypothesis:

H0:x∼PQ,H1:x∼PM,where M∈M.H_0: x \sim P_Q, \quad H_1: x \sim P_M, \quad \text{where } M \in \mathcal{M}.

Here, M\mathcal{M} is the set of any language models. The optimal likelihood ratio PM(x)/PQ(x)P_M(x)/P_Q(x) applies when the target distribution PMP_M is accessible. In the black-box setting, where MM is unknown, there is generally no uniformly most powerful test across all M∈MM \in \mathcal{M}.

Instead, existing methods often use a surrogate model SS to approximate the probability distribution of the target model MM, resulting in PS(x)/PQ(x)P_S(x)/P_Q(x). The effectiveness of this approximation depends on how closely PSP_S matches the true distribution PMP_M. In particular, the difference in detection advantages between the optimal likelihood ratio and the surrogate likelihood ratio is bounded by:

Advoptimal−Advsurrogate≤12[TV(PM,PQ)−(TV(PS,PQ)−TV(PM,PS))]≤TV(PM,PS)≤12DKL(PM∥PS),\mathrm{Adv}_{\mathrm{optimal}} - \mathrm{Adv}_{\mathrm{surrogate}} \le \frac{1}{2}\left[\mathrm{TV}(P_M, P_Q) - \left(\mathrm{TV}(P_S, P_Q) - \mathrm{TV}(P_M, P_S)\right)\right] \le \mathrm{TV}(P_M, P_S) \le \sqrt{\frac{1}{2} D_{\mathrm{KL}}(P_M \parallel P_S)},

where Advoptimal\mathrm{Adv}_{\mathrm{optimal}} and Advsurrogate\mathrm{Adv}_{\mathrm{surrogate}} denote the detection advantages of the likelihood ratio tests based on PM/PQP_M/P_Q and PS/PQP_S/P_Q, respectively. The inequalities follow from the definition of total variation distance, its triangle inequality, and Pinsker's inequality.

This provides a conditional guarantee, i.e., a surrogate model close to the true generator can achieve near-optimal classification power. Existing studies address this practical limitation in the black-box setting by using surrogate models [1,2] or pseudo likelihood estimation [3,4], but still these approximations do not necessarily preserve optimality (full references at the end of Part 2). We will incorporate this extension into the final version.

How would the analysis change if the negative class included text generated by other LLMs, or if the positive class consisted of text from multiple or unknown generators?

Thank you for the insightful question. When either class contains text from multiple or unknown generators, we can extend our formulation to a composite hypothesis test:

H0:x∼PA,PA∈PA,H1:x∼PB,PB∈PB,H_0: x \sim P_A, \quad P_A \in \mathcal{P}_A, \qquad H_1: x \sim P_B, \quad P_B \in \mathcal{P}_B,

where PA\mathcal{P}_A includes distributions of human text and other language models, while PB\mathcal{P}_B includes distributions of target language models. We agree that a uniformly most powerful test generally does not exist across all these distributions. However, for any fixed pair (PA,PB)(P_A, P_B), the Neyman–Pearson optimal statistic remains the likelihood ratio PB(x)/PA(x)P_B(x)/P_A(x), with PM(x)/PQ(x)P_M(x)/P_Q(x) as a special case.

AuthorsOpenReview ↗

Response to Reviewer eFtM (Part 2)

(Continued from Part 1)

How does the theory inform this realistic regime, where models are trained for roughly one epoch? A non-asymptotic analysis would be ideal, but even an empirical check would help...

Thank you for this suggestion. Looking into this we developed both a non-asymptotic theoretical analysis and the proposed empirical check.

Non-Asymptotic Analysis. Dropping the infinite-capacity and perfect-memorization assumptions, let D=(X1,…,XN)∼PQN\mathcal{D}=(X_1,\dots,X_N)\sim P_Q^N produce M←T(D)M \leftarrow \mathcal{T}(\mathcal{D}). For optimal statistics ΛMGTD(x;M)=PM(x)/PQ(x)\Lambda_{\mathrm{MGTD}}(x; M) = P_M(x)/P_Q(x) and ΛMIA(x;M)=P(M∣Xi=x)/P(M)\Lambda_{\mathrm{MIA}}(x; M) = P(M \mid X_i=x)/P(M), a score S(x;M)S(x; M) achieves positive expected transfer iff CovM(Λ,S)>0\mathrm{Cov}_M(\Lambda, S) > 0 under H0H_0. Since EM[ΛMIA]=1\mathbb{E}_M[\Lambda_{\mathrm{MIA}}] = 1, expanding the covariance yields:

CovM(ΛMIA,ΛMGTD)=EM∣Xi=x[PM(x)]−EM[PM(x)]PQ(x)=γ(x)PQ(x),\mathrm{Cov}_M(\Lambda_{\mathrm{MIA}}, \Lambda_{\mathrm{MGTD}}) = \frac{\mathbb{E}_{M \mid X_i=x}[P_M(x)] - \mathbb{E}_M[P_M(x)]}{P_Q(x)} = \frac{\gamma(x)}{P_Q(x)},

where γ(x)\gamma(x) is the expected probability gain induced by training on xx—often referred to as counterfactual memorization [5, 6]. Thus, positive transfer holds iff γ(x)>0\gamma(x) > 0 (small at 1 epoch, growing with epochs). Moreover, by Cauchy–Schwarz, for any score S(x;M)S(x; M):

CovM(ΛMIA,S)≥γ(x)PQ(x)−VarM(ΛMIA)⋅∥ΛMGTD−S∥L2.\mathrm{Cov}_M(\Lambda_{\mathrm{MIA}}, S) \ge \frac{\gamma(x)}{P_Q(x)} - \sqrt{\mathrm{Var}_M(\Lambda_{\mathrm{MIA}})} \cdot \lVert \Lambda_{\mathrm{MGTD}} - S \rVert_{L_2}.

This means any practical detector SS inherits the optimal detector's MIA advantage up to a penalty proportional to its L2L_2 approximation error to ΛMGTD\Lambda_{\mathrm{MGTD}} (and symmetrically for MGTD), formally grounding our hypothesis that better approximations transfer better. We will add this to the main paper.

Empirical Check. Using GPT-2 fine-tuned on WikiText-2 (EleutherAI/bergson-wikitext-2-gpt2; 400 members/non-members, 400 human/generated continuations, 1,000 bootstrap resamples), we evaluate all 15 methods:

EpochVal - train lossMIA AUROC (mean)Spearman ρ\rho (MIA vs. detection)
10.0990.595 [0.568, 0.622]0.64 [0.42, 0.79]
20.1760.677 [0.655, 0.703]0.73 [0.58, 0.83]
30.2460.740 [0.718, 0.763]0.79 [0.60, 0.83]

As training progresses, MIA performance improves and ρ\rho increases (0.64 to 0.73 to 0.79), while already reaching 0.64 at 1 epoch (mean AUROC 0.60), consistent with MIMIR and our non-asymptotic bound. We will add both results to the paper.

Could you provide evidence that a method’s approximation quality to Λ(x)\Lambda(x) correlates with its transferability?

Using 3-epoch WikiText-2 GPT-2 as PMP_M and Qwen2.5-14B as a proxy for PQP_Q, we measure standardized rank-MSE against ℓ^(x)=log⁡PM(x)−log⁡PQ(x)\hat{\ell}(x)=\log P_M(x)-\log P_Q(x) on 2,000 held-out C4 texts (excluding 13-gram WikiText-2 overlaps):

Performance metricSpearman ρ\rho [95% CI]
MIA AUROC-0.36 [-0.55, -0.06]
Detection AUROC-0.57 [-0.69, -0.34]
Harmonic mean of both AUROCs-0.52 [-0.65, -0.25]

Approximation error negatively correlates with both tasks and their harmonic mean, supporting our hypothesis that closeness to Λ(x)\Lambda(x) explains cross-task transferability.

References

[1] Sun et al. Minimizing Mismatch Risk: A Prototype-Based Routing Framework for Zero-shot LLM-generated Text Detection. ICML 2026.

[2] Bao et al. Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature. ICLR 2024.

[3] Zeng et al. DALD: Improving Logits-based Detector without Logits from Black-box LLMs. NeurIPS 2024.

[4] Kaneko et al. Sampling-based pseudo-likelihood for membership inference attacks. Findings of ACL 2025.

[5] Feldman & Zhang. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. NeurIPS 2020.

[6] Zhang et al. Counterfactual Memorization in Neural Language Models. NeurIPS 2023.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your careful consideration of our work. This is a gentle reminder that the discussion period ends soon.

To summarize our response to your concerns, we have extended our theoretical framework to black-box detection and mixed-generator settings, developed a non-asymptotic theoretical analysis, conducted an empirical check showing increasing cross-task correlation and MIA performance with training epochs, and provided empirical evidence that better likelihood ratio approximation is associated with stronger performance on both tasks.

We would be happy to discuss any remaining concerns and appreciate your time and guidance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.