acceptodds
Under review as a conference paper at ICLR 2027

Diana: Calibrating the Per-Token Null for Membership Inference on Fine-Tuned LLMs

Abstract

Membership inference attacks on language models detect whether a specific text was included in the training data. Fine-tuning on a domain complicates this task: an increase in a token’s log-probability over the pre-trained model, which we call its gain, may reflect domain-level learning rather than memorization of the target sequence itself, inflating false positives in reference-based attacks. We introduce *Diana* (Distribution-Informed Attacks with Null Adjustment), a plug-and-play calibration that isolates memorization from domain adaptation. Our two key observations are: (i) for non-members, the target model’s own next-token distribution approximates the domain distribution. This motivates centering: subtracting the target-weighted mean gain over possible next tokens from the observed gain. (ii) Under our modeling assumptions, the centered gain has zero conditional mean for non-members and positive conditional mean for members. To account for differences in background variability, *Diana* divides the centered gain by the regularized gain variance to obtain a calibrated gain. Hence, *Diana* requires only the log-probabilities of the target and pre-trained models—no auxiliary data or additional training—and integrates with any reference-based attacks, which then aggregate the calibrated gains under their original rules. Across six corpora, *Ratio**Diana* achieves the mean true-positive rate of *Ratio* at a false-positive rate and outperforms all evaluated prior attacks on every corpus.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
membership inference, large language models, privacy, privacy auditing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DE24OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses the challenge of distinguishing memorization from domain adaptation in membership inference on fine-tuned language models. An increase in token probability relative to a pretrained reference may reflect either exposure to the candidate sequence or learning patterns shared across the domain. The proposed method, DIANA, computes the target-weighted mean and variance of target-to-reference log-probability gains at each prefix. It subtracts this mean from the observed token’s gain and divides the result by a regularized variance. The calibration requires no auxiliary data or additional training and can be incorporated into reference-based attacks such as DIFF, RATIO, WBC, and EZ-MIA. Under idealized domain-matching and single-spike memorization assumptions, the paper analyzes non-member null centering, positive member means, and signal behavior near memorization saturation. Experiments cover six primary corpora, multiple model families, additional natural-text corpora, component ablations, restricted probability access, LoRA, and DP-SGD. In the primary evaluation, RATIO+DIANA increases mean TPR at 0.1% FPR from 11.2% to 47.4%.

Critical strengths
  1. Simple method with substantial gains and convincing ablations DIANA requires neither additional training nor auxiliary in-domain data. Across all six primary corpora, RATIO+DIANA achieves the highest point estimates on all reported primary metrics. Table 3 provides clear support for the design: centering alone increases mean TPR at 0.1% FPR from 0.112 to 0.418, while regularized variance weighting further improves it to 0.474. These results identify background subtraction as the main source of improvement, with variance weighting providing an additional benefit.

  2. Broad evaluation with useful mechanism diagnostics Beyond performance comparisons, Section 3.3 and Appendices M/N examine background estimation, variance prediction, and member signals during training. The target-weighted mean accounts for 91.1%–93.3% of the mean observed non-member gain, while members retain higher mean signals after centering. Experiments also cover multiple model families, natural-text corpora, and training and access settings. Although these diagnostics do not fully validate the modeling assumptions, they provide useful evidence about the calibration mechanism and its limitations.

Critical weaknesses
  1. The distinction between the theoretical guarantee and the implemented algorithm needs clarification. The proof of Proposition 1 uses the idealized mean prediction pⱼ⁻ and establishes null centering under pⱼ⁻ = pⱼᴰ, whereas the algorithm uses an individual trained model’s prediction pⱼᵀ. Agreement between the mean prediction and the domain distribution does not itself guarantee null centering for the actual model. Although the appendix acknowledges this distinction, the proposition’s direct reference to Equation (5) can suggest a guarantee for the implemented algorithm. Its scope should be stated explicitly and distinguished from empirical approximate calibration. This does not invalidate the experimental gains, but affects the precision of a central theoretical claim.

  2. The scope of the low-FPR gains remains insufficiently explained. On CC-News, RATIO+DIANA achieves an AUC of 0.928 but a TPR at 0.1% FPR of only 0.001, compared with 0.004 for uncalibrated RATIO. These point estimates do not establish significant degradation, but the discrepancy between strong overall discrimination and weak low-FPR performance deserves further explanation, given the paper’s emphasis on this operating regime.

Questions for authors
  1. Theoretical scope (regarding Weakness 1). Could the authors clarify the scope of Proposition 1 and discuss whether additional assumptions could extend the null-centering result from the idealized mean prediction to an individual trained model?

  2. Low-FPR performance (regarding Weakness 2). Could the authors explain the gap between strong AUC and weak performance at 0.1% FPR on CC-News? A comparison of centering-only and full DIANA, or an analysis of member and non-member score distributions near the decision threshold, would help clarify this behavior.

3 comments

We thank the reviewer for their feedback and constructive suggestions. We are encouraged that the reviewer found DIANA to be a "simple method with substantial gains and convincing ablations," and recognized our "broad evaluation with useful mechanism diagnostics." The revised paper has been uploaded, with changes highlighted in blue.


W1 & Q1: Theoretical Guarantee vs. Implemented Algorithm

W1: "The distinction between the theoretical guarantee and the implemented algorithm needs clarification. The proof of Proposition 1 uses the idealized mean prediction pj−p^-_j and establishes null centering under pj−=pjDp^-_j=p^D_j, whereas the algorithm uses an individual trained model's prediction pjTp^T_j. Agreement between the mean prediction and the domain distribution does not itself guarantee null centering for the actual model. Although the appendix acknowledges this distinction, the proposition's direct reference to Equation (5) can suggest a guarantee for the implemented algorithm. Its scope should be stated explicitly and distinguished from empirical approximate calibration."

Q1: "Could the authors clarify the scope of Proposition 1 and discuss whether additional assumptions could extend the null-centering result from the idealized mean prediction to an individual trained model?"

Proposition 1 is stated for the mean prediction pj−p^-_j. To extend it to the individual trained model that the algorithm uses, the assumption has to be made about that model: if the trained model's own prediction matches the domain distribution (pjT=pjDp^T_j=p^D_j), the same proof gives a zero non-member mean for the calibrated gain of Equation (5) as implemented. Agreement between the mean prediction and the domain distribution is not enough for this, as the reviewer notes. The revision states this scope directly after Proposition 1 (Section 3.2) and gives the result for a trained model after the proof (Appendix A.1).

A trained model does not meet this assumption with 100% accuracy, so in practice the calibration is approximate. The assumption guides the design of DIANA and helps explain why it works, and we measure how closely it holds. If the model's non-member predictive distribution exactly matches the domain distribution, then the baseline (the average gain under the fine-tuned model's own prediction) equals the gain that non-members actually receive. We test this on documents the model was not trained on. Their gain comes only from domain adaptation, so the baseline should remove all of it. On all six corpora, the baseline is 91–93% as large as the gain that non-members receive (Table 23). On Stanford, for example, subtracting it lowers the average non-member gain from 0.429 to 0.029 nats. After the full calibration of Equation (5), the non-member mean stays within 0.033 of zero at every level of gain variance (Section 3.3, Figure 2b). The same assumption also gives the variance of a non-member's centered gain, and the measured values match it with R2R^2 of 0.993–0.999 (Appendix M.2, Table 25).

W2: "The scope of the low-FPR gains remains insufficiently explained. On CC-News, RATIO+DIANA achieves an AUC of 0.928 but a TPR at 0.1% FPR of only 0.001, compared with 0.004 for uncalibrated RATIO. These point estimates do not establish significant degradation, but the discrepancy between strong overall discrimination and weak low-FPR performance deserves further explanation, given the paper's emphasis on this operating regime."

Q2: "Could the authors explain the gap between strong AUC and weak performance at 0.1% FPR on CC-News? A comparison of centering-only and full DIANA, or an analysis of member and non-member score distributions near the decision threshold, would help clarify this behavior."

We thank the reviewer for pointing this out. We re-examined CC-News and found that the low TPR at 0.1% FPR is caused by duplicated entries in the data. We drew the CC-News split from the public corpus, whose entries turn out not to be unique, so after the random split many of its non-members are copies of members: 3.8% are exact copies of a member article, and 8.9% share more than half of their 8-word sequences with the member set. The model was trained on these texts, so every attack gives them high scores. They are too few to change the AUC much, but they are more than enough to set the threshold at 0.1% FPR, which is why the AUC is high while the TPR at 0.1% FPR is low. The table shows CC-News before and after removing them. We also checked the other corpora, and this issue appears only in CC-News. The revision removes these copies from the CC-News split (Tables 2 and 10; procedure in Appendix D).

CC-NewsRATIO, TPR @ 1% FPRRATIO+DIANA, TPR @ 1% FPRRATIO, TPR @ 0.1% FPRRATIO+DIANA, TPR @ 0.1% FPR
Original split0.0170.0390.0040.001
Deduplicated split0.0520.7880.0130.519

Both analyses the reviewer suggests point to this cause. Near the threshold, the highest-scoring non-members are these copies: all of the 100 non-members with the highest RATIO+DIANA scores (the top 1%) share more than half of their 8-word sequences with the member set, and the same holds for 76 of the top 100 under RATIO and all of them under WBC. Centering only and full DIANA also fail in the same way on the original split, with a TPR of 0.001 at 0.1% FPR for both, and both recover on the deduplicated split, to 0.224 with centering only and 0.519 with full DIANA.

Thank you for the response. The authors have addressed most of my concerns. However, I do not find the paper strong enough to warrant a score of 4, so I will maintain my current score.

Reviewer_vhBqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Diana studies membership inference for fine-tuned autoregressive language models. It calibrates token-level likelihood gains using a predictive-distribution-based background estimate and variance weighting, aiming to distinguish memorization from domain adaptation. Experiments across multiple datasets report improved detection, including at low false-positive rates, supported by ablations and theoretical analysis under idealized assumptions.

Critical strengths

Simple, interpretable calibration that integrates with existing attacks without additional training. Strong empirical gains, with ablations showing that centering supplies most of the improvement.

Critical weaknesses

The calibration mechanism is not fully validated. DIANA estimates the non-member background from the target model itself, whose predictions reflect both domain adaptation and memorization. Calibration may therefore suppress genuine membership signals while removing background gains. The theoretical analysis relies on idealized assumptions, and aggregate moment diagnostics do not fully characterize the final sequence scores at low false-positive rates. Although the experiments support empirical effectiveness, the interpretation as separating adaptation from memorization requires more direct evidence. The advantage over strong baselines does not consistently generalize, and its applicability remains unclear. Despite substantial gains in the primary experiments, DIANA underperforms WBC in some cross-model and natural-text settings. The transfer experiments also omit EZ-MIA, the strongest prior baseline on average in the primary evaluation. The results therefore leave insufficiently explained when calibration helps, when it hurts detection, and which factors govern this behavior.

Questions for authors

Can the authors provide controlled inclusion/exclusion experiments with matched domain-training conditions to assess whether the background estimate captures non-member adaptation gains? How sensitive are the resulting scores to memorization-induced changes in that estimate?

At a fixed overall FPR, how does recall vary across memorization strengths or exposure frequencies compared with the baselines? Does the signal attenuation observed in Appendix N cause missed detections among strongly memorized members?

How should the decision threshold be selected without access to labeled evaluation candidates? Can the authors evaluate independently calibrated thresholds and report the resulting precision–recall trade-offs under plausible membership prevalences?

4 comments

We thank the reviewer for their feedback and constructive suggestions. We are encouraged that the reviewer found DIANA to be a "simple, interpretable calibration" with "strong empirical gains," and acknowledged that our ablations demonstrate the importance of centering. The revised paper has been uploaded, with changes highlighted in blue.


W1 & Q1: Validity of the Background Estimate

W1: "The calibration mechanism is not fully validated. DIANA estimates the non-member background from the target model itself, whose predictions reflect both domain adaptation and memorization. Calibration may therefore suppress genuine membership signals while removing background gains. The theoretical analysis relies on idealized assumptions, and aggregate moment diagnostics do not fully characterize the final sequence scores at low false-positive rates. Although the experiments support empirical effectiveness, the interpretation as separating adaptation from memorization requires more direct evidence."

Q1: "Can the authors provide controlled inclusion/exclusion experiments with matched domain-training conditions to assess whether the background estimate captures non-member adaptation gains? How sensitive are the resulting scores to memorization-induced changes in that estimate?"

We ran the inclusion/exclusion experiment the reviewer asks for. It shows that including a document in training changes the background estimate very little, so calibration removes little of the membership signal.

The experiment measures how much training on a document raises its observed gain and how much it raises its background estimate. This needs the same document scored twice: by a model that was trained on it, and by a model that was not. We therefore fine-tuned a second OLMoE model on a different random half of the same Stanford pool. 10,012 documents were in the training set of our main target but not of the second model. For each of them, we compute the observed gain and the background estimate under the main target and subtract the values under the second model. Two separately trained models also differ on text that neither has seen, so we subtract their difference on documents that neither was trained on. What remains is the change caused by training on the document. The table reports it next to the values for documents excluded from training (Table 23, Stanford).

Nats per tokenObserved gainBackground estimateBackground estimate as a share of observed gain
Documents excluded from training0.4290.40093.3%
Added by training on the document0.12910.013610.5%

For documents excluded from training, the observed gain comes only from domain adaptation, and the background estimate captures 93.3% of it (91–93% on all six corpora, Table 23). Training on a document raises its observed gain by 0.1291 nats per token and its background estimate by only 0.0136, so the background estimate takes away about one tenth of what membership adds (10.5%, 95% CI 9.7–11.3%). It rises so little because it is an average over all possible next tokens, so one memorized token changes it little unless the model puts almost all of its probability on that token.

Q2: Signal Attenuation and Recall Across Memorization Strengths

"At a fixed overall FPR, how does recall vary across memorization strengths or exposure frequencies compared with the baselines? Does the signal attenuation observed in Appendix N cause missed detections among strongly memorized members?"

The decrease of the centered member gain in Appendix N does not cause missed detections: the gain does not approach zero in practice, and its decrease does not reduce detection.

First, the centered member gain stays well above zero. It would reach zero only if the model predicted every member token with probability one. In the 20-epoch runs of Appendix N, the gain starts to fall only after epoch 6, when the model already overfits: its held-out perplexity has been rising since epoch 2 (table below). After 20 epochs, the model is far worse than the pre-trained model on held-out text (perplexity 16.85 against 4.93), yet the centered gain is still 0.089 nats per token (Table 26), about three times that of non-members in typical fine-tuning (0.025–0.035 nats, Table 24).

Second, the decrease does not reduce detection, because the spread of the gain falls as well. From epoch 6 to epoch 20, the member centered gain falls from 0.238 to 0.089 nats per token, and its standard deviation across member tokens falls from 0.755 to 0.298. Members in fact become easier to separate: the gap between the average RATIO+DIANA scores of members and non-members grows from 5.5 to 7.2 standard deviations. The table shows both: after epoch 6 the centered gain falls, while every attack still detects every member at 1% FPR.

Epoch12461020
Held-out perplexity (pre-trained: 4.93)3.263.193.343.826.2316.85
Centered member gain (Table 26)0.1210.1790.2290.2380.1860.089
RATIO, TPR @ 1% FPR0.1150.7831.0001.0001.0001.000
WBC, TPR @ 1% FPR0.2590.8781.0001.0001.0001.000
RATIO+DIANA, TPR @ 1% FPR0.6660.9851.0001.0001.0001.000

Held-out perplexity is measured on the 800 non-members of each checkpoint and averaged over the three seeds. Each epoch is one more exposure to every member, so the table also gives recall by exposure frequency.

We also ran the comparison across memorization strengths that the reviewer asks for. We use the experiment of Q1, which gives, for each member, how much training on it raised its observed gain. A larger rise means the document was memorized more strongly. We sort the members by this rise and split them into five equal groups. The table reports the share of members detected in each group, using one threshold per attack: the score that only 0.1% of all non-members exceed.

Share of members detected at 0.1% FPRLeast memorized 20%2nd3rd4thMost memorized 20%
RATIO0.0020.0270.1160.2210.337
WBC0.0010.0550.2500.5360.816
RATIO+DIANA0.0390.2880.6340.8390.937

RATIO+DIANA has the highest recall in every group, including the most strongly memorized members.

W2: Inconsistent Generalization and Missing Transfer Baseline

"The advantage over strong baselines does not consistently generalize, and its applicability remains unclear. Despite substantial gains in the primary experiments, DIANA underperforms WBC in some cross-model and natural-text settings. The transfer experiments also omit EZ-MIA, the strongest prior baseline on average in the primary evaluation. The results therefore leave insufficiently explained when calibration helps, when it hurts detection, and which factors govern this behavior."

DIANA is a calibration of the per-token signal that is applied to an existing attack, so the fair comparison is the same attack with and without DIANA: RATIO against RATIO+DIANA, and WBC against WBC+DIANA. RATIO and WBC turn token scores into a sequence score in different ways, so comparing RATIO+DIANA with WBC changes the calibration and the aggregation at once, and cannot show whether the calibration helps or hurts. The "+DIANA" entries in Table 10 are RATIO+DIANA. On the original CC-News split, they are better than RATIO in 39 of the 40 cells. We re-examined the only exception, CC-News at 0.1% FPR, and found that it is caused by duplicated entries in the data. We drew the CC-News split from the public corpus, whose entries turn out not to be unique, so after the random split many of its non-members are copies of members: 3.8% are exact copies of a member article, and 8.9% share more than half of their 8-word sequences with the member set. The model was trained on these texts, so every attack gives them high scores, and they set the threshold at low FPR. The table shows CC-News before and after removing them. We also checked the other corpora, and this issue appears only in CC-News. The revision removes these copies from the CC-News split (Tables 2 and 10; procedure in Appendix D).

CC-NewsRATIO, TPR @ 1% FPRRATIO+DIANA, TPR @ 1% FPRRATIO, TPR @ 0.1% FPRRATIO+DIANA, TPR @ 0.1% FPR
Original split0.0170.0390.0040.001
Deduplicated split0.0520.7880.0130.519

The same holds when DIANA is applied to WBC itself: WBC+DIANA is better than WBC in all 40 cells. This includes the settings the reviewer points to, where WBC is ahead of RATIO+DIANA, so in every row of Table 10 the best of these attacks is one calibrated with DIANA.

TPR @ 0.1% FPRRATIORATIO+DIANAWBCWBC+DIANA
Qwen2.5-1.5B0.0070.0140.0340.044
CC-News (deduplicated)0.0130.5190.6140.757
Wikipedia0.0190.7810.8360.863

We also ran EZ-MIA, with and without DIANA, on the other nine rows of Table 10, as the reviewer suggests.

AUCTPR @ 1% FPRTPR @ 0.1% FPR
Target / corpusEZ-MIA+DIANAEZ-MIA+DIANAEZ-MIA+DIANA
Llama-3.1-8B0.9500.9530.7980.8080.7160.733
Pythia-2.8B0.8010.8150.2430.3060.0980.180
Pythia-1.4B0.7850.8050.1960.2760.0770.144
OPT-1.3B0.7720.7920.1690.2500.0570.114
Qwen2.5-1.5B0.7350.7620.0970.1210.0190.029
Mamba-1.4B0.7740.7870.1400.2000.0390.061
XSum0.9590.9660.7730.8490.6050.755
CC-News (deduplicated)0.9690.9730.7980.8690.5890.677
Wikipedia0.9640.9710.8380.8780.7230.821

DIANA improves EZ-MIA in every row and on every metric in the table.

Therefore, with the CC-News split corrected, DIANA improves every attack it is applied to on every model and corpus of Table 10 and on all four metrics. We found no setting where it hurts detection.

Q3: Threshold Selection and Precision–Recall Trade-offs

"How should the decision threshold be selected without access to labeled evaluation candidates? Can the authors evaluate independently calibrated thresholds and report the resulting precision–recall trade-offs under plausible membership prevalences?"

The threshold for a target FPR can be chosen from a small set of known non-members, which is the standard practice in membership inference (Ye et al., 2022, cited in the paper). We tested it on the six primary corpora. For a target FPR α\alpha, we take 2,000 non-members and set the threshold so that a fraction α\alpha of them score above it (the ⌈(1−α)(2000+1)⌉\lceil(1-\alpha)(2000+1)\rceil-th smallest score, which is the standard conformal quantile). We apply this threshold to all remaining candidates, and repeat with 200 random choices of the 2,000. The table reports the realized FPR and TPR. It also reports precision, the fraction of flagged candidates that are members, when a fraction π\pi of the candidates are members: precision is π⋅TPR/(π⋅TPR+(1−π)⋅FPR)\pi\cdot\mathrm{TPR}/(\pi\cdot\mathrm{TPR}+(1-\pi)\cdot\mathrm{FPR}), and the table uses π\pi of 10%, 1% and 0.1%.

Target FPRAttackRealized FPRTPRPrecision at 10% membersat 1%at 0.1%
1%RATIO1.01%0.2980.7650.2290.029
1%WBC1.00%0.4470.8320.3110.043
1%RATIO+DIANA1.00%0.6230.8730.3850.059
0.1%RATIO0.099%0.1020.9200.5100.094
0.1%WBC0.099%0.2400.9640.7090.195
0.1%RATIO+DIANA0.105%0.4540.9800.8130.301

On average the thresholds give the intended FPR, and RATIO+DIANA remains the best attack with them. When 1% of the candidates are members, a threshold for 0.1% FPR gives 81% precision with RATIO+DIANA, compared with 71% for WBC and 51% for RATIO. The revision adds this procedure and the full table as Appendix P (Table 28).

Reviewer_VerpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DIANA, a calibration for reference-based membership inference attacks (MIAs) on fine-tuned language models. The idea is to remove domain-adaptation effects from per-token log-probability gains by subtracting the target-weighted mean across the vocabulary and then down-weighting high-variance samples. The paper provides conditions under which the calibrated gain has zero conditional mean for non-members and positive mean for members, and demonstrates empirical improvements across multiple models and datasets, especially at very low false-positive rates.

Critical strengths
  1. The paper provides evaluations across several datasets and model families.

  2. The method substantially improves performance at low false-positive rates.

  3. The ablations isolate the effects of centering and variance weighting, while the sensitivity analyses examine robustness to τ\tau and limited-output (top-k) access.

Critical weaknesses
  1. The novelty appears limited. DIANA is closely related to the normalization used in Min-K%++, applying predictive-distribution centering and variance-based weighting to target-reference gains rather than target log-probabilities. The main changes are applying this idea to reference-based attacks on fine-tuned models, using gain rather than log-probability, and introducing the regularization term τ\tau.

  2. Because DIANA uses the target predictive distribution to estimate the baseline, memorization itself changes the quantity being subtracted. As memorization becomes stronger, the estimated baseline can absorb part of the membership signal, and the paper itself shows that the centered member gain eventually decreases toward zero.

  3. The theoretical guarantees rely on restrictive assumptions. The zero-mean result for non-members assumes that the model's non-member predictive distribution exactly matches the domain distribution, while the member analysis assumes a single-spike memorization model. It is unclear how well these assumptions reflect real fine-tuning and memorization behavior.

  4. The low-FPR improvements are not consistent across all settings. In Appendix Table 10, DIANA is sometimes worse than the original reference-based baselines.

  5. The related-work discussion misses several closely related membership-inference and calibration methods, including SPV-MIA [1], DF-MIA [2], ACMIA [3], and InfoRMIA [4].

[1] Fu et al., Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration (SPV-MIA), NeurIPS 2024.

[2] Huang et al., DF-MIA: A Distribution-Free Membership Inference Attack on Fine-Tuned Large Language Models, AAAI 2025.

[3] Zare Zade et al., Automatic Calibration for Membership Inference Attack on Large Language Models, ECAI 2025.

[4] Tao and Shokri, Information-Theoretic Membership Inference for Granular Quantification of Memorization (InfoRMIA), ICLR 2026.

Questions for authors
  1. What is the main technical novelty of DIANA relative to Min-K%++ beyond applying similar normalization to target-reference gains?

  2. How realistic are the assumptions that the non-member predictive distribution matches the domain distribution and that memorization follows a single-spike model?

6 comments

We thank the reviewer for their feedback and constructive suggestions, and we appreciate that the reviewer recognized DIANA's substantial improvements at low false-positive rates. The revised paper has been uploaded, with changes highlighted in blue.


W1 & Q1: Novelty Compared with Min-K%++

W1: "The novelty appears limited. DIANA is closely related to the normalization used in Min-K%++, applying predictive-distribution centering and variance-based weighting to target-reference gains rather than target log-probabilities. The main changes are applying this idea to reference-based attacks on fine-tuned models, using gain rather than log-probability, and introducing the regularization term (τ\tau)."

Q1: "What is the main technical novelty of DIANA relative to Min-K%++ beyond applying similar normalization to target-reference gains?"

We respectfully disagree that DIANA applies the Min-K%++ normalization to target-reference gains. The two methods have very different motivations, and following the reasoning of Min-K%++ on normalization does not lead to DIANA.

Motivations & Formulations

Min-K%++ starts from members: maximum-likelihood training pushes training text toward a local maximum of the model's likelihood, so a member token should be a mode of the model's next-token prediction, or at least more probable than most alternatives. Its score therefore compares the observed token with the other tokens the same model could have predicted. DIANA starts from non-members: fine-tuning raises the probability of all in-domain text, so a non-member token also gains on average. DIANA estimates the gain a token would receive from domain adaptation alone, as if the sequence had not been used in training, and scores the token by how far its observed gain is above this estimate.

Written as formulas, the two token scores differ in what is subtracted and in what is divided by. Here xjx_j is the observed token, and WW is a token drawn from the model's next-token prediction, so each expectation and variance is taken over all possible next tokens.

Min-K%++:

log⁡pj(xj)−EW∼pj[log⁡pj(W)]VarW∼pj(log⁡pj(W))\frac{\log p_j(x_j)-\mathbb{E}_ {W\sim p_j}[\log p_j(W)]}{\sqrt{\mathrm{Var}_ {W\sim p_j}(\log p_j(W))}}

DIANA:

Gj(xj)−EW∼pjT[Gj(W)]τ+VarW∼pjT(Gj(W)),Gj(v)=log⁡pjT(v)−log⁡pjR(v)\frac{G_j(x_j)-\mathbb{E}_ {W\sim p^T_j}[G_j(W)]}{\tau+\mathrm{Var}_ {W\sim p^T_j}(G_j(W))},\qquad G_j(v)=\log p^T_j(v)-\log p^R_j(v)

  • The first difference is what is subtracted. Min-K%++ compares the observed token with the alternatives of the same model, so it subtracts a model's average log-probability under that model's own prediction. For the two models in the gain, this reasoning subtracts EpjT[log⁡pjT]−EpjR[log⁡pjR]\mathbb{E}_ {p^T_j}[\log p^T_j]-\mathbb{E}_ {p^R_j}[\log p^R_j]. DIANA subtracts EpjT[log⁡pjT]−EpjT[log⁡pjR]=KL(pjT∥pjR)\mathbb{E}_ {p^T_j}[\log p^T_j]-\mathbb{E}_ {p^T_j}[\log p^R_j]=\mathrm{KL}(p^T_j \Vert p^R_j), with both averages under the target's prediction. This is the gain that a non-member token receives from domain adaptation on average, so after it is subtracted a non-member has zero mean (Proposition 1): on the six corpora, the average non-member gain falls from 0.31–0.47 to 0.025–0.034 nats (Table 23). To answer Q1, this non-member baseline is DIANA's main technical novelty: shadow-model attacks obtain it by training models on auxiliary in-domain data, and DIANA computes it from the two models the attacker already has.

  • The second difference is what is divided by. Min-K%++ divides by the standard deviation to put every position on the same scale. DIANA divides by the variance plus τ\tau, so a position counts less when a non-member's gain fluctuates more there. The variance measures exactly this fluctuation: under Proposition 1, it is the variance of a non-member's centered gain at that position, and the measured values match it with R2R^2 of 0.993–0.999 (Appendix M.2, Table 25). τ\tau keeps the weight bounded, and Theorem 1 shows that it is needed to keep the member mean positive.

Empirical Results

The two also perform very differently on fine-tuned models: on the six corpora, Min-K%++ reaches a mean TPR of 0.050 at 0.1% FPR, and RATIO+DIANA reaches 0.474 (Table 1 and Appendix E).

Following the motivation of Min-K%++ does not lead to an effective design for fine-tuned models. We apply each step to RATIO and report the mean TPR at 0.1% FPR on the six corpora. For the subtraction, in the evaluation of Table 19 (Appendix H), the subtraction that follows Min-K%++ (the row "add entropy change") lowers the TPR from 0.112 to 0.074, while DIANA's subtraction raises it to 0.429. For the division, in Table 3, dividing DIANA's centered gain by the standard deviation as Min-K%++ does lowers the TPR from 0.418 to 0.373 (an additional run on the same splits), while DIANA's division raises it to 0.474; without τ\tau, it falls to 0.002.

W2: Memorization Contaminates the Background Estimate

"Because DIANA uses the target predictive distribution to estimate the baseline, memorization itself changes the quantity being subtracted. As memorization becomes stronger, the estimated baseline can absorb part of the membership signal, and the paper itself shows that the centered member gain eventually decreases toward zero."

We thank the reviewer for raising this point. The centered member gain does decrease after many epochs, but it does not approach zero in practice, and its decrease does not reduce detection. We will clarify both points in the revision.

First, the centered member gain stays well above zero. It would reach zero only if the model predicted every member token with probability one. In the 20-epoch runs of Appendix N, the gain starts to fall only after epoch 6, when the model already overfits: its held-out perplexity has been rising since epoch 2 (table below). After 20 epochs, the model is far worse than the pre-trained model on held-out text (perplexity 16.85 against 4.93), yet the centered gain is still 0.089 nats per token (Table 26), about three times that of non-members in typical fine-tuning (0.025–0.035 nats, Table 24).

Second, the decrease does not reduce detection, because the spread of the gain falls as well. From epoch 6 to epoch 20, the member centered gain falls from 0.238 to 0.089 nats per token, and its standard deviation across member tokens falls from 0.755 to 0.298. Members in fact become easier to separate: the gap between the average RATIO+DIANA scores of members and non-members grows from 5.5 to 7.2 standard deviations. The table shows both: after epoch 6 the centered gain falls, while every attack still detects every member at 1% FPR.

Epoch12461020
Held-out perplexity (pre-trained: 4.93)3.263.193.343.826.2316.85
Centered member gain (Table 26)0.1210.1790.2290.2380.1860.089
RATIO, TPR @ 1% FPR0.1150.7831.0001.0001.0001.000
WBC, TPR @ 1% FPR0.2590.8781.0001.0001.0001.000
RATIO+DIANA, TPR @ 1% FPR0.6660.9851.0001.0001.0001.000

Held-out perplexity is measured on the 800 non-members of each checkpoint and averaged over the three seeds.

W3 & Q2: Theoretical Assumptions

W3: "The theoretical guarantees rely on restrictive assumptions. The zero-mean result for non-members assumes that the model's non-member predictive distribution exactly matches the domain distribution, while the member analysis assumes a single-spike memorization model. It is unclear how well these assumptions reflect real fine-tuning and memorization behavior."

Q2: "How realistic are the assumptions that the non-member predictive distribution matches the domain distribution and that memorization follows a single-spike model?"

In the paper we test what each assumption predicts, and the results show that both describe real fine-tuning well. The assumptions do not hold with 100% accuracy, but they guide the design of DIANA and help explain why it works.

If the model's non-member predictive distribution exactly matches the domain distribution, then the baseline (the average gain under the fine-tuned model's own prediction) equals the gain that non-members actually receive. We test this on documents the model was not trained on. Their gain comes only from domain adaptation, so the baseline should remove all of it. On all six corpora, the baseline is 91–93% as large as the gain that non-members receive (Table 23). On Stanford, for example, subtracting it lowers the average non-member gain from 0.429 to 0.029 nats. We will further clarify this point in the revised paper.

The single-spike model says that training on a sequence raises the probability of the token that actually comes next and leaves the rest of the prediction otherwise unchanged. This is for sure a simplification, and a real LLM does not change its prediction in exactly this way; however, the single-spike model leads to two phenomena that we confirm in experiments. First, under the single-spike model, a member token should gain more than the baseline, because its probability was raised more than that of the other tokens. Members should therefore keep a positive gain after the baseline is subtracted, while non-members keep almost none. We measure 0.122–0.152 nats for members on the six corpora, against 0.025–0.035 for non-members (Table 24). This is one key insight motivating DIANA. Second, under the single-spike model, the gain left after subtraction is concave in the memorization strength (Proposition 2): it should first grow and then shrink as memorization becomes stronger. We trained for 20 epochs to test this. The gain left after subtraction grows until epoch 6 and then shrinks, and the curve the single-spike model predicts fits the measured one with R2=0.952R^2=0.952 (Section 3.3, Appendix N). These two properties are what our analysis of members relies on, so the single-spike model is a sound basis for it.

Comments on the paper

1 comment
Area_Chair_wA9gOpenReview ↗

Final feedback

Dear Reviewers,

As the discussion is close to the end, we encourage you to participate in the discussion to see whether your questions and concerns have been answered by the authors.

Thank you!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.