- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work addresses the challenge of distinguishing memorization from domain adaptation in membership inference on fine-tuned language models. An increase in token probability relative to a pretrained reference may reflect either exposure to the candidate sequence or learning patterns shared across the domain. The proposed method, DIANA, computes the target-weighted mean and variance of target-to-reference log-probability gains at each prefix. It subtracts this mean from the observed token’s gain and divides the result by a regularized variance. The calibration requires no auxiliary data or additional training and can be incorporated into reference-based attacks such as DIFF, RATIO, WBC, and EZ-MIA. Under idealized domain-matching and single-spike memorization assumptions, the paper analyzes non-member null centering, positive member means, and signal behavior near memorization saturation. Experiments cover six primary corpora, multiple model families, additional natural-text corpora, component ablations, restricted probability access, LoRA, and DP-SGD. In the primary evaluation, RATIO+DIANA increases mean TPR at 0.1% FPR from 11.2% to 47.4%.
Critical strengths
-
Simple method with substantial gains and convincing ablations DIANA requires neither additional training nor auxiliary in-domain data. Across all six primary corpora, RATIO+DIANA achieves the highest point estimates on all reported primary metrics. Table 3 provides clear support for the design: centering alone increases mean TPR at 0.1% FPR from 0.112 to 0.418, while regularized variance weighting further improves it to 0.474. These results identify background subtraction as the main source of improvement, with variance weighting providing an additional benefit.
-
Broad evaluation with useful mechanism diagnostics Beyond performance comparisons, Section 3.3 and Appendices M/N examine background estimation, variance prediction, and member signals during training. The target-weighted mean accounts for 91.1%–93.3% of the mean observed non-member gain, while members retain higher mean signals after centering. Experiments also cover multiple model families, natural-text corpora, and training and access settings. Although these diagnostics do not fully validate the modeling assumptions, they provide useful evidence about the calibration mechanism and its limitations.
Critical weaknesses
-
The distinction between the theoretical guarantee and the implemented algorithm needs clarification. The proof of Proposition 1 uses the idealized mean prediction pⱼ⁻ and establishes null centering under pⱼ⁻ = pⱼᴰ, whereas the algorithm uses an individual trained model’s prediction pⱼᵀ. Agreement between the mean prediction and the domain distribution does not itself guarantee null centering for the actual model. Although the appendix acknowledges this distinction, the proposition’s direct reference to Equation (5) can suggest a guarantee for the implemented algorithm. Its scope should be stated explicitly and distinguished from empirical approximate calibration. This does not invalidate the experimental gains, but affects the precision of a central theoretical claim.
-
The scope of the low-FPR gains remains insufficiently explained. On CC-News, RATIO+DIANA achieves an AUC of 0.928 but a TPR at 0.1% FPR of only 0.001, compared with 0.004 for uncalibrated RATIO. These point estimates do not establish significant degradation, but the discrepancy between strong overall discrimination and weak low-FPR performance deserves further explanation, given the paper’s emphasis on this operating regime.
Questions for authors
-
Theoretical scope (regarding Weakness 1). Could the authors clarify the scope of Proposition 1 and discuss whether additional assumptions could extend the null-centering result from the idealized mean prediction to an individual trained model?
-
Low-FPR performance (regarding Weakness 2). Could the authors explain the gap between strong AUC and weak performance at 0.1% FPR on CC-News? A comparison of centering-only and full DIANA, or an analysis of member and non-member score distributions near the decision threshold, would help clarify this behavior.
3 comments
W2: "The scope of the low-FPR gains remains insufficiently explained. On CC-News, RATIO+DIANA achieves an AUC of 0.928 but a TPR at 0.1% FPR of only 0.001, compared with 0.004 for uncalibrated RATIO. These point estimates do not establish significant degradation, but the discrepancy between strong overall discrimination and weak low-FPR performance deserves further explanation, given the paper's emphasis on this operating regime."
Q2: "Could the authors explain the gap between strong AUC and weak performance at 0.1% FPR on CC-News? A comparison of centering-only and full DIANA, or an analysis of member and non-member score distributions near the decision threshold, would help clarify this behavior."
We thank the reviewer for pointing this out. We re-examined CC-News and found that the low TPR at 0.1% FPR is caused by duplicated entries in the data. We drew the CC-News split from the public corpus, whose entries turn out not to be unique, so after the random split many of its non-members are copies of members: 3.8% are exact copies of a member article, and 8.9% share more than half of their 8-word sequences with the member set. The model was trained on these texts, so every attack gives them high scores. They are too few to change the AUC much, but they are more than enough to set the threshold at 0.1% FPR, which is why the AUC is high while the TPR at 0.1% FPR is low. The table shows CC-News before and after removing them. We also checked the other corpora, and this issue appears only in CC-News. The revision removes these copies from the CC-News split (Tables 2 and 10; procedure in Appendix D).
| CC-News | RATIO, TPR @ 1% FPR | RATIO+DIANA, TPR @ 1% FPR | RATIO, TPR @ 0.1% FPR | RATIO+DIANA, TPR @ 0.1% FPR |
|---|---|---|---|---|
| Original split | 0.017 | 0.039 | 0.004 | 0.001 |
| Deduplicated split | 0.052 | 0.788 | 0.013 | 0.519 |
Both analyses the reviewer suggests point to this cause. Near the threshold, the highest-scoring non-members are these copies: all of the 100 non-members with the highest RATIO+DIANA scores (the top 1%) share more than half of their 8-word sequences with the member set, and the same holds for 76 of the top 100 under RATIO and all of them under WBC. Centering only and full DIANA also fail in the same way on the original split, with a TPR of 0.001 at 0.1% FPR for both, and both recover on the deduplicated split, to 0.224 with centering only and 0.519 with full DIANA.
Thank you for the response. The authors have addressed most of my concerns. However, I do not find the paper strong enough to warrant a score of 4, so I will maintain my current score.
We thank the reviewer for their feedback and constructive suggestions. We are encouraged that the reviewer found DIANA to be a "simple method with substantial gains and convincing ablations," and recognized our "broad evaluation with useful mechanism diagnostics." The revised paper has been uploaded, with changes highlighted in blue.
W1 & Q1: Theoretical Guarantee vs. Implemented Algorithm
Proposition 1 is stated for the mean prediction pj−. To extend it to the individual trained model that the algorithm uses, the assumption has to be made about that model: if the trained model's own prediction matches the domain distribution (pjT=pjD), the same proof gives a zero non-member mean for the calibrated gain of Equation (5) as implemented. Agreement between the mean prediction and the domain distribution is not enough for this, as the reviewer notes. The revision states this scope directly after Proposition 1 (Section 3.2) and gives the result for a trained model after the proof (Appendix A.1).
A trained model does not meet this assumption with 100% accuracy, so in practice the calibration is approximate. The assumption guides the design of DIANA and helps explain why it works, and we measure how closely it holds. If the model's non-member predictive distribution exactly matches the domain distribution, then the baseline (the average gain under the fine-tuned model's own prediction) equals the gain that non-members actually receive. We test this on documents the model was not trained on. Their gain comes only from domain adaptation, so the baseline should remove all of it. On all six corpora, the baseline is 91–93% as large as the gain that non-members receive (Table 23). On Stanford, for example, subtracting it lowers the average non-member gain from 0.429 to 0.029 nats. After the full calibration of Equation (5), the non-member mean stays within 0.033 of zero at every level of gain variance (Section 3.3, Figure 2b). The same assumption also gives the variance of a non-member's centered gain, and the measured values match it with R2 of 0.993–0.999 (Appendix M.2, Table 25).