acceptodds
Under review as a conference paper at ICLR 2027

On Reliability of Membership Inference Vulnerability Evaluation

Abstract

Membership inference attacks (MIAs) are popular methods for empirically assessing the leakage of sensitive information in the training data through models or statistics learned from the data. The MI vulnerability is often evaluated through a binary classifier that tries to predict whether a particular sample was in the training data. In order to evaluate the effectiveness of MIAs multiple *shadow models* are trained using random partitions of a larger data set. After training the shadow models the MI vulnerability can be evaluated for all the samples for which we obtained shadow models. In order to evaluate the MI vulnerability reliably one needs a lot of shadow models which can be computationally infeasible. Therefore instead of reporting the actual sample level vulnerabilities aggregates over multiple samples are often reported in practice. We demonstrate two key weaknesses in typical MIA evaluation pipeline. First, we show that sampling the shadow data sets from a fixed superset leads to finite sample bias inflating the vulnerability estimates. Second, we show that evaluating the true positive rate (TPR) by concatenating MIA scores across multiple individuals, commonly used in the very low false positive rate (FPR) regime, is not calibrated across the per-sample FPRs. For both weaknesses we propose fixes that in the most simple approximate form do not incur any additional computation cost. We show that with additional computation one can further improve the reliability of the vulnerability estimation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
membership inference attacks, privacy, finite population bias, statistical calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sWBVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reliability of membership inference attack (MIA) vulnerability evaluation for machine learning models. The authors identify two important issues in common shadow-model-based MIA evaluation pipelines.

First, they show that sampling shadow datasets from a fixed finite superset introduces finite population bias, leading to underestimated in/out distribution variances and inflated vulnerability estimates. They propose finite population correction (FPC) methods, including a simple correction and a variance decomposition-based correction.

Second, they analyze the commonly used concatenation-based aggregation of MIA scores across multiple samples and models. They show that this approach only calibrates average FPR rather than per-sample FPR, potentially producing misleading vulnerability estimates. They propose a post-processing method to normalize scores and improve calibration.

The proposed methods are evaluated on Adult, CIFAR10, and PatchCamelyon datasets with TabPFN, ViT-B-16, and ResNet-based models. The experiments demonstrate that FPC reduces finite population bias and that post-processing improves the accuracy of aggregate vulnerability estimation.*

Critical strengths
  1. The paper identifies important statistical biases in existing MIA evaluation protocols. The finding that commonly used evaluation procedures may systematically overestimate privacy leakage is significant because it affects the reliability of privacy benchmarking and model comparison.

  2. The proposed corrections are practical and easy to integrate into existing shadow-model-based evaluation pipelines, improving the usability and potential impact of the work.

Critical weaknesses
  1. The empirical evaluation is somewhat limited compared with the broad claims about MIA evaluation reliability. More extensive validation across different MIA attacks, model architectures, and large-scale models would strengthen the conclusions.

  2. Some proposed corrections rely on assumptions such as Gaussian score distributions and variance decomposition approximations. The robustness of these assumptions in more diverse settings is not fully explored.

Questions for authors
  1. How robust are the proposed FPC corrections when the dataset distribution deviates from the assumed i.i.d. finite population sampling setting?

  2. Does the post-processing calibration method remain effective for MIA methods whose score distributions are not approximately Gaussian or do not rely on LiRA-style modeling?

  3. Can the authors evaluate whether applying these corrections changes the relative ranking of different models or privacy defenses compared with standard MIA evaluation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer sWBV

We thank the reviewer for their feedback. We address the concerns you have raised as follows:

Critical Weakness #1

The empirical evaluation is somewhat limited compared with the broad claims about MIA evaluation reliability. More extensive validation across different MIA attacks, model architectures, and large-scale models would strengthen the conclusions.

We have now added experiments for,

  • Different MIAs: We have added experiments with the direct variant of RMIA (Zarifzadeh et al., 2024) Appendix A.3. As with LiRA, the uncorrected mean per-sample AUC increasingly overestimates the population AUC as the sampling ratio (N/N+N/N_+) increases. FPC helps regulate the AUC bringing it closer to the population estimate.
  • Model architectures: We have added experiments with a MLP trained from scratch on the Purchase100 dataset in Appendix A.7. As in the fine-tuning and ICL experiments, FPC brings mean per-sample AUC closer to the population AUC.

Q1

How robust are the proposed FPC corrections when the dataset distribution deviates from the assumed i.i.d. finite population sampling setting?

We want to emphasize that the FPC is only needed when the shadow datasets are sampled from a finite population. Therefore, if the shadow data can be obtained from some other way, e.g. by collecting fresh samples from the population, the correction will not apply or might not be needed at all. However, since the population distribution is not available in realistic settings, the MIA methods are typically evaluated and compared in the finite frame setting we studied in our paper.


Q2 (and Critical Weakness #2)

Does the post-processing calibration method remain effective for MIA methods whose score distributions are not approximately Gaussian or do not rely on LiRA-style modeling?

We computed several statistics for the in/out distributions across our experiments to measure how close to a Gaussian the in/out distributions are. In Table A1 of the revised appendix, we show that in fact, the distributions can deviate somewhat heavily from the Gaussian given the non-zero kurtosis and skewness. Despite this, the post-processing still yields good FPR calibration as shown in our earlier results in the main paper. While the post-processing theory is derived for Gaussian distribution specifically, it is easy to see that if the out samples come from a location-scale family of distributions the post-processing transforms the out distributions into a shared distribution and the FPR calibration holds. We have written this down in Appendix Section A.6.


Q3

Can the authors evaluate whether applying these corrections changes the relative ranking of different models or privacy defenses compared with standard MIA evaluation?

This is an interesting question, and definitely highly relevant for the practical implications of our FPC suggestion. In section A.2 of the revised Appendix, we show when the order of the analytical AUC between two algorithms can change between the biased and the unbiased estimates. The bottom line is that the flip can happen depending on the relative ordering between the population and remaining variance sources in the algorithms. As an example, if algorithm Aj\mathcal{A}_j has a large population variance, i.e. it is heavily data dependent, the finite population bias can significantly pull this source of variance down. If another algorithm Ak\mathcal{A}_k is less sensitive to the data but has more residual variance than Aj\mathcal{A}_j, then the biased total variance of Aj\mathcal{A}_j might be larger than Ak\mathcal{A}_k but after FPC the order flips. Assuming the in/out mean gap of both Aj\mathcal{A}_j and Ak\mathcal{A}_k is similar (the algorithms converge to a similar solution on average) then the Aj\mathcal{A}_j looks less vulnerable in the biased setting where as Ak\mathcal{A}_k would in reality be less vulnerable.

To investigate whether FPC affects ranking of privacy defences, we attacked models trained with RelaxLoss (Chen et al. 2022) and SELENA (Tang et al. 2021). In Appendix A.4, we compare the uncorrected and FPC-calibrated MIA AUCs for models trained with RelaxLoss and SELENA. FPC changes the ordering within RelaxLoss. At several sampling ratios, τ=0.4\tau=0.4 yields higher uncorrected AUC but lower FPC-calibrated AUC than both τ=0.6\tau = 0.6 and τ=0.65\tau=0.65. This reversal is contrary to expectation because a higher τ\tau makes RelaxLoss counteract loss minimization at a higher training loss, which might be expected to provide stronger protection against MIAs. It also limits differences in AUC across different RelaxLoss settings.


References

  • Zarifzadeh et al. Low-Cost High-Power Membership Inference Attacks. ICML 2024.
  • Chen et al. RelaxLoss: Defending Membership Inference Attacks without Losing Utility. ICLR 2022.
  • Tang et al. Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture. USENIX Security 2022.
Reviewer_cDroOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors identify two issues affecting membership inference attack evaluation with limited shadow models and candidate samples and propose corresponding solutions.

  1. Finite Population Bias. Repeatedly constructing shadow model training sets from a fixed data pool may underestimate the true variability of model outputs, thereby overestimating membership inference risks.
  2. FPR Calibration Issues in Cross-Sample Aggregation. In concatenated membership inference attacks, aggregating attack scores across different samples to compute an overall ROC curve can control the global FPR, but results in individual samples being evaluated at different FPR.
Critical strengths

The authors identify two findings:

  1. The authors demonstrate that the construction of training datasets for shadow models can affect the estimated privacy risks associated with membership inference attacks.
  2. The authors highlight that global ROC curves obtained through pooled evaluation do not necessarily ensure properly calibrated FPRs at the individual-sample level, emphasizing the need for per-sample FPR calibration.
Critical weaknesses
  1. The overestimation of membership inference risks arising from the construction of shadow-model training datasets have limited implications for practical model security evaluations.
  2. Even after correction, the average TPR fails to fully capture the privacy risks faced by the most vulnerable samples, and the proposed correction offers limited improvements in assessing high-risk samples.
Questions for authors

How effective is the proposed Decomposed FPC correction method in evaluating the membership inference vulnerability of models trained with DP-SGD?

5 comments
AuthorsOpenReview ↗

Response to Reviewer cDro

We thank the reviewer for their feedback. We address the concerns you have raised as follows:

Critical Weakness #1

The overestimation of membership inference risks arising from the construction of shadow-model training datasets have limited implications for practical model security evaluations.

We want to emphasize that the overestimation of the risk due to finite population bias can make the comparison of samples or comparisons of defense mechanisms unreliable. See our revised Appendix A.4 where we studied the FPC effect on two privacy defenses, SELENA and RelaxLoss and demonstrated how the difference between RelaxLoss parameter choices has a more limited effect on the vulnerability after FPC. Thus, while a practical attack might not be directly affected, e.g. the calibration or choice of attack algorithm might be different when the calibration/choice is based on the biased evaluation.


Critical Weakness #2

Even after correction, the average TPR fails to fully capture the privacy risks faced by the most vulnerable samples, and the proposed correction offers limited improvements in assessing high-risk samples.

We agree that the average per-sample TPR does not capture the privacy risks faceted by the most vulnerable samples. However, our proposed finite population correction can still affect the absolute per-sample vulnerability evaluations even for the most vulnerable samples as shown in Figure 2, thus giving a more accurate understanding of the practical privacy leakage. Whether there are scenarios where the corrections change the absolute order of the vulnerability among the samples is an interesting question, and we explored this further in the revised Appendix A.6. We wrote the analysis in terms of comparing the vulnerability of two algorithms, but similar principles would likely extend to sample level evaluation as well. In summary, we showed that for two algorithms differing in their variance decomposition, the finite population bias can flip the vulnerability order compared to the non-biased case.


Q1

How effective is the proposed Decomposed FPC correction method in evaluating the membership inference vulnerability of models trained with DP-SGD?

This is a good question. If the reviewer is asking about the applicability of our proposed corrections w.r.t. auditing DP, then we want to point out that in typical DP auditing settings the neighboring datasets are fixed and known by the adversary. Therefore the FPC does not apply, as there is no “shadow data” in the evaluation. However, empirically assessing the vulnerability of an entire model release pipeline, such as in the typical MIA game, the D∖xD \setminus {x} is not known and the DD comprises real individuals instead of crafted canaries. Applying this vulnerability assessment on DP-SGD trained models, the FPC is affecting the evaluation due to the noise introduced from the shadow data. However, the DP introduced noise will affect the total variance of the in/out distributions and depending on the privacy level might lead to noise levels where the dataset sampling noise is drowned by the DP noise.

Reviewer_cDroOpenReview ↗

Response to Authors

Thank you very much for your response. Although the authors have addressed some of my concerns, the central issue remains: the significance of correcting an overestimated membership inference threat to a comparable level is still limited. The authors argue that this correction enables a more accurate comparison of different defense methods. However, practical security evaluations typically focus on the privacy protection a defense can provide under worst-case conditions, with the goal of strengthening defenses and reducing the risk of membership inference attacks as much as possible.

Nevertheless, I recognize the theoretical value of the authors’ work and will therefore maintain my current rating.

AuthorsOpenReview ↗

Follow-up Response to Reviewer cDro

However, practical security evaluations typically focus on the privacy protection a defense can provide under worst-case conditions, with the goal of strengthening defenses and reducing the risk of membership inference attacks as much as possible.

We respectfully disagree with the goal of reducing the privacy risk as much as possible. Real applications always have a privacy-utility tradeoff, and accurate estimates of the risk are vital for making an informed decision that balances this tradeoff. As a practical example, the US Census chose a level of DP that provides essentially no worst-case guarantee in order to have optimal privacy-utility tradeoff. Another important area of application is health, where too much privacy can mean worse health outcomes and even lost lives.

Reviewer_hr4EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies two issues in a typical MIA setup, leading to biased or over-estimated results. First, a typical real-world MIA setup trains shadow models on overlapping subsets of a finite dataset, while in theory it's assumed that those datasets are drawn independently from a certain distribution. Authors show that repeated sampling leads to lower variance in shadow model losses and inflates resulting MIA efficacy. Second, authors show that measuring TPR@low FPR require calibration to per-sample FPR rates, as the MIA risk is not uniform across the dataset.

Critical strengths

The repeated sampling result looks strong to me. Authors identified, quantified and corrected finite population bias, which is indeed a feature of many real-world MIA setups. Relevant theory seems solid, and empirical results (Fig 2) are both intuitive and interesting.

One caveat here is I haven't fully understand the variance decomposition, the empirical results (Fig 3) suggest that the proposed model might be too simple or doesn't fully explain the source of variance.

Critical weaknesses

Per-sample FPR calibration

On the per-sample FPR calibration I am not sure I agree with the proposed setup. Authors focus on TPR@ low FPR as the main MIA efficacy metric, but compute TPR@FPR for individual samples across many target models. The way I understand point of Carlini et al. (2022) on the importance of TPR@low FPR, it matters when TPR@low FPR helps us identify the most vulnerable points. The reason we care about TPR@low FPR is because an attacker can select only those samples it's confident in -- even if the attacker is bad on average, the risk fpr a minority of samples is significant. When we aggregate by sample and consider fixed sample and varying training seed/dataset, then AUC seems like the more appropriate metric to evaluate average risk faced by a given sample.

Dataset access

The risk overestimation from the finite population bias only applies to adversaries with access to the underlying distribution, but not the dataset of N+N_\text{+} samples. An attacker with (partial) access to the training data would have better performance. MIAs usually assume the strongest of the "realistic" attackers, so an overestimation of risk under some particular assumptions does not seem like a big issue to me.

Questions for authors
  • In what situations we want a good estimate of per-sample TPR@low FPR rather than AUC? Am I missing an important use case?
  • Do you imagine these estimation corrections be applied to MIA as DP auditing setups or when MIAs are used to evaluate empirical privacy risk? I'm a bit confused about the positioning here
3 comments
AuthorsOpenReview ↗

Response to Reviewer hr4E

We thank the reviewer for their feedback. We address the concerns you have raised as follows:

Q1 (and Critical Weakness #1)

In what situations we want a good estimate of per-sample TPR@low FPR rather than AUC? Am I missing an important use case?

In our work, we have used both the per-sample AUC as well as the TPR @ FPR as a measure for privacy. The reviewer is correct that the TPR @ (low) FPR was originally proposed by Carlini et al. as a metric to capture the “average” confident privacy leakage across individuals, and not as a sample-level metric. However, in the per-sample level, the TPR @ FPR gives complementary information about the ROC curve on top of the AUC which simply aggregates the whole curve into a single scalar. Since we cannot conveniently report the entire ROC curve we focused on a fixed FPR level, and decided to focus on the low FPR since that is the region which is not typically visible from the AUC. In summary, we are not arguing that TPR@low FPR is a better measure than AUC in any particular setting, but rather gives a complementary calibration point directly from the ROC curve. Finally, we want to highlight that we are not the first to suggest focusing on the per-sample TPR @ low FPR. For example Aerni et al. 2024 as well as Tobaben et al. 2025 used the per-sample TPR @ low FPR as a metric for studying the vulnerability.


Critical Weakness #2

The risk overestimation from the finite population bias only applies to adversaries with access to the underlying distribution, but not the dataset of samples. An attacker with (partial) access to the training data would have better performance. MIAs usually assume the strongest of the "realistic" attackers, so an overestimation of risk under some particular assumptions does not seem like a big issue to me

We would like to clarify the settings in which the FPC corrections apply. In an MIA, we are interested in whether a particular sample xtargetx_{\mathrm{target}} was used to train a model MtargetM_{\mathrm{target}}, i.e., whether xtarget∈Dtrainx_{\mathrm{target}} \in D_{\mathrm{train}}, where DtrainD_{\mathrm{train}} is the model's training set. The adversary does not know DtrainD_{\mathrm{train}} but aims to simulate the training process by repeatedly training shadow models on data drawn from the same distribution as the target training data.

In practice MIA algorithms are often evaluated on a fixed set of samples, which is (1) randomly partitioned into in/out datasets MM times, (2) models are trained on in partition and finally (3) the MIA vulnerability is evaluated using the trained models as target and shadow models. This approach makes sure that all the datasets come from the same distribution defined by the fixed (super)set of samples. What we show in our paper is that this approach, the practical MIA evaluation, suffers from the underestimated shadow model variance because the shadow datasets are drawn from the fixed set of samples.

While we agree that the partial knowledge of DtrainD_\mathrm{train} would likely improve the MIA performance, we want to highlight that this is not the typical setting where MIA algorithms are evaluated. Furthermore, even with the partial access to the training data, the unknown part of the training data still needs to be sampled from some distribution to learn the shadow models. And now if this is done with the finite set of samples, we again encounter the finite sample bias.


Q2

Do you imagine these estimation corrections be applied to MIA as DP auditing setups or when MIAs are used to evaluate empirical privacy risk? I'm a bit confused about the positioning here

In typical DP auditing settings the neighboring datasets DD and D’D’ are fixed and known by the adversary, and the goal is to understand whether the stochasticity arising from the algorithm satisfies the promised DP guarantees. In this setting there is no stochasticity arising from the data distribution and therefore the FPC is not needed. However, when empirically assessing vulnerability of the entire model release pipeline, such as in typical MIA game, the training set is typically unknown. When this stochasticity is simulated from a fixed pool of samples, the FPC becomes relevant.. Since this sampling design is common in shadow-model-based MIA evaluations, FPC is particularly relevant to those evaluations. We added some clarifications on the differences between auditing and MIA in the revised introduction.

I thank the authors for the response

In summary, we are not arguing that TPR@low FPR is a better measure than AUC in any particular setting, but rather gives a complementary calibration point directly from the ROC curve.

While it is a valid point, I would argue about the importance of that particular metric - and by extension methods that improve its approximation. As far as I understand, method in Section 4 is specific to correcting per-sample TPR@low FPR, which I do not currently see as particularly interesting.

While we agree that the partial knowledge of DtrainD_\text{train} would likely improve the MIA performance, we want to highlight that this is not the typical setting where MIA algorithms are evaluated.

My point was that MIAs, at least in non-DP-audit scenarios, are usually treated as an upper bound to other types of attacks (e.g. reconstruction or extraction) -- in practice we care about extraction more often than we do about MIA itself. In this spirit, slight overestimation of the risk is not a big issue, an underestimation would've been a more severe issue

My point was that MIAs, at least in non-DP-audit scenarios, are usually treated as an upper bound to other types of attacks (e.g. reconstruction or extraction) -- in practice we care about extraction more often than we do about MIA itself. In this spirit, slight overestimation of the risk is not a big issue, an underestimation would've been a more severe issue

We agree with the reviewer that MIA is typically only used as a proxy for types of attacks. However, we argue that the overestimation of vulnerability is harmful because of two reasons.

  1. Consider a data curator who chooses a defence method based on empirical work comparing defences against MIA in the biased evaluation setup. The curator is choosing the method based on some privacy/utility trade-off they are willing to accept. However, the curator could have chosen a method with better utility in absence of the bias with an equal privacy cost. In use cases such as healthcare, even small improvements in utility can lead to important differences in outcomes.
  2. The development of future attacks and defences. With the bias present, we cannot ever tell if the improvements proposed by a new method are due to actual algorithmic improvements the method actually makes or if they come from the biased variance estimation. Therefore our discovery is extremely important for any future work using the finite population based MIA evaluation in method development.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.