- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the reliability of membership inference attack (MIA) vulnerability evaluation for machine learning models. The authors identify two important issues in common shadow-model-based MIA evaluation pipelines.
First, they show that sampling shadow datasets from a fixed finite superset introduces finite population bias, leading to underestimated in/out distribution variances and inflated vulnerability estimates. They propose finite population correction (FPC) methods, including a simple correction and a variance decomposition-based correction.
Second, they analyze the commonly used concatenation-based aggregation of MIA scores across multiple samples and models. They show that this approach only calibrates average FPR rather than per-sample FPR, potentially producing misleading vulnerability estimates. They propose a post-processing method to normalize scores and improve calibration.
The proposed methods are evaluated on Adult, CIFAR10, and PatchCamelyon datasets with TabPFN, ViT-B-16, and ResNet-based models. The experiments demonstrate that FPC reduces finite population bias and that post-processing improves the accuracy of aggregate vulnerability estimation.*
Critical strengths
-
The paper identifies important statistical biases in existing MIA evaluation protocols. The finding that commonly used evaluation procedures may systematically overestimate privacy leakage is significant because it affects the reliability of privacy benchmarking and model comparison.
-
The proposed corrections are practical and easy to integrate into existing shadow-model-based evaluation pipelines, improving the usability and potential impact of the work.
Critical weaknesses
-
The empirical evaluation is somewhat limited compared with the broad claims about MIA evaluation reliability. More extensive validation across different MIA attacks, model architectures, and large-scale models would strengthen the conclusions.
-
Some proposed corrections rely on assumptions such as Gaussian score distributions and variance decomposition approximations. The robustness of these assumptions in more diverse settings is not fully explored.
Questions for authors
-
How robust are the proposed FPC corrections when the dataset distribution deviates from the assumed i.i.d. finite population sampling setting?
-
Does the post-processing calibration method remain effective for MIA methods whose score distributions are not approximately Gaussian or do not rely on LiRA-style modeling?
-
Can the authors evaluate whether applying these corrections changes the relative ranking of different models or privacy defenses compared with standard MIA evaluation?
Response to Reviewer sWBV
We thank the reviewer for their feedback. We address the concerns you have raised as follows:
Critical Weakness #1
We have now added experiments for,
Q1
We want to emphasize that the FPC is only needed when the shadow datasets are sampled from a finite population. Therefore, if the shadow data can be obtained from some other way, e.g. by collecting fresh samples from the population, the correction will not apply or might not be needed at all. However, since the population distribution is not available in realistic settings, the MIA methods are typically evaluated and compared in the finite frame setting we studied in our paper.
Q2 (and Critical Weakness #2)
We computed several statistics for the in/out distributions across our experiments to measure how close to a Gaussian the in/out distributions are. In Table A1 of the revised appendix, we show that in fact, the distributions can deviate somewhat heavily from the Gaussian given the non-zero kurtosis and skewness. Despite this, the post-processing still yields good FPR calibration as shown in our earlier results in the main paper. While the post-processing theory is derived for Gaussian distribution specifically, it is easy to see that if the out samples come from a location-scale family of distributions the post-processing transforms the out distributions into a shared distribution and the FPR calibration holds. We have written this down in Appendix Section A.6.
Q3
This is an interesting question, and definitely highly relevant for the practical implications of our FPC suggestion. In section A.2 of the revised Appendix, we show when the order of the analytical AUC between two algorithms can change between the biased and the unbiased estimates. The bottom line is that the flip can happen depending on the relative ordering between the population and remaining variance sources in the algorithms. As an example, if algorithm Aj has a large population variance, i.e. it is heavily data dependent, the finite population bias can significantly pull this source of variance down. If another algorithm Ak is less sensitive to the data but has more residual variance than Aj, then the biased total variance of Aj might be larger than Ak but after FPC the order flips. Assuming the in/out mean gap of both Aj and Ak is similar (the algorithms converge to a similar solution on average) then the Aj looks less vulnerable in the biased setting where as Ak would in reality be less vulnerable.
To investigate whether FPC affects ranking of privacy defences, we attacked models trained with RelaxLoss (Chen et al. 2022) and SELENA (Tang et al. 2021). In Appendix A.4, we compare the uncorrected and FPC-calibrated MIA AUCs for models trained with RelaxLoss and SELENA. FPC changes the ordering within RelaxLoss. At several sampling ratios, τ=0.4 yields higher uncorrected AUC but lower FPC-calibrated AUC than both τ=0.6 and τ=0.65. This reversal is contrary to expectation because a higher τ makes RelaxLoss counteract loss minimization at a higher training loss, which might be expected to provide stronger protection against MIAs. It also limits differences in AUC across different RelaxLoss settings.
References