- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper tackles the challenge of running membership inference attacks (MIAs) in an observational setting, where one only has black box access to the model being attacked, cannot retrain the model, and cannot intervene in the training process. MIAs require both training and non-training data to run, and a key challenge with running MIAs in observational settings is to obtain non-training members from the same distribution as the training members. A common issue e.g. with MIAs on LLMs was that the distribution shift between training and non-training members was unaccounted for, which inflated attack success scores.
To tackle the "inflation" caused by distribution shift, the paper proposes correcting the out-of-distribution (OOD) non-member attack scores to more closely match the in-distribution (ID) non-member scores. To achieve the correction, the member in-distribution scores are used as reference. A key assumption in the paper is that one can learn a representation of the data (both IID and OOD data) such that, conditioning on the representation, the score distribtution is identical among OOD and IID non-member samples. With this assumption, the authors propose learning a relationship between scores and representation , via either linear regression or quantile regression. Then, samples can be sampled from the represantion of member samples, and used to transport the non-member scores via the learned relationship between and scores.
Because the conditional independence assumption need not hold, the authors then propose a conservative correction on the scores. Their experimental results show that the final MIA success after the correction is closely beneath the "ground truth" MIA success with access to IID non-members.
Critical strengths
- The idea of correcting for the distribution shift by transporting scores from the out-of-distribution population to in-distribution population is appealing. It seems an easier feat than approaches which try to e.g., generate synthetic non-member samples.
- The experimental results clearly show the risk of MIA inflation due to distribution shift. The authors compare to relevant prior work. Results show the validity of the method in tackling the inflation and achieving conservative differential privacy lower bounds that are close to the ground truth.
Critical weaknesses
-
The key assumption/observation in the paper is the conditional independence of scores under a representation . However, there is not much intuition provided for why this should be the case. This requires more explanation and motivation since the whole methodology is based on this assumption. I also did not come away with a clear sense of why we should expect Assumptions 2-4 hold, and it was not clear if the correction methodology and its assumption are standard in the statistics literature. My main concern is that I did not come away with a clear intuition for why the transport should even work, given that all we have is member scores based on a model that memorizes.
-
Since the assumptions of the paper are hard to verify in practice, the validity of the methodology rests on experimental results. Running the experiments on a third dataset would be useful.
-
The section "source-to-target" displacement introduced a lot of new terminology without much explanation, and I could not follow. Maybe you can add additional explanation in the Appendix?
Questions for authors
- Given that linear and quantile transport give different results, in practice would you recommend running both of them and taking the max of the lower bounds produced by each method?
Response to Reviewer vypi
Thank you for the informative review. We address your concerns below.
Weaknesses:
W1
We have added an explanation of the intuition behind this assumption on page 4, lines 213–215. Our intuition is that, for a fixed target model, the MIA score of a non-member is determined by the input and the model's behavior. Therefore, when conditioning on the full input (X), the non-member score mechanism is domain-invariant for a deterministic scoring rule. However, directly modeling MIA scores as a function of high-dimensional inputs is challenging. We therefore introduce a pretrained representation and assume that it preserves sufficient score-relevant information for the conditional score distribution to remain invariant across domains (Assumption 1).
As discussed in Section 4.2, conditional invariance may fail when the representation does not preserve sufficient information or when the distributions have limited support overlap. This motivates our shift-aware correction. Since the true transport-induced audit error is unobservable, we construct within-OOD calibration tasks along the estimated OOD-to-IID shift direction, where both predicted and observed non-member scores are available.
This approach is motivated by the established statistical framework of counterfactual distribution inference [1], which estimates a counterfactual outcome distribution by combining a conditional outcome model with a different covariate distribution. This is also why we name Section 4.1 counterfactual score transport. Therefore, the transport has been explored in literature, while our proposed calibration tasks are novel and constitute the key contribution of this work.
We acknowledge that our assumptions are strong and may be hard to verify directly in practice. Our theorem therefore provides sufficient conditions for conservativeness relative to the oracle IID audit, while our experiments provide empirical evidence of the method’s effectiveness. We have clarified this limitation in the discussion section, lines 535-536.
We clarify that we have access to both member and OOD non-member data, which can be used to learn the IID non-member scores. Specifically, we learn the relationship between representations and scores from OOD non-members, and use member representations to characterize the target input distribution. The intuition is that, if representations capture the factors governing non-member scores, the relationship learned from OOD non-members can transfer to the target population. This is basically counterfactual distribution inference.
Because this transfer may be imperfect, we construct controlled transport tasks within the observed OOD data, where true scores are available, and measure the resulting audit errors. These tasks motivate a displacement-dependent correction.
W2
To further evaluate the generalizability of our method, we have conducted additional experiments on Criteo, extending our evaluation beyond image classification and language modeling to click-through rate prediction. Details can be found in Appendix C.4. Specifically, we consider February 15, 2015 as the source day. We introduce natural temporal distribution shifts by using OOD non-members collected 7 and 23 days after the source day.
The new Figure 7 shows that corrected quantile transport provides tighter bounds than corrected residual transport. With a 7-day gap, the distribution shift is relatively mild and has a negligible effect on the audit results. With a 23-day gap, the shift becomes more pronounced, and the uncorrected audit substantially overestimates privacy leakage. In contrast, our corrected methods produce conservative and tight bounds. These additional results further demonstrate the effectiveness of our method in a different application domain and under naturally occurring temporal distribution shifts.
W3
We have expanded the explanation of the source-to-target displacement section in Appendix D.5 to clarify each component in detail.
Questions:
Yes, this is exactly what we proposed in the discussion section, lines 528-530.
[1] Chernozhukov et al. . Inference on counterfactual distributions. Econometrica, 2013.