- Rating
- 3(+1)
- Confidence
- 2(−2)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a hierarchical Bayesian framework for membership inference that separates variation across stochastic queries, reference models, and data points. Its record-level instantiation models diffusion log losses with a Gaussian–Gamma hierarchy and computes a joint posterior-predictive likelihood ratio using Gibbs sampling and analytic marginalization. The group-level attack combines empirical-Bayes variance estimation with cross-fitted target-model adaptation to infer identity membership from potentially unseen images. Experiments cover DDPM and flow-matching models across four image datasets. The paper reports improved low-FPR attack performance and robustness when reference pools mix the two model families.
Critical strengths
-
Separating query randomness from variation across trained models is a sensible modeling choice. The conditionally conjugate construction and posterior-predictive derivations provide a concrete implementation of this idea.
-
The reported record-level improvements are substantial in several settings. For example, Table 1 reports 64.72% TPR at 0.1% FPR for flow matching on CIFAR10, compared with 51.45% for BASE and 48.94% for LiRA.
-
Inferring identity membership without querying training images is practically relevant. Table 2 explicitly separates this setting from one allowing a training image, and reports meaningful improvements over the adapted baselines.
Critical weaknesses
-
The experiments do not isolate the benefit of the proposed uncertainty modeling. The biggest concern for me is that the paper motivates its contribution through uncertainty propagation with finitely many reference models, but omits the most direct Bayesian comparison.
-
The explanation and scope of mismatch robustness are stronger than the evidence supports. Sections 4.1 and 5.3 explicitly treat the target as another draw from the same class-conditional population as the reference models. Integrating uncertainty within that population does not by itself establish robustness when the target comes from a different population.
-
The group-level uncertainty interpretation needs validation beyond ROC performance. The implemented group attack replaces the local predictive mixture with a moment-matched Gaussian and treats fitted factor quantities as fixed; Appendix D.6 appropriately acknowledges these approximations. It also assumes conditional independence between queried images after accounting for the global model coordinate. Consequently, the aggregate variance in Eq. (157) contains no residual covariance between images of the same person. Such dependence could affect the predictive tails, particularly when aggregating up to 200 images.
-
Important experimental choices are insufficiently specified in the manuscript. The attack fixes where the membership signal is strong, but neither its values nor the data used to select it are reported. This matters for comparing attacks without tuning on evaluation membership labels.
Questions for authors
See the weaknesses.
7 comments
Table 4. Mixed references: standardized pAUC at 1% FPR.
| Model / dataset / mode | BASE | LiRA | BaVarIA-t | BaVarIA-n | Ours |
|---|---|---|---|---|---|
| DDPM / CIFAR10 / online | 82.49 ± 2.97 | 78.35 ± 2.70 | 71.94 ± 2.05 | 78.81 ± 1.68 | 86.14 ± 1.65 |
| DDPM / CIFAR10 / offline | 71.43 ± 2.72 | 69.92 ± 2.64 | 63.25 ± 1.37 | 62.22 ± 1.03 | 81.35 ± 2.23 |
| DDPM / CelebA / online | 56.01 ± 0.82 | 55.22 ± 0.66 | 53.42 ± 0.40 | 53.48 ± 0.41 | 61.18 ± 1.34 |
| DDPM / CelebA / offline | 55.66 ± 0.73 | 54.94 ± 0.65 | 51.96 ± 0.21 | 51.77 ± 0.19 | 61.83 ± 1.51 |
| FM / CIFAR10 / online | 65.16 ± 2.20 | 76.60 ± 1.06 | 69.97 ± 0.79 | 74.46 ± 0.78 | 82.56 ± 1.33 |
| FM / CelebA / online | 62.69 ± 1.87 | 66.56 ± 1.13 | 63.64 ± 0.89 | 67.87 ± 1.12 | 74.23 ± 1.57 |
Tables 1–4 demonstrate that the BaVarIA attacks are performing substantially worse than our hierarchical Bayesian attack: for example, in the online setting, mean TPR at 0.1% FPR is 75.42% for our attack versus 26.27% and 44.50% for BaVarIA-t and BaVarIA-n on DDPM/CIFAR10 with matched references, and 35.20% versus 17.40% and 22.86%, respectively, on FM/CelebA with mixed references. In the matched setting, BaVarIA is even worse than LiRA and BASE for the majority of benchmarks. However, in the mixed setting, BaVarIA closes the gap to BASE and LiRA and sometimes overtakes them, further suggesting that the Bayesian modelling could increase robustness to this type of mismatch in the reference model population.
Note that the LiRA performance here differs somewhat from that reported in the submitted version; we have used the class-specific global-variance version instead of the shared local-variance version since it has the highest average rank rather than the highest average TPR at 0.1% FPR. Average rank is arguably a better metric to determine which variance computation is overall best across the different datasets, and as a result the LiRA performance is higher on CelebA and ImageNet. The updated LiRA performance is in most cases better, yet still worse than our hierarchical attack.
Weakness 2.
While it is true that Bayesian marginalization does only account for uncertainty within the assumed model, using a mixture of reference model types does not necessarily violate the assumption that the target and reference models follow the same class-conditional distribution. The assumed target model distribution reflects the adversary's prior belief and side-knowledge; a heterogeneous mixture of reference models can naturally represent adversarial uncertainty about the target model. Consequently, a heterogeneous pool of reference models is not inherently inconsistent with our framework. However, it is true that the particular modelling assumptions, such as the Gaussian-Gamma hierarchy used in the record-level attack, may be less accurate when applied with a heterogeneous mixture of reference models. The mechanisms underlying the empirically observed robustness to heterogeneous pools of reference models differ for the record-level and group-level attacks, so we separate our discussion of the two cases below.
Record-level attack. We believe the predictive posterior marginalization may contribute to the observed robustness when using a mixture of reference model types, relative to the baselines. Our Bayesian method places a prior on the distributional parameters, updates their posterior distributions by reference model loss observations, and integrates over these posterior distributions when computing the target posterior-predictive likelihood. The latter integration allows parameter configurations that better explain the observed target loss values to contribute more to the posterior-predictive likelihood, which could provide more flexibility under a heterogeneous reference population. We contrast this with other baseline approaches such as LiRA which commit to a Gaussian distribution fitted by statistics from reference model observations, or BASE which simply compares the target loss to a soft minimum of the reference losses, neither of which has a mechanism similar to that of our Bayesian attack. Of course, this does not guarantee in any way greater robustness under a heterogeneous reference population including mismatched model types. With too large of a difference between target and reference models, the reference-conditioned posterior distribution could assign negligible probability to suitable configurations, rendering the predictive posterior integration ineffective. Rather, this should be understood as a plausible explanation for our empirical observations.
Group-level attack. Our group-level attack combines Bayesian marginalization with estimated plug-in parameters, relying more heavily on the latter compared to our record-level attack. Therefore, we opt for a different approach and explicitly design the attack for robustness under a heterogeneous reference population. Specifically, we introduce a rank-one factor component to capture systematic shifts between observations across models, which could arise from differences in model type. The associated target model coordinate is then inferred from observations on other (non-target) groups, allowing the predictive distribution to adapt to systematic differences between target and reference models. Thus, for the group-level attack, we are primarily attributing the observed robustness in the mixed reference experiments to this rank-one factor component, rather than uncertainty integration. We investigate this experimentally by comparing the performance of the group-level attack with and without the rank-one factor component.
Effect of the rank-one factor. To isolate the factor's contribution, we compared the full group attack with a version retaining the same local Bayesian hierarchy but disabling factor fitting, reference residual refitting, and target-coordinate adaptation. We compare them over the benchmarks included in the paper, with the same settings. The results are presented in Table 5.
Table 5. Group attack with and without the rank-one factor: TPR at 0.1% FPR. Values are percentages, mean ± population standard deviation over ten target models. Differences are the full attack minus the no-factor variant TPR in percentage points. Audit modes “0 train” and “1 train” allow zero and at most one target training record per audited group, respectively.
| Dataset | References | Audit mode | Target | Full attack | No factor | Difference |
|---|---|---|---|---|---|---|
| CelebA | matched | 0 train | DDPM | 14.00 ± 2.01 | 14.86 ± 2.26 | −0.86 |
| CelebA | matched | 0 train | FM | 9.32 ± 2.29 | 9.49 ± 2.74 | −0.16 |
| CelebA | matched | 1 train | DDPM | 42.72 ± 4.98 | 44.12 ± 6.94 | −1.41 |
| CelebA | matched | 1 train | FM | 22.98 ± 4.90 | 22.70 ± 5.06 | +0.28 |
| CelebA | mixed | 0 train | DDPM | 15.27 ± 1.86 | 6.62 ± 0.85 | +8.65 |
| CelebA | mixed | 0 train | FM | 9.55 ± 2.05 | 6.79 ± 1.17 | +2.76 |
| CelebA | mixed | 1 train | DDPM | 43.82 ± 5.80 | 21.86 ± 3.54 | +21.96 |
| CelebA | mixed | 1 train | FM | 23.14 ± 3.83 | 14.43 ± 2.14 | +8.71 |
| VGGFace2 | matched | 0 train | FM | 8.60 ± 2.19 | 8.64 ± 2.07 | −0.04 |
The factor substantially improves performance with mixed references, while having little effect with matched references. Thus, it achieves its purpose and greatly improves robustness under mixed reference model types. However, we acknowledge that the group-level robustness claims are supported by limited evidence, with only one dataset and two model types due to resource constraints. We will include more results for the VGGFace2 dataset in the final version of the paper, if accepted. We will also clarify our discussion regarding robustness under a mixture of reference model types.
Weakness 3.
Our group-level attack should be seen as a modular semi-Bayesian approximation of a hierarchical model, with an additional rank-one factor component introduced to capture systematic shifts between models, arising from, e.g., mismatched reference models. Since the Bayesian marginalization is more restricted than in the record-level attack, the uncertainty integration qualities are more limited for this particular attack. However, our main goal is to make a strong and reasonably efficient attack, capable of inferring the training membership of individuals even without access to training records and outperforming prior attacks. Questions regarding how well probabilities are calibrated or if full covariance is represented become interesting only insofar as it helps us develop a stronger membership inference attack. Incidentally, this applies to the statements regarding full hierarchical Bayesian models about accounting for uncertainty due to limited queries or reference models; such qualities are important only to the extent to which they improve overall attack performance.
Thank you for the review and for providing very relevant feedback for improving our work. Below we have addressed the questions and concerns raised, including a comparison with BaVarIA (eq. (8)), clarifications regarding the robustness under mixed reference model types including a new ablation experiment, a discussion of our group-level attack and the omission of residual covariance between images of the same individual with a new experiment that investigates this, and finally a clarification on some experimental parameter settings that were missing in the submission, including the loss query time t∗.
Weakness 1.
We agree that our paper should include the most direct Bayesian comparison as a baseline, namely the BaVarIA attack (Eq. (8)). We have thus implemented its two variants, BaVarIA-t and BaVarIA-n, described in its original paper (Brännvall, Exponential-Family Membership Inference: From LiRA and RMIA to BaVarIA, 2026). We use κ₀ = 1 and α₀ = 2 for both variants, as suggested in their paper. The experimental settings are the same as described in our paper and we use the same log-loss samples and reference models for each attack.
All tabulated performance values are percentages, mean ± population standard deviation over ten targets. Bold marks the highest mean in each row. ImageNet is evaluated on a subset of 100k data points.
Table 1. Matched references: TPR at 0.1% FPR.
Table 2. Matched references: standardized pAUC at 1% FPR.
Table 3. Mixed references: TPR at 0.1% FPR.