- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisits the evaluation of high-resolution MIAs. It proposes simple loss-function and learning-rate adaptations, which significantly improve attack performance, showing that prior defense claims may overstate privacy protection under fixed attack setups. The authors also score the reconstructions with both standard and adversarially trained Inception evaluators, demonstrating a strong dependence of measured leakage on the choice of evaluator.
With these adaptations, they show that prior findings on the privacy-robustness trade-off in generic MIA defenses can be generalized to recent anti-gradient defenses as well. Extensive experiments show (1) improved attack performance across multiple datasets, defenses, attacks, and target model architectures, (2) disagreement between standard and robust evaluators, and (3) an empirical association between robustness and reconstruction quality.
Critical strengths
- Revealing overfitting issues in existing MIA defenses: This work reveals an issue of MIA defenses overfitting to fixed attack configurations, highlighting the need for adaptive and robust privacy evaluation.
- Extending the claim of a privacy–robustness trade-off: The paper extends the findings of prior work on the privacy-robustness trade-off to the most recent MIA defenses, highlighting an important future direction: protection against MIAs without sacrificing adversarial robustness.
Critical weaknesses
- Unclear interpretation of evaluator disagreement: While Section 1 states that neither evaluator is treated as ground truth, Section 3 interprets the lower AttAcc of the standard evaluator as evidence that it underestimates privacy leakage. The disagreement demonstrates sensitivity to evaluator choice, but it remains unclear how to interpret the higher scores of the robust evaluator and which better reflects the actual leakage.
- Lack of qualitative analysis: This work relies on quantitative metrics across all experiments. Providing qualitative analysis or visualizations of reconstructed samples could help readers better understand the improvement of adaptive attacks and the disagreement between standard and robust evaluators.
Questions for authors
- Generalizability of adaptive attacks: Table 5 shows that adaptive attacks only improve IF-GMI against RoLSS and NegLS. Is there any explanation for why the adaptation is much less effective for IF-GMI against other defenses?
- Effect of robust evaluators on non-AT models: In Tables 2 and 6, it seems that AttAcc increases for AT models when switching to robust evaluators, but the trend is reversed for many non-AT defenses. Is there any explanation or intuition behind this observation?
7 comments
- Re: Critical Weaknesses 1–2. Qualitative analysis and which evaluator better reflects actual leakage.
Thank you—we added qualitative face-reconstruction grids for every defense under vanilla PPA, adaptive PPA C-LR, and adaptive PPA N-LR.
Qualitative Results: LINK https://i.imgur.com/1RRuHGS.jpeg
The qualitative examples mirror the quantitative results: adaptive attacks recover more recognizable identity-bearing facial features than vanilla PPA for the defenses on which adaptation produces the largest leakage increases.
We further test this interpretation, and address evaluator concerns, using two independent qualitative evaluations:
- 2,225 human responses. We recruited human raters to assess identity, demographics, and family resemblance over 2,225 ratings.
-
(a) Adaptive attacks. Human raters score our adaptive attacks' reconstructions higher than those from vanilla PPA on the NegLS defense. Both adaptive attacks improve human ratings by approximately 7-17 percentage points across the three questions. Across both adaptive attacks, the improvements are statistically significant even after BH correction for the primary Same Individual and Same Demographics questions; Family Resemblance is also higher. Our two adaptive attacks obtain similar performance on each question (BH p 0.421). Human Evaluation of adaptive attack results plot: LINK https://i.imgur.com/FPbTMPr.png
-
(b) Robust evaluator. As described fully in our Response to Reviewer #1 (U2WM) above, we find that our Robust Inception feature-space distance evaluation correlates significantly better with human ratings than the vanilla Standard Inception evaluation model. For Inception distance, the improvement is statistically significant for both the primary Same Individual question (p = 0.008) and Family Resemblance question (p = 0.033). Evaluator confidence shows that our Robust Evaluator offers the same directional improvement over a standard evaluator. Note that this statistical significance was obtained despite the limited statistical power in a small-scale human study with just five human raters. Given the positive point estimates for confidence, we expect its differences to reach greater statistical significance in a larger-scale study. Human Evaluation of robust Inception plot: LINK https://i.imgur.com/rfWHzmx.png
- Comprehensive Gemini evaluations. Following Ho et al. (CVPR Findings 2026), we use Gemini to conduct additional qualitative comparisons across all 13 defenses × all 3 (vanilla + adaptive) attacks. As detailed in our Response to Reviewer #1 (U2WM) above, our adaptive attacks generally improve reconstruction success under Gemini as well. Results Plot: LINK https://i.imgur.com/PU70Eh0.png
- Re: Q2. “In Tables 2 and 6, it seems that AttAcc increases for AT models when switching to robust evaluators, but the trend is reversed for many non-AT defenses. Is there any explanation or intuition behind this observation?”
Our human study provides supporting evidence: Robust Inception distance correlates more strongly with human ratings on all three questions. But the broader contribution is not to replace one evaluator with another. This directional pattern is exactly what the evaluator feature-basis account predicts. Standard and robust Inception measure leakage through different feature bases: the standard evaluator places more weight on non-robust predictive features, while the robust evaluator places more weight on robust, visually meaningful features. This explains why evaluator substitution can lower measured leakage for models that rely more heavily on non-robust features while sharply increasing it for AT models.
Our manuscript states this point explicitly: “Neither evaluator is treated as ground truth; disagreement between them measures how strongly reported leakage depends on the evaluator’s feature basis” (lines 127–128). A low leakage score under one arbitrarily selected evaluator—such as the standard Inception evaluator used to assess virtually all defenses—therefore does not establish privacy when another evaluator detects significant leakage in the same reconstructions. As the manuscript concludes, reconstructed images should be “evaluated by more than one arbitrarily chosen classifier feature basis” (lines 158–160). Each evaluator detects leakage through one particular feature basis, and using multiple evaluators reveals leakage through different types of features (we will clarify the description of this point in Section 3--thank you for pointing out this opportunity to improve clarity).
We sincerely thank you for highlighting how our paper “reveals an issue of MIA defenses overfitting to fixed attack configurations.” We especially appreciate your recognition that this raises “an important future direction: protection against MIAs without sacrificing adversarial robustness.”
We understand that your main remaining reservations, and the reason for a score of 3 rather than a clear-accept 4, concern:
To address the qualitative and evaluator concerns, we add three complementary forms of qualitative evaluation (detailed in our comments below):
We strongly feel that this paper's results advance the MI/privacy field. Its main result shows that leading MIA defenses’ apparent privacy can reflect optimization and measurement failures, and that correcting these failures reveals a fundamental relationship between training-data privacy and adversarial robustness across defenses previously thought to be exceptions. We would appreciate it if you would consider improving your score in light of the new qualitative, human, and Gemini evaluations and the clarifications below.