acceptodds
Withdrawn from ICLR 2027

READ-OUTS ARE FILTERS, NOT FIREWALLS: MANIFOLD HEALTH SETS THE CEILING OF MEMBERSHIP INFERENCE PRIVACY

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Membership inference attacks (MIAs) exploit distinct channels of a trained model: representations, gradients, outputs, decision boundaries, and the per-sample training loss. Why do defenses that survive adaptive auditing on one dataset collapse on another? We attribute the surviving signal to a three-stage pipeline: data fingerprintability creates memorizable training samples; the representation encodes them; and the read-out decides how visibly the memorization surfaces in the observable scalar. One geometric property, the manifold health of the class-conditional distributions, splits datasets into three regimes and drives two degradation paths. In the fingerprint regime (scattered distributions: high intrinsic dimension, few samples per class), leakage resurfaces through any read-out as the shadow budget grows: on Location30 a density head read at 0.58 AUC with four shadows is read at 0.79 with eight. In the low-signal regime (mixed distributions), utility collapses faster than any defense can act: head-only DP-SGD on Texas100 falls from 61.9% to 12.6% accuracy. Within the manifold regime the read-out is decisive: linear logit heads carry a per-sample parameter imprint that LiRA amplifies (0.939 at 64 shadows vs. 0.832 with no defense), while a frozen-anchor density head suppresses the same channel to chance (0.5213 AUC at 94.9% accuracy on CIFAR-10; 0.4984 at 88.1% on PathMNIST) at zero ε cost. Read-outs are filters, not firewalls. Coarsening Location’s features from 446 to 112 dimensions confirms the fingerprint mechanism causally: leakage drops (0.885 → 0.811) as fingerprintability falls. We contribute a pre-deployment manifold audit (intrinsic dimension, class separation, samples per class) and a mechanism-matched, noise-matched, multi-seed protocol under which the three regimes (and the defense each one demands) become a measurement problem rather than trial and error.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Membership inference attacks (MIAs) ,DP-SGD,intrinsic dimension

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FjpbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates membership inference attacks (MIAs) and defenses. As far as I understand, the paper seeks to benchmark various MIAs and defenses and attempts to explain their performance differences across different data distributions. Specifically, it proposes a geometric interpretation of membership leakage by categorizing datasets into three regimes, investigates how different classifier heads affect privacy leakage, and proposes a density-based classifier head and a dual-track defense.

Critical strengths

Rigorously benchmarking different MIAs and defenses is a valuable effort. The paper attempts to provide a principled interpretation of why different methods perform differently across data distributions, which is a potentially interesting direction. If the authors can improve the clarity and organization of the paper to better communicate their key findings and contributions, the work could become more valuable.

Critical weaknesses

This paper reads like an AI-generated manuscript:

  1. It contains frequent and unnatural uses of semicolons and parentheses, which are stylistic features often associated with AI-generated writing.

  2. There are many nonstandard terms, such as "read-out", "manifold health", "shadow budget", "density head read","mixed distributions" that are directly used in the abstract and introduction without explaining their meanings, making it very difficult for readers to properly understand the key ideas of this submission

  3. The introduction and method sections are highly fragmented and lack a clear logical flow. Even after reading through the entire paper, I still cannot clearly identify its key ideas/findings/contributions.

Given these issues, I find it difficult to give a positive assessment of the paper in its current form, as the unclear presentation prevents me from thoroughly understanding and evaluating the technical details. The paper requires substantial restructuring to establish a clear and coherent logical flow and to make its key ideas and contributions understandable.

Questions for authors
  • The proposed method seems to be largely built upon CRISP (Anonymous, 2026), which is cited as an unpublished work under review. However, the reference does not provide a URL or any other means of accessing the manuscript. Could the authors clarify the relationship between CRISP and the proposed method, particularly which components are inherited from CRISP and which constitute the novel contributions of this paper? It would also be helpful to provide an accessible reference to CRISP.

  • Could the authors more clearly distinguish the main contributions of the paper, particularly the relationship among the proposed geometric framework, the dual-track defense, and the density-based classifier head?

  • Some statements appear to be imprecise or inappropriate. For example, "samples are scattered points rather than manifold elements" (line 94) is questionable, as scattered samples can still lie on an underlying manifold. The authors should carefully review the entire paper and clarify the intended meaning of such statements.

Reviewer_XB9oOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates why the same Membership Inference Attack (MIA) defense exhibits markedly different effectiveness across datasets. The authors decompose membership leakage into three stages—Data Fingerprintability, Representation Memorization, and Read-Out Visibility—and propose predicting a dataset's privacy-defense characteristics from its geometric properties, including Intrinsic Dimension, Class Separation, and Samples per Class.

Based on these metrics, the authors categorize datasets into three regimes—Manifold, Fingerprint, and Low-signal—and propose defense strategies targeting the corresponding stages. First, they employ Geometric Manifold Contraction to reduce representation-level membership signals. Second, they propose a Density Read-Out Head based on frozen synthetic anchors as a replacement for the conventional linear classification head, aiming to reduce loss-based membership leakage. In addition, they combine geometric contraction with head-only DP-SGD to form the Dual-Track Defense.

Experiments span CIFAR-10, PathMNIST, TinyImageNet, Location30, and Texas100, and evaluate the defenses using multiple MIAs, mechanism-matched shadow models, and varying privacy budgets. The results show that geometric contraction suppresses most representation-level attacks but may fail to defend against LiRA. On certain Manifold datasets, the Density Head reduces the LiRA AUC to nearly 0.5 while maintaining high classification accuracy. However, on Location30, the privacy protection of the Density Head degrades noticeably as the number of shadow models increases. Based on these findings, the authors argue that the geometric structure of the data determines the effective scope of privacy defenses, whereas the design of the read-out can only filter specific leakage signals and cannot fundamentally eliminate all membership information.

Critical strengths
  1. Analyzing the causes of MIA defense failure from the perspective of distinct leakage channels is of considerable research value. This paper not only examines the effectiveness of individual defense methods but also attempts to disentangle the contributions of Data Fingerprintability, Representation Memorization, and Read-Out Visibility to membership leakage. Notably, the authors find that Geometric Contraction substantially suppresses the majority of representation-level attacks, yet the loss-based LiRA remains effective. This observation suggests that merely altering the geometric structure of a model's internal representations may be insufficient to defend against all MIAs. Analyzing distinct leakage channels helps elucidate the limitations of existing defense methods.

  1. The paper presents a relatively systematic investigation of attack configurations and defense evaluation methodologies. The authors emphasize mechanism-matched and noise-matched shadow models, and investigate the impact of the shadow model budget and different random seeds on evaluation outcomes. In particular, Figure 3 shows that the LiRA AUC of the Density Head on Location30 increases substantially as the number of shadow models grows, indicating that a limited attack budget may overestimate the actual effectiveness of a defense. The paper further attempts to analyze the privacy–utility trade-off across different datasets. These experimental designs provide useful reference points for the evaluation of empirical MIA defenses.
Critical weaknesses
  1. The privacy guarantee of the proposed Dual-Track Defense is unclear and potentially misleading. In Section 3.4, the authors apply DP-SGD only to the Classification Head, while the Geometric Contraction Network is trained in a non-DP manner, and claim that the DP Head provides a formal privacy guarantee. However, if the Encoder and the Contraction Network are trained on the same private training data without DP, applying DP-SGD solely to the Classification Head does not directly guarantee that the entire model satisfies Differential Privacy. Prior work has shown that membership leakage is not confined to the final output layer of a model. [1] demonstrates that the hidden activations of intermediate layers can be directly exploited for membership inference, and that selecting neurons with strong membership-discriminative power can further amplify the attack. Moreover, [2,3] find, at the parameter level, that a model's privacy vulnerability is concentrated in a small subset of critical weights, which may reside in internal components such as convolution, attention, and MLP layers, rather than solely in the final classification layer. These studies further indicate that intermediate-layer representations and internal parameters may themselves encode membership-related information about the training data. Consequently, applying DP-SGD only to the final Classification Head cannot directly eliminate the membership-dependent information already learned by the upstream network. Although the authors employ Geometric Contraction to mitigate part of the representation-level leakage, such an empirical defense is not equivalent to a formal Differential Privacy guarantee. More importantly, when the upstream Encoder itself is trained on private data, the standard privacy analysis of head-only DP-SGD cannot be directly extended to the entire model without additional proof. The paper therefore needs to clearly distinguish between head-level DP and an end-to-end DP guarantee, and to provide the corresponding theoretical analysis. This issue directly affects the paper's central claim regarding the privacy guarantee of the Dual-Track Defense.

  1. The claimed mechanisms and predictive power of manifold health are not sufficiently validated. The paper attributes the privacy-protection effect of the Density Head to the fact that frozen synthetic anchors, unlike a Linear Head, do not accumulate per-sample parameter imprints. However, these anchors are still constructed from the training data, and the Encoder is still optimized on training samples. Therefore, merely freezing the anchors does not prove the absence of per-sample membership information. Furthermore, the explanation in Section 3.5—that a single sample's influence on the density score is O(1/n), whereas the loss change of a Linear Head is O(1)—lacks sufficient theoretical derivation and clearly stated assumptions. In addition, the authors validate the three proposed Manifold Health regimes using only five datasets, with the Fingerprint and Low-signal regimes each supported by a single dataset. This is insufficient to demonstrate that Intrinsic Dimension, Class Separation, and Samples per Class can universally predict defense effectiveness. The current experiments also fail to rule out confounding factors such as dataset characteristics, model architecture, and training configurations. These issues weaken the paper's explanation of the mechanism underlying the Density Head, as well as the support for the central conclusion that Manifold Health determines the privacy ceiling.

References

[1] Li et al. (ArXiv 2024). Unveiling the Unseen: Exploring Whitebox Membership Inference through the Lens of Explainability.

[2] Fang et al. (ICLR 2026). Learnability and Privacy Vulnerability are Entangled in a Few Critical Weights.

[3] Zhang et al. (TMLR 2026). Defending Membership Inference Attacks via Privacy-aware Sparsity Tuning

Questions for authors

Q1. Since the encoder is trained without DP, what formal privacy guarantee does the proposed Dual-Track Defense provide for the entire model?

Q2. Prior studies [1-3] show that hidden representations and internal weights can leak membership information. How does the proposed defense address these leakage channels beyond the classification head?

Q3. How can the claimed data-dependent privacy ceiling be established using only one fingerprint-regime dataset and a maximum of 16 shadow models?


References

[1] Li et al. (ArXiv 2024). Unveiling the Unseen: Exploring Whitebox Membership Inference through the Lens of Explainability.

[2] Fang et al. (ICLR 2026). Learnability and Privacy Vulnerability are Entangled in a Few Critical Weights.

[3] Zhang et al. (TMLR 2026). Defending Membership Inference Attacks via Privacy-aware Sparsity Tuning

Reviewer_opkXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to answer the questions 1) why does the same defense succeed on one dataset could collapse on another, and 2) what are the deciding factors. The paper investigate these questions by proposing a conceptual framework that links fingerprintability, representation memorization and readout visibility.

Critical strengths
  1. investigating the effectiveness of defense at a fine-grain view and link them to dataset structure and classifier design can be a very important and interesting question.
Critical weaknesses
  1. I had a really hard time understanding what the paper was trying to do. The research questions is clear, but the methodology and experiments are really confusing.

  2. The required AI usage statement is missing.

  3. numbers in evaluation are inconsistent. Zero ϵ\epsilon cost is very misleading with the DP methods also being used. The claim DP methods lacks proof and theoretical evidence of its correct usage.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.