acceptodds
Under review as a conference paper at ICLR 2027

ATTENMIA: READING MEMBERSHIP OFF THE ATTENTION MAPS OF LANGUAGE MODELS

Abstract

Large language models (LLMs) can reveal through attacks such as Membership Inference Attacks (MIAs) whether a sequence appeared in their training data, creating privacy risks. Many existing MIAs for LLMs derive their signal from model outputs. Motivated by recent research showing the important role attention plays in information flow within a model, in this paper, we instead ask whether membership signatures are visible in the model’s attention patterns. We introduce AttenMIA, a white-box MIA with two variants: Transitional features capture cross-layer attention changes, while Perturbation features measure attention shifts under input edits. Each variant uses a lightweight supervised classifier to convert its features into membership scores. We evaluate on MIMIR, a benchmark constructed to reduce known distributional pitfalls in membership evaluation. Across seven domains and three Pythia model sizes, AttenMIA (Perturbation) achieves the highest ROC AUC in 20 of 21 settings, with domain-averaged AUCs of 0.83–0.85, compared with 0.71 for the strongest evaluated baseline. When each domain is held out from attack training and selection, AttenMIA (Transitional) performs best among the evaluated methods across all three model sizes, averaging 0.73 ROC AUC and 26.7% TPR@1%FPR across domains and models. Under the same protocol, a supervised hidden-state control achieves an average AUC of 0.54, showing that the advantage arises from attention features. GPT-2 fine-tuning experiment shows that cross-layer transitional attention features, and perturbation-response features have different attention gradients for members and non-members. These results substantially improve on SoTA membership inference attacks, and demonstrate that self-attention provides a useful signal for training-membership inference that transfers across the evaluated domains.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
membership inference, privacy, large language models, attention, training data memorization, white-box attacks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5z6AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce a white-box membership inference attack that uses features from self attention maps instead of relying on the model outputs. The authors define two feature representations: 1) Transitional features: that measure the KL divergence between attention distributions of two adjacent layers, and 2) perturbation features: that capture how attention shifts when inputs are edited. They train a classifier on these features and find that it performs better than existing output-based MIA attacks.

Critical strengths
  • The proposed MIA is pretty novel. Most attacks rely on output probabilities or signals. Using self attention to detect membership information is interesting and novel.
  • The methodology to test for generalization is rigorous: The authors do a systematic controlled study by doing target-blind leave one domain out evaluations. This essentially shows that the attention-based membership signal transfers to unseen domains rather than just overfitting to specific training data.
  • The ablations and feature analysis are well reported, such as the contribution of different layers and perturbation methods.
Critical weaknesses
  • The paper empirically shows that the attention distributions differ for members and non-members but it does not isolate whether there are any specific factors (e.g., specific circuit types) that are driving the MIA signal.
  • The other concern I have is about the dimensionality overhead: extracting attention matrices across all layers and heads can be massive, as compared to the output-based MIA attacks. This definitely raises questions on the practicality of this attack.
Questions for authors
  • Did you find any specific mechanistic factors driving the MIA signal? also did you run any ablations to check if any specific attention heads disproportionately contribute to the perturbation and transitional signals?
  • Can you compare the scalability of the proposed MIA with output-based attacks, as I feel the output-based attacks would be much cheaper to run?
Reviewer_ehWQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies membership inference for pretrained LLMs. The authors propose a supervised white-box attack that predicts membership using attention-derived features from the target LLM. Experiments on MIMIR and WikiMIA show improvements over the evaluated baselines, including cross-domain performance on MIMIR.

Critical strengths
  1. The paper presents a concrete approach to exploiting attention features for membership inference, with ablations supporting the use of multiple perturbations and layers.
  2. The method shows empirical improvements on the evaluated baselines in most MIMIR settings and in cross-domain evaluation.
Critical weaknesses
  1. The baseline Hidden is the most relevant supervised baseline, but the paper does not specify its features, classifier architecture, or training procedure. This makes it difficult to assess whether attention features offer an advantage over other possible internal activations under comparable conditions.

  2. The authors provide motivation for using attention related features, but it does not clearly explain the choice of the specific attention features. The ablations show benefits from combining perturbations and layers, but do not establish whether the proposed feature construction improves over simpler attention statistics, such as the attention concentration statistic, when used under the same classifier.

Questions for authors
  1. The baseline method Hidden's feature extraction and classifier training need to be specified, including whether it uses the same objective and checkpoint selection as proposed method.

  2. A comparison with simpler attention statistics using the same classifier would help determine what the proposed cross-layer and perturbation features add.

  3. The numbers of members and non-members used for attack training, validation, and testing in each MIMIR domain are missing. Results by held-out domain and variability across seeds would also help assess the consistency of the cross-domain gains.

Reviewer_AN8nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes AttenMIA, a white-box membership inference attack built on attention maps. Transitional features measure the KL divergence between same-index heads in adjacent layers, and Perturbation features measure attention shifts under prefix insertion, word dropping, and word replacement. Each feature vector is fed to a supervised MLP trained on labeled members and non-members. On MIMIR with three Pythia models, the Perturbation variant reaches in-domain AUCs of 0.83 to 0.85 against 0.71 for the best baseline, the Transitional variant reaches 0.73 AUC in leave-one-domain-out transfer, and both variants reach about 0.99 AUC on WikiMIA.

Critical strengths
  • Using attention transitions and perturbation responses as membership features is a new angle, and the feature extraction needs only forward passes.

  • The cross-domain protocol is carefully designed, keeping the target domain out of preprocessing, training, and model selection, and the paper reports TPR at 1% FPR alongside AUC.

Critical weaknesses
  1. AttenMIA trains a classifier on labeled members and non-members, so it can learn any distribution difference between the two sets. Prior work shows that blind classifiers using only the text reach high AUC on many MIA benchmarks, including WikiMIA (Das et al., 2025; Duan et al., 2024). The 0.99 AUC on WikiMIA and on MIMIR GitHub strongly suggests such shortcuts, and the paper lacks the key control of running the same pipeline on a model that never saw the Pile.

  2. In-domain baselines use raw scalar scores without any classifier, so the comparison mixes supervised and unsupervised attacks. The only supervised control, Hidden, is described in one sentence, and stronger supervised baselines such as an MLP over all baseline scores (Maini et al., 2024) or linear probes on activations (Ibañez-Lissen et al., 2025) are missing.

  3. The paper does not state which MIMIR split is used, its n-gram overlap threshold, or the number of examples per domain, all of which strongly affect MIA results.

  4. The motivation from visual search and Hopfield networks is loose, and the GPT-2 gradient diagnostic in Figure 2 uses a fine-tuning setup different from the attack, so it gives limited support for claims.

Questions for authors
  1. What AUC does AttenMIA reach when the same pipeline runs on a model not trained on the Pile, such as a model released before the Pile or trained on a disjoint corpus?
  2. What AUC does a supervised bag-of-words or n-gram classifier reach on the same splits?
  3. Which MIMIR split and how many members and non-members per domain are used?
  4. What exactly are the Hidden features, and how do supervised baselines over the scalar scores perform in-domain?
1 comment

Thanks for the additional information. After reviewing the response, I keep my current review score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.