acceptodds
Under review as a conference paper at ICLR 2027

MA-LiRA: Memory-Aware Membership Inference Attacks

Abstract

Membership inference asks whether a candidate example participated in a model's training set and provides an important tool for empirical privacy auditing. Many strong attacks rely on statistics observed only at the final model checkpoint, potentially discarding information contained in how a candidate's behavior evolves during optimization. We introduce MA-LiRA, a checkpoint-aware membership-inference attack that exploits this training history. For each labeled candidate, MA-LiRA constructs a compact five-dimensional representation from its checkpoint-wise loss trajectory, capturing endpoint fit, overall learning progress, local trajectory irregularity, and temporally weighted changes. Each feature is calibrated using candidate-specific member and non-member shadow distributions, and the resulting likelihood-ratio coordinates are combined by a fusion model trained exclusively from shadow-derived pseudo-target examples, without using target membership labels for fitting or threshold selection. We evaluate MA-LiRA on CIFAR-10, CIFAR-100, and SVHN against a broad set of score-based, calibrated, likelihood-ratio, and trajectory-based membership-inference baselines. Across the evaluated datasets, MA-LiRA consistently improves overall membership discrimination and low-false-positive-rate detection under the stated checkpoint-aware access model. These results indicate that training-history information can provide membership evidence beyond that available from a final-checkpoint statistic alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
membership inference, privacy, machine learning privacy, training dynamics, memory-aware learning, likelihood-ratio attacks

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_eovgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a membership inference attack method that leverages training history information. They construct a five-dimensional feature vector and evaluate the effectiveness of the proposed attack on three datasets: CIFAR-10, CIFAR-100, and SVHN.

Critical strengths

Under the privacy auditing setting, the authors construct a five-dimensional feature vector to conduct attack evaluations.

Critical weaknesses
  1. The authors do not provide a detailed description of the attack scenario or the adversary's capabilities. The broadly defined privacy auditing setting lacks practical relevance and fails to reflect realistic attack scenarios.
  2. Several existing studies have already explored membership inference attacks that leverage training history information. The authors should clearly clarify how their proposed method differs from these prior works. [1]Jagielski, Matthew, et al. "How to combine membership-inference attacks on multiple updated machine learning models." Proceedings on Privacy Enhancing Technologies (2023). [2] Chen, Yuetian, et al. "Evaluating the Dynamics of Membership Privacy in Deep Learning." arXiv preprint arXiv:2507.23291 (2025). [3] Gu, Yuhao, et al. "CS-MIA: Membership inference attack based on prediction confidence series in federated learning." Journal of Information Security and Applications 67 (2022): 103201.
  3. The authors should conduct ablation studies on the proposed five-dimensional feature vector to evaluate the contribution and importance of each feature to the attack performance.
Questions for authors

The authors should provide a detailed discussion to clearly distinguish their proposed work from existing related works. [1]Jagielski, Matthew, et al. "How to combine membership-inference attacks on multiple updated machine learning models." Proceedings on Privacy Enhancing Technologies (2023). [2] Chen, Yuetian, et al. "Evaluating the Dynamics of Membership Privacy in Deep Learning." arXiv preprint arXiv:2507.23291 (2025). [3] Gu, Yuhao, et al. "CS-MIA: Membership inference attack based on prediction confidence series in federated learning." Journal of Information Security and Applications 67 (2022): 103201.

Reviewer_rT8VOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MA-LiRA, a membership inference attack that leverages intermediate training checkpoints to construct temporal features from per-example loss trajectories. These features are individually calibrated using candidate-specific shadow distributions and subsequently combined through a learned fusion model. The authors report improvements over several existing MIA baselines on CIFAR-10, CIFAR-100, and SVHN.

Critical strengths
  1. Good writing, easy to follow;
Critical weaknesses

Questionable Practicality of the Threat Model: MA-LiRA assumes access to multiple intermediate checkpoints of the target model, which is rarely feasible for realistic adversaries. Even under the proposed internal auditing scenario, the practical necessity of such an unusually privileged attack remains unclear. The paper would benefit from a more convincing justification of its threat model and real-world relevance.

Unfair Experimental Comparisons: MA-LiRA acesses multiple training checkpoints, while most baselines only observe the final model, resulting in an inherently unequal comparison. The reported gains may therefore stem primarily from additional information rather than methodological improvements. Without information-matched baselines, such as multi-checkpoint LiRA or raw trajectory classifiers, the claimed superiority remains insufficiently substantiated.

Incremental Technical Contribution: MA-LiRA largely combines existing LiRA calibration with handcrafted trajectory features and standard logistic regression. The proposed power-law memory weighting appears heuristic, with limited theoretical justification. Moreover, the lack of targeted ablations makes it difficult to establish its independent contribution, leaving the overall novelty rather modest.

Limited and Outdated Experimental Evaluation: The evaluation is restricted to CIFAR-10, CIFAR-100, and SVHN, with only one relatively small-scale architectures: ResNet-18. Such benchmarks seem insufficient for an ICLR 2027 submission, especially without experiments on pretrained or foundation models. The scalability of the checkpoint-intensive approach therefore remains questionable.

Insufficient Coverage of Related Work: The literature review focuses primarily on a small collection of representative MIA studies, with limited engagement with broader and recent developments. This is particularly concerning given the incremental nature of the contribution. While the authors' inclusion of the original ResNet paper demonstrates a commendable respect for foundational work, comparable attention to recent advances in membership inference would be more valuable.

Questions for authors

see weakness

Reviewer_ELk5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MA-LiRA that utilizes the loss trajectories of multiple checkpoints during model training. Unlike traditional LiRA, which primarily utilizes the final model output, MA-LiRA extracts five-dimensional features using the loss changes of candidate samples across multiple training stages. However, the paper does not prove that the memory mechanism is the main source of performance improvement and it lacks relevant experiments.

Critical strengths
  1. Unlike conventional post-training MIAs, the paper exploiting loss dynamics during training is well-motivated and practically valuable for checkpoint-based privacy auditing
  2. The paper moves beyond simply concatenating multi-checkpoint losses by constructing a complete framework that spans trajectory feature extraction, candidate-level likelihood ratio calibration, and feature fusion.
Critical weaknesses
  1. The paper takes memory-aware as its core innovation and constructs two memory features through power-law weighting. However, current experiments cannot prove the main contributing factors and lack key ablation experiments.
  2. The paper claims to outperform all compared methods, but the comparative experiments suffer from unfair access permissions. MA-LiRA can access multiple checkpoints throughout the target model's training process, while most comparative methods can only access the final model. It is recommended to add comparative experiments with equal checkpoint access permissions.
  3. The paper uses a fixed weighting coefficient of 0.7 to give higher weight to recent changes in loss. However, the paper does not adequately explain why recent training changes are more discriminative of membership than earlier changes, nor does it demonstrate that power-law decay is superior to other time-weighted mechanisms.
  4. The paper has some formatting issues, such as the title. Also, the overview section in section 3.2 Oerview, which uses only arrows, is somewhat crude and difficult to understand, especially the fusion module. It is recommended to use a diagram to present the information.
Questions for authors

In Table 2 of the paper, the average AUC of MA-LiRA's endpoint-only ablation is only 0.5249, while the full method is 0.7891. Why does endpoint-only MA-LiRA, which also uses candidate-level calibration, perform so poorly? The paper introduces two power-law memory-aware features as key components. However, it lacks systematic ablation experiments to verify their individual contributions. Can the authors demonstrate that these two memory features provide a significant performance improvement compared to using only the final loss, loss reduction, and trajectory roughness? The paper proposes a memory-aware feature using a power-law weighting function with the parameter α fixed at 0.7. What is the rationale for choosing this parameter? How much do different α values affect attack performance? Does power-law weighting consistently outperform uniform or exponential weighting across different datasets and checkpoint configurations? The paper reports good attack performance on three datasets, but primarily provides results from single experiments, lacking statistical analysis across random seeds.

Reviewer_zoaLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates membership inference attacks (MIAs), where an adversary attempts to determine whether a given query sample was used to train a target model. In particular, the paper considers a threat model in which the attacker has access to multiple intermediate checkpoints of the target model during training, rather than only the final trained model. The attacker then exploits the loss trajectory of each query sample across these checkpoints to infer whether it belongs to the target model's training dataset.

Critical strengths

Since I view this as a technical paper whose main contributions lie in exploring new problems and developing new methods, I place particular emphasis on the clarity of presentation, methodological novelty, and insights provided. Overall, the paper is clearly presented and easy to follow. Although the writing exhibits some signs of AI-assisted editing, such as the unusually frequent use of semicolons, the manuscript is generally well-organized and understandable.

Critical weaknesses

As this submission primarily focuses on developing a new attack method (and new setting), I place particular emphasis on the practicality and relevance of the assumed threat model. However, I have a major concern regarding the threat model considered in this paper. On the one hand, it appears too strong for conventional membership inference scenarios, where potential adversaries typically have access only to the final trained model, or even just its black-box interface. On the other hand, it appears unnecessarily restrictive for privacy auditing settings, where stronger adversarial capabilities are often considered. For example, in differential privacy (DP) auditing, the adversary may be allowed to observe individual gradient updates or even manipulate the construction of the training dataset. Such strong assumptions are motivated by the worst-case nature of DP guarantees, which must hold even when the adversary knows all but one individual's data and may consider adversarially constructed samples. In this regard, the proposed threat model seems to fall into an awkward middle ground: it is too strong to represent a typical real-world membership inference attack, yet it does not fully exploit the stronger capabilities commonly considered in privacy auditing. As a result, I find its practical relevance insufficiently motivated. The paper also lacks a thorough discussion of and comparison with the substantial body of privacy auditing literature, particularly work on auditing differential privacy guarantees.

Beyond the threat model, I also have concerns about the methodological design. While the paper provides several ablations, they do not sufficiently clarify which components drive the improvements and why they are effective. Overall, the experiments mainly demonstrate that access to additional checkpoint information improves membership inference, but provide limited insights into the effectiveness and necessity of the specific methodological choices.

Questions for authors
  • Practical motivation for the threat model. Could the authors further clarify the intended real-world application of the proposed checkpoint-aware threat model? In particular, under what practical scenarios would an adversary have access to multiple intermediate checkpoints of the target model, but not to other training information, such as individual gradient updates? If the method is primarily intended for authorized privacy auditing, what concrete auditing objectives does it serve beyond simply improving membership inference performance?

  • Connection to existing privacy auditing literature. Could the authors further discuss how the proposed method relates to the broader privacy auditing literature, particularly work on auditing differential privacy (DP) guarantees? In such settings, stronger adversarial capabilities are often considered, including access to individual gradient updates or control over the construction of training datasets, to empirically assess worst-case privacy leakage. Given these existing approaches, what is the motivation for adopting the intermediate checkpoint-access model proposed in this paper? More importantly, what additional privacy auditing insights or capabilities does this setting provide compared to existing approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.