acceptodds
Withdrawn from ICLR 2027

Exposing Membership Leakage in Test-Time Adaptation via Manifold Calibration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time adaptation (TTA) improves model robustness under distribution shifts by updating pretrained models on unlabeled test data during inference. However, test samples may leave detectable traces during model adaptation, raising a privacy concern that remains largely unexplored. Existing sequential membership inference attacks (MIAs) consider training or model-update trajectories, but do not account for the distribution-driven adaptation dynamics in TTA that can obscure candidate-specific membership traces. To address this gap, we propose the Manifold Calibration Attack (MCA), a label-free, trajectory-based attack designed for sequential TTA that separates candidate-specific update effects from shared adaptation dynamics. Specifically, MCA calibrates each candidate’s step-wise response against a local manifold neighborhood to remove shared adaptation motion, localizes candidate-relevant changes along the adaptation trajectory, and uses nuisance-aware Jacobian attribution to measure how strongly the selected updates align with the candidate’s functional subspace. Our analysis reveals that membership evidence is often stronger in early adaptation across different TTA methods. Extensive experiments across 13 settings spanning multiple datasets, model architectures, corruption streams, and representative TTA methods show that MCA achieves the best macro-average ROC-AUC and low-FPR detection performance among the evaluated attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine learning privacy, test-time adaptation, membership inference attacks, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bY4rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies membership inference attacks (MIA) in sequential Test-Time Adaptation (TTA). The authors point out that distribution-level adaptation can hide the membership signal of individual samples, making existing MIA methods less effective in this setting.

To address this problem, the authors propose Manifold Calibration Attack (MCA). The method first removes shared adaptation effects using local reference samples, identifies candidate-relevant steps along the adaptation trajectory, and then computes membership scores using Jacobian-based attribution.

Critical strengths
  • Interesting problem setting

I think the privacy problem considered in this paper is interesting. Unlike standard inference, TTA continuously updates the model using test samples, so these samples may leave some information in the model states. In particular, the authors point out that distribution-level adaptation and sample-specific effects are mixed together during TTA. This makes it difficult to directly apply existing MIA methods. I think this is a reasonable motivation for designing an attack specifically for TTA.

  • Experiments across different TTA settings

The experiments cover several datasets including CIFAR-10-C, CIFAR-100-C, and ImageNet-C, with different model architectures and TTA methods such as BN Adapt, TENT, SAR, and CoTTA. The authors also provide ablation studies and additional experiments on reference mismatch, adaptation hyperparameters, and differential privacy. These analyses are useful for understanding the proposed method. Some results on ImageNet-scale settings also show noticeable

Critical weaknesses
  • Limited novelty compared to existing approaches

The main method combines local manifold calibration, temporal localization, and Jacobian-based attribution. I think this combination is reasonable for the TTA setting, but I am not fully convinced about its methodological novelty.

Previous trajectory-based MIA studies have already considered membership signals from model updates over time. Gradient-based attribution has also been studied in previous works. In this sense, the proposed method seems to be more of a combination and adaptation of existing techniques, rather than a fundamentally new attack principle.

The authors provide ablation results showing that each component contributes to the performance. However, I think it is still unclear whether the full MCA pipeline is necessary compared to simpler approaches. For example, it would be useful to compare against a simpler temporal or gradient-based attack with local calibration, under the same access to model states and reference samples.

  • Relatively weak attack performance and practical applicability

According to Table 1, MCA achieves an average ROC-AUC of 0.561, compared to 0.544 for LiRA, which is the strongest baseline on average. Although MCA gives the best average result, the absolute AUC is still relatively close to random guessing (0.5).

Also, in several settings, MCA achieves AUC values close to 0.5, and sometimes performs worse than the existing baselines. For TPR@5%FPR, MCA improves the average result from 8.263% to 9.575%, but the overall detection performance is still limited.

I agree that some ImageNet-scale settings show more meaningful improvements. However, the performance varies considerably across different TTA methods. Considering the complexity of the proposed attack, I think the average improvement is rather small. It would be helpful to explain how consistently MCA improves over the strongest baselines, and how these results should be interpreted in terms of actual membership leakage.

  • Another concern is the practical attack setting.

MCA assumes that the attacker can access intermediate model states throughout the adaptation process. It also requires reference calibration and Jacobian computation for candidate samples. While this assumption can be reasonable in some collaborative TTA scenarios, I am not sure how realistic it is in more general deployment environments. The computational cost of the attack is also not sufficiently discussed.

Questions for authors

N/A

Reviewer_ejuDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates membership inference attacks in sequential test-time adaptation, where unlabeled test samples may leave detectable traces during model updates. The authors propose MCA, a label-free, trajectory-based attack that combines local manifold calibration, temporal localization, and Jacobian-based attribution to distinguish candidate-specific membership signals from shared adaptation dynamics. The method is evaluated across 13 experimental settings involving different datasets, model architectures, and TTA algorithms. Experimental results show that MCA achieves the highest average ROC-AUC (0.561) and TPR@5% FPR (9.575%) among the evaluated baselines.

Critical strengths
  1. This paper investigates an underexplored privacy risk associated with test-time adaptation. Unlike conventional membership inference attacks that primarily focuses on model training data, this work examines whether unlabeled test samples leave detectable membership traces during model adaptation.
  2. The method follows a coherent design. To address TTA-specific challenges, including shared adaptation dynamics that obscure candidate-specific membership signals and temporally sparse membership evidence, MCA combines manifold calibration, temporal localization, and Jacobian-based attribution.
Critical weaknesses
  1. The attack performance is weak. Although MCA achieves the best average performance across the 13 settings, its average ROC-AUC is only 0.561 (random baseline at 0.5), with a TPR of 9.575% at 5% FPR. Performance under several experimental settings is close to random guessing.
  2. The performance improvement over existing baselines is marginal. As shown in Table 1, MCA achieves only a 0.017 improvement in macro-average ROC-AUC and a 1.312 percentage-point improvement in TPR@5% FPR over LiRA, the strongest baseline in terms of aggregate performance. Moreover, the evaluation uses only 256 members and 256 non-members per setting. Although the authors report results over three random seeds, the statistical significance of these relatively small improvements remains unclear.
  3. The threat model assumes persistent attacker access to full model states, which limits real-world applicability. In typical online inference services, users can only query model outputs and cannot retrieve internal parameters.
  4. Presentation needs to be improved. For instance, Section 5.4 contains duplicated citation “Li et al. Li et al. (2026)”, and there is a formatting issue with citation placement after ImageNet-C in the dataset description of Section 5.1.
Questions for authors
  1. Could the authors provide more comprehensive statistical analyses, such as confidence intervals or significance tests, to assess the statistical significance of MCA’s improvements over the main baseline? In addition, could the authors enlarge the evaluation candidate pool, particularly the number of non-members, to improve the reliability of low-FPR evaluations?
  2. Could the authors provide concrete real-world deployment scenarios to justify the assumption that an attacker can continuously access intermediate model states?
Reviewer_hKrFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies membership inference against sequential test-time adaptation (TTA).

The authors propose Manifold Calibration Attack (MCA).

The authors evaluate this method on different combinations of datasets, architectures, and adaptation methods.

Critical strengths

The paper deals with the problem of Test Time Adaptation is a worthwhile privacy question.

Critical weaknesses

I found the paper quite difficult to follow. Many quantities and operations are introduced without explanation and the appendix is also very short and not giving a good idea of what is the contribution. I would have expected a more explanations of the algorithm, as well as further implementation details. In its current form, I find it difficult to fully understand and assess the proposed approach.

While the proposed method seems to achieve better results, the improvements are small.

Questions for authors

Could the authors provide a more detailed and mostly accessible explanation of the method, including how its main components are implemented in practice? Sections 3 and 4 are difficult to follow, and the appendix provides limited additional clarification.

Could the authors explain in more detail how the existing membership inference attacks were adapted (if adapted at all)?

The improvements over the baselines seem relatively modest on average. Could the authors comment on why the method works better in some settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.