acceptodds
Desk rejected at ICLR 2027

Image Membership Inference in Fine-Tuned Medical Vision Language Models via Decoding Surfaces

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Membership inference attacks (MIAs) are widely used to audit training-data leakage in fine-tuned medical vision–language models (VLMs). Existing VLM MIAs typically commit to a fixed decoding configuration or score family before evaluation, even though stochastic generation behavior can vary substantially with the inference policy. This restricted view may therefore miss stronger membership evidence elsewhere in the observable response space. We propose the Decoding-Surface Attack (DSA),, a calibrated black-box MIA that models the multi-temperature response surface at each output budget by combining within-policy consistency, cross-temperature geometry, and response behavior. Beyond attack measurement, DSA uses the resulting leakage score to guide surface-access restriction and privacy-aware checkpoint selection, forming an audit-to-mitigation workflow without claiming formal privacy guarantees. Extensive evaluations across medical VQA benchmarks and model families compare DSA with fixed-policy, partial-surface, and recent VLM MIA baselines. DSA achieves 0.811 macro AUC in our primary evaluation, improving over fixed-policy MIA by 25.5 AUC points and the strongest reproduced VLM MIA baseline by 18.1 points. Comprehensive component ablations and same-pool controls further verify the contribution of complementary surface evidence and rule out the tested dataset shortcuts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Membership Inference Attack, Black-box Attacks, Vision-Language Models, Medical Image Privacy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oXVzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper talks about membership inference attacks (MIA) on fine-tuned medical VLMs. The main argument is that how much a model leaks depends on the decoding setting, so auditing with one fixed setting can underestimate the risk. The authors propose DSA, which samples several responses at several temperatures and builds features from three parts: how consistent the responses are, how this consistency changes across temperatures, and simple response statistics. A logistic regression trained on a labeled calibration set gives the final score.

Overall, the paper is clearly written and the observation is interesting, but I am not fully convinced by the evidence for the main claim.

Critical strengths
  • The paper makes a point that is easy to overlook: the decoding setting matters for privacy audits. The best temperature changes with the model and output length, and the direction of the signal can even flip. I can see this being a useful warning for people who audit with one fixed setting.
Critical weaknesses
  • The comparison in the main results is not very fair, and the “decoding surface” part does not seem to be what drives the gain. DSA is a supervised classifier trained with about 390 labeled samples and 30–70 queries per candidate, while most baselines are single scores with 1–5 queries. The “+25.5 over fixed-policy” is measured against an average over all settings, which is a weak reference. To me the fair references are Validated-best (0.631) and Multi-temp LR (0.685). In the ablation, consistency + geometry only reaches 0.685, same as Multi-temp LR, while response statistics alone give 0.782. So most of the gain seems to come from length, uniqueness or entropy, which looks more like a style shift of the fine-tuned model than a property of the decoding surface.

  • I am not sure the signal is membership and not dataset artifacts. In the same-pool control, DSA drops from 0.915 to 0.690 on Qwen and from 0.706 to 0.626 on LLaVA, and this control is only run for one setting. The main tables and PMC-VQA are not under this protocol.

Questions for authors
  • How does DSA compare to a classifier with the same supervision on concatenated single-score features at the same query budget? This would help separate the benefit of the surface idea from supervision and more queries.
  • How much do the geometry features add beyond per-temperature features?
  • Could the same-pool control be run for all eight settings and PMC-VQA? And how were near-duplicate images handled?
Reviewer_1cHGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DSA, a black-box image membership inference attack on fine-tuned medical vision–language models. At each fixed output budget, it combines response consistency, cross-temperature statistics, and response behavior, using labeled calibration data to learn a membership score. Experiments report stronger attacks than the evaluated baselines and explore audit-guided decoding restrictions and checkpoint selection.

Critical strengths
  1. DSA treats decoding behavior as an observable source of membership evidence and combines complementary response measurements without requiring model internals. Its use of generated text and a simple calibration model provides a transparent approach to auditing systems where logits and training losses are unavailable.
  2. The method improves attack performance across the evaluated model families and is supported by an external benchmark, component ablations, and query-matched comparisons. The same-pool controls provide additional evidence of fine-tuning-related membership signal, giving the proposed audit a substantive empirical foundation.
Critical weaknesses
  1. The main comparative advantage has not been established under the stricter same-pool membership protocol. Although these controls demonstrate a surviving membership signal, they yield weaker attack performance and do not repeat the comparisons with competing MIAs. Because target training and query budgets also change, the current evidence does not establish whether the large relative gains reported in the primary evaluation persist when source-pool differences and other evaluation conditions are controlled more tightly.
  2. The specific contribution of multi-temperature modeling remains insufficiently isolated from the strength of the response-behavior features. The ablations show that response statistics account for much of the attack performance, and the low-query experiments demonstrate gains even without observing a full decoding surface. These findings leave open how much additional value comes from distributing a fixed query budget across temperatures, compared with a calibrated single-temperature attack using equally rich response features. This distinction is central to assessing the proposed methodological contribution.
Questions for authors
  1. Could the authors repeat the comparisons with the strongest competing attacks under the same-pool protocol, using the same target models and matched query budgets? This would clarify whether DSA retains its relative advantage under the stricter membership construction.
  2. How does DSA compare with a calibration-selected single-temperature attack using response-behavior and consistency features, the same calibration procedure, and the same total number of generations? This comparison would help isolate the additional benefit of multi-temperature modeling.
  3. Do the comparative conclusions remain consistent across independently fine-tuned target models and at low false-positive operating points, particularly under the same-pool protocol?
Reviewer_LqfoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates membership inference attacks (MIAs) against fine-tuned medical vision-language models (VLMs). The authors propose Decoding-Surface Attack (DSA), which exploits variations in model-generated responses across different decoding temperatures. The method extracts response consistency, cross-temperature statistics, and response-level behavioral features, and combines them using a logistic regression classifier trained on labeled member and non-member samples.

Critical strengths

Exploration of decoding configurations. The paper investigates how different decoding temperatures affect membership inference performance, highlighting a relatively under-explored dimension of privacy leakage in VLMs.

Combination of behavioral signals. The proposed method combines response consistency, cross-temperature statistics, and response-level features to improve membership inference performance.

Critical weaknesses

Strong assumptions about the attacker's prior knowledge. The proposed attack assumes that the adversary has access to a subset of the target model's fine-tuning data, a set of held-out in-distribution non-member samples, and black-box query access to the target model. Access to known members of the target fine-tuning dataset is not a standard assumption in the MIA literature. The authors should justify this assumption and evaluate DSA without labeled calibration data.

Missing important experimental details. The paper is missing important experiment setup, such as the size of the fine-tuning datasets, how member and non-member images are constructed, and how is the adversary's fine-tuning data constructed.

Experiment setup The paper reports weaker attack performance in the same-pool evaluation (Table 7) than in the main evaluation (Table 2). Since controlling for distributional differences between members and non-members is essential for a valid MIA evaluation, the same-pool construction should be adopted as the main experimental setup rather than treated as an additional validity control.

Questions for authors
  • Could the authors justify the assumption that the attacker has access to known training members and discuss whether this assumption is supported by prior MIA literature?
  • Do the compared VLM MIA baselines assume access to known members of the target training set? Could the authors clarify the differences in attacker knowledge assumptions and evaluate DSA under a setting without target-specific labeled calibration data?
  • Can the author report Table 2 in the same-pool experiment setup?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.