acceptodds
Under review as a conference paper at ICLR 2027

Black-Box Membership Inference Attack for Vision-Language Models Using Self-Memory Probing

Abstract

The rapid advancement of vision-language models (VLMs) increasingly depends on large-scale visual corpora, raising concerns about privacy risks and copyright compliance. Membership inference attacks (MIAs) provide a means to audit whether individual samples were used for training. However, existing VLM MIAs either require white-box or gray-box access to logits and token probabilities or rely on output-only proxies derived from task-specific visual interventions or surrogate models. These access and design dependencies limit their applicability to commercial VLMs deployed under the black-box Generation-as-a-Service (GaaS) paradigm, which exposes only final outputs. To address this gap, we propose Self-Memory Probing (SMP), an output-only black-box framework grounded in the self-memory effect. This effect manifests as greater consistency and concentration in responses to member images than in those to non-members. SMP probes this distinction by eliciting multiple outputs with semantically equivalent prompts to construct candidate-conditioned and non-member reference distributions. It then casts membership inference as a hypothesis test and uses Maximum Mean Discrepancy (MMD) to quantify the discrepancy between these distributions. Consequently, SMP derives membership evidence directly from the model’s native responses, without image perturbations, surrogate models, or token-level statistics.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Membership Inference Attack, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YhdYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a Membership Inference Attack for VLMs based on self-memory probe assumption, which is that the distances between two output distributions obtained using a member and non-member images will be larger than those obtained using two non-member images. The distance is measured using Maximal Mean Discrepancy and kernel tricks. The outputs are samples using a pool of textual prompts that aim to extract textual information from the given image. Since this is black-box output based technique, authors evaluate it on open-weight VLMs as well as online proprietary VLMs, with successful results.

Critical strengths
  • SMP assumption itself is intuitive and easy to understand.
  • I think it is nice that MIA can be done under black-box setting, and the evaluation of various kernels is also a helpful contribution to the field.
Critical weaknesses
  • The most critical weakness to me is that the evaluation framework is not clear. I may be missing something, but to perform and evaluate the MIA outlined in the paper, it is crucial that we have access to member and non-member images. Section 4.1 describes datasets, half of whose images are after specific cut-off dates, i.e., non-members. However, this also must mean that the other halves are member images for the models studied. However, there is no such explicit guarantee. VL-MIA/DALL-E (DALL-E) is said to contain images from "LAION CCS subset used for VLM pretraining", but is it true for all models studied, i.e., MiniGPT-4 Zhu et al. (2024), LLaVA-1.5 Liu et al. (2023b), and LLaMA-Adapter v2 Gao et al. (2023)? Without this, I think the whole evaluation becomes meaningless.

  • I am also slightly concerned whether non-members synthesized using DALL-E-2 can leak some membership indirectly.

Questions for authors
  • Are all the datasets described in Section 4.1 contain member images for all three studied models? This is not clear from the writing, but I think the whole paper depends on this.

  • In some sense, the paper is a validation of SMP assumption. I do not think this automatically connects to realistic use case. If we are to use this to investigate whether any of my own images have been used in training, what is the minimum input requirement? Am I right to to think that I need 1) my image and 2) a guaranteed non-member image?

Reviewer_M6wdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method, Self-Memory Probing (SMP), for output-only black-box membership inference attacks against vision-language models. Unlike existing white-box or gray-box MIAs against VLMs requiring access to internal model states, or black-box methods relying on image perturbations or surrogate models, SMP conducts membership inference under a strict black-box setting, namely only relying on model outputs without modifying the image or using surrogated-based score estimation. In order to achieve this, the paper proposes and utilizes self-memory effect: VLM’s responses tend to exhibit greater consistency and concentration against member images than non-members. Through obtaining multiple outputs with semantically equivalent prompts, SMP constructs candidate-conditioned and non-member reference distributions, and then uses Maximum Mean Discrepancy (MMD) with IDF-weighted Hamming kernel to quantify the discrepancy between these distributions. To evaluate the effectiveness and robustness of the proposed MIA framework, the paper conducts comprehensive comparison with existing SOTAs on both open-source and closed-source commercial VLMs. The experiment results show that SMP outperforms prior works and achieves significant improvement in TPR at low FPR.

Critical strengths
  1. A novel black-box MIA framework. By utilizing discrepancy of consistency and concentration in VLMs’ responses to member images and non-members, SMP identifies membership of a candidate effectively.
  2. Comprehensive comparison with existing SOTAs. The paper compares SMP with ten SOTAs in the evaluation, including six white-box/gray-box methods and four black-box methods, showing its outperformance, especially on TPR at low FPR, over prior works.
Critical weaknesses

Major Weaknesses:

  1. The practical necessity of performing output-only black-box MIA without image perturbations is unclear. For the motivation, the paper emphasizes the black-box Generation-as-a-service (GaaS) paradigm motivates MIA without accessing internal model states while existing black-box methods remain dependent on image perturbations or surrogate-based score estimation. While the motivation for output-only and black-box access is clear under the realistic GaaS setting, it is unclear why performing MIA on unmodified images represents an important gap (Section 1: direct membership inference from … elicited by unmodified images largely unexplored … To address this gap …). In what realistic scenarios would an attacker need to conduct output-only black-box MIA without modifying the input images?
  2. Some critical technical details are missing. The paper does not clearly explain how the non-member reference is selected. Since SMP relies on comparing the discrepancy of consistency and concentration in the VLM’s responses to the candidate against the responses to the non-member reference, the lack of clear reference selection criteria raises concerns. Throughout the paper, the non-member reference appears to be directly obtained. Similarly, Appendix D.3 assumes the non-member reference pool is directly available without detailing how this pool is constructed. More importantly, the last sentence of Appendix D.3 (‘This protocol measures robustness to … rather than robustness to distribution shifts across different reference sources.’) suggests that the robustness of SMP across different reference sources remains unclear. Please clarify whether there is a specific requirement for non-member references and provide robustness analysis under different reference sources.
  3. Some experimental settings are unclear. a) Configuration details of several existing works are missing. Some SOTAs involve hyperparameters or propose multiple attack variants, but the paper does not specify the exact settings or strategies used in the evaluation. For example, the values of K in Min-K%, K and α in MaxRényi-K%, α in ModRényi, and the specific LOMIA strategy used (Text-Text Feature Attack, Image-Text Feature Attack, and Dual Feature Attack) are not reported. Please clarify those configurations used for each baseline and if applicable, please also provide comparisons with those SOTAs under different settings or variants. b) The paper should also clarify the number of probing prompts used in the experiment. Appendix B only states ‘Each experiment uses only the required number of probing prompts’, which is not enough. And Section 4.4 also has the similar issue. Only the size of the pool is reported, but the exact size of the subset is unclear. Please improve the clarity to address above issues.

Minor Weaknesses:

  1. For the presentation, the font sizes in some figures are too small, particularly in Figure 2, Figure 5, Figure 7 and Figure 8, making them difficult to read.
  2. In Section 4.1, ‘… and further investigate factors that influence its effectiveness.All our experiments …’ should be corrected to ‘… and further investigate factors that influence its effectiveness. All our experiments …’.
Questions for authors

See weaknesses.

Reviewer_cHx2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work propose a black-box membership inference attack for VLMs. Based on responses distribution from multiple calibrated input prompts the method called self-memory probing aim to distinguish images of members from nonmembers. Authors present results, which show that for all tested approaches, their blackbox method is much better even than existing white box methods.

Critical strengths
  1. Tackling black-box MIA problem, and showing promising results compared to other, rather scarce black-box MIA methods.
  2. Although (in my opinion) not exactly clearly explained why, the Hamming kernel is the full and only insight that takes full responsibility for gains of the proposed method.
Critical weaknesses

W1. Authors do not explain the crucial part in MIA attacks, which is the dataset member vs non-member distributions.

This is a problem, since with no blind baseline (no target model, only classification to distinguish members/nonmembers) there is no way to tell if benchmarks do not contain a problem with distinguishing rather distribution shift than membership signal.

The information in this aspect is very lacking - e.g. Authors propose generating nonmembers without even a word of discussion that clearly they should mitigate for distribution shift between real and generated images, or images from around 2014 and 2024 (time split), or images from LAION2B and Flickr (own preposition of benchmark created by the Authors). There is a comment on following methodology of another work, but details are needed on how the authors went with deduplication/ distribution correction and in general distribution shift problem clearly existing in the used benchmarks.

W2 Authors put a lot of work to show that they use more than one prompt (which is a novelty part of the work if I am not mistaken): fig 2 on robustness and consistency, and a lot of space in the main text - but there is a gap why is it important, what is the motivation etc., it looks as assumption in 191 line, without literature backing, and the only figure showing this if later on figure 3, which has some problems - a) x axis looks like using cherry-picked values only 1, 5, 7, 8, 9, 10, 15, 20, b) title variation above each panel is misleading c) AUC results are close to random for everything but Hamming. d) after 10 prompts the performance is worse which is rather unexpected and not explained properly. e) why 10?

Other:

  • the flow of the paper was a bit hard to grasp for me - what is motivation, why this way, what other approaches may work and why are not better than this one, what is important, what is rather secondary in the method etc.
Questions for authors

See weaknesses for most important issues - W1. Then:

  1. Why 10? why it deteriorates (fig 3)
  2. Would the method work for LLMs, not only VLMs?
  3. Explanation for Hamming success, is there any other kernel that works?
  4. Why black-box method is better than any existing white-box in all tested cases?
  5. Blind baseline to be added.
  6. Results at 1% FPR to be added.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.