- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a Membership Inference Attack for VLMs based on self-memory probe assumption, which is that the distances between two output distributions obtained using a member and non-member images will be larger than those obtained using two non-member images. The distance is measured using Maximal Mean Discrepancy and kernel tricks. The outputs are samples using a pool of textual prompts that aim to extract textual information from the given image. Since this is black-box output based technique, authors evaluate it on open-weight VLMs as well as online proprietary VLMs, with successful results.
Critical strengths
- SMP assumption itself is intuitive and easy to understand.
- I think it is nice that MIA can be done under black-box setting, and the evaluation of various kernels is also a helpful contribution to the field.
Critical weaknesses
-
The most critical weakness to me is that the evaluation framework is not clear. I may be missing something, but to perform and evaluate the MIA outlined in the paper, it is crucial that we have access to member and non-member images. Section 4.1 describes datasets, half of whose images are after specific cut-off dates, i.e., non-members. However, this also must mean that the other halves are member images for the models studied. However, there is no such explicit guarantee. VL-MIA/DALL-E (DALL-E) is said to contain images from "LAION CCS subset used for VLM pretraining", but is it true for all models studied, i.e., MiniGPT-4 Zhu et al. (2024), LLaVA-1.5 Liu et al. (2023b), and LLaMA-Adapter v2 Gao et al. (2023)? Without this, I think the whole evaluation becomes meaningless.
-
I am also slightly concerned whether non-members synthesized using DALL-E-2 can leak some membership indirectly.
Questions for authors
-
Are all the datasets described in Section 4.1 contain member images for all three studied models? This is not clear from the writing, but I think the whole paper depends on this.
-
In some sense, the paper is a validation of SMP assumption. I do not think this automatically connects to realistic use case. If we are to use this to investigate whether any of my own images have been used in training, what is the minimum input requirement? Am I right to to think that I need 1) my image and 2) a guaranteed non-member image?