- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper talks about membership inference attacks (MIA) on fine-tuned medical VLMs. The main argument is that how much a model leaks depends on the decoding setting, so auditing with one fixed setting can underestimate the risk. The authors propose DSA, which samples several responses at several temperatures and builds features from three parts: how consistent the responses are, how this consistency changes across temperatures, and simple response statistics. A logistic regression trained on a labeled calibration set gives the final score.
Overall, the paper is clearly written and the observation is interesting, but I am not fully convinced by the evidence for the main claim.
Critical strengths
- The paper makes a point that is easy to overlook: the decoding setting matters for privacy audits. The best temperature changes with the model and output length, and the direction of the signal can even flip. I can see this being a useful warning for people who audit with one fixed setting.
Critical weaknesses
-
The comparison in the main results is not very fair, and the “decoding surface” part does not seem to be what drives the gain. DSA is a supervised classifier trained with about 390 labeled samples and 30–70 queries per candidate, while most baselines are single scores with 1–5 queries. The “+25.5 over fixed-policy” is measured against an average over all settings, which is a weak reference. To me the fair references are Validated-best (0.631) and Multi-temp LR (0.685). In the ablation, consistency + geometry only reaches 0.685, same as Multi-temp LR, while response statistics alone give 0.782. So most of the gain seems to come from length, uniqueness or entropy, which looks more like a style shift of the fine-tuned model than a property of the decoding surface.
-
I am not sure the signal is membership and not dataset artifacts. In the same-pool control, DSA drops from 0.915 to 0.690 on Qwen and from 0.706 to 0.626 on LLaVA, and this control is only run for one setting. The main tables and PMC-VQA are not under this protocol.
Questions for authors
- How does DSA compare to a classifier with the same supervision on concatenated single-score features at the same query budget? This would help separate the benefit of the surface idea from supervision and more queries.
- How much do the geometry features add beyond per-temperature features?
- Could the same-pool control be run for all eight settings and PMC-VQA? And how were near-duplicate images handled?