- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents M-Drift, a white-box membership inference method for vision-language and diffusion models. The method applies a single gradient-ascent parameter update and measures the resulting changes in output loss, internal representations, and cross-modal attention. Sparse autoencoders provide additional feature-level drift statistics, while diffusion models aggregate signals across sampled timesteps and noise realizations. A calibrated classifier combines these signals for instance-level and dataset-level auditing. Experiments on LLaVA, Qwen2-VL, Stable Diffusion, and DiT report improvements over several membership inference baselines, with additional ablations and localization analyses.
Critical strengths
1.The most convincing evidence is Appendix F.2, Table 10: under matched white-box access and scoring, M-Drift improves TPR@1%FPR from 84.6% to 94.9% on LLaVA and from 74.2% to 88.0% on Stable Diffusion relative to combined static features. This directly supports the utility of measuring post-intervention drift.
- The authors explicitly define controlled-continuation membership where historical training records are unavailable (Section 5.1 and Appendix E.1). The low-FPR uncertainty analysis (Table 11), component and parameter ablations (Table 12), and localization tests (Table 13) strengthen the empirical case. These experiments provide useful support for the proposed method.
Critical weaknesses
-
The scope of the membership evidence remains limited. For three of the four backbones, membership ground truth concerns controlled continuation training, not membership in the original pretraining corpus. The authors appropriately acknowledge this distinction. Nevertheless, clearer continuation-training details and an analysis of sensitivity to exposure frequency or calibration shift would help assess the method's applicability beyond the evaluated setting. This primarily limits the generality of the auditing conclusions.
-
Improvements over the strongest competing attacks are uneven. In Tables 1–2, the TPR@1%FPR advantage is only 0.1 percentage points over MaxR on LLaVA, although the gains are larger on Qwen and remain positive for the diffusion models. Table 11 provides uncertainty estimates for M-Drift, but paired uncertainty for the improvements over the strongest baselines would better establish their practical significance. End-to-end computational costs would also help contextualize these gains.
Questions for authors
1.Could the authors clarify the controlled continuation-training schedule and per-sample exposure frequency, and discuss the expected applicability to lower-exposure or shifted-calibration settings?
2.Could the authors report paired uncertainty for TPR@1%FPR improvements against MaxR/GSA and provide end-to-end inference costs, including the eight diffusion probing trials?
4 comments
Thanks again for the detailed review. We have revised the manuscript (see updated file), with new text highlighted in yellow, in line with your points:
- Scope of the membership claim (W1): The abstract, Sec. 5.1 and the Conclusion now limit our claims to the controlled-continuation labels (plus the exact DiT/ImageNet split) and do not claim membership in the original pretraining corpus. App. E.1 and App. H say the same, and that the blind controls apply only to the secondary diagnostic.
- Continuation schedule and exposure (Q1): App. E.1 and Table R5 now list the epochs, optimizer, learning rate, batch size and exposure per member for each backbone.
- Uneven gains and uncertainty (W2/Q2): Sec. 5.2 now states plainly that M-Drift ties MaxR on LLaVA at 1% FPR (94.9 vs. 94.8), where only the 0.1% FPR result separates them (80.2 vs. 71.8). It also says Qwen's gain is clearly significant, since MaxR's 84.1 lies outside our CI [87.4, 92.2], whereas the GSA margins on diffusion lie inside our CIs, so we do not claim significance there. Ties in Tables 1–2 are bolded jointly.
- Our central claim: The matched comparison against static white-box features (84.6 → 94.9 on LLaVA, 74.2 → 88.0 on SDv1.5) is now presented as the main evidence (Table 3, Sec. 5.3).
- Cost (Q2): Sec. 6 and Table R6 give the per-candidate costs, including the K = 8 diffusion trials.
There are some minor updates we will include them in the camera-ready version.
We hope the narrowed claims and added details resolve your concerns. If they do, we would be grateful if you would reconsider your score.
I appreciate the authors' detailed response and clarifications. However, some of my concerns remain unresolved, so I maintain my original rating.
Thank you for the follow-up, and for acknowledging the matched white-box comparison and the controlled-membership protocol.
These are the changes in the revised manuscript:
- Membership scope: the abstract, Sec. 5.1 and the Conclusion now limit the claims to verified continuation-training events, and do not claim membership in the original pretraining corpus.
- Continuation schedule and exposure: App. E.1, Table R5.
- Uneven gains: we now report the LLaVA 1%-FPR result against MaxR as a tie. We claim a significant gain only for Qwen, where MaxR lies outside M-Drift's 95% CI (Sec. 5.2).
- Cost: operation counts, including all K diffusion trials (Sec. 6, App. F.1, Table R6).
The central claim does not depend on uniform dominance over every baseline. It is that intervention-induced drift adds membership evidence beyond matched static white-box features: TPR@1%FPR rises from 84.6 to 94.9 on LLaVA and from 74.2 to 88.0 on SDv1.5 (Table 10).
We are committed to include any remaining references in final version. If any specific concern still seems unresolved, we would be grateful if you named it, so we can answer it directly. Thank you again for your time.
We thank the reviewer for recognizing our controlled-membership protocol and matched white-box comparison (Appendix F.2, Table 10). Following, we provide the response regarding generalization, paired uncertainty, and computational cost. We will update these in the manuscript and supplements.
W1: Membership scope and generalization. We explicitly distinguish controlled-continuation membership from unverified inclusion in historical pretraining. LLaVA, Qwen2-VL, and SDv1.5 use exact inclusion/exclusion labels for training continued from released checkpoints; DiT/ImageNet uses the known train/validation split (Sec. 5.1, App. E.1). We do not claim that a continuation non-member was absent from the original pretraining corpus. Source matching, deduplication, and blind temporal/metadata controls (Table 8) mitigate confounding, but cannot establish robustness to arbitrary calibration shift.
Table R4: Current Audit Scope and Future Work.
According to Table R4, we will narrow the abstract and conclusions to verified training events and distinguish present evidence from future generalization.
Q1: Scope and schedule. Labels are exact for a controlled continuation event (LLaVA, Qwen2-VL, SDv1.5) or the official split (DiT/ImageNet). We do not claim membership in the original pretraining corpus, and we will narrow the abstract and conclusions accordingly. Schedule details (to be added to App. E.1):
Table R5: Controlled continuation-training schedule and per-sample exposure.
As mentioned in Table R5, members are shuffled each epoch and seen exactly once per epoch; non-members are never used.
Lower exposure is expected to weaken all MIAs. We will add an exposure sweep (members seen 1/2/4 times) and a calibration-shift test (calibrator fitted on one source pool, audit on another), keeping the protocol fixed. Table 8 controls apply only to the secondary proxy-provenance diagnostic; they do not establish robustness to arbitrary shift.
W2/Q2: Uneven gains and uncertainty. On LLaVA at 1% FPR, M-Drift vs MaxR (94.9 vs 94.8) is a tie, and we will say so. Using Table 11, 95% bootstrap CIs:
Our central claim rests on the matched-access comparison (Table 10): drift improves over combined static white-box features, 84.6 → 94.9 (LLaVA) and 74.2 → 88.0 (SDv1.5). This isolates the effect of the intervention rather than implying uniform dominance.
Runtime. M-Drift costs ≈ 4 forward-equivalents for VLMs and K×(2F+1B) for diffusion with K=8 (K=4: 85.8 TPR@1%FPR). We will add measured V100 FP16, batch-1 end-to-end latency, and peak memory for M-Drift and MaxR/GSA, including the update/restore, readouts, and all K trials.
Summary: We will incorporate tables, schedule details, paired inference, and exposure/shift sensitivity into the rebuttal revision. The defensible contribution is that intervention-induced multimodal drift adds membership evidence beyond matched static white-box features, not that every setting yields a large margin or that continuation labels prove historical-pretraining membership.