- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies membership inference (MI) against text-to-video diffusion models. It makes three contributions: 1) A distribution-matched non-member set for VideoUFO. The authors re-run the VideoUFO collection pipeline on videos uploaded after June 2025, then pair the new clips with members by Hungarian matching. They report blind baselines on the result: the best blind family reaches 0.639 AUROC, compared with 0.999 on a cross-corpus split. 2) Adaptations of image and classifier MI attacks to T2V across black-box, gray-box and white-box access: loss, PIA, per-block gradient norms, representation read-out and VidLeaks. Results are scored with a “blind-adjusted AUROC,” which residualizes each attack score on the blind features. 3) The Reversal Gap (TRG), a gray-box attack. It scores the normalized loss difference between a clip and its time-reversed copy under shared noise, so per-clip difficulty cancels. The headline result is a supervised representation read-out at 0.717 AUROC (0.692 adjusted). TRG reaches 0.608 (0.630 adjusted), and a reference model trained on disjoint data reaches 0.587 adjusted. Overall, this is an interesting paper, working on membership inference attacks for video diffusion models, which proposes a simple but effective method.
Critical strengths
-
The evaluation methodology is the main contribution, and it is done carefully. The paper reports blind baselines before attacks and holds out a feature family (spectral) from the matching. It also runs cross-split transfer and two kinds of model control: a disjoint-data reference model and random-init weights. Few MI papers do this much.
-
TRG has clear intuition, and D.1/D.2 show it removes most of the per-clip difficulty component. Within-pair correlation drops from 0.77 to 0.24, and TRG gains nothing from reference calibration.
-
The ablations are solid: depth, timestep, frames, K, attack-model capacity, sample size, captions and re-encoding. According to the experimental results, the proposed method shows the effectiveness.
-
Overall, this paper is easy to follow and well written.
Critical weaknesses
-
The compute and averaging comparison is confounded. TRG uses 32 queries, averaged over K=8 noise draws and W windows. The loss baseline uses 3 queries and a single shared noise draw. D.3 shows that averaging alone lifts TRG from 0.628 to 0.675. To isolate the effect of the reversal contrast, the plain loss needs the same K×W averaging and an equal query budget.
-
Numbers are inconsistent across sections. Denoising loss: 0.514 in Table 1 versus 0.533 in D.2. TRG: Table 1 reports 0.608 under the stated K=8 protocol, but D.3 reports 0.675 at K=8, plus 0.029 more from a second window. D.3 also gives an extrapolated ceiling of 0.684 and a variance-decomposition bound of 0.728, which need reconciling with each other. Read-out: 0.717 in Tables 1 and 3, versus 0.716 with all timesteps and blocks and 0.710 at 500 pairs in D.3. These are small differences, but they suggest different runs or subsets. TRG query budget: the 32 queries are not decomposed into K, W, timesteps and directions.
-
Citations: CogVideo (Hong et al., 2022) is autoregressive, not a diffusion transformer. Kapoor et al. (2025) is a general adversarial-attacks survey and is a weak citation for “video MI is nascent.” Several works are cited under both architecture families in Section 2.1 (Wang 2025b, Chen 2023, Blattmann 2023). “Mira Ju et al. (2024)” has formatting errors.
Questions for authors
- Can you reconcile the TRG numbers (0.608 vs 0.675 at K=8) and the denoising-loss numbers (0.514 vs 0.533)?
- What does the plain loss achieve under the same K×W averaging and query budget as TRG?
- How do SecMI and CLiD perform on the matched split?
2 comments
Polite nudge
We once again thank the reviewer for their thoughts. Since the rebuttal period is nearing its end, we would really appreciate it if the reviewer would review our responses, which we believe have addressed all of their concerns, as outlined in the rebuttal above and included in the revised PDF. Thanks so much in advance!
Response to kDR3
We thank the reviewer for their thoughtful feedback and kind words, including that the evaluation methodology "is done carefully", that "few MI papers do this much", and that the paper "is easy to follow and well written".
We found the suggestions incredibly helpful and ran all requested experiments: (1) a compute-matched comparison with loss-based MIA, (2) a full reconciliation of the reported results, and (3) SecMI and CLiD on the matched split. We summarize the key findings below; the details have already been reflected in the revised manuscript.
"The plain loss needs the same K×W averaging and an equal query budget" (W1, Q2)
This is a great point. We reran the denoising loss using identical timesteps, windows, and noise draws as the reversal gap, with up to 32 forward queries. We found that the loss does not improve with averaging (0.532-0.533, 4 -> 32 queries). With equivalent query budgets, the reversal gap is better by +0.045 [0.005, 0.082] at 8 queries, +0.062 [0.022, 0.100] at 16 queries, and +0.075 [0.036, 0.111] at 32 queries (see main response, point #3). Therefore, its advantage does NOT come from the number of queries, but from the contrast with the time-reversed clip. We added the curve of AUROC against the number of queries for both to App. D.3.
"Can you reconcile the TRG numbers (0.608 vs 0.675 at K=8) and the denoising-loss numbers (0.514 vs 0.533)?" (W2, Q1)
We thank the reviewer for noticing this! We traced all results back to the experiments that produced them, using our W&B logs, and added a table to the appendix that maps experiments to utilized subsets and settings (see main response, point #4).
Citations (W3)
We thank the reviewer for catching these and apologize for the oversight! All of them have been fixed in the revised manuscript: CogVideo (Hong et al., 2022) is no longer grouped with the diffusion transformers, Kapoor et al. (2025) has been replaced with a citation specific to video MI, the double listings in Sec. 2.1 (Wang 2025b, Chen 2023, Blattmann 2023) have been removed, and the Ju et al. (2024) entry has been corrected.
"How do SecMI and CLiD perform on the matched split?" (Q3)
This is an excellent question and we thank the reviewer for the suggestion! We adapted both methods for T2V and, for a fair comparison, used the same preprocessing, windows, and seeded noise as for the reversal gap. SecMI reaches 0.476 [0.455, 0.496] and CLiD 0.539 [0.505, 0.574]. Both hence score below the reversal gap at 0.608 [0.576, 0.639], even though CLiD uses twice as many queries (64 vs. 32). Both baselines are now included in the revised manuscript for more comprehensive evaluation (see main response, point #6).
Ethics flag
We thank the reviewer for raising this! While we believe there are many important and ethical reasons to pursue MI (e.g., transparency and safety of training data), we agree that it could also be misused to reveal whether a specific person’s video was used to train a model. We, thus, added an ethics statement on dual use to the paper. To further mitigate this risk, we release the non-member set and model weights only under gated access (with reviewed requests), together with video identifiers, URLs, and the collection metadata.
If the reviewer’s concerns have been addressed, we would appreciate it if they considered increasing their score.