- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper treats visually tampered segments in videos of people speaking not as the classification of independent short segments, but as the structural learning of a sequence of segment labels covering the entire video. Candidate segments are scored using VideoMAE/LoRA, and learning is performed using the distribution function of all candidate paths and the path that matches the ground-truth label. During inference only, a prior distribution derived from Whisper’s word timestamps is incorporated, and tampered segments are output based on frame-level marginal probabilities. In the ArEnAV evaluation of 212 videos, compared to independently trained models using the same structured decoder, AP75 improved by 18.37 points with Diff2Lip training and by 17.36 points with LatentSync training.
Critical strengths
- A comparative design that separates the effects of learning and inference is preferable. The independent learning models are also provided with a shared decoder and a speech prior distribution, and a comparison is included where calibration was performed only on the source side. This makes it easier to verify the effectiveness of structural learning than a comparison where advanced post-processing is added solely to the proposed method.
- The improvements over the main baselines are significant and appear across multiple metrics. In Table 1 (p. 6), under both training conditions, the model outperforms the independent learning model with a shared decoder not only in AP50 and AP75 but also in frame AP and AR50. It is encouraging that these improvements are not limited to a single metric.
Critical weaknesses
- It is difficult to verify the independence between the training data’s provenance and the evaluation results. While the appendix states that past training included LAV-DF material, the LAV-DF results are presented as a transfer evaluation. It is necessary to explicitly state the segmentation used, the lack of overlap between parent videos, and the training history at each checkpoint. This does not imply that data leakage has been confirmed; rather, the issue is that readers cannot definitively determine the meaning of the transfer evaluation.
- The effects of key design elements are not sufficiently disentangled. There are no results from a direct comparison on the main evaluation set between U+S, which uses an audio prior distribution, and U+N, which uses a neutral prior distribution. Furthermore, in the comparison at 1,000 updates, U-max sometimes outperforms U in interval AP; while LatentSync’s AP75 improves from 13.63 to 22.06, its frame AP decreases. While this does not negate structural learning itself, further explanation is needed regarding the benefits of a design that sums over adaptation paths. A comparison at 4,000 updates—the same as in the main results—would also be desirable.
- There are concerns regarding the consistency and reproducibility of the baseline. For the “J native” model under the “Diff2Lip” condition in Table 1, the AP50 and AP75 values are 21.41 and 5.70, respectively, whereas under the “clean” condition in Table 4, they are 19.99 and 6.17. An explanation is needed as to whether this discrepancy is due to differences in checkpoints, training conditions, or evaluation methods.
Questions for authors
- Could you provide the training history for each major checkpoint, the number and composition of source-fit, modern-fit, and source-hold instances, and clearly indicate the lack of overlap at the parent-video level with the LAV-DF evaluation videos?
- Could you define the J independent training loss mathematically? Specifically, how are the teacher labels and weights for candidate intervals that straddle the truth boundary determined? Also, what is the reason for the difference in the “clean” values for J native in Tables 1 and 4?