acceptodds
Under review as a conference paper at ICLR 2027

Beyond Isolated Clips: Structured Learning for Temporal Forgery Localization

Abstract

Temporal forgery localization must identify the extent of a manipulated interval, not only assign suspicious scores to short clips. We study speech-assisted visual localization in talking-person video, where the input is one audio-video clip and the output is a set of visual manipulation intervals on its original timeline. Our structured localizer scores a common graph of 25-Hz candidate segments, trains the scores through a globally normalized distribution over complete temporal labelings, and uses an input-derived speech prior only at inference. This separates the learning objective from the choice of speech-guided readout. On a locked 212-video ArEnAV preview, the resulting U+S system reaches 46.59/33.76/2.65 AP at IoU 0.50/0.75/0.95 for Diff2Lip-trained checkpoints and 34.03/23.91/2.62 for LatentSync-trained checkpoints. A source-calibrated independent model given the same decoder reaches 28.42/15.39/0.10 and 26.87/6.55/0.08, respectively; the AP75 gains are 18.37 and 17.36 percentage points. Cross-generator direction rows and a strict visual LAV-DF transfer evaluation provide complementary evidence, while a matched U-max control exposes a segment-versus-frame ranking trade-off. Together, these results support globally normalized segment learning as a practical complement to independent clip scoring under the fixed protocol, with each evaluation scope reported separately.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
temporal forgery localization, audiovisual forgery detection, structured prediction, segmental models, speech-guided inference, dynamic programming

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LA9zOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper treats visually tampered segments in videos of people speaking not as the classification of independent short segments, but as the structural learning of a sequence of segment labels covering the entire video. Candidate segments are scored using VideoMAE/LoRA, and learning is performed using the distribution function of all candidate paths and the path that matches the ground-truth label. During inference only, a prior distribution derived from Whisper’s word timestamps is incorporated, and tampered segments are output based on frame-level marginal probabilities. In the ArEnAV evaluation of 212 videos, compared to independently trained models using the same structured decoder, AP75 improved by 18.37 points with Diff2Lip training and by 17.36 points with LatentSync training.

Critical strengths
  • A comparative design that separates the effects of learning and inference is preferable. The independent learning models are also provided with a shared decoder and a speech prior distribution, and a comparison is included where calibration was performed only on the source side. This makes it easier to verify the effectiveness of structural learning than a comparison where advanced post-processing is added solely to the proposed method.
  • The improvements over the main baselines are significant and appear across multiple metrics. In Table 1 (p. 6), under both training conditions, the model outperforms the independent learning model with a shared decoder not only in AP50 and AP75 but also in frame AP and AR50. It is encouraging that these improvements are not limited to a single metric.
Critical weaknesses
  • It is difficult to verify the independence between the training data’s provenance and the evaluation results. While the appendix states that past training included LAV-DF material, the LAV-DF results are presented as a transfer evaluation. It is necessary to explicitly state the segmentation used, the lack of overlap between parent videos, and the training history at each checkpoint. This does not imply that data leakage has been confirmed; rather, the issue is that readers cannot definitively determine the meaning of the transfer evaluation.
  • The effects of key design elements are not sufficiently disentangled. There are no results from a direct comparison on the main evaluation set between U+S, which uses an audio prior distribution, and U+N, which uses a neutral prior distribution. Furthermore, in the comparison at 1,000 updates, U-max sometimes outperforms U in interval AP; while LatentSync’s AP75 improves from 13.63 to 22.06, its frame AP decreases. While this does not negate structural learning itself, further explanation is needed regarding the benefits of a design that sums over adaptation paths. A comparison at 4,000 updates—the same as in the main results—would also be desirable.
  • There are concerns regarding the consistency and reproducibility of the baseline. For the “J native” model under the “Diff2Lip” condition in Table 1, the AP50 and AP75 values are 21.41 and 5.70, respectively, whereas under the “clean” condition in Table 4, they are 19.99 and 6.17. An explanation is needed as to whether this discrepancy is due to differences in checkpoints, training conditions, or evaluation methods.
Questions for authors
  • Could you provide the training history for each major checkpoint, the number and composition of source-fit, modern-fit, and source-hold instances, and clearly indicate the lack of overlap at the parent-video level with the LAV-DF evaluation videos?
  • Could you define the J independent training loss mathematically? Specifically, how are the teacher labels and weights for candidate intervals that straddle the truth boundary determined? Also, what is the reason for the difference in the “clean” values for J native in Tables 1 and 4?
Reviewer_KGcFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies temporal forgery localization using a method that combines VideoMAE [1], LoRA [2], and Whisper [3]. The method is evaluated on ArEnAV [4] and LAV-DF [5, 6] and compared with WAFL [7] and AuViRe [8].

[1] Tong et al. (2022). VideoMAE: Masked Autoencoders Are Data-Efficient Learners for Self-Supervised Video Pre-Training.

[2] Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models.

[3] Radford et al. (2023). Robust Speech Recognition via Large-Scale Weak Supervision.

[4] Kuckreja et al. (2026). Tell Me Habibi, Is It Real or Fake?

[5] Cai et al. (2022). Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization.

[6] Cai et al. (2023). Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization.

[7] Wang et al. (2026). Word-Anchored Temporal Forgery Localization.

[8] Koutlis and Papadopoulos (2026). AuViRe: Audio-Visual Speech Representation Reconstruction for Deepfake Temporal Localization.

Critical strengths

None.

Critical weaknesses

The presentation of this paper is difficult to follow.

  1. Although the abstract briefly describes the structured objective, it moves too quickly into numerous numerical results before establishing an intuitive understanding of the approach. Labels such as U+S, LAV-DF, and U-max appear without explanation, and the meaning of “speech-guided readout” is unclear at this point.
  2. The Introduction reports improvements for Diff2Lip- and LatentSync-trained configurations without explaining or citing.
  3. Section 3.1 moves abruptly from the task definition to detailed implementation choices and hyperparameters.
  4. Although the structured localizer is named U in the Introduction, the Method section does not explicitly connect this label to the formulation being presented. This makes it difficult to map the equations to the experimental variants. The relationships among U, U+S, U-max, and the baseline variants should be explained clearly and consistently.

The bibliography contains only 12 references and omits the original Diff2Lip and LatentSync papers despite their central role in the experiments. These omissions raise concerns about the completeness of the attribution and discussion of relevant prior work. Finally, the methodological novelty is not sufficiently established. The authors should explain more clearly how this formulation differs from existing semi-Markov and segmental models, and what new methodological insight it provides beyond their application to temporal forgery localization.

Questions for authors

None.

Reviewer_nuaCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a semi-Markov-style structured learning framework for speech-assisted visual forgery localization, replacing independent segment classification with globally normalized training over complete temporal labelings and demonstrating substantial improvements in temporal interval detection under a controlled, speech-guided decoding protocol.

Critical strengths
  1. Well-motivated formulation: The paper addresses a real limitation of independent clip classification by learning complete temporal labelings. The structured objective is technically sound and well suited to temporal forgery localization.
  2. Convincing experimental improvements: The comparison with J-Decode+S is carefully designed, using the same backbone, candidate segments, and decoder. The AP75 improvements of 18.37 and 17.36 points provide strong evidence for the benefit of structured training.
  3. Useful additional analysis: The paper includes cross-generator evaluation, LAV-DF transfer results, and ablation studies. The U-max comparison is particularly useful in showing the trade-off between segment-level and frame-level performance.
Critical weaknesses
  1. Limited evaluation scale: The main results are based on only 212 ArEnAV videos, with one annotated manipulation interval per fake video. It remains unclear whether the method generalizes to larger datasets or videos containing multiple manipulated intervals.
  2. Unclear transfer evaluation: Appendix A.1 mentions earlier training exposure to LAV-DF, while Section 4.3 presents LAV-DF as a transfer benchmark. The authors should clarify the training data and confirm that no evaluation videos or source videos were previously used.
  3. Poor boundary precision and robustness: Despite the strong AP75 improvements, AP95 remains very low (2.65 and 2.62). Performance also drops sharply under video compression, raising concerns about precise boundary localization in practical settings.
  4. Incomplete ablation studies: The contribution of the speech prior is not directly evaluated. A comparison of U+N versus U+S would help clarify its effect. Comparisons with simpler structured alternatives, such as linear-chain CRFs, would also strengthen the methodological contribution.
  5. Experimental inconsistencies: The clean J-native results differ between Tables 1 and 4, without an explanation. The paper also lacks multi-seed results or confidence intervals to support the stability of the reported improvements.
Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.