acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Temporal Forgery Localization: From Semantic-Oriented Dense Detection to Forgery-Sensitive Sparse Localization

Abstract

Temporal Forgery Localization (TFL) aims to identify the temporal boundaries of manipulated segments within otherwise authentic content. Given the commonalities between TFL and Temporal Action Localization (TAL), many TFL methods build on TAL-derived feature modeling and localization designs, with further forgery-specific adaptations. However, these adaptations often address representation learning and localization separately, leaving the preservation of manipulation-sensitive responses and the modeling of sparse interval structure insufficiently explored jointly. We propose DiST-TFL, a Diffusion-guided Sparse Temporal Forgery Localization framework that jointly adapts representation learning and temporal localization to TFL-specific characteristics. Diffusion-based feature refinement provides a reference for revealing manipulation-sensitive responses, which are propagated across scales through a hierarchical temporal feature pyramid. A sparse query-based localization module then predicts a compact set of forged intervals without Non-Maximum Suppression (NMS). Experiments on LAV-DF and AV-Deepfake1M demonstrate substantial improvements in overall localization performance, while ablation studies validate the complementary roles of diffusion-guided feature modeling and sparse localization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Temporal Forgery Localization (TFL), Audio-visual Deepfake Detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_XpjzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits Temporal Forgery Localization (TFL) by highlighting its key differences from Temporal Action Localization (TAL) and adapting the model architecture accordingly. The authors propose DiST-TFL, a Diffusion-guided Sparse Temporal Forgery Localization framework that jointly addresses forgery-sensitive representation learning and sparse temporal localization. Experimental results on two benchmark datasets demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. The paper highlights the differences between Temporal Forgery Localization (TFL) and Temporal Action Localization (TAL), particularly the subtle forgery cues and sparse temporal distribution of manipulated segments. This provides a reasonable motivation for revisiting existing localization architectures.
  2. The proposed method achieves substantial improvements in mAP on both LAV-DF and AV-Deepfake1M compared with the reported baselines, demonstrating its effectiveness on established audio-visual forgery localization benchmarks.
Critical weaknesses
  1. The proposed framework relies on concatenated audio-visual features extracted by AV-HuBERT. However, its core components, including diffusion-guided feature refinement and sparse query-based localization, do not explicitly model cross-modal interactions. It would be valuable to evaluate the framework under visual-only and audio-only settings to better understand the contribution of each modality and whether the proposed architecture remains effective with single-modal inputs.
  2. The experiments are conducted only on LAV-DF and AV-Deepfake1M, both of which focus on audio-visual forgery localization. Since the proposed framework is not inherently restricted to audio-visual inputs, evaluating it on the visual-only ActivityForensics benchmark would help demonstrate its generalizability to different forgery localization settings. Furthermore, a direct comparison with TADiff on this benchmark would provide stronger evidence for the effectiveness of the proposed diffusion-guided design over existing diffusion-based approaches.
Questions for authors

Please refer to weakness part.

Reviewer_v41ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiST-TFL for audio-visual temporal forgery localization. It applies diffusion-inspired feature refinement before hierarchical temporal modeling, then uses sparse queries and one-to-one matching to predict forged intervals without non-maximum suppression. The method achieves mAP of 93.13 on LAV-DF and 84.17 on AV-Deepfake1M. The evaluation includes component ablations, recall under limited prediction budgets, diffusion-step sensitivity, and inference efficiency.

Critical strengths
  1. The architecture in Figure 2 coherently combines feature refinement, multi-scale modeling, and sparse interval prediction, as detailed in Sections 3.2-3.4.
  2. Table 1 shows improved mAP on both benchmarks, while Table 4 demonstrates advantages under limited prediction budgets.
  3. Component ablations in Table 2, sensitivity tests in Table 7, and latency comparisons in Tables 3 and 8-9 help characterize the method's benefits and costs. Table 10 further examines refinement placement.
Critical weaknesses
  1. Diffusion-specific benefits remain unclear. Table 9 shows substantial differences in parameters and FLOPs between early and late refinement, limiting attribution of the improvements in Table 10 to refinement placement.
  2. Table 2 shows that the full model underperforms the no-query variant at [email protected] on both datasets, despite higher mAP. Section 4.3 acknowledges this boundary-precision trade-off but provides limited explanation.
  3. The diffusion description differs from the code. Section 6.5 specifies a fixed training start at t = 2 for the refinement process defined by Equation 1, whereas the supplementary DiffusionRefiner.forward implementation samples the corruption timestep before reversing from the maximum timestep.
  4. The supplied AV-Deepfake1M configuration differs from Table 5 in learning rate, decoder depth, proposal range, and GIoU weight, making the reported setup difficult to reproduce.
Questions for authors
  1. Could the authors provide capacity-matched comparisons for the refinement variants in Tables 9-10 and explain the [email protected] degradation in Table 2, particularly for short forged segments?
  2. Which diffusion initialization procedure generated the results? Please reconcile the procedure described in Equation 1 and Section 6.5 with the supplied implementation.
  3. Could the authors release the exact configuration and checkpoint corresponding to Table 1, resolving the discrepancies with Table 5?
  4. In Section 2, please cite and contextualize Learning Discriminative Noise Guidance for Image Forgery Detection and Localization, published at AAAI 2024, ForgeryGPT: Multimodal Large Language Model For Explainable Image Forgery Detection and Localization, and Threat in Frequency: Unveiling Hidden Attack Surfaces by Exploiting Stealthy Backdoors on Diffusion Models. Clarify their connections to noise guidance, explainability, and security while distinguishing their tasks from temporal localization; direct cross-task experiments are unnecessary.
Reviewer_Y8DtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes DiST-TFL for audio-visual temporal forgery localization. A diffusion-style refinement of the input features serves as a reference in cross-attention, placed before a temporal feature pyramid. A sparse query decoder trained with one-to-one matching then predicts forged intervals without NMS. Experiments on LAV-DF and AV-Deepfake1M show consistent gains over prior methods, particularly at strict IoU thresholds, and are supported by ablations and an efficiency analysis.

Critical strengths

A new way to expose manipulation cues through a self-generated reference. Prior TFL methods mainly look for evidence of forgery in cross-modal inconsistency or cross-modal reconstruction error, which implicitly assumes that one modality remains a reliable reference. This paper instead derives a reference from the features themselves through generative refinement, and lets the original features interact with this reference through cross-attention, without computing an explicit reconstruction residual. The interaction happens before any temporal aggregation, so subtle local traces can be emphasized before pooling dilutes them. The idea does not rely on cross-modal disagreement or a reconstruction target, and it could carry over to other localization tasks where the targets are semantically close to the background. I regard this as the paper's main contribution and the primary reason for my positive assessment, even though its mechanism needs stronger support (see Weakness 1).

Rethinking TFL as a problem distinct from action localization, at both the representation and the prediction level. Existing TFL methods largely inherit the TAL pipeline and adapt either the features or the localization head in isolation. The paper identifies two properties that separate the tasks: forged content stays semantically similar to authentic content, and forged intervals are sparse. It then derives a design for each level from these properties. The sparse decoder also anchors its queries at high-confidence locations in the multi-scale memory while keeping the query content learnable, which ties the sparse prediction to the forgery-sensitive features rather than treating the two as independent parts. This framing gives the field a clearer basis for moving beyond TAL-derived designs.

Critical weaknesses

The role of diffusion guidance is not sufficiently substantiated. The refinement module has no diffusion or reconstruction objective, starts from a low noise level, and is trained only through the localization loss. In effect, it behaves like a learned residual branch with mild noise injection. Performance is largely insensitive to the number of denoising steps, and the paper does not show that the refined reference or the resulting attention emphasizes manipulated regions. The module also adds considerable parameters and computation, so part of the gain may come from extra capacity. Since this is the main contribution, I would expect stronger evidence.

The ablations do not clearly attribute the gains. Table 2 suggests that on AV-Deepfake1M most of the improvement comes from the query decoder, which follows existing DETR-style localization and RegQAV. On LAV-DF, the authors' own dense variant is already competitive, and several partial variants collapse at strict IoU. This looks more like optimization instability than the component interdependence claimed in the paper, and no variance across runs is reported. The reproduced baselines are also lower than their originally reported results, and recall at larger budgets from the standard protocol is not reported.

Questions for authors
  1. How does the model perform if the diffusion refinement is replaced by plain self-attention, or by a residual branch of the same architecture without noise injection, under a matched parameter budget? This would show whether the gain comes from the diffusion process or from added capacity.

  2. Why does removing the diffusion refinement from the query-based model cause a sharp drop at strict IoU on LAV-DF but not on AV-Deepfake1M? An explanation would clarify how the two components interact.

Reviewer_NXuuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates temporal forgery localization (TFL) in audio-visual content, focusing on preserving subtle forgery evidence and reducing redundant temporal predictions. The authors propose DiST-TFL, which combines diffusion-guided hierarchical feature modeling with sparse query-based localization to identify manipulated intervals without NMS. Experiments on LAV-DF and AV-Deepfake1M demonstrate significant improvements over existing methods, achieving mAP scores of 93.13% and 84.17%, respectively. Ablation studies further validate the effectiveness of the proposed components.

Critical strengths
  1. The work is well-targeted: it identifies two genuine TFL-vs-TAL differences, weak forgery traces diluted across scales and sparse forged intervals mismatched with dense prediction and designs matched components, including diffusion-guided cross-attention before hierarchical modeling and NMS-free sparse query-based localization. Ablations show these components are complementary.
  2. Experiments are comprehensive and consistently strong: on two widely used benchmarks, the method achieves mAP gains of +5.89 and +10.20 over the second-best, along with efficiency analysis showing NMS-free inference and competitive end-to-end latency, diffusion-placement and sensitivity studies, and qualitative paired comparisons.
Critical weaknesses
  1. The main comparison is not strictly dependent on feature control. In Table 1, ba-tfd/ba-tfd+/ummaformer/auvire uses AV Hubert features to reproduce, but retains their respective superparameters, while dimodif and regqav use their native pre training to represent the values of the original paper reported; In addition, the two datasets use different optimizer/decoder/proposal scope/geographic information volume (giou) settings. The paper acknowledges this ("rather than strict feature control evaluation"), but the main improvement (+5.89/+10.20) is based on the hybrid protocol, so the claimed improvement range needs to be verified under the unified input and protocol settings.
  2. The diffusion-guidance mechanism is not well explained and is highly sensitive to localization. The paper does not analyze what the fine-grained references actually learn, nor how cross-attention highlights forgery-related differences. The extreme gap between early and late diffusion placements on LAV-DF (93.13 vs. 22.28 mAP, Table 10) is striking: the roughly 71-point collapse suggests that the late variant essentially fails and requires further explanation.
Questions for authors
  1. Could you provide a main comparison of strictly controlled functions? All baselines (including regqav and dimodif) are trained with AV Hubert function under unified protocol? Under this setting, how will the reported gain (+5.89/+10.20) change?
  2. What does the diffusion refinement reference actually encode? Can you visualize X̃ or cross-attention weights to prove that it reveals an operationally sensitive response? How do you explain the extreme difference between morning and night on lav-df (93.13 vs 22.28)? Is the collapse of late variants an optimization problem or the mechanism itself?
  3. The two datasets used different learning rates, decoding depths (4 vs 2), proposal ranges (1.0 vs 0.5), and giou weights (2.0 vs 1.0). Are these tuned for each dataset? What is the best result in a single unified configuration?
1 comment
AuthorsOpenReview ↗

Response to Q1: Fairness of Baseline Comparisons.

We thank the reviewer for highlighting the importance of a feature-controlled comparison. To address this concern, we have additionally reproduced DiMoDif and RegQAV using the same AV-HuBERT input features as DiST-TFL and the other baselines, while retaining their respective model architectures and training configurations.

During the rebuttal period, we also identified an oversight in our original LAV-DF experiments: a portion of the extracted LAV-DF features had inadvertently been omitted when training DiST-TFL. To ensure experimental rigor and fairness, we retrained DiST-TFL using the complete LAV-DF feature set and updated the corresponding results. Consequently, its mAP on LAV-DF increases from 93.13% to 94.996%.

Under the unified AV-HuBERT features, DiMoDif achieves mAP scores of 74.85% and 56.15%, while RegQAV achieves 77.28% and 51.81% on LAV-DF and AV-Deepfake1M, respectively. DiST-TFL continues to achieve the highest mAP on both datasets (94.996% and 84.17%). Importantly, AuViRe remains the second-best method on both benchmarks, with mAP scores of 87.24% and 73.97%. Therefore, DiST-TFL maintains improvements of +7.76 and +10.20 percentage points, respectively, under the unified feature setting.

Regarding RegQAV and DiMoDif, both methods rely on specialized feature representation strategies. RegQAV introduces register-enhanced representation learning to improve audio-visual feature modeling, while DiMoDif employs independently pretrained Visual Speech Recognition (VSR) and Audio Speech Recognition (ASR) encoders to capture cross-modal speech discrepancies. We initially retained their originally reported results out of concern that replacing these specialized representations with unified AV-HuBERT features might affect their distinctive feature modeling mechanisms and potentially influence their performance.

Compared with the originally reported results, RegQAV's mAP on AV-Deepfake1M decreases from 56.66% to 51.81% (-4.85 points), although it increases from 69.90% to 77.28% (+7.38 points) on LAV-DF. For DiMoDif, the original paper does not report mAP, so we compare [email protected] instead. Interestingly, DiMoDif's [email protected] improves from 95.50% to 98.26% (+2.76 points) on LAV-DF and from 86.93% to 96.46% (+9.53 points) on AV-Deepfake1M. Nevertheless, we have now reproduced both methods under the unified AV-HuBERT feature setting to provide a more controlled and transparent comparison.

We have updated Table 1 and Table 2, their corresponding descriptions, and the experimental protocol in Section 4.2 to reflect the corrected LAV-DF results and the additional baseline experiments. All modifications are highlighted in blue in the revised manuscript. The corresponding updated checkpoint will also be provided in the revised supplementary materials.

Overall, these additional experiments confirm that DiST-TFL maintains its performance advantages under a feature-controlled comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.