acceptodds
Withdrawn from ICLR 2027

AV-FABLE: Audio-Visual Manipulation Forensics in Long-Form Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Localized AI manipulations in long-form videos often affect brief events within otherwise authentic content, making video-level classification insufficient for forensic analysis. Existing audio-visual deepfake benchmarks largely focus on short talking-head clips, while long-form temporal forensics remains predominantly visual. We introduce AV-FABLE, a dual-timeline benchmark for sparse manipulations in open-scene long-form audio-visual videos. AV-FABLE contains 16,650 pristine–manipulated twin pairs (998 hours) created with 27 editing configurations, with separate audio and visual timelines that enable modality-specific temporal localization and expose errors hidden by joint-timeline evaluation. On AV-FABLE, zero-shot prompting of audio-visual large language models (AV-LLMs) yields near-chance detection and poor temporal localization. In contrast, supervised readouts of their frozen perception features recover strong forensic signals, suggesting manipulation evidence is encoded in these representations but not effectively exposed through language outputs. Motivated by this observation, we propose NFSR(Native Forensics via Separated Readouts), which reads frozen AV-LLM features with separate audio and visual temporal decoders. NFSR achieves the best audio-visual localization performance in-domain and across all three out-of-distribution settings, including transfer to held-out videos from an external long-video forensic benchmark. AV-FABLE, code, and model weights will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
audio-visual forensics, temporal localization, manipulation detection, multimodal benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YzH6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AV-FABLE, a benchmark for detecting and temporally localizing sparse audio-visual manipulations in long-form videos. The benchmark contains 16,650 pristine–manipulated video pairs and adopts separate audio and visual timelines to support modality-specific localization. The authors further investigate the forensic capabilities of audio-visual large language models (AV-LLMs), showing that their zero-shot language outputs perform poorly despite useful forensic information being accessible from frozen perceptual representations. Motivated by this observation, they propose NFSR, which combines intermediate-layer and interface features, host-conditioned modulation, and modality-separated temporal decoders. Experiments demonstrate strong in-domain performance and improvements over existing baselines across three out-of-distribution evaluation settings.

Critical strengths
  1. Meaningful benchmark setting and comprehensive dataset construction. AV-FABLE extends audio-visual manipulation localization to long-form, open-scene videos. Its dual-timeline evaluation, diverse editing configurations, pristine–manipulated pairs, and human-reviewed annotations provide a valuable resource for forensic research.

  2. Interesting findings on forensic information in AV-LLM representations. The contrast between near-chance zero-shot detection and strong supervised readouts of frozen representations is insightful. The layer-wise analyses further highlight the distinction between forensic information accessibility and downstream utilization.

  3. Strong empirical results and extensive evaluations. NFSR achieves strong in-domain and OOD performance, outperforming UMMAFormer under matched feature and training settings. Comprehensive ablations, encoder comparisons, and multi-seed experiments further support its effectiveness.

Critical weaknesses
  1. Insufficient connection between the research motivations and the proposed methodological solution.

The paper highlights two important but somewhat distinct problems: the need for modality-specific temporal localization in asynchronous audio-visual manipulations, and the gap between forensic information encoded in AV-LLM representations and its accessibility through language outputs. However, the relationship between these problems and the necessity of the proposed architecture is not sufficiently established.

In particular, while separate audio and visual timelines are well motivated as an evaluation protocol, it remains unclear what distinctive modeling difficulty is introduced by asynchronous manipulations beyond requiring independent predictions for each modality. NFSR largely addresses this through independently parameterized unimodal temporal decoders. The paper does not sufficiently analyze whether asynchronous events are inherently more challenging than temporally aligned events, or whether the proposed architecture provides specific advantages under such conditions. Consequently, the benchmark motivation and representation-readout methodology appear somewhat loosely connected rather than forming a fully unified research question.

  1. The extent to which localization performance depends on dataset-specific generation artifacts remains unclear.

Although the pristine–manipulated twin design and matched export procedures help reduce potential shortcuts, these controls primarily address content and file-level processing differences. They do not fully exclude generator- or editing-pipeline-specific artifacts within manipulated intervals, such as local resampling traces, synthesis textures, or acoustic discontinuities.

This distinction matters because high localization accuracy could partly arise from detecting such artifacts rather than learning forensic evidence that generalizes across manipulation processes. The substantial performance degradation under held-out generators and certain media corruptions further motivates investigating this issue, although these results do not themselves establish shortcut learning. More targeted controls, such as artifact-matched pristine segments, would strengthen the validity and generalizability of the benchmark conclusions.

  1. The methodological novelty and interpretation of the representation findings could be further strengthened.

NFSR primarily combines established components, including gated feature projection, FiLM-based conditioning, multi-scale temporal localization, and independent modality-specific decoders. While the combination is empirically effective, the paper provides limited evidence of a fundamentally new modeling principle beyond adapting existing techniques to the proposed benchmark.

Moreover, the comparison between zero-shot language outputs and supervised linear probes involves different supervision conditions and prediction mechanisms. Therefore, it establishes a useful descriptive performance gap but does not fully isolate whether the limitation lies in the language pathway, insufficient task adaptation, or the output formulation. A more systematic comparison between representation accessibility and downstream supervised adaptation would strengthen this central interpretation.

Additionally, although AV-FABLE supports explanation evaluation, NFSR itself does not generate forensic explanations, and the separately evaluated explanation models remain relatively weak. This leaves a gap between the broader forensic capabilities supported by the benchmark and those demonstrated by the proposed method.

Questions for authors

Q1. Could the authors provide a more detailed analysis of localization performance on temporally aligned versus asynchronous audio-visual manipulations? In particular, does NFSR offer greater advantages as the temporal mismatch between modalities increases?

Q2. Could the authors further investigate whether the high localization performance depends on generator- or editing-pipeline-specific artifacts? For example, would artifact-matched pristine controls help distinguish transferable forensic evidence from construction-specific shortcuts?

Q3. How strongly does layer-wise linear-probe performance correlate with downstream supervised adaptation performance? Would the layers achieving the highest linear-probe AUROC also yield the best temporal localization or language-based detection performance after matched supervised adaptation?

Q4. Given that AV-FABLE includes explanation as a benchmark task, how do the authors envision integrating evidence-grounded explanations with NFSR? Could the authors clarify whether explanation is intended as a separate future research direction or part of the proposed forensic framework?

6 comments
AuthorsOpenReview ↗

Response to Reviewer YzH6 (1/5): W1, Q1

W1\color{blue}{\text{W1}}: This comment led us to state the link between the two problems, which our submission left implicit. We see them as one question asked at two levels: where should a forensic decision be read from?

At the level of evaluation, the dual timeline says that a decision must be made per modality, because a correct time on the wrong track is an error. At the level of representation, our analysis says that the evidence itself is per modality and sits inside each encoder: it is most accessible at intermediate depth and weakens toward the fused language interface. NFSR follows from putting the two together: read each modality's evidence where it is strongest, and decide on that modality's own timeline.

You ask whether asynchronous edits are harder to model, beyond needing two outputs. Our analysis shows that they are. We grouped the both-track test videos by how much their audio and visual edits overlap in time (three seeds per model):

Audio/visual overlapPairsUMMAFormer (same features)Cross-modal variantNFSR
High (tIoU ≥ 0.5)71697.5197.1996.43
Partial (0 < tIoU < 0.5)1180.1392.4393.65
Disjoint (tIoU = 0)9167.8185.1786.32
Drop from high to disjoint−29.70−12.02−10.11

Values are AV-mAP (%). Every model loses accuracy when the two edits separate in time, so asynchrony is a difficulty of its own and not only a matter of output format. The loss is three times larger for UMMAFormer, a standard localizer given the same features, than for NFSR. When the edits coincide, all three models are equivalent; the architecture matters exactly when the tracks diverge.

Our response to Q1 gives the paired intervals.

Revision. Sections 1 and 4 now present the argument in this order: dual-timeline task, where the evidence lives, and the readout that follows.

Q1\color{blue}{\text{Q1}}: Thank you for suggesting this analysis. The answer is yes: NFSR's advantage over a standard localizer on the same features appears when the audio and visual edits separate in time, and it is absent when they coincide.

We grouped the both-track ID test videos by the temporal IoU between their audio and visual edit intervals and evaluated three seeds per model. Each cell is AV-mAP / Event-IoU (%).

Audio/visual overlapPairsUMMAFormer (same features)Cross-modal variantNFSR
High (tIoU ≥ 0.5)71697.51 / 92.3597.19 / 92.1496.43 / 88.86
Partial (0 < tIoU < 0.5)1180.13 / 57.4192.43 / 78.9193.65 / 83.57
Disjoint (tIoU = 0)9167.81 / 52.4485.17 / 73.5886.32 / 70.80

Paired differences in Event-IoU, in percentage points, with 95% bootstrap intervals over source videos:

Audio/visual overlapNFSR − UMMAFormerNFSR − cross-modal variant
High−3.49 [−4.71, −2.27]−3.29 [−4.07, −2.49]
Partial+26.16 [9.06, 42.45]+4.66 [1.35, 8.37]
Disjoint+18.36 [13.25, 23.37]−2.78 [−5.73, 0.12]

Against the standard localizer, the difference moves from −3.5 points when the edits coincide to +18 and +26 points when they do not. In AV-mAP, UMMAFormer loses 29.7 points between high overlap and disjoint, and NFSR loses 10.1.

Against our cross-modal variant, which already decodes each track separately and adds cross-attention between the branches, NFSR is on par for disjoint edits (the interval includes zero) and slightly behind when the edits coincide. This comparison isolates what matters: per-track decoding is what handles asynchrony. Letting the branches exchange information adds nothing for asynchronous edits, gives a small gain for co-timed ones, and has a cost that we measured separately: on audio-only speech edits the cross-modal visual head raises 10.5 points more false alarms than on pristine twins (95% CI [6.7, 14.6]), while the separated head raises none. We chose separation for that reason.

The partial group has 11 pairs and we draw no trend from it alone.

Revision. Both tables are in the appendix, and Section 5.3 summarizes them.

AuthorsOpenReview ↗

Response to Reviewer YzH6 (2/5): W2, Q2

W2\color{blue}{\text{W2}}: We agree that twin pairs and matched export control content and file-level differences, not what happens inside the edited interval, and we have run the artifact-matched pristine controls you propose. The results separate three things that our submission left entangled: the processing applied around an edit, the synthetic content itself, and the identity of the generator.

Artifact-matched pristine segments. Pristine content goes through the same processing as an edit, with no generated content.

Pristine content, processed like an editVideos flagged at the processed window (three checkpoints)
Audio: original signal through the same cut, reinsertion, and export0/59, 0/59, 0/59
Visual: same retiming and reinsertion0/24, 1/24, 1/24
Visual: same, plus encode–decode through the generator's VAE (Wan2.1)0/24, 1/24, 1/24

Resampling, splicing, loudness matching, re-export, and even the generator's own VAE do not trigger the detector. When speech in those same audio windows is resynthesized, 56–58 of the 59 videos are flagged. Detection responds to synthesized content and not to the pipeline that inserts it.

Nothing outside the edit carries the label. A probe restricted to unedited context, at least 2 s from any edit, is at chance on every test set, while the same probe on edit-overlapping windows is not.

Test setPairsUnedited context only, AUROC [95% CI]Overlapping the edit, AUROC [95% CI]
ID2,20849.80 [49.49, 50.13]82.57 [81.63, 83.54]
HDTF35950.74 [49.88, 51.62]64.18 [62.39, 66.51]
Held-out generators24750.04 [49.46, 50.65]71.30 [68.88, 74.07]
Held-out TI2V24750.00 [50.00, 50.00]66.84 [63.89, 69.71]

The evidence transfers across manipulation processes. This is the part of your concern that bears on generalization, so we tested it with matched leave-one-generator-out training: two models with identical data size, sources, editing operation, and budget, one of which never sees the target generator.

Withheld generatorTest pairsSeen in trainingNever seen
MMAudio (audio mAP)18884.1071.52
Wan2.2 Fun (visual mAP)14993.9291.13

The model keeps 85% and 97% of its accuracy on a generator it never saw. Two further checks point the same way. A speech synthesizer absent from training (IndexTTS2), applied to unseen HDTF speakers, is flagged in 22–26 of 37 videos with the transcript unchanged and in 29–33 of 38 with the transcript changed. And on TASLE, built by another group with a different pipeline, NFSR obtains 54.03 Loc-F1 on the commercial-generator subset against 30.85 for the TASLE-trained MSLoc-PG.

On robustness to corruption. Re-encoding the whole video with H.264 at CRF 28 recompresses edited and unedited regions alike: 107 of 184 visual edits are still found at the correct location (144 before), with 2 alarms on the 184 pristine twins. Heavier corruption lowers recall, as the robustness table reports.

In summary, what NFSR detects is synthesized content inside the edited interval; this evidence is absent from processed pristine material and from unedited context, and most of it carries over to generators and a benchmark that the model never saw.

Revision. Section 5.2 now contains a paragraph, Controls for construction cues, with the full tables and intervals in the appendix.

Q2\color{blue}{\text{Q2}}: Yes, such controls help, and we have run them. The full results are in our response to W2 above; the table below summarizes them.

QuestionControlResult
Does the processing around an edit trigger detection?Pristine audio through the same cut, reinsertion, and export0 of 59 videos flagged, all three checkpoints
Pristine video through the same retiming and reinsertion, with and without the generator's VAE0–1 of 24 flagged in both conditions
Is there a cue outside the edit?Probe on unedited context only49.8–50.7 AUROC (chance) on all four test sets
Is the evidence specific to the training generators?Matched leave-one-generator-out71.52 / 84.10 audio mAP (MMAudio) and 91.13 / 93.92 visual mAP (Wan2.2 Fun), never seen / seen
A synthesizer absent from training, on unseen speakersflagged in 22–26 of 37 videos

The outcome of the artifact-matched pristine controls is clear: processing alone does not trigger detection, while synthesized content in the same windows does. They therefore do what you intended, which is to separate construction-specific shortcuts from evidence carried by the manipulated content, and the leave-one-generator-out results show that most of that evidence transfers to generators the model never saw.

Revision. We have added these controls to Section 5.2 and the appendix, and the release will include the processed-pristine clips so that others can run the same check on their own detectors.

AuthorsOpenReview ↗

Response to Reviewer YzH6 (3/5): W3a

W3a\color{blue}{\text{W3a}}: This is an important question for the paper, and we would like to answer it directly. We chose standard components deliberately; what NFSR contributes is a principle about where and how a frozen AV-LLM should be read for forensics, together with evidence that the principle matters. It has three parts, each tied to a measurement.

  1. Read below the language interface, at the depth where the evidence is most accessible. Linear-probe AUROC peaks at intermediate layers (visual block 6: 88.44) and falls toward the final block that feeds the language model (80.55). Removing the intermediate-layer features costs 6.55 AV-mAP in-domain and 8.72 under shift.
  2. Judge local evidence relative to the host recording. Replacing a video's statistics with those of another recording from the same dataset and of the same length lowers OOD AV-mAP from 68.98 to 43.79 and nearly triples false alarms.
  3. Keep the evidence path separate for each modality. A visual head that can see the audio fires on audio-only speech edits 10.5 points more often than on their pristine twins (95% CI [6.7, 14.6]); the separated head shows no increase.

That the principle carries weight, and not the particular blocks, is shown by a comparison in which the features are identical and only the way they are read changes:

Same frozen Qwen3 features, same budgetID AV-mAPHeld-out generators AV-mAPHeld-out TI2V V-mAP
UMMAFormer, a standard multimodal localizer84.7633.9836.06
NFSR91.3049.1268.07

The alternatives that go through the language model do not come close: fine-tuning Qwen3-Omni to output timestamps reaches 37.80 AV-mAP, and using it to verify or re-score NFSR's candidates leaves localization unchanged (73.60–73.62 mean mAP). The principle is also not tied to one model: the same readout works on Qwen2.5-Omni and on BEATs + CLIP.

We therefore see the novelty in the finding-driven design and in the demonstration that, for forensics, an AV-LLM is better read natively than queried.

Revision. Section 4 now opens with this statement, and each component cites the prior work it builds on.

Reviewer_7fsnOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work builds AV-FABLE, the first audio-visual dual-timeline benchmark for sparse tampering in long videos (16,650 twin pairs, 27 edit configurations, with pristine-twin controls and metadata shortcut checks). It shows that current AV-LLMs fail at forensics not because the representations lack evidence (a linear probe on the same frozen features reaches 90.8 audio AUROC) but because the evidence is not exposed through the language interface. The proposed method, NFSR, which uses separated native readouts, achieves the best results on ID and all three OOD settings. The work also confirms that a merged timeline systematically masks modality misattribution.

Critical strengths
  1. The problem setup hits the mark: dual-timeline evaluation is a genuine finding in itself. Existing audio-visual forensic benchmarks either use a single merged timeline or cover vision only. The authors argue that a merged timeline rewards predictions that find the right moment but the wrong modality. They therefore design a native-track evaluation in which visual predictions cannot match audio events. This exposes real problems: An AV-FABLE-trained ActionFormer reaches 89.22 V-mAP but 0 A-mAP, a flaw that Joint-mAP completely hides. Conversely, NFSR with visual predictions alone gets 64.04 Joint-mAP, higher than every vision-only baseline. 561 dual-track videos have fully non-overlapping audio and visual events, so per-track annotation is not a formality.

  2. Data construction is more rigorous than comparable work. Pristine twins: Each tampered video has a source-matched original that went through identical cropping, reinsertion and export (container, codec, resolution, frame rate, sample rate). This suppresses content and processing shortcuts at the source. Shortcut testing (Appendix A.5): Logistic regression and GBDT on file-level metadata alone give multi-field AUROC of only 53.9–54.6 (ID) and 51.0–53.0 (OOD). The strongest single field reaches 51.9, which is near chance. Many dataset papers skip this step. Annotation quality: Two reviewers independently audited all 22,153 events, with κ=0.84 (whether the edit is realized within the recorded interval) and κ=0.82 (whether the evidence statement is supported by the media). A third party arbitrated the 7.7% of disagreements. Leakage-free splits: Source videos never cross splits, and 4 of the 27 edit configurations are held out entirely as OOD.

  3. The core finding is insightful and broadly valuable: evidence exists in the representations but is not exposed through the language interface. Zero-shot AV-LLMs (Qwen2.5-Omni-7B, Qwen3-Omni-30B, video-SALMONN 2+) have detection AUROC of 49–52, which is chance level, and mAP ≤0.30. On the same frozen perceptual features, linear probes reach ID AUROC of 90.8 (audio) and 80.8 (visual). Unsupervised feature-deviation and temporal-jump scores are far worse than the probes. A layer-depth sweep shows audio peaking at layer 12 (89.6 vs. 89.1 at layer 16) and vision peaking at block 6 (88.44), dropping to 80.55 at the last block. The intuition that deeper encoders make forensic evidence more linearly separable is refuted. This is the most valuable part of the paper and goes beyond the benchmark itself.

  4. NFSR's components map one-to-one onto the observations, and the ablations are well run. The three components (intermediate-layer features, interface features, host-conditioned FiLM) each correspond to one finding, so the design is not an ad hoc stack. Ablations use 3-seed mean ± std with a unified 12-epoch training and model-selection protocol. Removing intermediate-layer features lowers AV-mAP by 6.55 (ID) and 8.72 (OOD). Removing interface features barely changes ID, but lowers OOD by 4.07 and more than doubles FAR. The conclusion that the components matter more under distribution shift is convincing. NFSR is best on all three OOD settings (source transfer to HDTF/TASLE, held-out generator, held-out TI2V route) without threshold recalibration.

  5. The evaluation protocol is honest and guards against metric gaming. The threshold is calibrated once, on 256 pristine source validation videos at a 5% FAR budget, then fixed for ID and all OOD sets. NFSR's actual FAR stays at 2.97–4.67%. The authors note that low FAR alone is meaningless, since a conservative detector can lower it by missing events. They therefore require it to be read together with Attribution F1 and Event-IoU. They openly disclose that DAV-Det covers more visual events under source transfer but has FAR as high as 24.03%/40.45%. They state that all architecture, layer selection, hyperparameter and model-selection choices were made without touching OOD test results.

Critical weaknesses
  1. The "long-form" claim is overstated, and all data is synthetic, with no real-world validation. The median duration is only 88 s, and 80% of videos fall between 33 and 207 s. This is close in magnitude to LAV-DF and AV-Deepfake1M, which are criticized as "short clips". The title, abstract and "minute-scale" wording are exaggerated. The "998 hours" figure counts both ends of every twin pair. After deduplication it is only 499.1 hours, which the authors mention in a single line in Appendix A.1. All data is generator-synthesized. There are no in-the-wild tampered samples, and NFSR is not validated on any real forensic data, so its practical value is left hanging.

  2. The most central doubt is not ruled out: the model may be learning generator fingerprints rather than tampering semantics. In A.5 the authors admit the metadata probe only covers file-level attributes and does not exclude decoding-signal traces the generator introduces within edited intervals, such as local resampling, generator-specific texture and synthetic acoustic residue. This is exactly what NFSR is most likely to exploit. It reads intermediate-layer features from the frozen encoders of a single backbone (Qwen3-Omni), and those layers retain much low-level texture and spectral detail, where fingerprints most easily survive. The collapse pattern supports this. ID performance is 96.04 AUROC / 91.30 AV-mAP, but it falls to 68.07 AV-mAP on held-out TI2V, and source-transfer V-IoU is only 26.42–41.21. The authors attribute the OOD degradation to difficulty under distribution shift, but run no discriminating experiments, such as training on unedited regions only, cross-generator validation, or adversarial removal of generator identity. The core claim that the forensic signal is evidence of semantic tampering is therefore not actually demonstrated. This is the most serious flaw in the paper.

  3. Baseline comparisons are unfair, and the main-table scores conflate capability with output-interface fit. An AV-FABLE-trained AASIST reaches 75.95 AUROC but only 0.06 A-mAP after window→interval conversion. AuViRe and ICS-Sync output unified predictions with no modality labels, so their A/V-mAP is recorded as "–", and their Joint-mAP is only 0.42 and 5.47. These methods thus nearly "don't exist" in the main table, not because they cannot do forensics but because their output format doesn't match the evaluation. Despite a brief note, ranking them beside NFSR with bold/underline "best" marks is still misleading. Only NFSR, UMMAFormer and TASLE-ActionFormer report 3-seed means. All others are single runs, so the differences in Tables 2/3 have no confidence intervals. NFSR uses frozen features from a 30B-class multimodal model, while Xception, AASIST and ActionFormer are small-to-mid-sized models trained from scratch. The confound of parameter count and pretraining scale is never separated out.

  4. "Explanation" as one of the four evaluation dimensions is hollow. The paper claims to evaluate localization, detection, attribution and explanation, and the dataset does annotate 20,529 candidate evidence records. But Tables 2 and 3 contain no quantitative explanation metric beyond Attribution F1. The material is pushed to Appendix H, and the main text offers one qualitative sentence: current baselines produce well-formed output but often misidentify the operation or target. The dual-reviewer evidence audit (κ=0.82) yields no comparable number in the main text, so readers cannot verify this dimension's contribution.

  5. Argument strength, writing density and self-criticism are insufficient. Thin key ablations: The support for "separated readouts beat cross-modal readout" is only a 2.9% vs. 11.8–15.1% false-trigger rate comparison, with no significance test or confidence interval. The motivation for host-conditioned FiLM (the same local change means different things under different video conditions) rests on a 4.73-point OOD gain. The ID gain is only 1.44 points (89.86→91.30), nearly noise given a ±0.86 std. Suspected OOD contamination: Table 3 marks TASLE with † as overlapping the model's training distribution and admits that TASLE-trained models lead in source-transfer V-mAP because those events come almost entirely from their training distribution. The source-transfer OOD is therefore not a clean held-out setting, yet it is averaged with the other two OOD sets into "OOD-All". Over-reliance on the appendix: A.5, C, D.3, F.5, F.6, H and Tables 8/10/11/12/13/15 carry almost all key arguments. After the 9-page main text, readers get conclusions but not the evidence chain, which is unfriendly to reviewers. Perfunctory limitations section: It is only 4 lines and reads like a future-work wish list ("extend to broader real editing pipelines", "improve robustness to media corruption"). It omits the three points most worth discussing: dependence on a single backbone, the gap between synthetic edits and real tampering, and the generator-fingerprint confound. Data release is also restricted by third-party licenses (only event-level annotations can be released), so the barrier to reproduction is much higher than the abstract's promise to publicly release the dataset, code and weights suggests.

Questions for authors

none

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7fsn (1/5): W1

W1\color{blue}{\text{W1}}: Your comment raises three separate points, and we address them in turn.

Duration. We respectfully disagree that AV-FABLE is close in magnitude to LAV-DF and AV-Deepfake1M. The difference is an order of magnitude.

BenchmarkMean video duration
LAV-DF8.6 s [1]
AV-Deepfake1M++9.6 s [2]
AV-FABLE108 s (median 88 s)

Statistical sources and scope: LAV-DF—DeformTrace [1], “Datasets and Evaluation Metrics” (full test set); AV-Deepfake1M++—Kukanov and Ng [2], Section 5.1, Table 1 (training split).

In the AV-FABLE test split, 73% of videos run longer than one minute and 18% longer than three. A single manipulated event (median about 9 s) lasts as long as an entire LAV-DF clip, and edits occupy a median of 9.45% of the timeline. We use "long-form" in the same sense as the closest prior work, TASLE (Feng et al., ICML 2026, Explainable Forensics of Manipulated Segments in Untrimmed Long Videos [3]): sparse local edits embedded in much longer authentic footage. You are right that readers should be able to judge this for themselves, so the abstract now states the median duration, and Section 3.3 and Table 1 give the distribution.

Length also changes the difficulty of the task. Three-seed means on the ID test set:

Video durationPairsNFSR AV-mAPUMMAFormer (same features)Gap
< 60 s62394.3789.46+4.91
60–120 s78792.4786.41+6.06
120–180 s50586.8678.68+8.18
≥ 180 s43288.6880.50+8.18
Edited share of the videoPairsNFSR AV-mAPUMMAFormer (same features)Gap
< 5%42977.7866.18+11.60
5–10%82191.1786.23+4.94
10–20%69495.5489.69+5.85
≥ 20%40393.4787.14+6.33

Both models lose accuracy as videos lengthen and edits become sparser, and NFSR's margin is widest in exactly those conditions. These tables are now in the appendix.

Hours. You are right, and we have corrected this. The abstract, the introduction, and Section 3 now give 499.1 hours, counting each pair once. The 998.1-hour figure for both twins is kept only as a secondary note.

Synthetic data and real-world validation. Controlled generation is what gives this benchmark frame-accurate labels on two separate tracks and a pristine twin for every sample. Videos collected in the wild offer neither, which is why temporal-localization benchmarks, including LAV-DF, AV-Deepfake1M, and TASLE, are constructed this way. Within that constraint we keep the data close to practice and we test outside our own pipeline. The host footage is real, unscripted video. The 27 configurations include current open and commercial generators. And with no adaptation NFSR transfers to TASLE, which another group built with different generators: 50.80 Loc-F1 on its official test against 63.62 for the TASLE-trained MSLoc-PG, and 54.03 against 30.85 on the TASLE subset produced with the commercial generators Kling and Jimeng, where NFSR outperforms the model trained on TASLE itself. These results are now in Section 5.2. Evaluation on manipulated videos gathered from the wild is the next step, and the Limitations paragraph says so.

References

[1] Xiaodong Zhu, Suting Wang, Yuanming Zheng, Junqi Yang, Yangxu Liao, Yuhong Yang, Weiping Tu, and Zhongyuan Wang. DeformTrace: A Deformable State Space Model with Relay Tokens for Temporal Forgery Localization. Proceedings of the AAAI Conference on Artificial Intelligence, 40(42):36110–36118, 2026. DOI: 10.1609/aaai.v40i42.40928. URL: https://ojs.aaai.org/index.php/AAAI/article/view/40928

[2] Ivan Kukanov and Jun Wah Ng. KLASSify to Verify: Audio-Visual Deepfake Detection Using SSL-based Audio and Handcrafted Visual Features. arXiv:2508.07337, 2025. URL: https://arxiv.org/abs/2508.07337

[3] Yue Feng, Jingjing Li, Qijia Lu, Wei Ji, Jingrou Zhang, Fei Shen, Xiao Li, Yizhen Jia, Qiang Chen, Limin Wang, Wentong Li, and Jie Qin. Explainable Forensics of Manipulated Segments in Untrimmed Long Videos. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306:30115–30131, 2026. URL: https://proceedings.mlr.press/v306/feng26g.html

AuthorsOpenReview ↗

Response to Reviewer 7fsn (2/5): W2

W2\color{blue}{\text{W2}}: Thank you for stating this concern so precisely. We took it more seriously than any other point, and we ran the experiments you proposed. Before presenting them, we would like to clarify our claim, because the wording in our submission invited a stronger reading than we intended.

We do not claim that NFSR recognizes semantic implausibility. The task is to localize where content was synthetically replaced or inserted, and traces left by synthesis are legitimate evidence for that task, as they are for a human analyst. What would undermine the benchmark is a different kind of evidence: cues that exist only because of our construction pipeline, or cues tied to particular generators that disappear with the next one. Both hypotheses are testable, and we tested each.

A. Is there any cue outside the edit? (your "training on unedited regions only")

We train a probe that sees only unedited context, at least 2 s away from any edit, and compare it with the same probe on windows that overlap the edit. Training and test sources are disjoint.

Test setPairsUnedited context only, AUROC [95% CI]Overlapping the edit, AUROC [95% CI]
ID2,20849.80 [49.49, 50.13]82.57 [81.63, 83.54]
HDTF35950.74 [49.88, 51.62]64.18 [62.39, 66.51]
Held-out generators24750.04 [49.46, 50.65]71.30 [68.88, 74.07]
Held-out TI2V24750.00 [50.00, 50.00]66.84 [63.89, 69.71]

Outside the edit the probe is at chance on every test set. NFSR behaves the same way: in the unedited regions of manipulated videos it raises no more alarms than on their pristine twins (audio 2.36% against 3.00%, visual 1.66% against 2.23%; 2,303 ID pairs, three seeds). Nothing in the export, the container, or the surrounding footage reveals the label.

B. Do the processing steps, rather than the generated content, trigger detection?

Here we pass pristine content through the same processing as an edit, with no generated content.

Pristine content, processed like an editVideos flagged at the processed window (three checkpoints)
Audio: original signal through the same cut, reinsertion, and export0/59, 0/59, 0/59
Visual: same retiming and reinsertion0/24, 1/24, 1/24
Visual: same, plus encode–decode through the generator's VAE (Wan2.1)0/24, 1/24, 1/24

The processing chain does not trigger the detector, and neither does the generator's own VAE. When speech in those same audio windows is resynthesized, 56–58 of the 59 videos are flagged. We also re-encoded entire videos with H.264 at CRF 28, which recompresses edited and unedited regions alike: 107 of 184 visual edits are still found at the correct location (144 before re-encoding), with 2 alarms on the 184 pristine twins.

C. Are the cues tied to particular generators? (your "cross-generator validation")

TestResult
Leave-one-generator-out, matched data and budget: MMAudio84.10 audio mAP when seen in training, 71.52 when never seen
Leave-one-generator-out, matched data and budget: Wan2.2 Fun93.92 visual mAP when seen, 91.13 when never seen
A speech synthesizer absent from training (IndexTTS2), on unseen HDTF speakersflagged in 22–26 of 37 videos with the transcript unchanged, and in 29–33 of 38 with the transcript changed
Unseen route and generator (TI2V)68.07 V-mAP; UMMAFormer on the same features: 36.06
Another group's benchmark, commercial generators (TASLE, Kling/Jimeng)54.03 Loc-F1; MSLoc-PG trained on TASLE: 30.85

The model keeps 85% and 97% of its accuracy on generators it never saw, detects a synthesizer it never saw on speakers it never saw, and on TASLE's commercial-generator subset outperforms a model trained on TASLE.

On the drop from ID to OOD. Three observations indicate that it reflects the difficulty of the shift, not a dependence of NFSR on low-level fingerprints. A standard localizer on the same features falls much further (UMMAFormer: 84.76 in-domain, then 33.98 and 36.06, against 49.12 and 68.07 for NFSR). Removing the intermediate-layer features makes transfer worse, not better (OOD-All AV-mAP 62.45 → 53.73), the opposite of what one expects if those layers mainly carried generator-specific fingerprints. And reading the layers at which the linear probe peaks improves held-out-generator mAP by 9–14 points (see our reply to Reviewer YzH6, Q3).

Revision. Section 5.2 has a new paragraph, Controls for construction cues, with full tables and intervals in the appendix, where the task is stated in these terms: localizing synthesized content, whether or not the meaning changes. The Limitations paragraph names the open case: manipulations with no synthesis step, such as splicing real footage, lie outside what this benchmark tests.

AuthorsOpenReview ↗

Response to Reviewer 7fsn (3/5): W3, W5c

W3\color{blue}{\text{W3}}: We agree with you on the presentation and have changed the tables accordingly. On the confound of model scale, we can offer direct evidence, which we summarize below.

Presentation. Tables 2 and 3 no longer rank methods across output interfaces. The best and second-best marks are removed, and the captions state that rows with window-level, single-track, unified-timeline, or text outputs are task references, while the like-for-like comparison is among methods that output separate audio and visual intervals. AASIST's 75.95 AUROC is a detection result and is discussed as such; its 0.06 A-mAP after window-to-interval conversion reflects the output form, not its ability to detect.

Is the gain due to a 30B-scale model? We believe it is not, for three reasons.

NFSR does not run the language model. It reads only the frozen audio and visual encoders of Qwen3-Omni. The 30B parameters belong to the language model, which NFSR never executes.

With features held fixed, the readout design matters. UMMAFormer receives the same frozen Qwen3 features, splits, and training budget.

Same frozen featuresID AV-mAPID AUROCOOD-All AV-mAP
UMMAFormer + Qwen384.76 ± 0.1989.18 ± 0.2943.50 ± 0.64
NFSR + Qwen391.30 ± 0.1396.04 ± 0.1062.45 ± 0.08

With much smaller encoders, the readout is still ahead of every baseline. The same readout on other frozen encoders (native-track mAP, %):

Frozen encoderID audioID visualHDTF audioHeld-out gen. audioHeld-out gen. visualTI2V visual
Whisper-large-v377.3n/a86.831.4n/an/a
BEATs + CLIP93.983.485.956.149.249.6
Qwen2.5-Omni-7B encoders87.593.794.637.258.872.3
Qwen3-Omni-30B encoders92.290.498.448.949.368.1

With BEATs + CLIP, encoders two orders of magnitude smaller than a 30B model, the readout averages 88.7 AV-mAP in-domain, above every baseline in Table 2, including UMMAFormer on Qwen3 features (84.76), and it gives the best held-out-generator audio result in the table. The largest encoder is not the best on every column, so scale does not explain the result.

We also note that the small ActionFormer trained from scratch on AV-FABLE is competitive on its own track (89.22 V-mAP against 90.36 for NFSR), as we state in the paper. What it lacks is the audio track, not capacity.

Seeds. NFSR, UMMAFormer, and the TASLE-trained ActionFormer are three-seed means; across seeds the standard deviation of mAP is at most 0.28 in-domain. The NFSR–UMMAFormer gaps (6.54 points in-domain, 18.95 under shift) are more than twenty times that. The remaining baselines are single runs, and the table caption now says so explicitly; their gaps to NFSR on native-track mAP are tens of points, so no conclusion in the paper depends on single-run differences.

Revision. Section 5.1 now makes these points in the main text.

W5c\color{blue}{\text{W5c}}: We would like to clarify that the source-transfer setting is not contaminated; our † note was unclear and gave that impression. The † marks the baselines that were trained on TASLE (MSLoc-PG and the TASLE-trained ActionFormer). For those two rows the TASLE portion of source transfer is in-domain, which is an advantage for those baselines, not for NFSR. NFSR is never trained on TASLE, and we additionally removed the 54 TASLE examples whose 15 underlying EPIC-KITCHENS recordings also appear in AV-FABLE training.

To show that no conclusion rests on TASLE, we report the results with TASLE removed entirely (three-seed means, identical features and budget):

EvaluationMetricNFSRUMMAFormer
HDTFaudio mAP98.3798.91
Held-out generatorsAV-mAP49.1233.98
Held-out TI2Vvisual mAP68.0736.06
Equal-weight mean of the three71.8556.31

HDTF audio is saturated for both models; on the generator and route shifts NFSR leads by 15 and 32 points. OOD-All is used only as a one-number summary in the ablation table; every claim about a specific kind of shift is made from the per-setting columns of Table 3. Results on the full official TASLE test use that benchmark's own metric and are reported separately in Section 5.2.

Revision. The caption of Table 3 and Section 3.3 now state what the † marks and describe the de-duplication.

Reviewer_orcaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a new dataset, called AV-FABLE, designed to study localized AU manipulations in long-form videos. The dataset is constructed from a few existing datasets and, different from prior datasets, includes both audio and video manipulations in part of the videos that are not necessarily synchronous The main idea behind the dataset is to provide data for benchmarking manipulation detectors on various tasks, including temporal localization, manipulation detection, modality attribution, and potentially also explanations. The focus, however, is clearly on the temporal localization task. Several protocols are also introduced to ensure reproducible testing of future manipulation localization detection models and their generalization abilities. Additionally, the authors also introduce a strong baseline model, named NFSR, which learns to localize audio and video manipulations using frozen features from a pretrained audio-visual LLM and separate decoders for each modality. The model is tested on the proposed benchmark and in comparison to several selected baselines and is shown to lead to strong performance on the evaluated tasks.

Critical strengths

The paper has several strengths that could be highlighted:

  • The paper makes a good case for a novel dataset that includes long-term videos with (potentially asynchronous) AI manipulations both in the video as well as the audio stream. They then introduce AV-FABLE, the proposed dataset, which is reasonably well designed in terms of tasks and experimental protocols. The dataset has the potential to have good impact within the media-forensics community if adopted. Given the unique nature of the dataset there is reasonable probability that this will be the case. The dataset construction procedure is well designed in reasonably well documented and the dataset and accompanying labels have been validated by two reviewers.

  • The paper also introduces a strong baseline, which is expected these days by dataset papers. The proposed baseline model, NFSR, uses features extracted from the Qwen3-Omni model and is shown to yield highly competitive performance on the proposed dataset. The model could be important, but its performance on other related datasets would still need to be validated more comprehensively.

Critical weaknesses

While I do find the paper interesting and see that the dataset and baseline model could be important contributions to the community, I also have several concerns. Below are some that I feel are most critical:

  • The paper was clearly written or heavily edited by an LLM. LLMs tend to make the writing very concise and condensed, which then makes it difficult to read the paper. This is also the case with this submission, where certain parts are written in such a compact manner that I did not understand what the authors were trying to say. As a result, I had to read everything multiple times, look at the somewhat extended sections in the appendix and still fill in some gaps on my own. While I usually don’t like to comment on the language, the condensed writing in this paper, took away from the technical content. Below are a few examples:

o There is a lot of terminology that I feel is non-standard and is used in the paper without proper explanations. The reader eventually figures out what these are, but terms like these should be properly introduced and explained, e.g.: interface features, host conditioning, editing routes, etc.

o In section 3.2, it was not really clear what the sentence was trying to say “ActivityNet (Heilbron et al., 2015) and EPIC-KITCHENS-100 (Damen et al., 2022) provide third person activities and egocentric interactions; HDTF (Zhang et al., 2021) supplies held-out talking-head videos.” My first impression was that some related work slipped into this paragraph, as this sentence was not really related to anything. I needed to look at the supplementary to realize that existing datasets were used as sources of the data. A simple sentence along the lines: “The video material for AV-FABLE was sourced from existing datasets.” would have made the transition much more readable. There are a lot of examples like these in the paper, where the writing is not really-self contained.

o I did not understand how the experimental protocols work. I am aware that there are explanations in the supplementary, but from the main paper and this sentence “We evaluate three OOD settings (Section 5.2): source transfer to HDTF and a source-disjoint TASLE subset, held-out generators on seen routes, and heldout TI2V, a text-and-image-conditioned video generation route withheld together with its generator” this is not really clear.

  • Dataset contributions are important and may have a significant contribution, but only if the datasets are made publicly available. As I understand from the statements at the end of the paper, the dataset, as described, will not be made available, but protocols, timestamps, metadata, generated assets, evaluation code and model configurations will be made available among others. I don’t’ completely understand how much effort will be required to reconstruct the dataset on the end of a researcher using this material. Also, the generation pipeline relies on some existing AI models, which may disappear over time, be updated, change capabilities and so. Is it possible to ensure that exactly the same dataset can be reconstructed, so that experiments are comparable?

  • During the construction of the dataset, the authors used Gemini to plan the manipulations. I did not completely understand if the videos were uploaded for analysis or something else? This needs to be specified more precisely. If video were uploaded to a commercial third party system, is this allowed and were permissions obtained. The EPIC-KITCHENS-100 dataset, for example, includes recordings of people in their everyday activities. It would important to know if the original consents also include third-party AI processing.

  • NFSR performs the best in all experiments on the proposed dataset and the authors highlight its OOD performance. However, the experiments are still conducted within the same dataset, but different OOD scenarios. If I look at table 21 of the supplementary, the model is being outperformed by MSLoc on another dataset. This may need a little more discussion.

  • The paper claims that the context of the entire video needs to be considered by the local manipulation evidence and then proposes a conditioning based a global mean and standard deviation of the features. Are these statistics capturing meaningful contextual information? There is an ablation that shows that this helps performance, but whether this really captured context is not clear.

Questions for authors

I would welcome responses to questions related to the weaknesses highlighted above:

  1. Could the authors provide information on the availability of the dataset in line with the comment made in the section above?

  2. I would appreciate feedback and details on the use of Gemini and the data that was uploaded for planning.

  3. Please comment on the generalization capabilities of the proposed model.

5 comments
AuthorsOpenReview ↗

Response to Reviewer orca (1/4): W1, W2

W1\color{blue}{\text{W1}}: We are grateful for the concrete examples; they showed us exactly where the writing failed. We agree that our main text was too compressed to be read without the appendix, and we have rewritten it. The three passages you quote now read as follows.

  • Terminology. Every term is defined where it first appears, and a new terminology table in the appendix collects them. Interface features are the encoder outputs that the AV-LLM passes to its language model; intermediate-layer features come from a fixed earlier layer of the same encoder. The host video is the real recording that receives an edit, and host conditioning modulates local features with that recording's own feature statistics. An editing route is the type of editing operation together with the input the generator receives (for example masked video-to-video, or text-to-audio); an editing configuration is one generator paired with one route.
  • Section 3.2. The paragraph now opens with: "The source videos come from three existing datasets: ActivityNet for third-person activities, EPIC-KITCHENS-100 for egocentric interactions, and HDTF for held-out talking-head videos."
  • OOD protocol, Section 3.3. The single sentence became one sentence per setting: "Source transfer tests HDTF recordings and a TASLE subset whose source recordings are absent from training. Held-out generators uses unseen generators for editing operations represented in training. Held-out TI2V withholds both a text-and-image-conditioned video generation operation and its generator."

Beyond these three passages, we revised Sections 3–5 so that each paragraph can be read without the appendix. Section 4 now says what each component does, and why, before its equation (we also completed Eqs. 5–6 with the quality and ranking terms used in our implementation; no result changes). Section 5 now reports the numbers behind each claim in the main text: the external TASLE test, the processing controls, the host-conditioning intervention, and the explanation baselines.

Revision. These edits are highlighted in the revised PDF (Section 1, p. 2; Sections 3.2–3.3, pp. 4–5; Section 4, pp. 5–7; Section 5, pp. 8–10).

W2\color{blue}{\text{W2}}: We agree that a dataset contribution only counts if the data is available, and we are glad to clarify this. AV-FABLE will be public, and every researcher will be able to rebuild exactly the dataset used in our experiments. Our original wording, "generated assets", did not make this clear, so we describe below what we release and what a user needs to do.

What we release.

  1. Every edited audio, visual, and audio-visual clip used in our experiments, as fixed files.
  2. A manifest that gives, for each of the 22,153 events, the source-video ID, insertion timestamps, track, generator, route, and labels.
  3. A batch reconstruction script with a pinned environment.
  4. The split, OOD-setting, and threshold-calibration lists.
  5. Evaluation code, model code, and NFSR weights.

The manifest, the lists in (4), the evaluator with its unit tests, and the model code are already in the anonymized supplementary material.

What a user does. Download the source videos from their official providers and run the script. It reinserts our fixed clips at the recorded coordinates and applies the common export profile. No generator and no API is called at any point, so model updates or service shutdowns cannot alter the data. The script checks every rebuilt pair against the manifest. On a 64-core CPU server without a GPU, the full rebuild takes about 26–30 hours.

Why clips plus a script instead of full videos. The source datasets ask users to obtain the original footage from the providers, and we follow that. The edited clips are ours to release. Edits occupy a median of 9.45% of each video, so this route also spares users a 1.71 TB download that would consist mostly of footage they already have.

Revision. Section 3.3 and the Ethics statement now state this release plan explicitly.

AuthorsOpenReview ↗

Response to Reviewer orca (2/4): W3

W3\color{blue}{\text{W3}}: Thank you for raising this and for flagging it for ethics review; we should have stated the data flow precisely in the submission. To answer your question directly: yes, in both steps short clips of source footage were sent to the Gemini API, and this use is permitted by the source licenses.We give the details below.

What Gemini received.

StepInput to GeminiSource media involved?
Edit planningA short clip around the event from the original video together with its caption and event descriptionYes, source-video short clip
Evidence drafting (20,529 events)A short clip around the event from the original video, the same clip from the edited video, and the edit instructionYes, event-centered excerpts only

No full source video was uploaded. Gemini's output never becomes a label on its own: temporal labels come from the executed edits, and every evidence record is checked by two human reviewers against the media.

Why this is permitted.

  • License. EPIC-KITCHENS-100 is released under CC BY-NC 4.0 [1] and HDTF under CC BY 4.0 [2]. Both licenses allow anyone to copy, redistribute, and adapt the material; EPIC-KITCHENS adds the non-commercial condition. Our use is non-commercial academic research, and Creative Commons' own guidance [3] ties "non-commercial" to the purpose of the use, not to whether a paid tool is involved. Sending excerpts to an analysis service is a narrower act than the redistribution and adaptation these licenses already grant.
  • Consent. EPIC-KITCHENS was collected under University of Bristol faculty ethics approval with signed participant consent. Participants reviewed their own footage before it was published, and it was published for open research use under the license above. The recordings are egocentric views of kitchen activity and participants are anonymous. Non-commercial research on this publicly released footage, with no attempt to identify anyone, is the use the participants agreed to.
  • Data handling. We called the API through a paid project. Under the Gemini API terms for paid services, prompts and outputs are not used to train or improve Google's models and are processed under Google's data-processing addendum.
  • Established practice. Peer-reviewed work applies the same kind of commercial multimodal API to the same sources: OSCaR (NAACL 2024) [4] uses GPT-4V on EPIC-KITCHENS content, and Zhang et al. (NAACL 2025) [5] uses GPT-4V on ActivityNet frames.

Revision. Section 3.2 and the AI-use statement now describe both steps and their inputs, and the Ethics statement states the licenses and the data handling. We do not redistribute any source footage. Because no benchmark label depends on Gemini, the evidence-drafting step can also be re-run with a locally hosted model; we release the prompts for this purpose. We welcome the ethics review and are glad to supply any further documentation it needs.

References

[1] EPIC-KITCHENS. EPIC-KITCHENS Dataset. Official dataset website. Accessed November 12, 2026. URL: https://epic-kitchens.github.io/2026

[2] HDTF contributors. HDTF: Dataset and Code for “Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset”. GitHub repository. Accessed November 12, 2026. URL: https://github.com/MRzzm/HDTF

[3] Creative Commons. Frequently Asked Questions, “Does my use violate the NonCommercial clause of the licenses?” Accessed November 12, 2026. URL: https://creativecommons.org/faq/#does-my-use-violate-the-noncommercial-clause-of-the-licenses

[4] Nguyen Nguyen, Jing Bi, Ali Vosoughi, Yapeng Tian, Pooyan Fazli, and Chenliang Xu. OSCaR: Object State Captioning and State Change Representation. Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3565–3576, 2024. DOI: 10.18653/v1/2024.findings-naacl.226. URL: https://aclanthology.org/2024.findings-naacl.226/

[5] Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander G. Hauptmann, Yonatan Bisk, and Yiming Yang. Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward. Proceedings of NAACL 2025: Human Language Technologies (Volume 1: Long Papers), pp. 694–717, 2025. DOI: 10.18653/v1/2025.naacl-long.30. URL: https://aclanthology.org/2025.naacl-long.30/

AuthorsOpenReview ↗

Response to Reviewer orca (3/4): W4, W5, Q1

W4\color{blue}{\text{W4}}: You are right that Table 21 deserved discussion in the main text. The table compares two different settings: NFSR is tested on TASLE without having seen any TASLE data, whereas MSLoc-PG was trained on TASLE and is tested on its own benchmark. It therefore compares cross-dataset transfer with an in-domain specialist. With this in mind, the results support three conclusions, which we now state in Section 5.2.

1. Transfer to TASLE works, and it is stronger than transfer in the other direction.

DirectionModel (training data)TestResult
AV-FABLE → TASLENFSR (AV-FABLE)Official TASLE test, Loc-F150.80
in-domain referenceMSLoc-PG (TASLE)same63.62
TASLE → AV-FABLEMSLoc-PG (TASLE)AV-FABLE ID / held-out generators / TI2V, V-mAP1.90 / 7.65 / 0.92
TASLE → AV-FABLENFSR visual readout (TASLE)AV-FABLE ID, V-mAP25.2
in-domain referenceNFSR visual readout (AV-FABLE)same91.0

Without any TASLE data, NFSR reaches 80% of the score of the TASLE-trained model on TASLE's own test. In the opposite direction, TASLE-trained models retain a small fraction of in-domain accuracy on AV-FABLE.

2. On the parts of the TASLE test that are furthest from its training data, NFSR is ahead. On the subset edited purely with the commercial generators Kling and Jimeng, NFSR obtains 54.03 Loc-F1 against 30.85 for MSLoc-PG; on the TVSum-sourced videos, 43.75 against 38.96 (both in Table 21).

3. The gap on the full TASLE test comes from the training data, not from the architecture. The same readout trained on TASLE reaches 98.30 AUROC and 93.71 visual mAP on TASLE (appendix, training-corpus comparison).

Regarding your remark that the OOD experiments stay within one dataset, we would like to clarify that two of the three settings go beyond it. Source transfer tests HDTF talking-head recordings, a domain absent from training, and a subset of TASLE, which another group built with its own generators and pipeline. The other two settings withhold generators, and in one case the entire editing route.

Revision. The full TASLE result and the key points of this discussion are now in Section 5.2.

W5\color{blue}{\text{W5}}: This is a fair question. As you note, the ablation shows that the module helps, but not whether it helps because it uses information about the specific video. We tested this directly with an intervention.

We freeze one trained checkpoint, the local features, and all thresholds, and change only the statistics fed to FiLM. In the donor condition each video receives the mean and standard deviation of a different recording from the same source dataset with matched duration. In the constant condition every video receives training-set average statistics.

Statistics given to FiLMID AV-mAPID FAROOD AV-mAPOOD FAR
The video's own93.053.1268.983.91
Another video's (same source, matched duration)83.983.9143.7910.94
Training-set average83.245.4752.947.03

128 pairs per split; all values in %. With the video's own statistics, Event-IoU is higher than with donor statistics by 9.88 points on ID (95% CI [5.75, 14.29]) and 13.65 points on OOD ([7.78, 19.86]).

These results show two things. First, the decoder depends on these statistics: giving it another video's values costs 25 AV-mAP points under shift and nearly triples false alarms. Second, what it uses is specific to the individual recording. The donor comes from the same dataset and has the same length, so dataset identity and duration are held fixed, and performance still falls to the level of a constant input, or below it.

This is what we mean by "context". The mean and standard deviation of a recording's features summarize its baseline: capture conditions, compression, acoustic environment, and overall scene content. The same local deviation can be ordinary in one recording and anomalous in another, and FiLM lets the decoder judge it against the right baseline. The effect is largest under distribution shift, where baselines differ most from training, which matches the retraining ablation (+4.73 OOD AV-mAP against +1.44 in-domain).

Revision. Section 4.2 now states this interpretation, and Section 5.3 reports the intervention.

Q1\color{blue}{\text{Q1}}: Yes. As detailed in our response to W2 above, we release every edited clip used in the experiments as fixed files, the full event manifest, the reconstruction script, the split and calibration lists, the evaluation and model code, and the NFSR weights. A user downloads the source videos from their official providers and runs the script; nothing is regenerated and no external service is called, so every user obtains the same dataset. The rebuild takes about 26–30 hours on one CPU server. The manifest, lists, evaluator, and model code are already included in the anonymized supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.