- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AV-FABLE, a benchmark for detecting and temporally localizing sparse audio-visual manipulations in long-form videos. The benchmark contains 16,650 pristine–manipulated video pairs and adopts separate audio and visual timelines to support modality-specific localization. The authors further investigate the forensic capabilities of audio-visual large language models (AV-LLMs), showing that their zero-shot language outputs perform poorly despite useful forensic information being accessible from frozen perceptual representations. Motivated by this observation, they propose NFSR, which combines intermediate-layer and interface features, host-conditioned modulation, and modality-separated temporal decoders. Experiments demonstrate strong in-domain performance and improvements over existing baselines across three out-of-distribution evaluation settings.
Critical strengths
-
Meaningful benchmark setting and comprehensive dataset construction. AV-FABLE extends audio-visual manipulation localization to long-form, open-scene videos. Its dual-timeline evaluation, diverse editing configurations, pristine–manipulated pairs, and human-reviewed annotations provide a valuable resource for forensic research.
-
Interesting findings on forensic information in AV-LLM representations. The contrast between near-chance zero-shot detection and strong supervised readouts of frozen representations is insightful. The layer-wise analyses further highlight the distinction between forensic information accessibility and downstream utilization.
-
Strong empirical results and extensive evaluations. NFSR achieves strong in-domain and OOD performance, outperforming UMMAFormer under matched feature and training settings. Comprehensive ablations, encoder comparisons, and multi-seed experiments further support its effectiveness.
Critical weaknesses
- Insufficient connection between the research motivations and the proposed methodological solution.
The paper highlights two important but somewhat distinct problems: the need for modality-specific temporal localization in asynchronous audio-visual manipulations, and the gap between forensic information encoded in AV-LLM representations and its accessibility through language outputs. However, the relationship between these problems and the necessity of the proposed architecture is not sufficiently established.
In particular, while separate audio and visual timelines are well motivated as an evaluation protocol, it remains unclear what distinctive modeling difficulty is introduced by asynchronous manipulations beyond requiring independent predictions for each modality. NFSR largely addresses this through independently parameterized unimodal temporal decoders. The paper does not sufficiently analyze whether asynchronous events are inherently more challenging than temporally aligned events, or whether the proposed architecture provides specific advantages under such conditions. Consequently, the benchmark motivation and representation-readout methodology appear somewhat loosely connected rather than forming a fully unified research question.
- The extent to which localization performance depends on dataset-specific generation artifacts remains unclear.
Although the pristine–manipulated twin design and matched export procedures help reduce potential shortcuts, these controls primarily address content and file-level processing differences. They do not fully exclude generator- or editing-pipeline-specific artifacts within manipulated intervals, such as local resampling traces, synthesis textures, or acoustic discontinuities.
This distinction matters because high localization accuracy could partly arise from detecting such artifacts rather than learning forensic evidence that generalizes across manipulation processes. The substantial performance degradation under held-out generators and certain media corruptions further motivates investigating this issue, although these results do not themselves establish shortcut learning. More targeted controls, such as artifact-matched pristine segments, would strengthen the validity and generalizability of the benchmark conclusions.
- The methodological novelty and interpretation of the representation findings could be further strengthened.
NFSR primarily combines established components, including gated feature projection, FiLM-based conditioning, multi-scale temporal localization, and independent modality-specific decoders. While the combination is empirically effective, the paper provides limited evidence of a fundamentally new modeling principle beyond adapting existing techniques to the proposed benchmark.
Moreover, the comparison between zero-shot language outputs and supervised linear probes involves different supervision conditions and prediction mechanisms. Therefore, it establishes a useful descriptive performance gap but does not fully isolate whether the limitation lies in the language pathway, insufficient task adaptation, or the output formulation. A more systematic comparison between representation accessibility and downstream supervised adaptation would strengthen this central interpretation.
Additionally, although AV-FABLE supports explanation evaluation, NFSR itself does not generate forensic explanations, and the separately evaluated explanation models remain relatively weak. This leaves a gap between the broader forensic capabilities supported by the benchmark and those demonstrated by the proposed method.
Questions for authors
Q1. Could the authors provide a more detailed analysis of localization performance on temporally aligned versus asynchronous audio-visual manipulations? In particular, does NFSR offer greater advantages as the temporal mismatch between modalities increases?
Q2. Could the authors further investigate whether the high localization performance depends on generator- or editing-pipeline-specific artifacts? For example, would artifact-matched pristine controls help distinguish transferable forensic evidence from construction-specific shortcuts?
Q3. How strongly does layer-wise linear-probe performance correlate with downstream supervised adaptation performance? Would the layers achieving the highest linear-probe AUROC also yield the best temporal localization or language-based detection performance after matched supervised adaptation?
Q4. Given that AV-FABLE includes explanation as a benchmark task, how do the authors envision integrating evidence-grounded explanations with NFSR? Could the authors clarify whether explanation is intended as a separate future research direction or part of the proposed forensic framework?
6 comments
Response to Reviewer YzH6 (2/5): W2, Q2
: We agree that twin pairs and matched export control content and file-level differences, not what happens inside the edited interval, and we have run the artifact-matched pristine controls you propose. The results separate three things that our submission left entangled: the processing applied around an edit, the synthetic content itself, and the identity of the generator.
Artifact-matched pristine segments. Pristine content goes through the same processing as an edit, with no generated content.
| Pristine content, processed like an edit | Videos flagged at the processed window (three checkpoints) |
|---|---|
| Audio: original signal through the same cut, reinsertion, and export | 0/59, 0/59, 0/59 |
| Visual: same retiming and reinsertion | 0/24, 1/24, 1/24 |
| Visual: same, plus encode–decode through the generator's VAE (Wan2.1) | 0/24, 1/24, 1/24 |
Resampling, splicing, loudness matching, re-export, and even the generator's own VAE do not trigger the detector. When speech in those same audio windows is resynthesized, 56–58 of the 59 videos are flagged. Detection responds to synthesized content and not to the pipeline that inserts it.
Nothing outside the edit carries the label. A probe restricted to unedited context, at least 2 s from any edit, is at chance on every test set, while the same probe on edit-overlapping windows is not.
| Test set | Pairs | Unedited context only, AUROC [95% CI] | Overlapping the edit, AUROC [95% CI] |
|---|---|---|---|
| ID | 2,208 | 49.80 [49.49, 50.13] | 82.57 [81.63, 83.54] |
| HDTF | 359 | 50.74 [49.88, 51.62] | 64.18 [62.39, 66.51] |
| Held-out generators | 247 | 50.04 [49.46, 50.65] | 71.30 [68.88, 74.07] |
| Held-out TI2V | 247 | 50.00 [50.00, 50.00] | 66.84 [63.89, 69.71] |
The evidence transfers across manipulation processes. This is the part of your concern that bears on generalization, so we tested it with matched leave-one-generator-out training: two models with identical data size, sources, editing operation, and budget, one of which never sees the target generator.
| Withheld generator | Test pairs | Seen in training | Never seen |
|---|---|---|---|
| MMAudio (audio mAP) | 188 | 84.10 | 71.52 |
| Wan2.2 Fun (visual mAP) | 149 | 93.92 | 91.13 |
The model keeps 85% and 97% of its accuracy on a generator it never saw. Two further checks point the same way. A speech synthesizer absent from training (IndexTTS2), applied to unseen HDTF speakers, is flagged in 22–26 of 37 videos with the transcript unchanged and in 29–33 of 38 with the transcript changed. And on TASLE, built by another group with a different pipeline, NFSR obtains 54.03 Loc-F1 on the commercial-generator subset against 30.85 for the TASLE-trained MSLoc-PG.
On robustness to corruption. Re-encoding the whole video with H.264 at CRF 28 recompresses edited and unedited regions alike: 107 of 184 visual edits are still found at the correct location (144 before), with 2 alarms on the 184 pristine twins. Heavier corruption lowers recall, as the robustness table reports.
In summary, what NFSR detects is synthesized content inside the edited interval; this evidence is absent from processed pristine material and from unedited context, and most of it carries over to generators and a benchmark that the model never saw.
Revision. Section 5.2 now contains a paragraph, Controls for construction cues, with the full tables and intervals in the appendix.
: Yes, such controls help, and we have run them. The full results are in our response to W2 above; the table below summarizes them.
| Question | Control | Result |
|---|---|---|
| Does the processing around an edit trigger detection? | Pristine audio through the same cut, reinsertion, and export | 0 of 59 videos flagged, all three checkpoints |
| Pristine video through the same retiming and reinsertion, with and without the generator's VAE | 0–1 of 24 flagged in both conditions | |
| Is there a cue outside the edit? | Probe on unedited context only | 49.8–50.7 AUROC (chance) on all four test sets |
| Is the evidence specific to the training generators? | Matched leave-one-generator-out | 71.52 / 84.10 audio mAP (MMAudio) and 91.13 / 93.92 visual mAP (Wan2.2 Fun), never seen / seen |
| A synthesizer absent from training, on unseen speakers | flagged in 22–26 of 37 videos |
The outcome of the artifact-matched pristine controls is clear: processing alone does not trigger detection, while synthesized content in the same windows does. They therefore do what you intended, which is to separate construction-specific shortcuts from evidence carried by the manipulated content, and the leave-one-generator-out results show that most of that evidence transfers to generators the model never saw.
Revision. We have added these controls to Section 5.2 and the appendix, and the release will include the processed-pristine clips so that others can run the same check on their own detectors.
Response to Reviewer YzH6 (3/5): W3a
: This is an important question for the paper, and we would like to answer it directly. We chose standard components deliberately; what NFSR contributes is a principle about where and how a frozen AV-LLM should be read for forensics, together with evidence that the principle matters. It has three parts, each tied to a measurement.
- Read below the language interface, at the depth where the evidence is most accessible. Linear-probe AUROC peaks at intermediate layers (visual block 6: 88.44) and falls toward the final block that feeds the language model (80.55). Removing the intermediate-layer features costs 6.55 AV-mAP in-domain and 8.72 under shift.
- Judge local evidence relative to the host recording. Replacing a video's statistics with those of another recording from the same dataset and of the same length lowers OOD AV-mAP from 68.98 to 43.79 and nearly triples false alarms.
- Keep the evidence path separate for each modality. A visual head that can see the audio fires on audio-only speech edits 10.5 points more often than on their pristine twins (95% CI [6.7, 14.6]); the separated head shows no increase.
That the principle carries weight, and not the particular blocks, is shown by a comparison in which the features are identical and only the way they are read changes:
| Same frozen Qwen3 features, same budget | ID AV-mAP | Held-out generators AV-mAP | Held-out TI2V V-mAP |
|---|---|---|---|
| UMMAFormer, a standard multimodal localizer | 84.76 | 33.98 | 36.06 |
| NFSR | 91.30 | 49.12 | 68.07 |
The alternatives that go through the language model do not come close: fine-tuning Qwen3-Omni to output timestamps reaches 37.80 AV-mAP, and using it to verify or re-score NFSR's candidates leaves localization unchanged (73.60–73.62 mean mAP). The principle is also not tied to one model: the same readout works on Qwen2.5-Omni and on BEATs + CLIP.
We therefore see the novelty in the finding-driven design and in the demonstration that, for forensics, an AV-LLM is better read natively than queried.
Revision. Section 4 now opens with this statement, and each component cites the prior work it builds on.
Response to Reviewer YzH6 (1/5): W1, Q1
W1: This comment led us to state the link between the two problems, which our submission left implicit. We see them as one question asked at two levels: where should a forensic decision be read from?
At the level of evaluation, the dual timeline says that a decision must be made per modality, because a correct time on the wrong track is an error. At the level of representation, our analysis says that the evidence itself is per modality and sits inside each encoder: it is most accessible at intermediate depth and weakens toward the fused language interface. NFSR follows from putting the two together: read each modality's evidence where it is strongest, and decide on that modality's own timeline.
You ask whether asynchronous edits are harder to model, beyond needing two outputs. Our analysis shows that they are. We grouped the both-track test videos by how much their audio and visual edits overlap in time (three seeds per model):
Values are AV-mAP (%). Every model loses accuracy when the two edits separate in time, so asynchrony is a difficulty of its own and not only a matter of output format. The loss is three times larger for UMMAFormer, a standard localizer given the same features, than for NFSR. When the edits coincide, all three models are equivalent; the architecture matters exactly when the tracks diverge.
Our response to Q1 gives the paired intervals.
Revision. Sections 1 and 4 now present the argument in this order: dual-timeline task, where the evidence lives, and the readout that follows.
Q1: Thank you for suggesting this analysis. The answer is yes: NFSR's advantage over a standard localizer on the same features appears when the audio and visual edits separate in time, and it is absent when they coincide.
We grouped the both-track ID test videos by the temporal IoU between their audio and visual edit intervals and evaluated three seeds per model. Each cell is AV-mAP / Event-IoU (%).
Paired differences in Event-IoU, in percentage points, with 95% bootstrap intervals over source videos:
Against the standard localizer, the difference moves from −3.5 points when the edits coincide to +18 and +26 points when they do not. In AV-mAP, UMMAFormer loses 29.7 points between high overlap and disjoint, and NFSR loses 10.1.
Against our cross-modal variant, which already decodes each track separately and adds cross-attention between the branches, NFSR is on par for disjoint edits (the interval includes zero) and slightly behind when the edits coincide. This comparison isolates what matters: per-track decoding is what handles asynchrony. Letting the branches exchange information adds nothing for asynchronous edits, gives a small gain for co-timed ones, and has a cost that we measured separately: on audio-only speech edits the cross-modal visual head raises 10.5 points more false alarms than on pristine twins (95% CI [6.7, 14.6]), while the separated head raises none. We chose separation for that reason.
The partial group has 11 pairs and we draw no trend from it alone.
Revision. Both tables are in the appendix, and Section 5.3 summarizes them.