- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SAFER studies whether an agentic video detector should spend extra computation acquiring spatial, spectral, or latent forensic features. It combines a VLM frontline, a dispatcher, and an arbiter, and compares tool-assisted predictions with a same-reasoner no-tool condition. On Chrono-66k and four external benchmarks, the paper reports limited incremental value from tools and substantial gains from target-domain threshold recalibration.
Critical strengths
The matched no-tool comparison avoids the common mistake of comparing a tool-using system with a different model and then attributing every gain to the tools.
The paper also asks a practically relevant question about computation rather than reporting accuracy alone.
Critical weaknesses
-
The main result is that the learned dispatcher almost always declines to acquire evidence. This makes SAFER look less like a general cost-aware agent and more like a carefully measured negative result for one frontline model and one tool suite. The paper should not imply that it has learned a transferable principle for when forensic evidence is worth acquiring unless that principle survives changes in detector, tools, costs, and domains.
-
The term “forensic evidence” is also doing too much work. Tool outputs are treated as evidence, but the paper does not establish that they correspond to causal traces of generation. They may simply be another set of dataset, codec, or generator correlations.
-
In addition, the strongest cross-domain improvement appears to come from recalibrating the decision threshold. If so, the central contribution is calibration under shift, while the agentic routing machinery adds complexity without independent value.
-
Please be rigorous about referencing. For example, since ViF-Bench is heavily used, why only referencing its huggingface link rather than its original paper?
Questions for authors
- Does SAFER outperform a no-tool model plus a simple confidence gate, temperature scaling, or abstention policy?
- What fraction of test videos actually triggers each tool group, and is the gain statistically different from always skipping tools?
- Are the forensic measurements stable across codecs, sources, and generators, or do they merely correlate with benchmark construction?
3 comments
Q1 (continued). Confidence gate under shift. With the in-domain threshold on the default's margin, the gate sends 10 to 25% of external videos to all tools and is significantly worse than always skipping on three of four benchmarks (fixed / broken: ViF-Bench 0 / 37, FakePartsBench 67 / 116, GenBuster 25 / 66, all p < 0.001; GenVidBench 27 / 37, p = 0.26). Frontline confidence does locate the videos where tools change a decision (AUC 0.83 to 0.99 under generator-disjoint cross-validation). A probe trained on the target benchmark's own labels can also separate fixes from breaks among those changes (AUC 0.83 to 0.99 on three benchmarks, 0.61 to 0.70 on GenVidBench). But the direction of the change alone does as well (AUC 0.82 to 0.99 on the same three, 0.70 on GenVidBench): changes toward "generated" are mostly fixes and changes toward "real" are mostly breaks, because unseen generators push scores toward real. That is the operating-point shift again, and re-selecting the threshold corrects it without calling any tool (reply to rZyQ, W2).
Q2. Tool shares and statistical tests. In-domain (Appendix C), the Ultra-Fast and Balanced dispatchers send 100% of test videos to the no-tool exit; Max-Security sends 47.8% to tools (28.9% latent, 17.4% spectral, 1.4% spatial). The confidence gate sends 0.8% of in-domain videos and 6.5 to 18.7% of external videos to tools (Table 13). Exact McNemar tests against always skipping, from the Table 13 counts:
| Test set | All-tools evidence mode: fixed / broken | p | Confidence gate: fixed / broken | p |
|---|---|---|---|---|
| Chrono-66k test | 7 / 1 | 0.07 | 2 / 0 | 0.50 |
| ViF-Bench | 0 / 37 | < 0.001 | 0 / 14 | < 0.001 |
| FakePartsBench | 69 / 116 | < 0.001 | 18 / 34 | 0.04 |
| GenBuster | 25 / 66 | < 0.001 | 9 / 28 | 0.003 |
| GenVidBench | 28 / 37 | 0.32 | 3 / 13 | 0.02 |
In-domain no gain is significant. Under shift, tools make significantly more decisions worse than better on three of four benchmarks. The routing oracle is the only router significantly better than always skipping (12 to 13 errors, p ≤ 0.016).
Q3. Stability of the measurements. We ran an intervention test for this response. 346 held-out real Chrono-66k clips (test and validation splits) were passed, with content fixed, through the Wan2.1 and CogVideoX video VAEs (neither is the SVD VAE the latent tool uses) and, as a control, re-encoded with H.264 at CRF 18, 28 and 35 and H.265 at CRF 28; every copy then went through the same normalization. No feature moves by more than 0.23 SD of its spread over real clips, and every feature the VAEs move, re-encoding moves in the same direction by a comparable amount (high-frequency energy: -0.23 SD for CogVideoX, -0.13 for H.265 at CRF 28; DIRE: -0.20 and -0.09). None of the round-tripped real clips is called generated. The largest real-versus-generated gaps on the benchmarks (about 1 SD for the high-frequency and DIRE features on GenBuster) are four to five times what a VAE round-trip produces. After normalization, the features thus register a loss of high-frequency detail, which compression also causes, rather than a trace specific to a generator's VAE. The test covers VAE decoding only, not diffusion sampling. We also tested whether each feature's direction is stable, on the external rerun with the features saved: 10 of the 23 keep one direction on every benchmark where they are significant, 12 reverse between benchmarks (spatial, spectral and latent alike), and 1 is not significant on two or more. The measurements are therefore not stable traces across sources. The limitations now say they may be dataset and codec correlates, and that the signal they carry with target labels (reply to t7Rq, Q3) does not transfer from training.
Thanks for the following up experiments. I will raise my rating to 2.
Thank you for the direct review. We accept your description of the main result: a carefully measured negative result for one frontline and one tool suite. The revision says that, and the new analyses below test the routing machinery you questioned.
W1. A negative result, not a transferable principle. We removed the claim that SAFER learns a transferable principle. What transfers is the protocol: measure tool value against the same reasoner on the same frames, and record every action's outcome per video. It matters because agentic detectors such as LAVID, Hermes and TUE-Detector report tool gains against a different baseline. In our data that comparison gives a strong pattern (ρ = -0.49 across categories: tools appear to help exactly where the detector is weak) that disappears against the same reasoner (ρ = +0.05, p = 0.75 against scanner AUC; ρ = +0.23, p = 0.13 against arbiter AUC). Figure 3(b) shows this contrast.
W2. "Forensic evidence" does too much work. Agreed. We do not show that the 23 measurements are causal traces of generation, and Appendix F already notes that the cue statistics mix generation traces with encoding differences. The limitations now say so. If they are dataset correlates, that is consistent with our main finding that their value does not transfer. A rerun of the four external benchmarks with the features saved supports this: 12 of the 23 reverse direction between benchmarks, in all three families, and the DIRE features point the way their design assumes (lower reconstruction error for generated videos) only on GenBuster. An intervention test on real clips (Q3) finds no feature that responds to a VAE round-trip beyond what re-encoding does.
W3. The real contribution is calibration under shift. Partly agreed. The routing machinery is not an independent contribution; contribution 1 now describes the dispatcher as a test of whether tool use can be routed. What remains is (1) the protocol, (2) the negative result on tool value, and (3) the split of errors under shift into a threshold-fixable part and a ranking-limited part, with tools fixing neither. Without per-action outcomes and the oracle we could not bound what any router can gain, which is what makes (2) more than "tools did not help in our setup".
W4. Referencing. Thank you. ViF-Bench is now cited to its paper (Li et al., "Skyra: AI-Generated Video Detection via Grounded Artifact Reasoning", arXiv:2512.15693) alongside the dataset release. We also corrected four other entries (two incomplete author lists, two venues).
Q1. Confidence gate, temperature scaling, abstention.