acceptodds
Under review as a conference paper at ICLR 2027

SAFER: WHEN IS ACQUIRING FORENSIC EVIDENCE WORTH THE COST IN AGENTIC AI-GENERATED VIDEO DETECTION?

Abstract

Agentic approaches have recently emerged for AI-generated video detection, allowing vision-language models (VLMs) to acquire and reason over specialized forensic evidence. However, forensic tool calls incur additional computational latency, while their marginal utility relative to this cost remains unquantified. To close this gap, we propose SAFER, a cost-aware agentic framework that treats forensic analysis as evidence acquisition under a computational budget. SAFER combines a VLM frontline with a dispatcher that decides whether additional spatial, spectral, or latent forensic evidence is worth acquiring, and an arbiter that integrates the acquired evidence. A same-reasoner no-tool prediction on every forward pass provides a controlled baseline for measuring the marginal value of forensic evidence without confounding tool utility with differences between models. Experiments on Chrono-66k and four external benchmarks show that the forensic tools contain discriminative signal in isolation, yet provide limited incremental value beyond the VLM frontline. In-domain, full tool acquisition fixes only 6 of 20 remaining errors while increasing latency by 1.71; across 44 unseen generator and editing categories, its mean AUC contribution is . The cost-aware dispatcher consequently learns to bypass unnecessary tool calls, matching or exceeding full-tool baselines while operating at approximately 0.60 their latency. Finally, we demonstrate that generalization failures under domain shift stem primarily from miscalibrated scoring rather than missing forensic evidence; on three of four external benchmarks, simple decision-threshold recalibration using a small target sample improves balanced accuracy by 4.3 to 12.9 points, outperforming adapted tool-routing strategies.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Forensics, Agentic AI, Vision-Language models, Spatial-spectral representation learning, LoRA fine-tuning, Frequency-domain analysis, AI-generated video detection, forensic evidence, multimodal forensics, tool-augmented reasoning, adaptive computation, cost-aware inference, domain shift, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4jrvOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SAFER studies whether an agentic video detector should spend extra computation acquiring spatial, spectral, or latent forensic features. It combines a VLM frontline, a dispatcher, and an arbiter, and compares tool-assisted predictions with a same-reasoner no-tool condition. On Chrono-66k and four external benchmarks, the paper reports limited incremental value from tools and substantial gains from target-domain threshold recalibration.

Critical strengths

The matched no-tool comparison avoids the common mistake of comparing a tool-using system with a different model and then attributing every gain to the tools.

The paper also asks a practically relevant question about computation rather than reporting accuracy alone.

Critical weaknesses
  1. The main result is that the learned dispatcher almost always declines to acquire evidence. This makes SAFER look less like a general cost-aware agent and more like a carefully measured negative result for one frontline model and one tool suite. The paper should not imply that it has learned a transferable principle for when forensic evidence is worth acquiring unless that principle survives changes in detector, tools, costs, and domains.

  2. The term “forensic evidence” is also doing too much work. Tool outputs are treated as evidence, but the paper does not establish that they correspond to causal traces of generation. They may simply be another set of dataset, codec, or generator correlations.

  3. In addition, the strongest cross-domain improvement appears to come from recalibrating the decision threshold. If so, the central contribution is calibration under shift, while the agentic routing machinery adds complexity without independent value.

  4. Please be rigorous about referencing. For example, since ViF-Bench is heavily used, why only referencing its huggingface link rather than its original paper?

Questions for authors
  1. Does SAFER outperform a no-tool model plus a simple confidence gate, temperature scaling, or abstention policy?
  2. What fraction of test videos actually triggers each tool group, and is the gain statistically different from always skipping tools?
  3. Are the forensic measurements stable across codecs, sources, and generators, or do they merely correlate with benchmark construction?
3 comments

Thank you for the direct review. We accept your description of the main result: a carefully measured negative result for one frontline and one tool suite. The revision says that, and the new analyses below test the routing machinery you questioned.

W1. A negative result, not a transferable principle. We removed the claim that SAFER learns a transferable principle. What transfers is the protocol: measure tool value against the same reasoner on the same frames, and record every action's outcome per video. It matters because agentic detectors such as LAVID, Hermes and TUE-Detector report tool gains against a different baseline. In our data that comparison gives a strong pattern (ρ = -0.49 across categories: tools appear to help exactly where the detector is weak) that disappears against the same reasoner (ρ = +0.05, p = 0.75 against scanner AUC; ρ = +0.23, p = 0.13 against arbiter AUC). Figure 3(b) shows this contrast.

W2. "Forensic evidence" does too much work. Agreed. We do not show that the 23 measurements are causal traces of generation, and Appendix F already notes that the cue statistics mix generation traces with encoding differences. The limitations now say so. If they are dataset correlates, that is consistent with our main finding that their value does not transfer. A rerun of the four external benchmarks with the features saved supports this: 12 of the 23 reverse direction between benchmarks, in all three families, and the DIRE features point the way their design assumes (lower reconstruction error for generated videos) only on GenBuster. An intervention test on real clips (Q3) finds no feature that responds to a VAE round-trip beyond what re-encoding does.

W3. The real contribution is calibration under shift. Partly agreed. The routing machinery is not an independent contribution; contribution 1 now describes the dispatcher as a test of whether tool use can be routed. What remains is (1) the protocol, (2) the negative result on tool value, and (3) the split of errors under shift into a threshold-fixable part and a ranking-limited part, with tools fixing neither. Without per-action outcomes and the oracle we could not bound what any router can gain, which is what makes (2) more than "tools did not help in our setup".

W4. Referencing. Thank you. ViF-Bench is now cited to its paper (Li et al., "Skyra: AI-Generated Video Detection via Grounded Artifact Reasoning", arXiv:2512.15693) alongside the dataset release. We also corrected four other entries (two incomplete author lists, two venues).

Q1. Confidence gate, temperature scaling, abstention.

  • Confidence gate. SAFER's learned dispatcher does not beat it. In the submitted Table 3 the gate (a supervised router at λ = 0.05) makes 18 errors against 20 (2 fixed, 0 broken, p = 0.50). A plain margin threshold on frontline confidence, chosen on validation with matched outputs, makes 15 errors calling tools on 0.8% of videos (6 fixed, 1 broken against always skipping, p = 0.13); the best learned router makes 16 (p = 0.22), and GRPO skips tools at every λ ≥ 0.0005 (reply to t7Rq, W2). A probe explains why: frontline confidence predicts where tools change a decision (AUC 0.97, 38 test videos) but not whether the change is a fix (AUC 0.69, CI [0.53, 0.83], 15 fixes vs. 58 breaks pooled). Routing on uncertainty finds the right videos; it cannot tell which of them tools will help.
  • Temperature scaling. It changes no decision at the 0.5 threshold. In-domain it lowers ECE from 0.026 to 0.002 for the default (fitted T = 0.42; the raw scores are under-confident) with errors unchanged at 20. The balanced-accuracy gains in Table 4 need a threshold shift, which is why we now call them an operating-point shift.
  • Abstention. Abstaining on the least confident videos, the default makes 8 errors at 99% coverage and none at 95% (the all-tools evidence mode: 6 and 0). In-domain, abstention removes the remaining errors at far lower cost than tools.

Q1 (continued). Confidence gate under shift. With the in-domain threshold on the default's margin, the gate sends 10 to 25% of external videos to all tools and is significantly worse than always skipping on three of four benchmarks (fixed / broken: ViF-Bench 0 / 37, FakePartsBench 67 / 116, GenBuster 25 / 66, all p < 0.001; GenVidBench 27 / 37, p = 0.26). Frontline confidence does locate the videos where tools change a decision (AUC 0.83 to 0.99 under generator-disjoint cross-validation). A probe trained on the target benchmark's own labels can also separate fixes from breaks among those changes (AUC 0.83 to 0.99 on three benchmarks, 0.61 to 0.70 on GenVidBench). But the direction of the change alone does as well (AUC 0.82 to 0.99 on the same three, 0.70 on GenVidBench): changes toward "generated" are mostly fixes and changes toward "real" are mostly breaks, because unseen generators push scores toward real. That is the operating-point shift again, and re-selecting the threshold corrects it without calling any tool (reply to rZyQ, W2).

Q2. Tool shares and statistical tests. In-domain (Appendix C), the Ultra-Fast and Balanced dispatchers send 100% of test videos to the no-tool exit; Max-Security sends 47.8% to tools (28.9% latent, 17.4% spectral, 1.4% spatial). The confidence gate sends 0.8% of in-domain videos and 6.5 to 18.7% of external videos to tools (Table 13). Exact McNemar tests against always skipping, from the Table 13 counts:

Test setAll-tools evidence mode: fixed / brokenpConfidence gate: fixed / brokenp
Chrono-66k test7 / 10.072 / 00.50
ViF-Bench0 / 37< 0.0010 / 14< 0.001
FakePartsBench69 / 116< 0.00118 / 340.04
GenBuster25 / 66< 0.0019 / 280.003
GenVidBench28 / 370.323 / 130.02

In-domain no gain is significant. Under shift, tools make significantly more decisions worse than better on three of four benchmarks. The routing oracle is the only router significantly better than always skipping (12 to 13 errors, p ≤ 0.016).

Q3. Stability of the measurements. We ran an intervention test for this response. 346 held-out real Chrono-66k clips (test and validation splits) were passed, with content fixed, through the Wan2.1 and CogVideoX video VAEs (neither is the SVD VAE the latent tool uses) and, as a control, re-encoded with H.264 at CRF 18, 28 and 35 and H.265 at CRF 28; every copy then went through the same normalization. No feature moves by more than 0.23 SD of its spread over real clips, and every feature the VAEs move, re-encoding moves in the same direction by a comparable amount (high-frequency energy: -0.23 SD for CogVideoX, -0.13 for H.265 at CRF 28; DIRE: -0.20 and -0.09). None of the round-tripped real clips is called generated. The largest real-versus-generated gaps on the benchmarks (about 1 SD for the high-frequency and DIRE features on GenBuster) are four to five times what a VAE round-trip produces. After normalization, the features thus register a loss of high-frequency detail, which compression also causes, rather than a trace specific to a generator's VAE. The test covers VAE decoding only, not diffusion sampling. We also tested whether each feature's direction is stable, on the external rerun with the features saved: 10 of the 23 keep one direction on every benchmark where they are significant, 12 reverse between benchmarks (spatial, spectral and latent alike), and 1 is not significant on two or more. The measurements are therefore not stable traces across sources. The limitations now say they may be dataset and codec correlates, and that the signal they carry with target labels (reply to t7Rq, Q3) does not transfer from training.

Thanks for the following up experiments. I will raise my rating to 2.

Reviewer_rZyQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when forensic tools are worth their latency cost in AI-generated video detection. The authors propose SAFER, a cost-aware agentic framework that treats forensic analysis as evidence acquisition under a computational budget. SAFER runs a VLM frontline with a dispatcher that decides whether additional spatial, spectral, or latent forensic evidence is worth acquiring. The same reasoner is also run without tools, giving a controlled marginal-value estimate. Experiments across Chrono-66k and four external benchmarks show that tools help only marginally in the in-domain setting, add roughly zero AUC under domain shift.

Critical strengths
  • The same-reasoner counterfactual is a clean and useful design.

  • The format-normalization setting is careful, and the domain-shift evaluation over 44 unseen categories is more informative.

  • The analysis of negative result on tool redundancy is valuable, and the recalibration comparison is insightful.

Critical weaknesses
  • I think that the central claim, forensic tools are largely redundant, is overclaimed. It is only established for one scanner/arbiter pair, one hand-crafted 23-feature pool, and one evidence-graph fusion scheme. More experiments should be provided to support the claim.

  • In-domain performance is already ver high, so there is little room for tools to help. In the setting of domain shift, the AUC analysis is threshold-independent, but the practical conclusion rests on balanced accuracy at a fixed threshold, and the recalibration-versus-routing comparison uses labeled target data differently.

Questions for authors

Please add at least one more backbone pair and a stronger tool-fusion variant to strengthen the experiment.

2 comments

Thank you for the review. We agree that the redundancy claim was stated too broadly, and the revision scopes it.

W1 / Q1. One backbone pair, one feature pool, one fusion scheme. Agreed. The abstract, Sec. 1 and the conclusion now state the result for the detector we study, and the extended limitations section lists one backbone pair, one hand-crafted feature pool read through one evidence graph, untested learned tools, two training generators and a single training seed. Latency is now compared in one harness: re-timed at batch size 1, all tools cost 1.69x the no-tool exit in-domain and 1.67 to 1.69x under shift (Table 1 reports 1.71x at batch size 4). Under the same protocol, the external training-free detector D3 reaches 0.32 to 0.61 AUC against 0.64 to 0.86 for SAFER (reply to t7Rq, Q3). We did not train a second backbone pair for this response. What the paper does show beyond one fixed pipeline is the reason for the null: the features carry signal alone (96.39% macro-F1) and the arbiter reads them (tools change 185 decisions on FakePartsBench under shift), but against the same reasoner their mean value over 44 unseen categories is -0.007 AUC with a CI containing zero. That null is zero-shot. Fit on the target's own labels, the same features add 0.03 to 0.19 AUC over the default (reply to t7Rq, Q3), but 12 of the 23 reverse direction between benchmarks, so the gain needs target labels and is out of reach for a fusion learned in-domain. The revision states the claim in that form. The measurement protocol, which is the transferable part, applies unchanged to other pairs.

W2. In-domain ceiling, threshold dependence, unequal use of labels.

In-domain ceiling. Agreed: at 99.70% macro-F1 there is little room, and we no longer draw a general conclusion from in-domain. Under shift, default balanced accuracy is 53.7 to 73.8%, so room is large, and the routing oracle still bounds any router at 1.6 to 7.2 points (Table 4). Tools also help least where room is largest: without GenBuster, same-reasoner tool value rises with the arbiter's AUC (ρ = +0.40, p = 0.01, Appendix E).

Threshold dependence. The tool result does not rest on the 0.5 threshold. The -0.007 ΔAUC over 44 categories is threshold-free. At each score's own best threshold (Table 13, "Recal."), the all-tools evidence mode and the default reach 62.4 vs. 62.2 on ViF-Bench, 67.5 vs. 67.4 on FakePartsBench, 77.0 vs. 81.2 on GenBuster and 74.0 vs. 75.3 on GenVidBench. In-domain, the new threshold-free metrics are saturated for every mode (TPR at 1% FPR 0.995 to 0.999, AUC ≥ 0.9996), so there is no low-FPR regime in which tools separate.

One number to flag before it misleads: at recalibrated thresholds, tools fix 264 and break 49 decisions on ViF-Bench (Table 13). This comes from ViF-Bench's 95% share of fakes; balanced accuracy moves only from 62.2 to 62.4.

Unequal use of labels. Both arms in Table 4 see the same labeled half: recalibration fits one threshold, routing refit fits a six-way policy. The capacity gap is the finding: a one-parameter fix recovers more than a policy trained on the same labels, because tool-fixable videos are rare (8 of 6,000 in validation; Table 9). The revision states this explicitly in Sec. 4.4.

Recalibrate, then route. We added the combined arm you asked about: on the fitting half, re-select the threshold of every action, then refit the router (and the oracle) on the recalibrated outputs. Balanced-accuracy gain over the default at 0.5, mean over the 10 splits:

BenchmarkRe-selected defaultRe-selected all-tools evidence modeRefit router on re-selected outputsOracle on re-selected outputs
ViF-Bench+7.6+8.1+6.6+20.2
FakePartsBench+4.8+5.1+3.7+11.4
GenBuster+12.5+8.0+12.4+16.0
GenVidBench-1.0-2.9-3.2+4.0

Once thresholds are re-selected, all tools add 0.5 and 0.3 points on two benchmarks and cost 4.5 and 1.9 on the other two, and the refit router falls below the re-selected default on all four. The oracle shows that a per-video choice made with the label known could gain more; no router trained on the same labels captures it.

Presentation. Sec. 3.2 now defines each operating mode once, including that the evidence mode averages scanner and arbiter as the no-tool exit does, and the tables use those names.

Thanks for your response. I keep my score, due to the limited technical novelty.

Reviewer_t7RqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAFER studies cost-aware forensic evidence acquisition for AI-generated video detection. It combines a VLM scanner and arbiter with a GRPO-trained dispatcher that selectively invokes 23 forensic measurements. A same-arbiter no-tool baseline enables controlled evaluation of marginal evidence utility. Experiments on Chrono-66k and four normalized external benchmarks suggest limited incremental benefits from forensic tools, frequent tool skipping, and greater improvements from target-domain threshold adaptation than from routing.

Critical strengths
  • Controlled measurement of forensic utility. The same-reasoner counterfactual (Section 3.3) reduces model-capacity confounding, while Figure 3 demonstrates the misleading conclusions arising from heterogeneous baseline comparisons. This provides a valuable methodological perspective on tool-augmented detection.

  • Comprehensive diagnostic analysis. The external evaluation across 44 unseen categories, format-normalization audit, oracle analysis, and accuracy–latency comparisons provide useful evidence about the limitations of forensic tool acquisition under domain shift.

Critical weaknesses
  • The central calibration conclusion exceeds what the adaptation protocol establishes. Section 4.4 and Table 4 fit thresholds using one generator-disjoint half of each target benchmark, potentially involving thousands of labeled examples. No label-budget analysis supports the rem that a small or minimal sample suffices. More fundamentally, improved balanced accuracy after threshold adjustment demonstrates an operating-point mismatch, not necessarily probabilistic miscalibration. The paper does not distinguish these mechanisms through calibration diagnostics. Moreover, the recalibrated system still achieves only 61.1% balanced accuracy on ViF-Bench (Appendix Table 12). These limitations materially weaken the claim that miscalibrpeated claiation, rather than inadequate transferable representations, primarily explains generalization failures.

  • The experiments do not establish meaningful learned routing beyond a cost-induced no-tool policy. The default dispatchers select no-tool exit for 100% of in-domain samples (Appendix C). Appendix E reports that skipping tools becomes preferable at a cost weight of approximately 0.001, whereas the smallest positive weight used in training is 0.05. Thus, the observed policy may largely follow from reward scaling rather than learning instance-specific evidence utility. Indeed, a simple confidence gate achieves 18 errors versus the default dispatcher's 20 at nearly identical latency (Table 3). Furthermore, tool actions use arbiter-only predictions while no-tool exit averages scanner and arbiter scores, confounding action comparisons. The claim of matching or exceeding full-tool baselines is also imprecise: the strongest all-tools evidence mode achieves 14 versus 20 errors, albeit at 1.71× latency (Table 1). These results support a useful cost–accuracy trade-off, but not a demonstrated advantage of GRPO-based adaptive routing. Broader claims of forensic redundancy should also remain restricted to the evaluated VLM pair and evidence interface.

Questions for authors
  • What are the exact labeled sample sizes and threshold-selection criteria in Table 4? Do the gains persist under smaller, fixed label budgets? Can calibration diagnostics distinguish probability miscalibration from operating-point mismatch?
  • Does GRPO outperform simple confidence-based or supervised routing under matched action outputs and comparable latency budgets, particularly near the empirical cost break-even point?
  • Can the authors evaluate an alternative forensic fusion architecture or strong trace-enhanced detector under the same normalization protocol to substantiate the broader redundancy claim?
  • How much of the near-saturated in-domain performance persists after controlling for residual metadata and dataset-source shortcuts? Can the authors clarify the accuracy–latency claim relative to the strongest all-tools evidence mode?
3 comments

Thank you for the careful reading. Both weaknesses are correct as stated, and we changed the claims to match. W = weakness, Q = question.

W1 / Q1. The calibration claim goes beyond the protocol.

Label counts. Table 4 fits on one generator-disjoint half of each benchmark, about half of its videos (on average over the 10 splits, 82 real and 1,438 generated videos on ViF-Bench, 1,000 and 2,789 on FakePartsBench, 500 and 830 on GenBuster, 250 and 500 on GenVidBench). That is not a small sample. We removed "small" and "minimal" from the abstract, Sec. 1, Sec. 4.4 and the conclusion, and Sec. 4.4 now reports how few labels suffice (below). Appendix D already reports one low-label case: a threshold set at 95% specificity from labeled real videos only gains +2.7 points on ViF-Bench and +5.3 on GenBuster without any labeled fake.

Label budget. We drew k labeled videos from the fitting half (half real, half generated; 50 draws for each of the 10 splits) and re-selected the threshold on them. Balanced-accuracy gain on the held-out half:

k labeled videosViF-BenchFakePartsBenchGenBuster
10+4.5+1.4+7.9
25+5.4+2.8+10.3
50+6.0+3.3+11.3
100+6.3+3.8+11.9
500+7.3+4.4+12.5
Full half+7.6+4.8+12.5

With 50 labeled videos, threshold re-selection recovers 79%, 69% and 90% of its full-half gain; with 100, it recovers 83%, 79% and 95%. A router refit on the same k labels gains at most +2.0 points (GenBuster, k = 50) and about zero elsewhere. On GenVidBench the default is already near its best threshold, and re-selection does not help (-1.0 ± 1.7 points with the full half). The full-half values are re-derived from the per-video tables with the paper's splits and agree with Table 4 within 0.5 points.

Operating point, not miscalibration. We agree, and the revision says "operating-point shift" throughout. We also ran the diagnostics you asked for, in-domain (new appendix paragraph "Probability calibration in-domain"):

ScoreECE rawECE after temperatureFitted TErrors (raw / after)
SAFER default0.0260.0020.4220 / 20
No-tool arbiter0.0240.0030.5643 / 43
All-tools evidence mode0.0270.0020.3614 / 14

So the in-domain scores are under-confident probabilities (T < 1) but correctly thresholded: the Youden threshold fit on validation is 0.49 for the default and 0.48 to 0.51 for the other arbiter scores, and temperature scaling changes no decision. Under shift the picture reverses: specificity on real videos stays at 88.8 to 99.4% while fake recall falls to 8.1 to 58.9% (Table 11), and re-selecting the threshold gains 4.3 to 12.9 points (Table 4). This is a location shift of the operating point.

Calibration under shift. We ran the same diagnostics on the external benchmarks, fitting temperature on the fitting half and evaluating on the other half. Under shift the scores are also miscalibrated, in the opposite direction: they are over-confident (fitted T = 3.3 to 148; ECE 0.41 to 0.83 on ViF-Bench, FakePartsBench and GenBuster, and 0.21 on GenVidBench). Temperature scaling roughly halves ECE on the first three but changes no decision, so balanced accuracy stays at 53.3, 62.3 and 67.5%. Re-selecting the threshold moves it to 60.9, 67.1 and 80.0%. Both effects are present under shift, and only the operating-point shift accounts for the loss in accuracy. The revision says exactly this in place of "miscalibrated scoring".

ViF-Bench at 61.1%. Agreed. Threshold re-selection does not change AUC, and ViF-Bench AUC is 0.646, so ranking is the limit there. The revision replaces "miscalibrated scoring" with "operating-point shift", and per benchmark the picture is: on GenBuster and FakePartsBench most thresholded errors are fixed by a threshold shift; on ViF-Bench ranking limits accuracy; tools fix neither, at the default or at recalibrated thresholds (Table 13, "Recal." column).

W2 / Q2. Routing beyond a cost-induced no-tool policy.

We agree, and we no longer claim the dispatcher learned instance-level evidence utility. You were right that we trained only λ ≥ 0.05. We retrained the GRPO dispatcher for λ ∈ {0, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2} with three new seeds each (Sec. 4.3 and the new Appendix E paragraph "Latency-weight sweep"):

  • For every λ ≥ 0.0005, all three seeds send every test video to the no-tool exit (20 errors). This matches the analytic break-even near λ = 0.001 in Appendix E: with 8 tool-fixable videos out of 6,000 in validation, skipping is optimal at almost any non-zero cost.
  • At λ = 0, the dispatchers call tools on 52 to 98% of videos and make 21 to 25 errors (fixed / broken against always skipping: 3/8, 5/7, 4/5; all p ≥ 0.22).

Matched outputs. You were also right that tool actions returned the arbiter alone while the no-tool exit averaged scanner and arbiter. We retrained every router with matched outputs (every action returns the scanner-arbiter average):

Router (matched outputs)ErrorsTool usep50 latencyFixed / broken vs. always skipMcNemar p
Always skip (SAFER default)200%239 ms——
GRPO, λ = 0 (3 seeds)15–1699–100%325–344 ms5–6 / 1–2≥ 0.12
GRPO, λ ≥ 0.0005 (3 seeds)200%239 ms0 / 01
Supervised router, best λ (0.01)162.1%240 ms5 / 10.22
Margin threshold on frontline confidence, chosen on validation150.8%240 ms6 / 10.13
Routing oracle120.1%239 ms8 / 00.008

No router differs significantly from always skipping; the oracle does. In-domain, the simplest router, a margin threshold that sends the least confident videos to all tools, does as well as any learned one. Under shift the same kind of gate breaks about three decisions for each one it fixes (89 broken, 30 fixed across the four benchmarks; Table 13), so for this detector no router is worth deploying over skipping, and the dispatcher serves as the measuring instrument. (The submitted Table 3 "confidence gate" is the supervised router at λ = 0.05: 18 errors, 2 fixed / 0 broken.)

Accuracy-latency claim. "Matching or exceeding full-tool baselines" was imprecise. The abstract now states the exact claim: the default matches or beats the all-tools evidence mode in AUC on all four external benchmarks at 0.60x its latency. In-domain, against the static all-tools arbiter, the default makes 20 vs. 31 errors at 0.69x its latency; against the matched all-tools evidence mode it makes 20 vs. 14 errors (7 fixed, 1 broken, p = 0.07) at 0.58x the latency.

Scope. The redundancy claim is now stated for the detector we study (one scanner/arbiter pair, one feature pool, one fusion), in the abstract, Sec. 1 and the limitations.

Q3. Alternative fusion and a trace-enhanced detector. We did not add a second fusion scheme or a learned trace detector to SAFER; the limitations now state that learned tools are not tested and that the results concern the features as this arbiter reads them. To bound what a better fusion could gain, we reran the four external benchmarks with the 23 features saved (same normalization; on ViF-Bench, where the video IDs match, 99.7% of decisions are unchanged) and fit a fusion on each target's own labels, training on half of the generators and testing on the unseen half (10 splits). With those labels the features do add signal: gradient-boosted trees on the VLM scores plus the features reach an AUC of 0.67, 0.92, 0.94 and 0.94 on ViF-Bench, FakePartsBench, GenBuster and GenVidBench, against 0.64, 0.72, 0.85 and 0.87 for the default with no fitting. That signal is not portable: 12 of the 23 features reverse direction (generated vs. real) between benchmarks, so a fusion learned in-domain cannot use it, and the gain needs labeled target videos. The bound is also optimistic, because each benchmark has a single source of real videos, shared by both halves, so the fit can partly learn that source. The revision therefore scopes the redundancy claim to the zero-shot setting, in which the detector sees no target labels, and reports the bound in the appendix and the limitations. The arbiter does read the features in that setting (under shift, tools change 185 decisions on FakePartsBench and 91 on GenBuster, Table 13); they just do not help it. We also ran D3 (ICCV 2025), a training-free trace detector, on the same normalized videos with its official code (XCLIP-B/16 encoder). It reaches an AUC of 0.56, 0.54, 0.61 and 0.32 on ViF-Bench, FakePartsBench, GenBuster and GenVidBench, against 0.64, 0.71, 0.85 and 0.86 for the SAFER default on the same videos (Table 17). Its direction reverses on GenVidBench, as 12 of our 23 features do between benchmarks. Added to the target-label fusion, it gains at most 0.03 AUC (FakePartsBench) and at most 0.001 on ViF-Bench and GenBuster, and it is unstable on GenVidBench, where its direction differs between the two generators.

Q4. In-domain shortcuts. After normalization the metadata classifier still reaches 0.80 AUC on Chrono-66k (Table 15), and real and generated clips differ in content (Kinetics actions vs. VidProM prompts). We agree this inflates in-domain accuracy; the revised Sec. 4.1 no longer calls the in-domain results "robust against shortcut learning" and states that some format signal remains. This is why the main claims rest on the external benchmarks, where normalized metadata AUC is 0.18 to 0.50 and default balanced accuracy is 53.7 to 73.8%. The in-domain numbers serve as a ceiling: even there, with an AUC of 0.9999, the remaining errors sit in the least confident 1% of videos (8 errors at 99% coverage, 0 at 95%).

Comments on the paper

1 comment

We thank the reviewers for close and specific reviews. All three accept the same-reasoner counterfactual as the right way to measure tool value, and all three object to claims that go past what our experiments show. We agree with most of these objections. The revised PDF narrows the claims and adds new analyses; the individual replies give the details.

What the paper now claims. SAFER is a controlled measurement of when forensic evidence is worth its cost in a VLM-based detector. It contributes (1) a measurement protocol: a same-reasoner no-tool prediction on every pass plus per-action outcome tables; (2) the finding that, for the detector we study, tools add little beyond the VLM frontline in-domain (6 of 20 errors removed, p = 0.07, at 1.71x the latency) and nothing measurable on average across 44 unseen categories (-0.007 AUC, 95% CI [-0.017, 0.003]); and (3) a split of errors under shift into a part that re-selecting the decision threshold fixes and a part limited by ranking, with tool routing fixing neither. The GRPO dispatcher is one of several routers we use to test (2) and (3), not a method contribution. The title is now "SAFER: Measuring When Forensic Evidence Is Worth Its Cost in VLM-Based AI-Generated Video Detection".

What we removed or corrected. "Matching or exceeding full-tool baselines" (now the exact comparison), "small" and "minimal" target sample, "mathematically optimal policy", "miscalibrated scoring" where we showed an operating-point shift, about twenty further overclaims, and the ViF-Bench citation (now Li et al., Skyra, arXiv:2512.15693). We added a definition of the operating modes to Sec. 3.2 and an extended limitations section.

New analyses in the revision (in-domain: Chrono-66k test split, 6,676 videos; external: the four benchmarks, 10 generator-disjoint splits):

ConcernAnalysisResult
Constant policy forced by λ (t7Rq, 4jrv)GRPO retrained for 10 values of λ from 0 to 0.2, 3 seeds each, native and matched outputs; McNemar against always skippingEvery λ ≥ 0.0005 skips tools on every video (20 errors). At λ = 0, 15 to 25 errors, all p ≥ 0.12
Matched outputs (t7Rq)Every action returns the scanner-arbiter average; GRPO and the supervised router retrained; a margin threshold on frontline confidence addedBest learned router: 16 errors (p = 0.22). Margin threshold chosen on validation: 15 errors at 0.8% tool use (p = 0.13)
Is any router significant? (4jrv)Exact McNemar of every router against always skippingNo router; only the per-video oracle (12 to 13 errors, p ≤ 0.016)
Calibration vs. operating point (t7Rq, 4jrv)ECE, Brier, NLL before and after temperature, Platt and threshold shiftECE 0.024–0.027 falls to 0.002–0.003 with temperature; every decision unchanged; validation thresholds already near 0.5
Can tool value be predicted? (4jrv)Probes on the frontline stateConfidence predicts where tools change a decision (AUC 0.97), not whether the change helps (0.69, CI [0.53, 0.83])
Abstention (4jrv)Risk-coverage8 errors at 99% coverage, 0 at 95%
Threshold-free metrics (rZyQ)TPR at 1% and 5% FPR, partial AUCAll modes saturated (TPR at 1% FPR ≥ 0.995)
How few labels suffice (t7Rq)Threshold re-selection and router refit from k = 10 to 500 labeled videos50 labels recover 69 to 90% of the full-half gain; a router on the same labels gains at most 2 points
Calibration under shift (t7Rq, 4jrv)ECE before and after temperature; balanced accuracy after temperature vs. thresholdOver-confident (ECE 0.21 to 0.83); temperature changes no decision; the threshold gains 4.8 to 12.5 points
Equal label use (rZyQ)Re-select every action's threshold, then refit the routerRouter below the re-selected default on all four benchmarks
Routing under shift (4jrv)Confidence gate vs. always skip; fix-vs-break probesGate significantly worse on 3 of 4; what predicts fixes is the direction of the score shift

Numbers already in the submitted paper are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.