acceptodds
Under review as a conference paper at ICLR 2027

Verus-Insight: Towards Explainable Artifact Detection in AI-Generated Videos

Abstract

Recent advances in video generation have made synthetic videos increasingly realistic, placing greater demands on reliable AI-generated video detection. Existing artifact-based detectors typically identify generation artifacts as evidence for source prediction, yet they may either fail to recognize subtle artifacts or reach the correct source decision without faithfully identifying the evidence underlying it. In this work, we systematically investigate how to design and train effective artifact detectors and formulate artifact detection as a structured process comprising three complementary capabilities: Inspection Planning, which determines what should be examined; Evidence Verification, which verifies whether the inspected content exhibits genuine generation artifacts; and Evidence Aggregation, which integrates the verified evidence into the final source decision. Building on this formulation, we introduce Verus-Insight, trained with human-anchored full-process supervision and progressive capability training to strengthen both fine-grained artifact detection and video-level detection. We further introduce Verus-Bench, a benchmark with fine-grained human annotations of perceptible artifacts for evaluating artifact detection. Extensive experiments across diverse benchmarks demonstrate that Verus-Insight achieves strong performance in both fine-grained artifact analysis and AI-generated video detection, with substantial gains in recovering human-perceived artifacts. Further analyses reveal that the three capabilities are complementary yet distinct, and that Verus-Insight more reliably converts relevant inspection targets into supported artifact evidence without increasing spurious artifact claims.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, Video Generation, Artifact Detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EmVUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Verus-Insight, an AI-generated video detector that works in three steps. It first plans what to inspect, then checks each target for generation artifacts, and finally combines the evidence into a real/fake verdict. The model is built on Qwen3-VL-8B and trained with warm-up tasks, supervised fine-tuning on human-annotated artifact data, and reinforcement learning. The authors also construct Verus-Bench, a benchmark with fine-grained human artifact annotations. Results show clear gains over the base model in both detection accuracy and artifact identification.

Critical strengths
  • A clear three-stage decomposition of artifact detection. The paper splits detection into Inspection Planning, Evidence Verification and Evidence Aggregation, and evaluates each stage separately. This shows that existing detectors often reach the right verdict with the wrong evidence, which is a useful insight.
  • A fine-grained benchmark. Verus-Bench gives each artifact a human description, a time interval and bounding boxes. This makes it possible to test whether detectors find the actual artifacts, not just the correct label.
Critical weaknesses
  • The same model is the reward judge, the evaluator, and a baseline. Gemini 3.1 Pro computes the RL reward (Eq. 1, App. C.2), scores Plan and Finding F1 for all models (App. E). Verus-Insight is therefore trained against the judge that later grades it, so part of the Finding F1 gain may reflect fitting this judge rather than better artifact recovery.
  • Spatiotemporal grounding is never evaluated. The method relies on explicit temporal and spatial grounding, and Verus-Bench provides frame-wise boxes. Yet all metrics are text-only: time and boxes are explicitly excluded from matching (App. E.2). The paper never shows that the predicted locations are correct.
Questions for authors

Please address the points raised in the Weaknesses section (I would be willing to raise my score if the authors provide evaluation with an independent judge and localization metrics for matched findings.).

2 comments
AuthorsOpenReview ↗

Response to W1: Evaluation with Independent Judges

W1: The same model is the reward judge, the evaluator, and a baseline. Gemini 3.1 Pro computes the RL reward (Eq. 1, App. C.2), scores Plan and Finding F1 for all models (App. E). Verus-Insight is therefore trained against the judge that later grades it, so part of the Finding F1 gain may reflect fitting this judge rather than better artifact recovery.

Thank you for highlighting the potential evaluation bias arising from using Gemini 3.1 Pro for both RL reward computation and fine-grained evaluation. We agree that this overlap warrants independent verification.

To address this concern, we re-evaluated all six models using two independent semantic-matching judges: Gemini 4 Argon and GPT-6.1 Sol, neither of which was used for RL reward optimization. We kept the original model predictions, extracted findings, ground-truth annotations, and evaluation samples fixed, replacing only the semantic-matching judge. The evaluation covers 674 videos and 4,044 model–video predictions across Verus-Bench and RewardData.

Table 1: Independent-Judge Evaluation on Verus-Bench (F1, %)

ModelGemini 3.1 PlanGemini 3.1 FindingGemini 4 PlanGemini 4 FindingGPT-6.1 PlanGPT-6.1 Finding
Verus-Insight24.7015.9727.7118.6421.6915.97
GPT-5.531.2823.5333.8726.9222.6518.55
Qwen3-VL-8B17.087.5621.129.5812.496.05
Skyra15.627.2620.038.9711.545.56
BusterX++10.173.1511.333.785.321.57
VideoVeritas14.225.3014.908.399.035.08

Table 2: Independent-Judge Evaluation on RewardData (F1, %)

ModelGemini 3.1 PlanGemini 3.1 FindingGemini 4 PlanGemini 4 FindingGPT-6.1 PlanGPT-6.1 Finding
Verus-Insight31.3335.4230.9738.5426.5928.65
GPT-5.524.0218.0824.8218.8120.2215.71
Qwen3-VL-8B18.673.3818.133.5914.222.56
Skyra26.4614.1127.1617.4220.8110.21
BusterX++8.172.739.163.525.781.59
VideoVeritas17.888.4720.979.7517.227.57

The results show consistent conclusions across all three judges. In particular, the Finding F1 ranking of all six models remains unchanged on both benchmarks. Verus-Insight consistently ranks second on Verus-Bench and first on RewardData, outperforming all evaluated specialized detectors under both independent judges. Its improvements over the Qwen3-VL-8B backbone also remain substantial, indicating that the gains are not specific to Gemini 3.1 Pro's matching decisions.

Absolute F1 values vary across judges, reflecting differences in semantic matching strictness. Importantly, this experiment isolates the evaluation judge: previously extracted model outputs remain fixed, and the original Gemini-based extraction procedure for applicable baselines is retained. We therefore interpret these results as independent validation of semantic-matching robustness, rather than a fully independent end-to-end evaluation pipeline.

AuthorsOpenReview ↗

Response to W2: Spatiotemporal Grounding Evaluation

W2: Spatiotemporal grounding is never evaluated. The method relies on explicit temporal and spatial grounding, and Verus-Bench provides frame-wise boxes. Yet all metrics are text-only: time and boxes are explicitly excluded from matching (App. E.2). The paper never shows that the predicted locations are correct.

We agree that semantic matching alone does not establish whether the predicted artifacts are correctly localized. To address this, we conduct an additional spatiotemporal grounding evaluation on Verus-Bench (191 videos, 321 GT findings) and RewardData (483 videos, 182 GT findings), covering six models. We keep the original model predictions and one-to-one semantic matches fixed, and independently measure temporal and spatial localization accuracy.

Table: Spatiotemporal Grounding of Semantically Matched Findings

DatasetModelMatchedTemporal ValidtIoU ↑Spatial ValidsIoU ↑
Verus-BenchVerus-Insight4848/480.50369/480.4428
GPT-5.51047/1040.46830NA
Qwen3-VL-8B4515/450.31460NA
Skyra1713/170.42139/170.3015
BusterX++204/200.39170NA
VideoVeritas242/240.33330NA
RewardDataVerus-Insight6867/680.721060/680.3348
GPT-5.59910/990.48270NA
Qwen3-VL-8B335/330.40150NA
Skyra4737/470.593235/470.3678
BusterX++240/24NA0NA
VideoVeritas664/660.06750NA

Temporal Valid and Spatial Valid indicate the number of semantically matched findings with evaluable temporal intervals and spatial boxes, respectively. tIoU is averaged over valid finding-level intervals; sIoU is averaged over aligned frame pairs (within 0.25 s), comprising 38/885 frames for Verus-Insight and 41/410 frames for Skyra on Verus-Bench/RewardData. Missing or unalignable locations are excluded from conditional IoU averages and reflected in the reported coverage.

  • Temporal grounding. Verus-Insight provides evaluable temporal localization for 48/48 matched findings on Verus-Bench and 67/68 on RewardData, achieving mean tIoU scores of 0.5036 and 0.7210, respectively. This demonstrates substantial temporal localization coverage and measurable agreement with the annotated artifact intervals.
  • Spatial grounding. Verus-Insight provides evaluable bounding-box localization for 9/48 matched findings on Verus-Bench and 60/68 on RewardData, achieving frame-level sIoU scores of 0.4428 and 0.3348, respectively. Notably, on RewardData, Verus-Insight achieves substantially higher spatial coverage than Skyra (88.2% vs. 74.5%), while maintaining comparable localization accuracy (0.3348 vs. 0.3678 sIoU). On Verus-Bench, Verus-Insight also achieves higher conditional sIoU than Skyra (0.4428 vs. 0.3015), although spatial coverage remains limited. These results demonstrate Verus-Insight's ability to provide spatially grounded artifact evidence with a favorable balance between localization coverage and accuracy.

For reproducibility, native localization fields are extracted deterministically. For baselines with unstructured responses, GPT-6.1 Sol extracts only explicitly stated locations from the original responses, without access to videos or GT annotations. All IoU values are then computed deterministically, without modifying the original semantic matches.

Reviewer_Syn9OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Verus-Insight treats AI-generated video detection as inspection planning, evidence verification, and evidence aggregation. It uses human-anchored supervision and progressive training, and introduces Verus-Bench with fine-grained artifact annotations. The paper reports gains in artifact finding and source classification across several benchmarks.

Critical strengths

The decomposition into planning, finding, and aggregation gives the paper a useful vocabulary for separating source prediction from the reasons offered for that prediction.

The authors also report more than a single final accuracy number, which is helpful for diagnosing where a detector fails.

Critical weaknesses
  1. The paper has not yet shown that artifact explanations are necessary or even sufficient for reliable authenticity detection. Its artifact targets are human-perceived abnormalities defined by a rubric, not independently verified traces of the generation process. A model can therefore improve Finding F1 by learning the annotation style, while still producing explanations that are irrelevant, incomplete, or wrong. The paper's use of “explainable” should be treated cautiously unless the findings improve independent human decisions or survive counterfactual tests.

  2. The three-stage architecture may also be a presentation of the desired output rather than a demonstrated scientific necessity. If most gains come from a stronger backbone, extra supervision, or longer reasoning, the staged design does not by itself establish a new detection principle. Finally, the asymmetric reward encourages empty reports on real videos and label-compatible reports on fake videos. This creates a direct risk of optimizing plausible-looking explanations rather than verifiable evidence.

Questions for authors
  1. Are training and evaluation separated by source video, near-duplicate content, prompt, generator, and annotation lineage?
  2. What happens when false-positive artifact reports on real videos receive the same cost as missed artifacts on fake videos?
  3. Do independent annotators or downstream reviewers make better authenticity decisions using the generated findings?
  4. Which gains remain on unseen generators and artifact categories, rather than on the same annotation vocabulary?
9 comments
AuthorsOpenReview ↗

Response to Q1. Training–Evaluation Separation and Data Leakage

Q1: Are training and evaluation separated by source video, near-duplicate content, prompt, generator, and annotation lineage?

We conducted a leakage audit between our training instances and four evaluation sets: **Verus-Bench (192 videos), RewardData (484), Artifact-Bench T1 (500).

The audit covers the following aspects:

  • Video-source separation: We compared sample identifiers, original source-video paths, temporal clip intervals, and frame-level SHA-256 hashes. We found zero exact overlaps or shared source-video segments identifiable through these checks.
  • Annotation lineage: We traced the original video_id and annotation_id records and verified that no evaluation annotations or their derived supervision were reused in training.
  • Prompt separation: VLM task instructions are intentionally shared across splits, but these are not the original video-generation prompts. Since the latter are unavailable, we do not claim generation-prompt disjointness.
  • Generator separation: The main benchmarks are not designed to be generator-disjoint. We therefore do not attribute their results solely to unseen-generator generalization. We separately examine CogVideoX, which is absent from our task-specific training data, in Q4.

These checks establish exact video-content and annotation-lineage separation for the four audited datasets. They do not constitute exhaustive perceptual near-duplicate detection, nor do they establish prompt- or generator-disjointness across all evaluations. The detailed audit is limited to the four sets named above; we do not extend its conclusions to other evaluation subsets without corresponding checks.

AuthorsOpenReview ↗

Response to Q2. Asymmetric Reward Design and False-Positive Artifact Reporting

Q2: What happens when false-positive artifact reports on real videos receive the same cost as missed artifacts on fake videos?

We agree that equalizing the cost of false-positive artifact reports on real videos and missed artifacts on generated videos is a meaningful alternative. We can clarify the design of our current reward and examine whether it exhibits the reporting shortcut raised by the reviewer.

As defined in Eq. (1), our reward is gated by both output validity and source correctness. For correctly classified generated videos, Finding F1 rewards recovery of annotated artifacts while penalizing unmatched predictions (FP) and missed annotations (FN). For correctly classified real videos, the reward favors the absence of affirmative artifact reports. These two branches are not equivalent to assigning identical marginal costs to individual errors.

Importantly, a correct Fake label alone does not guarantee a high artifact-recovery reward: predicted findings must semantically match annotated artifacts to receive matching credit. Conversely, empty artifact reports on generated videos cannot recover annotated findings.

Our existing results provide three relevant observations:

  • Low false-positive reporting on real videos. Verus-Insight achieves a strict artifact false-positive rate of 1.20% on real RewardData videos.
  • Simultaneous improvements in artifact precision and recall. On RewardData, Finding Precision and Recall improve from 1.87% / 18.13% for Qwen3-VL-8B to 33.66% / 37.36% for Verus-Insight. Thus, the improved precision is accompanied by substantially better recovery of annotated artifacts, rather than merely suppressing artifact reports.
  • Source decisions are sensitive to intermediate anomaly evidence. In controlled inference-time interventions, neutralizing matched anomaly evidence changes 41/79 (51.9%) Fake predictions to Real, whereas neutralizing normal evidence produces 0/79 such changes on the same fake-video subset. This indicates that the downstream verdict is sensitive to the content of the artifact evidence.

Together, these observations do not support the concern that our current model simply learns to issue empty reports or label-compatible artifact descriptions. However, they characterize the behavior of our proposed asymmetric objective, rather than establishing equivalence with a retrained equal-cost alternative. We will make this distinction explicit in the revised manuscript.

AuthorsOpenReview ↗

Response to Q3. Human Utility and Verifiability of Generated Findings

Q3: Do independent annotators or downstream reviewers make better authenticity decisions using the generated findings?

We conducted a preliminary, small-scale exploratory assessment of whether presenting generated findings could assist human authenticity judgments. We did not observe a clear additional benefit over video-only inspection. Since this assessment was exploratory rather than a sufficiently powered controlled study, we do not claim that our findings improve independent human classification accuracy. For context, Artifact-Bench [1] reports 93.6% authenticity-classification accuracy for human experts, suggesting that high video-only performance may limit the measurable room for improvement, although this does not establish a ceiling effect in our setting. Our principal evaluation of explanations instead concerns whether the reported artifact evidence can be independently verified against the video.

To examine this, we conducted a blinded human audit covering 1,496 unmatched findings and 142 originally matched prediction–GT pairs from five models on the same 60 fake videos. Two annotators independently evaluated visual support and perceptual salience without access to GT annotations or model identities. GT coverage and matching errors were examined only after these judgments were fixed, with disagreements resolved through third-annotator adjudication.

The audit yields the following results:

  • Human validation of matched findings: 122/142 (85.9%) originally matched predictions were confirmed. For Verus-Insight specifically, 28/31 (90.3%) matched findings were confirmed as visually supported.
  • Annotation omissions: 74/1,496 (4.9%) unmatched prediction occurrences were attributed to genuine artifacts missing from the GT.
  • Semantic-matching errors: Another 110/1,496 (7.4%) unmatched predictions corresponded to annotated artifacts but were missed by the automatic matcher.
  • Remaining unmatched findings: These included duplicate reports (22.3%), non-salient observations (21.8%), unsupported claims (27.1%), and visually indeterminate cases (16.4%).

These results show that a substantial fraction of automatically matched artifact findings can be independently verified by human inspection. They also demonstrate that matching against the annotation vocabulary is not equivalent to establishing visual correctness, which motivates our separate human audit.

Our claim of explainability is therefore grounded in the production of structured, visually inspectable artifact evidence, rather than a demonstrated improvement in human decisions or complete causal faithfulness. Further details of the audit are reported in our response to Reviewer e5nD (Q3).

[1] Tang, Y., Shi, Y., Zhang, Z., et al. Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos. arXiv:2605.18984, 2026.

Reviewer_e5nDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Verus-Insight for explainable AI-generated video detection, breaking the task into Inspection Planning, Evidence Verification, and Evidence Aggregation with progressive training and human-annotated supervision. The authors also introduce the Verus-Bench benchmark, and experiments across multiple datasets show strong detection and artifact-analysis performance.

Critical strengths
  • The dataset and method designs are well-motivated, and guiding the MLLM through region inspection, evidence verification, and evidence aggregation is logical.
  • The authors conduct thorough and extensive experiments, which validate the strong detection performance of Verus-Insight.
Critical weaknesses
  • The authors provide enough motivation for how the dataset is annotated and built, but they do not fully explain the implementation details of the annotation process. For example, in stages P1–P4, it is unclear which specific models are used at each step and how much human review is involved. Considering that different proprietary models show large performance gaps in both source prediction and artifact analysis in Tables 2 and 3, I think the authors need to provide more details to support the reliability of the results.
  • Comparing Tables 2 and 3, Gemini-3.1-Pro gets higher Plan F1 and Finding F1 than Verus-Insight on Verus-Bench (26.87 vs. 24.70 and 18.16 vs. 15.97), but its BA is much lower than Verus-Insight (51.04 vs. 89.06). This shows that the metrics for inspection planning and evidence verification are not always consistent with the final detection performance. Meanwhile, given the generally low Plan and Finding scores in Table 3 (all F1 scores are below 40%), although Verus-Insight brings some improvements, the hallucination problem still seems to exist. Taking these together, I am still not convinced that the proposed inspection planning and evidence verification actually support the improvement in final detection performance.
Questions for authors
  • What specific models are used in each step of P1–P4 during dataset construction, and how much human review is involved in checking the generated outputs?
  • Since higher Plan F1 and Finding F1 do not always lead to higher BA (as seen with Gemini-3.1-Pro vs. Verus-Insight in Tables 2 and 3, and removing the Stage-II rubric in Table 4), and the overall Plan and Finding F1 scores remain below 40%, how can the authors show that inspection planning and evidence verification actually drive the final detection gains?
  • The original dataset construction relies on human annotations, which mainly focus on "human-visible" artifacts. Could this limit the model's ability to capture low-level traces that are hard for humans to notice (such as frequency-domain statistics or upsampling artifacts)? Also, could this be one reason why the Plan and Finding F1 scores are generally low (i.e., the artifacts found by the model do exist, but are not covered by the human annotations)?
  • The authors only compare MLLM-based detection methods and do not report results where baselines are retrained on the same training data. I think the authors could add some non-MLLM methods on the same datasets for comparison.
6 comments
AuthorsOpenReview ↗

Response to Q1. Dataset Construction Models and Human Quality Control

Weakness 1 + Q1:What specific models are used in each step of P1–P4 during dataset construction, and how much human review is involved in checking the generated outputs?

We clarify the models used in each construction stage (P1–P4) and distinguish human-annotated evidence from model-generated supervision.

Models used in P1–P4:

StagePurposeModel
P1: Rubric GenerationConvert human-annotated artifacts into outcome-neutral inspection criteriaGemini 3.1 Pro
P2: Normal ProposalPropose content-specific normal checks without access to artifact GT or source labelsQwen3-VL-235B
P3: Normal VerificationIndependently verify the proposed observations and their visual supportGemini 3.1 Pro + GPT-5.6 Sol
P4: Target EnrichmentComplete scene descriptions, inspection domains, and conditional standards without altering verified evidenceGPT-5.6 Sol

Human annotation and quality control. The original artifact annotations undergo three human stages: initial annotation followed by secondary and tertiary review. These annotations provide the evidence anchors for subsequent supervision construction. For the model-generated normal checks, we apply additional safeguards:

  1. GT-conflict filtering: Candidates overlapping annotated artifacts within a ±0.25s temporal margin and referring to the same subject are rejected. Exact duplicates are also removed.
  2. Independent verification: Gemini 3.1 Pro and GPT-5.6 Sol independently examine each candidate without access to each other's decisions. Only candidates verified as normal by both models are retained.
  3. Final selection: Verified candidates undergo quota-based selection, yielding 24,359 normal checks in the final training set (21,577 from Verus-Annotation and 2,782 from RewardData, as reported in Table 5).
  4. Human spot-check: We additionally manually inspected 100 retained normal checks. Their observations and inspection criteria were generally consistent with the corresponding video content, providing a qualitative check on the automatically constructed supervision. Importantly, the model-generated stages do not overwrite the original human artifact evidence, and we do not claim exhaustive human verification of every generated normal check. Our construction instead combines human-reviewed artifact anchors with independently verified normal observations and additional manual quality checks.
AuthorsOpenReview ↗

Response to Q2. Do Intermediate Capabilities Contribute to Final Detection?

Q2: Since higher Plan F1 and Finding F1 do not always lead to higher BA (as seen with Gemini-3.1-Pro vs. Verus-Insight in Tables 2 and 3, and removing the Stage-II rubric in Table 4), and the overall Plan and Finding F1 scores remain below 40%, how can the authors show that inspection planning and evidence verification actually drive the final detection gains?

We agree that higher Plan/Finding F1 does not necessarily imply higher Balanced Accuracy (BA). These metrics evaluate distinct capabilities: Plan F1 measures coverage of artifact-relevant inspection targets, Finding F1 measures recovery of annotated artifacts, and BA measures final source discrimination. Indeed, Table 4 shows that removing the Stage-II rubric slightly improves BA while reducing both intermediate F1 scores.

To examine whether intermediate content actually influences downstream predictions, rather than merely providing additional output structure, we conducted controlled inference-time interventions on Verus-Bench and RewardData. The experiments involve 79 eligible fake videos with previously matched artifacts (19 from Verus-Bench and 60 from RewardData), together with 79 real-video controls. We modify intermediate plans or evidence while keeping the input videos and model parameters fixed, and regenerate the subsequent stages.

1. Inspection Planning → Artifact Recovery We replace artifact-relevant inspection criteria with off-target criteria and use irrelevant edits as controls.

DatasetPlan ConditionTarget Recovery ↑Finding F1 (%) ↑Fake → Real
Verus-Bench (19)Original19/1935.510/19
Off-target1/194.219/19
Irrelevant edit19/1934.860/19
RewardData (60)Original56/6076.710/60
Off-target1/602.7819/60
Irrelevant edit57/6078.080/60
Overall (79)Original75/7959.290/79
Off-target2/793.3528/79
Irrelevant edit76/7959.610/79

Target Recovery measures whether the artifact targeted by the intervention is recovered. Finding F1 is computed on this selected diagnostic subset rather than the full benchmark.

Redirecting artifact-relevant inspection criteria reduces target recovery from 75/79 to 2/79 (a 92.4-percentage-point decrease) and Finding F1 from 59.29% to 3.35%. It also changes 28/79 fake-video verdicts to Real. In contrast, irrelevant edits preserve artifact recovery, Finding F1, and final source predictions. This indicates that the content of inspection planning matters for subsequent artifact recovery, rather than simply the presence of a rubric.

2. Evidence Verification → Source Decision

We further neutralize a GT-matched anomaly observation or a normal observation, then regenerate the subsequent evidence assessment, aggregation, and verdict.

Ground TruthEvidence ConditionNPred. RealPred. FakeInconclusive
FakeOriginal evidence790790
FakeAnomaly neutralized79411622
FakeNormal neutralized790790
RealOriginal evidence797720
RealNormal neutralized7948229

Neutralizing matched anomaly evidence changes 41/79 (51.9%) fake-video predictions to Real, compared with 0/79 when normal evidence is neutralized. The 22 inconclusive outputs are reported separately and are not counted as successful classification flips. For real-video controls, normal-evidence neutralization introduces no additional Fake predictions, although 29 outputs become inconclusive. Unmodified prefix continuations reproduce all 158/158 historical verdicts in the replay controls.

These interventions provide direct evidence that inspection plans influence artifact recovery and that intermediate evidence affects subsequent source decisions. They complement the component ablations by establishing the functional relevance of intermediate content in the tested continuation setting. We acknowledge that they do not, by themselves, isolate how much of the overall BA improvement is attributable to staging rather than other training factors, nor do they imply a monotonic relationship between intermediate F1 and BA.

AuthorsOpenReview ↗

Response to Q3. Human-Perceived Artifacts and Annotation Completeness

Q3: The original dataset construction relies on human annotations, which mainly focus on "human-visible" artifacts. Could this limit the model's ability to capture low-level traces that are hard for humans to notice (such as frequency-domain statistics or upsampling artifacts)? Also, could this be one reason why the Plan and Finding F1 scores are generally low (i.e., the artifacts found by the model do exist, but are not covered by the human annotations)?

We agree that relying on human annotations introduces a specific scope limitation. Our work intentionally targets human-perceptible visual and temporal artifacts, rather than exhaustive forensic signatures such as frequency-domain statistics or upsampling traces. These less perceptible signals may provide complementary detection cues, but they fall outside the scope of our human-grounded artifact evaluation.

To directly examine whether incomplete annotations contribute to the relatively low Finding F1 scores, we conducted a human audit of all 1,496 unmatched findings and 142 matched prediction–GT pairs produced by five models on the same 60 fake videos.

Two annotators independently assessed visual support and perceptual salience without access to GT annotations or model identities. After these judgments were fixed, they examined GT coverage and semantic-matching outcomes, with disagreements resolved through third-annotator adjudication.

Human Audit of Artifact Findings Across Five Models

ModelUnmatchedGT OmissionMatcher MissDuplicateNon-salientUnsupportedIndeterminateValid Matched
Verus-Insight1248121420383228/31
GPT-5.528828355065654558/66
VideoVeritas377122580751058012/16
BusterX++61920301801501706914/18
Skyra88681016282010/11
Total1,49674110334326406246122/142

Counts refer to prediction occurrences across models, not necessarily distinct artifact events. Duplicate denotes repeated reporting of the same GT event; Indeterminate denotes insufficient evidence for reliable human verification. Valid Matched reports the human-confirmed fraction of originally matched predictions.

The audit reveals three findings.

  • Incomplete annotations and matching errors do account for some genuine detections being penalized. Among unmatched predictions, 74/1,496 (4.9%) were attributed to GT omissions, while 110/1,496 (7.4%) corresponded to artifacts covered by GT but missed by semantic matching. Together, these account for 12.3% of unmatched predictions.
  • Missing GT annotations are not the primary explanation for low Finding F1. The remaining unmatched predictions include duplicate reports (22.3%), non-salient observations (21.8%), unsupported claims (27.1%), and visually indeterminate cases (16.4%). These categories have different implications: duplicates may describe real but repeatedly reported evidence, while unsupported claims lack sufficient visual support.
  • The automatically matched findings are largely supported by independent human inspection. Human annotators confirmed 122/142 (85.9%) originally matched predictions across the five models. For Verus-Insight specifically, 28/31 (90.3%) matched findings were confirmed.

These results indicate that annotation incompleteness and matcher errors partially depress measured Finding F1, but do not justify attributing the generally low scores primarily to missing annotations. Plan F1 also depends on the annotated inspection targets, although this audit directly quantifies the issue for findings rather than planning. Our evaluation should therefore be interpreted as measuring recovery of annotated, human-perceptible artifacts, not exhaustive detection of all possible forensic traces.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.