acceptodds
Under review as a conference paper at ICLR 2027

Auditing the Ground Truth: A Field-Aware Audit of Video Anomaly Understanding Benchmarks

Abstract

Language annotations in video anomaly understanding benchmarks guide model training and evaluation. Their quality must be assessed in relation to field responsibilities, temporal scopes, and granularity. We introduce a contract-aware audit framework that translates native field requirements into diagnostic questions and specifies the permitted evidence. Four tasks assess video quality, standalone text quality, native cross-text consistency, and video–text alignment. To determine whether a video–text pair meets multiple quality requirements simultaneously, we link video, standalone text, and alignment judgments to compute conditional joint usability. Our study includes 7,928 video entries and 56,103 video–text pairs across nine benchmarks. In the model-assisted audit, conditional joint usability ranges from 22.07% to 74.20% across benchmarks under field-macro aggregation. The predominant failure pattern is that video and text pass standalone checks, but their pairing fails alignment. To test whether audit-guided annotation revision improves performance, we revise CUVA descriptions and conduct controlled experiments with LVLMs, evaluating both description generation and multiple-choice question answering on VALU. The results show downstream benefits from revision, with effects varying by model and evaluation metric. Field-level audits further reveal quality differences within individual benchmarks. These diagnostics identify annotations requiring review and, together with native task coverage, guide the selection of benchmarks and fields suited to the evaluation goal and temporal granularity.

Primary area:
datasets and benchmarks
Keywords:
Video Anomaly Understanding, Benchmark Auditing, Annotation Quality, Video–Text Alignment, Vision–Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ptA2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the quality of ground-truth annotations in video anomaly understanding benchmarks. The authors propose four audit tasks to examine individual samples and use LVLMs to score existing benchmarks, constructing three quality profiles: QS, MCR, and JUR. Based on the diagnostic results, the authors further explore audit-guided revision and compare it against the original data and direct rewriting, finding that audit-guided revision leads to performance improvements. Finally, the paper investigates benchmark selection and shows how diagnostic results can guide more targeted selection at both the task and field levels.

Critical strengths

This paper investigates the reliability of reference annotations themselves, covering nine benchmarks, 7,928 videos, and 56,103 video-text pairs. This is an important issue for the reliability of evaluation as a whole, making the work a valuable piece of benchmark meta-research. Moreover, the authors’ decomposition of annotation quality is convincing and more rigorous than simply asking an LLM, “Is this annotation correct?” Building on their diagnostic findings, the authors further explore annotation repair through controlled experiments and analyze how to select appropriate benchmarks. Overall, the paper presents a well-rounded study with a relatively complete research loop.

Critical weaknesses

The downstream validation in RQ2 is relatively limited in scope, as annotation revision is conducted only on CUVA and involves just 693 training examples. This raises concerns about the robustness and generalizability of the experimental results.

Questions for authors
  1. The paper defines four tasks with distinct evidence requirements. Why does the JUR metric incorporate only three of these four audit tasks?
  2. In Table 1, how are the various scores for the candidate evaluators against the human reference annotations calculated? What does each metric measure?
  3. Were the 693 CUVA training examples revised during annotation correction? Under the Original Data, Direct Rewrite, and Audit-guided settings, was the same revised evaluation dataset used for evaluation across all three settings?
  4. What is the relationship between the seven fields and the four audit tasks? It appears that the first three audit tasks are independent of the specific fields. Could the authors clarify how the field definitions are incorporated into these tasks?
1 comment
AuthorsOpenReview ↗

A brief note before the discussion closes

Dear Reviewer ptA2, We understand how busy this stage of the review process is for reviewers, and we appreciate the time you have already devoted to our paper. As the discussion period draws to a close, we would be happy to answer any remaining questions about the UCF-CAP results or our evaluation protocol.

Best regards, Authors of Submission 9191

Reviewer_io1KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a field-aware quality auditing framework for video anomaly understanding datasets, evaluating nine major benchmarks across four dimensions: video quality, text quality, cross-text consistency, and video–text alignment. It identifies widespread video–text alignment issues and introduces an audit-guided annotation revision method to address these data quality problems.

Critical strengths
  1. The study covers a broad range of datasets.

  2. The research questions are of practical value to the evaluation and training of Video LLMs.

Critical weaknesses
  1. The problem itself is not novel. Many prior works have identified and attempted to address this issue, while this paper provides almost no quantitative comparisons with previous auditing or correction methods.

  2. The current ablation studies are limited to comparisons of annotation strategies under "Training Labels". There are almost no ablation studies on the specific design details of the proposed data annotation method (e.g., Field Contract, but not only). The experiments only demonstrate that the proposed auditing and correction methods are useful, without exploring the independent contributions of individual stages to the final results.

  3. Based on the above points, although the paper demonstrates considerable effort in investigating video–text alignment issues and validating the effectiveness of the proposed methods through downstream experiments, it lacks many necessary comparisons with similar methods, making it difficult to position the work within the existing literature. Furthermore, the lack of ablation studies on the internal stages and components of the proposed auditing and correction methods makes it difficult to establish a causal relationship between the proposed design choices and the downstream results.

  4. The presentation needs improvement. For example, some figures appear overly designed, with excessive decorative elements and relatively limited information, making it difficult to quickly understand and grasp the key points of the paper from the figures alone.

Questions for authors

Please refer to Weaknesses.

3 comments
AuthorsOpenReview ↗

A brief note before the discussion closes

Dear Reviewer io1K, We know this is a busy time and appreciate the time you have given our paper. As the discussion draws to a close, please let us know if you have any remaining questions about the comparisons or ablation results. We would be happy to clarify them promptly.

Best regards, Authors of Submission 9191

Thanks for your response. The new experiments address some of my concerns. However, unifying video, standalone text, and alignment judgments into audit criteria is still a combination of existing concepts. In addition, the manuscript has not undergone substantial revisions. Therefore, I will maintain my original assessment.

AuthorsOpenReview ↗

Clarification of our field-aware audit and manuscript revisions

Dear Reviewer io1K,

Thank you for reviewing the additional experiments and for clarifying your concern about novelty. Combining video quality, text quality, and video–text alignment scores still leaves a central question unresolved: what conditions must an annotation meet to serve as a valid reference for its field? In ECVA video 26, for example, the footage shows a man toppling a sculpture. It supports a description of his action, but it does not establish the claim in the Cause field that he was intoxicated.

The technical challenge is to apply a common audit procedure across benchmarks while preserving each field’s requirements for a correct answer. Our framework translates those requirements into audit criteria before scoring. For each field, the Field Contract specifies what an annotation must convey, which claims require verification, and what evidence the evaluator may use. For video–text alignment, we decompose the annotation into independently verifiable atomic claims, identify the claim central to the field, and check each claim against the permitted footage. The scoring rules distinguish insufficient evidence from a direct contradiction. In the ECVA example, this allows the evaluator to credit the observable action while identifying the causal explanation as unverified. We applied this framework across nine benchmarks. On the 400-video validation set with human reference judgments, the full framework achieved an F1F_1 of 95.2095.20\\% for major-concern detection. Removing field-specific semantic rules reduced F1F_1 to 88.6888.68\\%, and removing explicit atomic-claim decomposition reduced it to 88.8588.85\\%, drops of 6.52 and 6.35 percentage points, respectively.

These claim-level records could also inform the design of training data for process reward models and judge models. Each claim and its verification outcome could provide supervision for an intermediate verification step. Human-verified cases such as the ECVA example could also be used to train and evaluate judge models on whether they can identify an unsupported inference within an otherwise accurate account.

Prior work has examined how the choice of anomaly annotations affects the evaluation of video anomaly detectors. Our study asks whether the reference text itself fulfills its field’s intended role given the available evidence. If you have a specific study in mind that addresses a similar question, we would appreciate the reference and welcome a direct comparison of the approaches.

We had planned to finalize the manuscript after further discussion with the reviewers so that we could incorporate their feedback. We have now uploaded the revised PDF to OpenReview. Section 3.4 (p. 6, Table 2) compares our full audit with four external methods on major-concern detection and reports ablations that remove field-specific semantic rules, explicit atomic-claim decomposition, or both. Appendix I (p. 47, Table 13) gives precision, recall, F1, and false-positive and false-negative counts for each method and ablation.

Best regards,
Authors of Submission 9191

Reviewer_EQm4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this submission, the authors introduce a field-aware framework for auditing language annotations in video anomaly understanding benchmarks. It evaluates dimensions including Video Evidence Quality (V), Standalone Text Quality (T-local), Cross-text Consistency (T-relation), and Video–Text Alignment (A). The main contribution includes a field-aware audit framework, cross-benchmark quality findings, and audit-guided revision and use.

Critical strengths
  1. Important research problem. In this submission, the authors study the ground-truth annotations, an important but often overlooked issue in video anomaly understanding.

  2. Practical validation. The authors study annotation revision using diverse models, multiple training seeds, and both description generation and multiple-choice evaluation.

  3. Good Writing. The paper is well-structured and easy to follow.

Critical weaknesses
  1. Limited breakdown of evaluator validation.

Table 1 reports strong overall agreement with human annotations but lacks per-dimension results. Reporting alignment-specific agreement would further strengthen the reliability of the paper's main findings.

  1. Limited downstream improvements.

The improvements on VALU-ADC are relatively small, particularly for VAD-DPO, while Qwen3-VL gains only 1.64 (Table 3). These modest and model-dependent gains raise concerns about whether the proposed design provides consistent and meaningful downstream benefits.

  1. Insufficient validation of the field-aware design.

The authors validate the overall auditing protocol in Table 1, but the paper does not ablate key components such as Field Contracts and atomic-claim decomposition, leaving their individual contributions to auditing reliability unclear.

Questions for authors
  1. Could the authors report evaluator–human agreement separately for each audit dimension, particularly video–text alignment?

  2. Given the relatively small improvements on VALU-ADC, could the authors provide further evidence that proposed design yields reliable and meaningful downstream benefits?

  3. Could the authors provide ablation studies on field contracts and atomic-claim decomposition?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.