- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the quality of ground-truth annotations in video anomaly understanding benchmarks. The authors propose four audit tasks to examine individual samples and use LVLMs to score existing benchmarks, constructing three quality profiles: QS, MCR, and JUR. Based on the diagnostic results, the authors further explore audit-guided revision and compare it against the original data and direct rewriting, finding that audit-guided revision leads to performance improvements. Finally, the paper investigates benchmark selection and shows how diagnostic results can guide more targeted selection at both the task and field levels.
Critical strengths
This paper investigates the reliability of reference annotations themselves, covering nine benchmarks, 7,928 videos, and 56,103 video-text pairs. This is an important issue for the reliability of evaluation as a whole, making the work a valuable piece of benchmark meta-research. Moreover, the authors’ decomposition of annotation quality is convincing and more rigorous than simply asking an LLM, “Is this annotation correct?” Building on their diagnostic findings, the authors further explore annotation repair through controlled experiments and analyze how to select appropriate benchmarks. Overall, the paper presents a well-rounded study with a relatively complete research loop.
Critical weaknesses
The downstream validation in RQ2 is relatively limited in scope, as annotation revision is conducted only on CUVA and involves just 693 training examples. This raises concerns about the robustness and generalizability of the experimental results.
Questions for authors
- The paper defines four tasks with distinct evidence requirements. Why does the JUR metric incorporate only three of these four audit tasks?
- In Table 1, how are the various scores for the candidate evaluators against the human reference annotations calculated? What does each metric measure?
- Were the 693 CUVA training examples revised during annotation correction? Under the Original Data, Direct Rewrite, and Audit-guided settings, was the same revised evaluation dataset used for evaluation across all three settings?
- What is the relationship between the seven fields and the four audit tasks? It appears that the first three audit tasks are independent of the specific fields. Could the authors clarify how the field definitions are incorporated into these tasks?
A brief note before the discussion closes
Dear Reviewer ptA2, We understand how busy this stage of the review process is for reviewers, and we appreciate the time you have already devoted to our paper. As the discussion period draws to a close, we would be happy to answer any remaining questions about the UCF-CAP results or our evaluation protocol.
Best regards, Authors of Submission 9191