- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper argues that automated judges for open-ended video tasks are usually validated only by how well their final scores correlate with human scores, which can hide errors in the intermediate judgments. The authors propose a paradigm with four stages: a shared scoring procedure that humans and models both follow and that exposes intermediate outputs, a small calibration panel selected to cover the variety of inputs and responses, human annotations split into an expert reference and a human baseline, and a certification rule that requires an automated judge to match or beat the human baseline at every intermediate and final step. They apply it to detailed video captioning, where captions are split into short factors that are judged for correctness and coverage to give precision, recall and F1. On a panel of 64 captions annotated by four researchers, they compare 35 single-pass and agentic VLM judges and find that only Gemini 3.7 Flash with agentic execution is certified, while most judges beat the human baseline on F1 correlation but fall short on precision.
Critical strengths
- The paper shows that final score correlation alone is not enough to validate an automated judge. Every evaluation in Table 2 beats the human baseline on F1 correlation, yet most fail on precision, which is a useful and well supported point.
- The shared procedure, fixed factor segmentation and leave-one-annotator-out baseline give humans and models a common basis for comparison, and the prompts and implementation details are well documented.
- The comparison is broad, covering 35 evaluators in single-pass and agentic form, text metrics and a holistic VLM judge.
Critical weaknesses
- The paper presents a paradigm for open-ended video understanding, but tests it only on detailed captioning with CaReBench. It is not clear how it would apply to other tasks such as video question answering, where splitting responses into factors may be harder.
- The paper does not report the cost or runtime of agentic execution, even though it suggests choosing among certified judges by efficiency.
- I think the novelty is modest. The paper builds on established components, which it cites, including splitting outputs into claims (FActScore), scoring precision and recall over units (SPICE, AutoDQ), comparing a model with a held-out human annotator (Calderon et al., 2025) and averaging judges (Verga et al., 2024). Step-level validation against experts also appears in MME-Emotion. The new element is requiring the human baseline comparison at every intermediate step.
Questions for authors
- Can you apply the paradigm to another open-ended video task, such as video question answering, or test whether the certified judges remain certified on a second captioning dataset?
- In Table 4, all 15 evaluations exceed the human baseline on recall correlation, most by a wide margin (about 0.76 to 0.87 versus 0.72). Does this indicate that a single annotator is too weak a bar for certification, and how much does the human baseline vary across the four leave-one-out assignments?
Response to Reviewer Wt9U
Thank you for highlighting the limitations of final-score-only validation and the need to clarify applicability, cost, and the human baseline.
W1 and Q1: Application beyond captioning
Appendix N shows how the paradigm extends beyond captioning through task-specific alignment points. Two worked video-QA examples adapt AutoEval-Video's question-specific rules: a dog's sequence of seat changes (Appendix N.1, Figure 9 and Table 21) and the reason a car stops (Appendix N.2, Figure 10 and Table 22). Humans and automated evaluators would apply the same rules, with final correctness requiring every rule to pass.
AutoEval-Video already provides a valid instantiation with a final alignment point: its GPT-4V-response subset reports 96.6% automated versus 94.9% human agreement, while its mixed panel reports 97.0% versus 97.5% (Chen et al., 2024). Our illustration adds per-rule comparisons and emphasizes renewing human evidence when the evaluated response distribution changes. The two examples have shown the application of our paradigm is general enough to go beyond evaluating captioning, such as evaluating open-ended video question answering tasks.
W2: Evaluation cost
We estimated monetary cost on 64 captions with two runs per method and model (Appendix O, Table 23). Mean single-pass/agentic cost is $0.0125/$0.2707 for Gemini 3.8 Flash and $0.0636/$0.6777 for Gemini 3.1 Pro Preview—approximately 22-fold and 11-fold increases. These estimates include recorded scoring, inspection, vision-query, and retry usage at the stated reference prices.(Appendix M, Tables 19–20).
W3: Novelty of the evaluation paradigm
Please see overall response G1 for our contribution and scope. Appendix I (Table 10) compares documented fulfillment of the paradigm’s four components across video evaluation protocols. These comparisons clarify how our contribution organizes established practices into general, executable guidance for designing evaluations and substantiating scoped human-level alignment claims.
Q2: Is the human recall baseline too weak?
The four held-out-human recall correlations are 0.8001, 0.7716, 0.5802, and 0.7092 (H1–H4), averaging 0.7153 (Appendix K, Figure 7). Thus the mean contains substantial annotator variation. It represents individual human execution under this procedure, not agreement with an infallible oracle or with an expert committee acting as the evaluator.
The paradigm aims to support choosing a certified automated evaluator in place of individual human evaluation within the evaluated scope. Certification requires human-level alignment at every specified alignment point. Additional effort can support aggregation of human judgments, and automated judgments can likewise be combined, as examined in our assembly study in Section 5 (Table 4). Our instantiation focuses on certification relative to individual human performance. Appendix K shows each held-out annotator’s factor-level and caption-score alignment against a reference formed from the other three, together with the four-assignment mean (Figure 7). It also provides pairwise confusion matrices for precision and recall judgments (Figures 5–6), making the underlying variation in human judgments explicit.
Reference
Xiuyuan Chen, Yuan Lin, Yuchen Zhang, and Weiran Huang. 2024. AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering. Computer Vision – ECCV 2024, pp. 179–195. Springer Nature Switzerland.