acceptodds
Under review as a conference paper at ICLR 2027

A Paradigm for Designing and Certifying Evaluation for Open-Ended Video Understanding

Abstract

Evaluating open-ended video understanding tasks is difficult because videos span diverse scenes and events, while valid responses vary in the information they convey and how they express it. Automated scoring can introduce errors that remain overlooked and obscure the capabilities being measured. We propose a paradigm for designing and certifying evaluation for these tasks. It combines a scoring procedure shared across evaluators with expert references and human baselines on a calibration panel. Certification requires an automated evaluation to match or exceed human-baseline agreement with the expert reference at every intermediate and final alignment point. Procedure design aims to reduce avoidable evaluation errors, while these comparisons make remaining disagreements visible. We instantiate the paradigm in detailed video captioning, assessing factual correctness and reference coverage through factor judgments and precision, recall, and F1 scores. In this instantiation, Gemini 3.7 Flash with agentic execution is the only certified individual evaluation among those tested. The paradigm also helps us compare and improve automated evaluations: averaging agentic outputs with those from a single scoring call to the same VLM improves alignment throughout the scoring procedure. Other evaluations exceed the human baseline in final-score correlation yet fail intermediate checks. These findings demonstrate how procedure-level analysis identifies evaluation weaknesses and guides further design and refinement.

Primary area:
datasets and benchmarks
Keywords:
open-ended video understanding, evaluation design, evaluation certification, meta-evaluation, human evaluation, video captioning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Wt9UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that automated judges for open-ended video tasks are usually validated only by how well their final scores correlate with human scores, which can hide errors in the intermediate judgments. The authors propose a paradigm with four stages: a shared scoring procedure that humans and models both follow and that exposes intermediate outputs, a small calibration panel selected to cover the variety of inputs and responses, human annotations split into an expert reference and a human baseline, and a certification rule that requires an automated judge to match or beat the human baseline at every intermediate and final step. They apply it to detailed video captioning, where captions are split into short factors that are judged for correctness and coverage to give precision, recall and F1. On a panel of 64 captions annotated by four researchers, they compare 35 single-pass and agentic VLM judges and find that only Gemini 3.7 Flash with agentic execution is certified, while most judges beat the human baseline on F1 correlation but fall short on precision.

Critical strengths
  • The paper shows that final score correlation alone is not enough to validate an automated judge. Every evaluation in Table 2 beats the human baseline on F1 correlation, yet most fail on precision, which is a useful and well supported point.
  • The shared procedure, fixed factor segmentation and leave-one-annotator-out baseline give humans and models a common basis for comparison, and the prompts and implementation details are well documented.
  • The comparison is broad, covering 35 evaluators in single-pass and agentic form, text metrics and a holistic VLM judge.
Critical weaknesses
  • The paper presents a paradigm for open-ended video understanding, but tests it only on detailed captioning with CaReBench. It is not clear how it would apply to other tasks such as video question answering, where splitting responses into factors may be harder.
  • The paper does not report the cost or runtime of agentic execution, even though it suggests choosing among certified judges by efficiency.
  • I think the novelty is modest. The paper builds on established components, which it cites, including splitting outputs into claims (FActScore), scoring precision and recall over units (SPICE, AutoDQ), comparing a model with a held-out human annotator (Calderon et al., 2025) and averaging judges (Verga et al., 2024). Step-level validation against experts also appears in MME-Emotion. The new element is requiring the human baseline comparison at every intermediate step.
Questions for authors
  • Can you apply the paradigm to another open-ended video task, such as video question answering, or test whether the certified judges remain certified on a second captioning dataset?
  • In Table 4, all 15 evaluations exceed the human baseline on recall correlation, most by a wide margin (about 0.76 to 0.87 versus 0.72). Does this indicate that a single annotator is too weak a bar for certification, and how much does the human baseline vary across the four leave-one-out assignments?
1 comment

Response to Reviewer Wt9U

Thank you for highlighting the limitations of final-score-only validation and the need to clarify applicability, cost, and the human baseline.

W1 and Q1: Application beyond captioning

Appendix N shows how the paradigm extends beyond captioning through task-specific alignment points. Two worked video-QA examples adapt AutoEval-Video's question-specific rules: a dog's sequence of seat changes (Appendix N.1, Figure 9 and Table 21) and the reason a car stops (Appendix N.2, Figure 10 and Table 22). Humans and automated evaluators would apply the same rules, with final correctness requiring every rule to pass.

AutoEval-Video already provides a valid instantiation with a final alignment point: its GPT-4V-response subset reports 96.6% automated versus 94.9% human agreement, while its mixed panel reports 97.0% versus 97.5% (Chen et al., 2024). Our illustration adds per-rule comparisons and emphasizes renewing human evidence when the evaluated response distribution changes. The two examples have shown the application of our paradigm is general enough to go beyond evaluating captioning, such as evaluating open-ended video question answering tasks.

W2: Evaluation cost

We estimated monetary cost on 64 captions with two runs per method and model (Appendix O, Table 23). Mean single-pass/agentic cost is $0.0125/$0.2707 for Gemini 3.8 Flash and $0.0636/$0.6777 for Gemini 3.1 Pro Preview—approximately 22-fold and 11-fold increases. These estimates include recorded scoring, inspection, vision-query, and retry usage at the stated reference prices.(Appendix M, Tables 19–20).

W3: Novelty of the evaluation paradigm

Please see overall response G1 for our contribution and scope. Appendix I (Table 10) compares documented fulfillment of the paradigm’s four components across video evaluation protocols. These comparisons clarify how our contribution organizes established practices into general, executable guidance for designing evaluations and substantiating scoped human-level alignment claims.

Q2: Is the human recall baseline too weak?

The four held-out-human recall correlations are 0.8001, 0.7716, 0.5802, and 0.7092 (H1–H4), averaging 0.7153 (Appendix K, Figure 7). Thus the mean contains substantial annotator variation. It represents individual human execution under this procedure, not agreement with an infallible oracle or with an expert committee acting as the evaluator.

The paradigm aims to support choosing a certified automated evaluator in place of individual human evaluation within the evaluated scope. Certification requires human-level alignment at every specified alignment point. Additional effort can support aggregation of human judgments, and automated judgments can likewise be combined, as examined in our assembly study in Section 5 (Table 4). Our instantiation focuses on certification relative to individual human performance. Appendix K shows each held-out annotator’s factor-level and caption-score alignment against a reference formed from the other three, together with the four-assignment mean (Figure 7). It also provides pairwise confusion matrices for precision and recall judgments (Figures 5–6), making the underlying variation in human judgments explicit.

Reference

Xiuyuan Chen, Yuan Lin, Yuchen Zhang, and Weiran Huang. 2024. AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering. Computer Vision – ECCV 2024, pp. 179–195. Springer Nature Switzerland.

Reviewer_FVsyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for certifying automated evaluators of open-ended video understanding by measuring their agreement with human experts at both intermediate and final scoring stages. The framework consists of a shared scoring procedure, calibration panel construction, human reference and baseline establishment, and evaluator certification. The method assesses factor-level factual precision and reference coverage and report caption-level precision, recall, and F1 scores. Experiments show that fine-grained evaluation improves alignment with human judgments over direct VLM scoring. Gemini 3.7 Flash with agentic execution is the only individual evaluator meeting all certification criteria, while combining agentic and single-pass outputs further improves alignment.

Critical strengths
  1. The intuition of having an automated evaluation for open-ended video understanding is important and the topic is timely.
  2. The proposed method provides a structured way to analyze evaluator reliability at different stages.
Critical weaknesses
  1. The main components of the proposed framework are quite similar to existing approaches. Decomposing generated text into factual units and evaluating individual claims is similar to existing fine-grained evaluation methods such as FActScore. Similarly, measuring agreement with human annotators and using human-level agreement as a qualification criterion have been extensively explored in prior work.
  2. A central claim of the paper is that automated evaluators can be certified as achieving human-level alignment, following the procedure described in the paperr. However, the certification experiment only has 64 annotated caption samples as human evaluator inputs, which is quite limited.
  3. The comparisons in the experiments mainly include text-based metrics and direct VLM-as-a-Judge baselines. However, the related work discusses several specialized video caption evaluation approaches, including FactVC, AutoDQ, and CaReBench's CapST scoring, which are not discussed here.
Questions for authors

See weaknesses

1 comment

Response to Reviewer FVsy

Thank you for recognizing the importance of evaluator reliability and the need for rigorous assessment in open-ended video understanding. We also appreciate your constructive suggestions for clarifying the contribution and strengthening the supporting evidence.

W1: Relationship to established components

Please see overall response G1 for our contribution and scope. To address the relationship to established components, Appendix I (Table 10) compares evaluation protocols against the paradigm’s four components. This comparison clarifies how our contribution organizes established practices into an executable process for designing evaluations and substantiating scoped human-level alignment claims.

W2: Evidence from 64 captions

Please see overall response G2 for the study-scale evidence. Appendix J.1 documents panel composition (Tables 11–12), and Appendix J.2 shows caption vocabulary before and after selection (Figure 4), while Appendix J.3 reports annotation workload and timing (Tables 13–14). Appendix J.4 compares the 48-, 64-, and 80-caption panels across the same 31 evaluation configurations: Table 15 summarizes decision-change rates, and Table 16 illustrates selected changing alignment-point decisions. Together, these analyses document the panel’s coverage, human annotation effort, and sensitivity to study scale, providing evidence of the robustness at current study scale.

W3: Specialized video-caption evaluation baselines

We extended the baseline study with CapST (Xu et al., 2026), preserving its released prompts and testing GPT-4.1 and a Gemini 3.8 Flash judge on all 64 caption pairs. All four configurations yield lower precision, recall, and F1 correlations than the VLM-as-a-judge baselines. The table below extends the original comparison in Section 5 (Table 3) with CapST.

MethodEvaluator / metricP score Pearson r ↑R score Pearson r ↑F1 score Pearson r ↑
Human-0.71030.71530.7285
Text-onlyBLEU--0.4064
Text-onlyROUGE-L-0.13220.7193†0.5937
Text-onlyBERTScore-0.00550.7583†0.3837
CapSTGPT-4.1 / events0.12690.38570.2258
CapSTGPT-4.1 / objects0.04270.34210.1716
CapSTGemini 3.8 Flash / events0.07380.42370.2195
CapSTGemini 3.8 Flash / objects-0.05060.30590.1559
VLM-as-a-judgeGemini 3.7 Flash0.61210.67210.6675
VLM-as-a-judgeGPT-5.40.62300.65620.6081
VLM-as-a-judgeClaude Sonnet 4.60.54550.7801†0.7830†
VLM-as-a-judgeQwen 3.8 27B0.57670.8196†0.7926†
VLM-as-a-judgeKimi K30.62280.7862†0.7795†
Single-passGemini 3.7 Flash0.68490.8479†0.8485†
AgenticGemini 3.7 Flash‡0.7303†0.8531†0.8673†

Pearson r against human scores, averaged over four leave-one-out assignments. BLEU is compared with human F1; CapST F1 is derived per caption. † meets the human baseline; ‡ denotes five-point certification. Bold/italics mark the best/second-best automated scores. Non-CapST results are from the original study.

CapST evaluates text-based event/object entailment, whereas our human rubric covers a broader range of caption details and incorporates video evidence. More generally, methods designed for different evaluation criteria may also misalign with the rubric curated for our open-ended caption study, underscoring the need for task-specific human validation through a separate instantiation of the paradigm for each task to support fair comparison.

Reference

Xu, Yifan, Xinhao Li, Yichun Yang, Desen Meng, Rui Huang, and Limin Wang. 2026. CaReBench: A Fine-grained Benchmark for Video Captioning and Retrieval. International Conference on Learning Representations, pp. 51124–51150.

Reviewer_FpzaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a framework for evaluating the reliability of automated judges for video captioning tasks. It compares both intermediate judgments and overall scores against expert annotations. On 64 caption samples, the paper find that Genimi 3.7 Flash is the only individual certified evaluator

Critical strengths

1.The paper evaluates both intermediate judgments and final scores, revealing errors that overall score correlations alone can miss.

2.The paper highlights that judgment errors can cancel out, producing an accurate final score, which may be also a good insight to other domains.

3.The leave-one-out setup is a sensible choice, it measures how well one human agrees with the others and uses that agreement as a baseline for automated evaluators.

Critical weaknesses

1.Only 64 captions from 61 videos are used to certify Gemini 3.7 Flash. A larger evaluation set and repeated runs would provide stronger evidence to support this certification claim.

2.It would be helpful to report the model distribution of the 64 captions, as overrepresentation of one model could bias the certification results.

3.I understand that by using leave-one-out, one human baseline can be compared to the average of the other three. It would be helpful to report the inter-annotator consistency, such as pairwise agreement rates and the proportion of factors on which all annotators agree, to clarify the reliability.

4.The paper reports that the average of single-pass and agentic evaluation offers the best performance. I assume it is that fast response (single-pass) and slow-thinking (agentic) attend to different parts in the evaluation, and the average of them can complement each other. Concretely, what type of error patterns does each approach handle better, and when should one be preferred over the other? Would averaging repeated single-pass or repeated agentic evaluations yield similar gains?

5.In Line 1335 VLM model in the vision-query skill is shared across all agentic evaluation runs. Can the author explain the design choice of this? Using the same OpenCode harness is reasonable, it also supports vision understanding using the hosted model. Qwen run on OpenCode can use Qwen itself as the vision understanding backend. As this is video captioning tasks, performance difference mostly depends on the different model’s vision part. A shared VLM may mask the differences.

Questions for authors

see weakness

2 comments

Response to Reviewer Fpza

Thank you for recognizing our finding that errors in intermediate judgments can cancel out in aggregate scores. This highlights the importance of assessing evaluator reliability at both intermediate and final stages, a central principle of our proposed paradigm. We also appreciate your constructive suggestions on panel composition, human agreement, execution variability, and agentic design.

W1: Study scale

We obtained annotations from the same four researchers for 16 additional captions, extending the panel to 80. Please see overall response G2 and Appendix J.4, Table 15, for the comparison of the 48-, 64-, and 80-caption panels across 31 shared evaluation configurations. Alignment-point decisions change in 5/155 cases (3.2%) and 3/155 cases (1.9%) across successive expansions. These results show limited changes across the tested panel sizes.

Gemini 3.7 Flash was deprecated on October 8th, which prevented fresh replication of the original version. But the limited changes in alignment-point decisions across the tested panel sizes indicates the increase in panel size does not bring drastic changes for certification.

W2: Caption distribution and panel selection

As summarized in Section 4.2 (Table 1), we constructed the candidate pool from 1,000 video–reference pairs using eight caption-generating models, three frame rates, and three prompts. Section 4.2 and Appendix D explain how selection seeks coverage of joint video, reference-caption, and generated-caption embeddings while balancing generation settings. This addresses diversity in the outputs themselves: different models can produce similar captions, so balanced model counts alone need not ensure broad coverage.

Please see overall response G2 for the broader study-scale discussion. For detailed distribution evidence, Appendix J.1 reports coverage across generation and selection stages (Table 11) and the annotated panel’s generator, prompt, and frame-rate composition (Table 12). Appendix J.2 (Figure 4) presents caption word clouds before and after selection, providing a qualitative comparison of vocabulary distributions.

W3: Inter-annotator consistency

The table below reports exact pairwise agreement. All four annotators agree on 45.8% of precision factors (374/816) and 30.3% of recall factors (303/1,001), including matching abstentions. Such disagreement is natural in open-ended video evaluation, where judgments can admit different interpretations.

Appendix K provides complementary views of this variation: pairwise confusion matrices for precision and recall factors (Figures 5–6), and each held-out annotator’s factor-level and caption-score alignment against a reference formed from the other three (Figure 7). These comparisons make the variation underlying the mean human baseline explicit, supporting a scoped interpretation of certification relative to individual human performance.

FactorH1–H2H1–H3H1–H4H2–H3H2–H4H3–H4
Precision74.3%64.3%66.7%65.6%68.3%66.9%
Recall59.3%55.8%63.4%50.6%48.2%64.4%

W4: Comparing execution methods

Error patterns. To examine error patterns, we conducted a case study and present a representative ladder-climbing example (Appendix L: factor judgments in Table 17; whole-caption scores and errors in Table 18). Agentic rejected “preparing to climb,” accepted by all humans and single-pass, but matched human partial coverage for “two sections joined together,” which single-pass fully credited despite missing detail. Its lower recall-factor but higher precision-factor error illustrates complementary strengths.

When to prefer each method. Our paradigm leaves method choice to each use case, with certification providing a core selection criterion. Other factors may also influence the final choice. We illustrate this by examining the relationship between video duration and evaluation error across 11 models (Appendix P, Figure 11). Averaged across models, the smoothed error differences favor agentic for recall and F1 throughout the observed duration range; precision favors agentic for shorter and longer videos and single-pass in between. The individual-model panels show variation across models (Appendix P, Figures 12–14), so selection should reflect the chosen model and application requirements. This auxiliary study offers guidance without prescribing a universal choice.

Repeated averaging. To test repeated averaging, we ran each method twice for Gemini 3.8 Flash and Gemini 3.1 Pro Preview on 64 captions (Appendix M, Tables 19–20). Same-method averaging reduces both factor errors; mixed pairs generally improve them further, while scalar gains vary by model and pairing. These gains are consistent with reduced run-to-run variability and additional error cancellation across methods.

W5: Agentic design and the shared vision backend

We agree that the reviewer's example, “Qwen run on OpenCode can use Qwen itself as the vision understanding backend,” is a meaningful design option. However, our design examines the underlying model's agentic capability on video tasks. Our shared Gemini 3.1 Pro Preview vision-query skill extends participation to LLMs without native vision, allowing us to examine agentic reasoning and tool use for video understanding. It provides an additional visual interface while allowing agents to choose their inspection strategies. Models with native vision can also use the file-reading tool to inspect images directly, a behavior observed in the execution traces of several models. Appendix F (the agentic implementation description) documents the shared tool interface; Appendix L (Figure 8) illustrates direct frame inspection in the paired case. This leaves the underlying model to decide how to use the visual interface, as part of its agentic reasoning and tool use.

Comments on the paper

1 comment

Overall response

We thank the reviewers for their careful reading and for recognizing the value of the paradigm and its instantiation. We have revised the manuscript and added supplementary experiments to address their concerns and suggestions. Below, we clarify two shared concerns; the individual responses address reviewer-specific comments.

G1. Contribution and scope

We reiterate that our core contribution is a general, executable paradigm for designing and certifying evaluation of open-ended video understanding tasks (Section 3). Establishing the reliability and boundaries of an evaluation is a prerequisite to interpreting its scores and applying it to wider response distributions. Appendix I (Table 10) shows that few reviewed video protocols document the complete process.

The captioning instantiation demonstrates that the paradigm is executable and yields meaningful evidence about both qualifying evaluations and remaining limitations. We hope its empirical findings and design lessons encourage comparable studies in the community. Each open-ended task requires its own instantiation and human evidence; our work does not seek to solve every task's evaluation. The paradigm provides a structured process for pursuing human-level alignment, establishing its scope, and iterating when an evaluation falls short.

G2. Study scale

To assess sensitivity to study scale, we extended the human panel to 80 captions and compared it with the original 64-caption panel and the fixed random 48-caption subset (Appendix J.4, Table 15). Across the same 31 evaluation configurations, alignment-point decisions—whether an evaluation meets the corresponding human baseline—change in 5/155 cases (3.2%) from 48 to 64 captions and 3/155 cases (1.9%) from 64 to 80. Most pointwise decisions therefore remain stable across the tested panel sizes; Appendix J.4 also illustrates changing decisions (Table 16).

This sensitivity analysis complements the panel coverage and annotation effort documented in Appendices J.1–J.3 (Tables 11–14; Figure 4). The original 64 captions span 61 videos and all eight caption-generating models, three prompts, and three frame rates represented in the candidate pool. Four researchers independently evaluated every caption, contributing 7,268 factor judgments. Recorded annotation intervals yield a median-based workload estimate of approximately 54 hours (Appendix J.3, Tables 13–14). The same researchers annotated the additional 16 captions. Certification remains specific to the evaluated panel and must be reassessed as the panel changes, consistent with the paradigm’s emphasis on establishing the boundaries of evaluation reliability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.