acceptodds
Under review as a conference paper at ICLR 2027

SEA-Bench: Self-Evolving Agentic Benchmark for Video Generation

Abstract

Video generation models are advancing rapidly, while their evaluation benchmarks remain largely static. Existing benchmarks typically fix the evaluated capability dimensions and task distributions, causing previously informative tests to saturate and newly emerging capabilities to remain unmeasured. We introduce SEA-Bench, a self-evolving benchmark construction framework that enables agents to discover and operationalize evaluation dimensions from observed model behaviors. Starting from an existing benchmark, SEA-Bench iteratively proposes capability hypotheses, designs executable prompts and evaluation procedures, validates them on target models, and uses an evidence-based decision agent to accept benchmark updates supported by the resulting evaluation evidence. Unlike prior dynamic evaluation methods that adapt instances or evaluation trajectories within a predefined capability space, SEA-Bench evolves the evaluation space itself. We instantiate SEA-Bench for text-to-video generation and evolve benchmarks for model pools with different capability profiles. Retrospective comparisons with independently developed video benchmarks assess the community alignment of retained dimensions, while human studies validate their meaningfulness, prompt faithfulness, and agreement between executable evaluators and human preferences. We compare aggregate model discrimination before and after benchmark evolution in the two evaluated model pools.

Primary area:
datasets and benchmarks
Keywords:
self-evolving agent, video model benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WYJVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors propose SEA-Bench, an agentic framework that evolves the evaluation dimensions of video generation benchmarks based on model feedback. It follows a propose–execute–decide pipeline to add, refine, or remove evaluation dimensions. Starting from VBench, the authors evaluate the framework on text-to-video models and a task-specific memory benchmark. The evolved benchmarks are validated through cross-pool comparisons, alignment with existing benchmarks, human evaluation, and aggregate model discrimination.

Critical strengths
  1. Adaptive benchmark construction. Unlike existing methods that mainly generate harder evaluation instances, the proposed SEA-Bench updates the evaluation dimensions themselves. This provides a practical direction for evaluating emerging capabilities not covered by existing benchmarks.
  2. Adaptation to different model capabilities. The authors adopt two model pools for different evaluation dimensions. The cross-pool results in Table 3 further show that selected Group-B dimensions are more discriminative for Group B, supporting capability-dependent benchmark construction.
  3. Human validation of the discovered dimensions. The authors evaluate dimension meaningfulness, prompt faithfulness, and evaluator–human agreement using held-out examples, whose results provide initial evidence that the constructed evaluations are meaningful.
  4. Support for task-specific evaluation goals. Beyond open-ended evolution, SEA-Bench constructs eight memory-related dimensions from a user-specified goal, demonstrating its flexibility.
Critical weaknesses
  1. Lack of baselines and ablation studies. The paper does not compare SEA-Bench against simpler benchmark expansion strategies or ablate the proposed agent components. It is unclear how much the feedback-driven evolution and Decision Agent contribute to the reported improvements.
  2. Limited generalization of the evolved benchmarks. The improvements in model discrimination are evaluated on the same model pools used during benchmark evolution, each containing only four models. Without evaluation on unseen models, it is unclear whether the discovered dimensions capture general capabilities or mainly distinguish the target models.
  3. Limited human validation. Only eight selected dimensions are validated, and evaluator–human agreement is 72.3% over 47 non-tied pairs. This is a relatively small validation set given the number of dimensions generated by the framework, leaving the reliability of the full benchmark unclear
  4. Insufficient validation of the task-specific benchmark. The memory benchmark is evaluated mainly through qualitative correspondence with MBench. No quantitative results are provided to establish whether the eight memory dimensions reliably measure the intended capabilities or improve model evaluation.
Questions for authors
  1. Could the authors provide an ablation or a matched-budget baseline to clarify the contribution of the feedback-driven evolution and Decision Agent, beyond simply expanding the benchmark?
  2. Do the improvements in Table 6 hold across independent evolution runs or on unseen models? This would help establish whether the evolved dimensions generalize beyond the target model pool.
  3. Could the authors provide per-dimension human agreement results, along with the number of annotators and excluded ties? This would help assess the reliability of the newly constructed evaluators.
  4. Is there quantitative validation of the memory-specific benchmark, beyond the qualitative comparison with MBench in Table 7?
Reviewer_Sz99OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SEA-Bench is an agentic framework for evolving video-generation benchmarks. Starting from VBench, a Proposal Agent suggests additions, refinements, or deletions of capability dimensions. An Execution Agent constructs prompts and executable evaluators, tests the target models, and collects evidence. A separate Decision Agent accepts or rejects benchmark updates, while accumulated construction records inform subsequent iterations. The experiments evolve benchmarks for two pools of four video-generation models and explore a memory-focused evaluation goal. Validation includes capability coverage, retrospective alignment with existing benchmarks, cross-pool testing of three discovered dimensions, held-out human annotations, and changes in the coefficient of variation of aggregate model scores.

Critical strengths
  1. Meaningful benchmark-evolution formulation. Treating capability dimensions, prompts, and evaluators as revisable benchmark components addresses coverage gaps that cannot be resolved merely by generating harder prompts. The proposal–execution–decision workflow provides a concrete mechanism for operationalizing these updates.

  2. Held-out human validation. The human study excludes validation prompts and videos from evolution and evaluator development, while hiding model identities and evaluator scores.

Critical weaknesses
  1. Discrimination gains do not yet establish more reliable evaluation. Candidate acceptance considers model discrimination, but Table 6 does not specify independent evaluation prompts/generations or uncertainty estimates.

  2. The contribution of execution-driven evolution is not isolated. There is no matched-budget feedback-free baseline or repeated-run analysis showing that observed model behavior drives better dimension discovery.

Questions for authors
  1. Were the Table 6 prompts and videos excluded from benchmark construction and selection? Please provide the score-normalization rules, raw aggregate model scores, and uncertainty estimates on fresh evaluation samples.

  2. What construction and generation budgets were used, and how does SEA-Bench compare with the same agent operating without execution feedback under matched budgets across repeated runs?

Reviewer_vbHxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes SEA-Bench, an agentic framework that iteratively evolves an existing video-generation benchmark. Starting from VBench, a Proposal Agent (deepseek-v4.1-flash) proposes ADD/REFINE/DELETE operations on capability dimensions based on the current benchmark and an accumulated construction context (prior hypotheses, evaluation scores, and past decisions); an Execution Agent operationalizes each ADD/REFINE proposal by constructing generation prompts and selecting an evaluation mechanism (existing tools, VQA with Qwen3.8-Flash, or customized code), producing a per-dimension score vector over the target model pool; an independent Decision Agent returns a binary ACCEPT/REJECT for each proposal based on distinctness, operational validity, and model discrimination. Experiments use two pools of four commercial text-to-video models (cost-efficient Group A vs. higher-capability Group B), with one benchmark evolved per pool. Reported evidence includes: (i) pool-specific discovered capability families (e.g., counting, constraint following, fine-grained motion for Group A; camera control, text rendering, temporal event sequencing for Group B), with cross-pool discrimination gaps when Group-B dimensions are re-evaluated on Group A (e.g., 0.0872 vs. 0.4001 for temporal event sequencing); (ii) post-hoc community alignment of retained dimensions with independently developed benchmarks (VBench-2.0, VBench++, W-Bench); (iii) human validation of a subset of new dimensions (100% endorsement and prompt faithfulness with Krippendorff's alpha = 1.000; evaluator-human agreement of 72.3% over 47 non-tied pairs, alpha = 0.776); (iv) aggregate model discrimination (coefficient of variation of aggregate scores) changing from 0.0142 to 0.0171 for Group A and from 0.0135 to 0.0808 for Group B; and (v) a task-specified run that produces an eight-dimension memory benchmark, qualitatively compared with MBench.

Critical strengths

S1. The paper addresses a real and timely problem — static benchmarks saturating under rapidly advancing generative models — and provides, to my knowledge, the first instantiation of evidence-based benchmark self-evolution in the video domain. Its central design choice, that every benchmark update must be backed by executed evaluation evidence (a score vector over the target pool) rather than LLM conjecture, and that an independent agent issues explicit ACCEPT/REJECT decisions, is a reasonable mechanism for keeping benchmark evolution grounded. The post-hoc community alignment (Table 4) further shows that several discovered directions coincide with capabilities independently emphasized by later benchmarks (VBench-2.0, W-Bench), suggesting the discoveries are not arbitrary. S2. The pool-adaptivity finding is supported by the paper's most quantitative experiment: dimensions discovered for the higher-capability pool show systematically higher discrimination within that pool than in the cost-efficient pool under frozen evaluation specifications (Table 3), and the two evolved benchmarks emphasize different capability profiles. This is evidence that the construction adapts to the observed model population rather than converging to a fixed dimension set.

Critical weaknesses

W1. The paper provides no controlled evidence that the agentic framework itself, rather than the underlying LLM, is responsible for the reported discoveries. There is no baseline or ablation of the loop: no comparison against a simpler pipeline (e.g., single-prompt dimension proposal without execution evidence, without the Decision Agent, or without the construction context), and no component-level ablation. Moreover, the quality metrics are self-defined (coefficient-of-variation discrimination) and computed on the benchmark the framework itself produced. The human validation — the only human-grounded evidence — is anomalous: Krippendorff's alpha = 1.000 for both dimension endorsement and prompt faithfulness (perfect agreement is implausible and suggests a very small or degenerate annotation design), and evaluator-human agreement is 72.3% on only 47 non-tied pairs. Group A's aggregate discrimination is essentially unchanged (+0.0029), so the claimed discriminative benefit holds only for one of the two pools. W2. The actual evolution process is never demonstrated. The paper does not report the model pool's original per-dimension results on VBench, any concrete iteration trace (which limitation was diagnosed from which scores, what ADD/REFINE/DELETE operation was proposed in response, and why the Decision Agent accepted or rejected it), or full per-model per-dimension score tables on the evolved benchmark — only radar-style dimension profiles (Figure 3) and aggregate discrimination (Table 6). Community alignment is post-hoc and qualitative. As a result, the central claim that the framework discovers valid, diagnostic dimensions beyond VBench cannot be verified from the paper's tables alone, and the reader cannot distinguish a principled evolution from an opaque LLM-driven re-description of the benchmark.

Questions for authors

Q1. [Clarification] Agent usage details. What exactly does the Proposal Agent receive at each iteration, and how is the model pool's performance information summarized and fed back (per-dimension scores, failure cases, evaluator rationales)? Please provide the actual prompt templates for all three agents, and specify how the agent is instructed to derive new hypotheses from prior evaluation outcomes, beyond the qualitative statement in Appendix A.2. Q2. [Experiment request] VBench-case experiment details. Please report: (a) the two model pools' original per-dimension scores on VBench; (b) concrete iteration traces of the evolution — which limitations were identified from which scores, what operations were proposed in response, and the Decision Agent's accept/reject rationales; (c) full per-model, per-dimension score tables on the evolved benchmark (not only the aggregate discrimination in Table 6 and the profiles in Figure 3); and (d) the number of iterations, number of proposals by operation type, acceptance rates, and the compute/token cost of the full evolution. Q3. [Clarification] Human validation protocol. How many annotators and how many items underlie the alpha = 1.000 results in Table 5, and how were the validated dimensions selected? If the endorsement set is small, the perfect agreement is uninformative and should be scaled up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.