- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In this paper, the authors propose SEA-Bench, an agentic framework that evolves the evaluation dimensions of video generation benchmarks based on model feedback. It follows a propose–execute–decide pipeline to add, refine, or remove evaluation dimensions. Starting from VBench, the authors evaluate the framework on text-to-video models and a task-specific memory benchmark. The evolved benchmarks are validated through cross-pool comparisons, alignment with existing benchmarks, human evaluation, and aggregate model discrimination.
Critical strengths
- Adaptive benchmark construction. Unlike existing methods that mainly generate harder evaluation instances, the proposed SEA-Bench updates the evaluation dimensions themselves. This provides a practical direction for evaluating emerging capabilities not covered by existing benchmarks.
- Adaptation to different model capabilities. The authors adopt two model pools for different evaluation dimensions. The cross-pool results in Table 3 further show that selected Group-B dimensions are more discriminative for Group B, supporting capability-dependent benchmark construction.
- Human validation of the discovered dimensions. The authors evaluate dimension meaningfulness, prompt faithfulness, and evaluator–human agreement using held-out examples, whose results provide initial evidence that the constructed evaluations are meaningful.
- Support for task-specific evaluation goals. Beyond open-ended evolution, SEA-Bench constructs eight memory-related dimensions from a user-specified goal, demonstrating its flexibility.
Critical weaknesses
- Lack of baselines and ablation studies. The paper does not compare SEA-Bench against simpler benchmark expansion strategies or ablate the proposed agent components. It is unclear how much the feedback-driven evolution and Decision Agent contribute to the reported improvements.
- Limited generalization of the evolved benchmarks. The improvements in model discrimination are evaluated on the same model pools used during benchmark evolution, each containing only four models. Without evaluation on unseen models, it is unclear whether the discovered dimensions capture general capabilities or mainly distinguish the target models.
- Limited human validation. Only eight selected dimensions are validated, and evaluator–human agreement is 72.3% over 47 non-tied pairs. This is a relatively small validation set given the number of dimensions generated by the framework, leaving the reliability of the full benchmark unclear
- Insufficient validation of the task-specific benchmark. The memory benchmark is evaluated mainly through qualitative correspondence with MBench. No quantitative results are provided to establish whether the eight memory dimensions reliably measure the intended capabilities or improve model evaluation.
Questions for authors
- Could the authors provide an ablation or a matched-budget baseline to clarify the contribution of the feedback-driven evolution and Decision Agent, beyond simply expanding the benchmark?
- Do the improvements in Table 6 hold across independent evolution runs or on unseen models? This would help establish whether the evolved dimensions generalize beyond the target model pool.
- Could the authors provide per-dimension human agreement results, along with the number of annotators and excluded ties? This would help assess the reliability of the newly constructed evaluators.
- Is there quantitative validation of the memory-specific benchmark, beyond the qualitative comparison with MBench in Table 7?