- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a new metric, "Divid", to evaluate the diversity of generated videos. Unlike existing metrics, Divid aims to assess diversity across multiple dimensions. The authors argue and demonstrate that a single score is insufficient to fully capture video diversity. The paper also identifies two fundamental bottlenecks to improving diversity: default mode convergence and realization gap, and suggests a future research direction to address these challenges.
Critical strengths
- The idea of separating diversity into different aspects for evaluation is interesting. This provides users with a more comprehensive understanding of how the model behaves across different dimensions and helps identify areas for further improvement. In particular, Figure 1 provides a strong starting point and clearly illustrates the motivation behind this work.
Critical weaknesses
- Although the motivation is clear, the evaluation results for the proposed metrics are relatively weak. Specifically, the Spearman correlations with human evaluations are generally weak to moderate (Table 5), while an ideal correlation would be above 0.8. Similarly, Krippendorff's alpha values are also low, indicating limited agreement among human annotators. These results raise concerns about how reliably the proposed metrics can reflect human judgments of diversity.
- I understand that this is primarily a benchmark paper, so proposing a new approach to enhance diversity is not necessarily required. However, the paper lacks concrete experiments demonstrating how existing diversity enhancement techniques (e.g., prompt rewriting, as mentioned by the authors) can actually improve diversity and how these improvements are reflected in the proposed metrics. The authors attempt to provide some "actionable" directions for improving models, but these discussions are somewhat incomplete. For instance, the discussion at the end of "Model Convergence" in Section 5.2 does not really suggest a concrete solution. Overall, while the authors are not expected to develop a new diversity enhancement method, I believe the paper would be strengthened if they applied existing techniques and examined how the resulting changes are captured by the proposed metrics. This would help validate whether the identified diversity issues are indeed reflected in practice and whether the proposed metrics can reliably measure improvements in these areas.
Questions for authors
- It would be helpful if the authors could further explain why the proposed metrics are still considered reliable despite their relatively low correlations with human evaluations, and whether this limitation affects the validity of their findings.
- I recommend that the authors include additional experiments using existing diversity enhancement techniques to demonstrate whether these approaches can effectively address the identified issues and lead to measurable improvements in the corresponding Divid dimension scores.
7 comments
Authors' Response: W1&Q1 (2/4)
2) The additional stratified analysis shows that DiVid correlates strongly with human judgments when annotators agree; lower correlations mainly come from items that even humans find hard to distinguish.
We analyze the 90 items rated by two annotators separately for each of the six dimensions. We encode the five preference levels as −2, −1, 0, 1, and 2. Items whose two ratings differ by at most one level form the high-agreement group; those whose ratings differ by two or more levels form the low-agreement group. For diversity differences that humans can distinguish consistently, DiVid shows high correlations with human judgments; lower correlations are mainly observed in the group with greater disagreement between annotators. For example, in the case for Wan2.2-5B, DiVid assigns HappyHorse a slightly higher subject diversity score, while one human rater considers Wan2.2-5B more diverse and the other favors HappyHorse. In fact, the two sets exhibit comparable subject diversity.
| Dimension | Full-sample DiVid–human ρ | High-agreement DiVid–human ρ | High-agreement human–human ρ | Low-agreement DiVid–human ρ | Low-agreement human–human ρ |
|---|---|---|---|---|---|
| Semantic | 0.604 | 0.694 | 0.949 | 0.340 | −0.208 |
| Style | 0.563 | 0.697 | 0.882 | 0.325 | −0.137 |
| Subject | 0.549 | 0.764 | 0.914 | 0.217 | −0.190 |
| Scene | 0.541 | 0.742 | 0.910 | 0.330 | 0.077 |
| Motion | 0.531 | 0.753 | 0.892 | 0.205 | −0.291 |
| Camera | 0.556 | 0.720 | 0.874 | 0.276 | −0.165 |
3) Quality control and complementary validations support metric validity.
- Annotation quality control (Appendix B.4). The original study incorporates standardized training, blinding and randomization, separate displays for different dimensions, and video-viewing requirements. All annotators correctly answered every gold item, providing direct support for annotation quality.
- Complementary experiments further support metric validity. Table 4 shows that Motion responds as expected to differences in direction, speed, and action. Table 8 shows that, when variation in the other factors is small, Motion and Camera identify video pairs with pronounced differences in the target factor for 92.0% and 93.3% of prompts, respectively, supporting their ability to provide distinct diagnostic information. Moreover, the leading model in each factor remains unchanged under stricter validity filtering (Appendix B.3), and the key cross-dimensional ranking reversals are supported by paired statistical tests (Appendix C.1).
Together, these findings support DiVid’s use for model-level diversity comparisons and factor-level diagnosis, as well as our central finding that models exhibit different strengths across diversity dimensions.
Authors' Response: W2&Q2 (3/4)
W2&Q2: Need to display the results of diversity enhancement techniques, showing how DiVid captures their improvements across dimensions; need to show whether DiVid can capture the intended improvements in diversity dimensions; better show more concrete directions for improving current methods.
Re: Thank you for the suggestion. To address your concerns, we: 1) add experiments showing that existing techniques produce measurable diversity gains, and that DiVid captures both these improvements and their dimension-specific effects; 2) provide controlled, single-factor interventions to test whether the corresponding DiVid dimension responds as intended; 3) give more concrete recommendations.
1) Dimension-level diagnosis of existing methods.
We evaluate sampling-based methods (Bell CFG, FreqCFG, CADS) and two prompt-rewriting baselines (DPP, DIVA) on Wan2.2-5B across all 206 prompts, generating 10 videos per prompt for each method. We report six-dimensional diversity alongside quality and faithfulness, with IF evaluated against the original prompt.
- Sampling-based interventions: Bell CFG (TV-CFG) uses a guidance schedule that first increases and then decreases; FreqCFG applies separate guidance strengths to low-frequency structure and high-frequency details; CADS perturbs the text conditioning early in sampling and gradually restores it.
- Prompt rewriting: DPP, an inference-time variant of DPP-GRPO, that selects candidates using the original method’s diversity and semantic-relevance scoring, replacing GRPO training because the code and the trained rewriting model are not open-source; DIVA, our re-implementation of its Director → Architect → Critic → Producer workflow. Both use DeepSeek-v4.1-flash-1017.
| Method | Semantic | Style | Subject | Scene | Motion | Camera | TCE | TIE | IF | AQ | IQ | TF | MS |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CFG=5 | 0.169 | 0.234 | 0.447 | 0.426 | 0.326 | 0.097 | 5.29 | 13.36 | 0.848 | 0.543 | 0.672 | 0.980 | 0.989 |
| Bell CFG | 0.203 | 0.246 | 0.484 | 0.493 | 0.373 | 0.119 | 6.41 | 14.32 | 0.842 | 0.536 | 0.673 | 0.975 | 0.988 |
| FreqCFG | 0.195 | 0.248 | 0.477 | 0.477 | 0.361 | 0.104 | 6.08 | 13.91 | 0.843 | 0.536 | 0.671 | 0.979 | 0.989 |
| CADS | 0.243 | 0.330 | 0.560 | 0.586 | 0.357 | 0.054 | 7.06 | 15.75 | 0.760 | 0.515 | 0.700 | 0.985 | 0.990 |
| DPP | 0.240 | 0.279 | 0.492 | 0.546 | 0.301 | 0.034 | 6.74 | 14.43 | 0.893 | 0.599 | 0.702 | 0.988 | 0.993 |
| DIVA | 0.216 | 0.230 | 0.434 | 0.466 | 0.306 | 0.021 | 6.19 | 13.94 | 0.897 | 0.583 | 0.715 | 0.988 | 0.994 |
Bold indicates superior performance over the CFG=5 baseline.
The results directly address whether existing techniques can alleviate the identified issues:
- Bell CFG and FreqCFG improve all six DiVid dimensions. For example, Bell CFG increases Motion from 0.326 to 0.373 and Camera from 0.097 to 0.119, demonstrating measurable improvements in temporal diversity.
- Prompt rewriting improves diversity while preserving quality and faithfulness. DPP increases Semantic from 0.169 to 0.240 and Scene from 0.426 to 0.546, while increasing IF from 0.848 to 0.893 and improving all four quality metrics. DIVA also improves Semantic, Scene, IF, and all four quality metrics.
- DiVid identifies the scope and trade-offs of these improvements. Although DPP and DIVA improve overall diversity and quality, their Motion and Camera diversity decreases. CADS improves five dimensions but reduces Camera, IF, and AQ. In particular, CADS, DPP, and DIVA all increase TCE/TIE while reducing Camera diversity—a distinction that the overall scores do not reveal.
Authors' Response: W1&Q1 (1/4)
Dear Reviewer yYb3, thank you for your careful comments and suggestions. We address your concerns as follows:
Re: Thank you for raising this concern about metric reliability. We would like to clarify that our correlations and annotator agreement are in line with existing evaluation work. We further support the reliability of both the metrics and the human annotations through an additional stratified analysis and the validation already presented in the paper.
1) Our metric–human correlations (ρ) and human–human agreement (α) are comparable to those reported in existing evaluations of open-ended generation.
For subjective, open-ended judgments, ρ>0.8 is not a universal reliability threshold. Under Cohen’s (1988) convention, a correlation of 0.5 represents a large effect. This convention is routinely applied to rank correlations in evaluation research, including CodeJudge (Tong and Zhang, 2024) and PASTA (Yang et al., 2026). Moreover, our correlations are computed at the item level (two models’ videos for one prompt), which is stricter than the model-level win-ratio correlations used by video benchmarks such as VBench. The table below lists ρ and α reported in recent evaluation studies: our ρ (0.531–0.604) and α (0.679–0.703) lie within the reported range.