- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This is a benchmark paper that aims to evaluate whether characters and props can preserve their identities while carrying out the intended story in reference-conditioned animated video generation. It's core contribution is that it proposed a systematic list of questions for quantitative evaluation, covering various fine-grained aspects that previous evaluations have rarely covered comprehensively. As for validation, their main method is that they adopt an annotate-then-review protocol. Beyond that, they additionally conduct independent annotation by 3 annotators on a subset of 176 clips to assess agreement.
Critical strengths
The main strength of this paper is that it constructs a systematic and fine-gained list of evaluation questions and conducts a large-scale human evaluation on videos generated by multiple models. This evaluation framework covers multiple specific capability dimensions, providing clearer evaluation criteria and research directions for future model improvements research.
Critical weaknesses
- The evaluation question list is comprehensive, but the cost of conducting an evaluation is too high: each video requires 20–30 minutes of annotation, and annotators are paid $20 per hour. Because of that, the benchmark is difficult to use for frequent model iteration and comparison and for reproduction by other laboratories. I thus doubt its practical utility.
- The paper seems to contain some inconsistencies: the human-evaluation ranking in Figure 7 differs from the ranking reported in Section 3.1, and although A.1 states that the supplementary ZIP contains outputs from all six models, the submitted one contains outputs from only three models.
Questions for authors
Could you explain why the human-evaluation ranking in Figure 7 differs from the ranking reported in Section 3.1? The paper states that the supplementary ZIP contains all six models' outputs, but the submitted archive contains outputs from only three models. Could you clarify this?
4 comments
Follow-up: the cost of a single evaluation run
Our earlier reply gave corpus-level savings. Here is what a single evaluation run costs.
Frequent model iteration and comparison. A new model adds one clip per story, so 100 clips on the released split. What that costs depends on what you need from it:
- USD 22 for a model ranking: the judge alone, at USD 0.22 per clip, unattended at 256 s each.
- USD 458–676 for human labels under the hybrid protocol, with 4 of the 100 clips needing no human at all.
- USD 667–1,000 to annotate every clip by hand, at the 20–30 minutes and USD 20 per hour (cited rate).
On the full 317-story corpus the same three are about USD 70, USD 1,453–2,143 and USD 2,114–3,170. Ranking a new checkpoint is therefore a per-run cost rather than a per-study one.
Reproduction by other laboratories. Reproducing our published results requires no annotation at all. The human labels for 300 video clips are released now and 600 at camera-ready, with the code that produces every figure and table from them, and the released labels let the full aggregation be re-run end to end.
We hope this addresses your concern, and are glad to provide further detail.
Gentle Follow-Up Before Discussion Deadline
We thank the reviewer for their questions and requests for clarification. We have also posted a summary of our revisions and additional supporting results in the global comment for ease of reference.
As the discussion deadline approaches, we would sincerely appreciate the opportunity to engage with the reviewer and hear whether our responses have adequately addressed their concerns.
Thank you for the effort put into the detailed rebuttal and additional analyses. These address part of my concerns. However, the hybrid protocol still requires substantial human effort for fine-grained evaluation. The reported median delegation rate is 0% for interaction, handover, and action coverage. The low-cost AI ranking option does not replace these detailed assessments. Moreover, the time savings are estimated from the fraction of delegated items, rather than measured annotation time. I therefore remain concerned about the benchmark’s practicality for frequent model iteration and maintain my weak rejection recommendation.
Rebuttal Response to the concerns.
We thank the reviewer for their thoughtful and constructive feedback.
Regarding the supplementary archive, we acknowledge this oversight in its description. We now provide an anonymous repository (https://anonymous.4open.science/r/ANIMACT-4526/ ) containing human annotations for 300 clips across all six models, reference assets, 48 sample clips (8 stories × 6 models), and reproducibility code for the aggregation, ranking comparisons, and main analyses.
What the protocol saves, and what it costs. Our protocol removes 34.6% of question-items from the human annotation queue, saving 219–329 annotator-hours and USD 3,961–6,152 (31–32% of the human-only cost of USD 12,680–19,020), after accounting for the USD 421 AI judge panel cost. These savings come at the cost of a five-percentage-point reduction in defect visibility, to 89.9%. We further illustrate this trade-off through a case study in Figure Fig-Usecase (available in our repository), showing how the share of question-items eligible for delegation varies across jobs, ranging from 4% to 49%.
Release. Our repository (https://anonymous.4open.science/r/ANIMACT-4526/) includes annotations, gating code, and threshold-sweep tool (which a practitioner can use to tune their threshold depending on the use case and rubrics preference); all 100 stories with full annotations will be released at camera-ready.