acceptodds
Under review as a conference paper at ICLR 2027

ANIMACT: What Breaks When Characters Meet? A Capability Taxonomy for Multi-Reference Animated Video Generation

Abstract

Reference-conditioned animated video generation requires models to preserve visually specified characters and props while executing a short story with the correct actions, interactions, and bindings. Animation makes this evaluation particularly challenging: stylized deformation and exaggerated motion may be intentional, while identity, contact, weight, and who-acted-on-whom must remain visually clear. Existing benchmarks capture parts of this problem, but do not jointly evaluate reference-grounded story execution within a single generated clip. We introduce , a holistic human evaluation benchmark that combines 14 overall criteria with eight targeted capability probes, covering reference fidelity, action execution, role assignment, interaction, identity preservation, and temporal, spatial, and physical stability. The benchmark contains 317 human-verified stories spanning 64 characters and 13 props; 95% are multi-character, and each conditions on 1–4 reference assets together with evaluation-only visual and action annotations. Across 1,902 clips from six reference-to-video systems, Seedance-2.5 is the clear overall leader, yet substantial failures remain: 40% of scripted character-to-character interaction edges do not occur, appearance consistency degrades as reference load increases while entity presence remains stable, and only of clips pass all applicable evaluation axes. Finally, we show that AI judges recover aggregate model trends much better than individual human judgments, making them useful for relative model comparisons but not yet a replacement for fine-grained human evaluation.

Primary area:
datasets and benchmarks
Keywords:
multi-reference video generation, benchmark, human evaluation, character consistency, identity preservation, animated video generation, video generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Zjf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This is a benchmark paper that aims to evaluate whether characters and props can preserve their identities while carrying out the intended story in reference-conditioned animated video generation. It's core contribution is that it proposed a systematic list of questions for quantitative evaluation, covering various fine-grained aspects that previous evaluations have rarely covered comprehensively. As for validation, their main method is that they adopt an annotate-then-review protocol. Beyond that, they additionally conduct independent annotation by 3 annotators on a subset of 176 clips to assess agreement.

Critical strengths

The main strength of this paper is that it constructs a systematic and fine-gained list of evaluation questions and conducts a large-scale human evaluation on videos generated by multiple models. This evaluation framework covers multiple specific capability dimensions, providing clearer evaluation criteria and research directions for future model improvements research.

Critical weaknesses
  1. The evaluation question list is comprehensive, but the cost of conducting an evaluation is too high: each video requires 20–30 minutes of annotation, and annotators are paid $20 per hour. Because of that, the benchmark is difficult to use for frequent model iteration and comparison and for reproduction by other laboratories. I thus doubt its practical utility.
  2. The paper seems to contain some inconsistencies: the human-evaluation ranking in Figure 7 differs from the ranking reported in Section 3.1, and although A.1 states that the supplementary ZIP contains outputs from all six models, the submitted one contains outputs from only three models.
Questions for authors

Could you explain why the human-evaluation ranking in Figure 7 differs from the ranking reported in Section 3.1? The paper states that the supplementary ZIP contains all six models' outputs, but the submitted archive contains outputs from only three models. Could you clarify this?

4 comments
AuthorsOpenReview ↗

Rebuttal Response to the concerns.

We thank the reviewer for their thoughtful and constructive feedback.

  1. Ranking differences and supplementary archive. We thank the reviewer for pointing this out. The ranking differences arise from how scores are aggregated, not from the evaluation subset. Section 3.1 and Table 7 report the mean clip-level score over all 1,902 clips, weighting every scored item equally, whereas Figure 7 takes a question- then axis-weighted mean over the 1,876 clips carrying both a human and an AI-judge label. Under the clip-level mean MiniMax-H3 leads HappyHorse 1.1 by 0.017; under axis weighting the two reverse and sit 0.015 apart. Recomputing the axis-weighted aggregation on the full 1,902-clip corpus gives the same reversal (0.004), so it follows from the weighting rather than from the smaller judge-annotated subset. Seedance-2.5 is first under both aggregations. We have clarified these definitions in Sections 3.1 and 4, and we refer to Figure 7 consistently as the judge-comparison ranking.

Regarding the supplementary archive, we acknowledge this oversight in its description. We now provide an anonymous repository (https://anonymous.4open.science/r/ANIMACT-4526/ ) containing human annotations for 300 clips across all six models, reference assets, 48 sample clips (8 stories × 6 models), and reproducibility code for the aggregation, ranking comparisons, and main analyses.

  1. Utility. We thank the reviewer for raising this concern. ANIMACT enables approximate model comparisons without requiring full human re-annotation. Our results show that AI judges recover aggregate model rankings well (ρ = 0.886; 78.3% question-level ordering agreement), supporting their use for relative model comparisons. We have clarified this distinction across Sections 2–4 and also present a hybrid AI–human evaluation protocol (more details in Shared Response) to reduce the human annotation burden for fine-grained capability assessment.

What the protocol saves, and what it costs. Our protocol removes 34.6% of question-items from the human annotation queue, saving 219–329 annotator-hours and USD 3,961–6,152 (31–32% of the human-only cost of USD 12,680–19,020), after accounting for the USD 421 AI judge panel cost. These savings come at the cost of a five-percentage-point reduction in defect visibility, to 89.9%. We further illustrate this trade-off through a case study in Figure Fig-Usecase (available in our repository), showing how the share of question-items eligible for delegation varies across jobs, ranging from 4% to 49%.

Release. Our repository (https://anonymous.4open.science/r/ANIMACT-4526/) includes annotations, gating code, and threshold-sweep tool (which a practitioner can use to tune their threshold depending on the use case and rubrics preference); all 100 stories with full annotations will be released at camera-ready.

AuthorsOpenReview ↗

Follow-up: the cost of a single evaluation run

Our earlier reply gave corpus-level savings. Here is what a single evaluation run costs.

Frequent model iteration and comparison. A new model adds one clip per story, so 100 clips on the released split. What that costs depends on what you need from it:

  • USD 22 for a model ranking: the judge alone, at USD 0.22 per clip, unattended at 256 s each.
  • USD 458–676 for human labels under the hybrid protocol, with 4 of the 100 clips needing no human at all.
  • USD 667–1,000 to annotate every clip by hand, at the 20–30 minutes and USD 20 per hour (cited rate).

On the full 317-story corpus the same three are about USD 70, USD 1,453–2,143 and USD 2,114–3,170. Ranking a new checkpoint is therefore a per-run cost rather than a per-study one.

Reproduction by other laboratories. Reproducing our published results requires no annotation at all. The human labels for 300 video clips are released now and 600 at camera-ready, with the code that produces every figure and table from them, and the released labels let the full aggregation be re-run end to end.

We hope this addresses your concern, and are glad to provide further detail.

AuthorsOpenReview ↗

Gentle Follow-Up Before Discussion Deadline

We thank the reviewer for their questions and requests for clarification. We have also posted a summary of our revisions and additional supporting results in the global comment for ease of reference.

As the discussion deadline approaches, we would sincerely appreciate the opportunity to engage with the reviewer and hear whether our responses have adequately addressed their concerns.

Thank you for the effort put into the detailed rebuttal and additional analyses. These address part of my concerns. However, the hybrid protocol still requires substantial human effort for fine-grained evaluation. The reported median delegation rate is 0% for interaction, handover, and action coverage. The low-cost AI ranking option does not replace these detailed assessments. Moreover, the time savings are estimated from the fraction of delegated items, rather than measured annotation time. I therefore remain concerned about the benchmark’s practicality for frequent model iteration and maintain my weak rejection recommendation.

Reviewer_5NHBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

ANIMACT evaluates whether video generation systems can preserve reference characters while carrying out a specified story. The benchmark contains 317 stories, 64 characters, and 13 props, with 1,902 clips generated by six systems. Human annotators assess 14 overall criteria and 27 questions across eight capability families. These cover identity, action execution, role assignment, physical contact, and consistency across space and time. The paper reports frequent failures in interactions and compares automated judges with human judgments, finding that agreement on aggregate results does not necessarily translate into reliable decisions on individual clips.

Critical strengths
  1. The scenarios target concrete and recognizable failures. Keeping track of two similar characters, preserving identity through occlusion, and transferring an object to the correct recipient require several aspects of a story to remain consistent. The detailed questions help distinguish these failures from general visual quality problems and could be useful when comparing model revisions.
  2. The human annotation effort is valuable. An independent set of 176 clips is annotated by three raters, with a reported Gwet’s AC1 of 0.917. The results on physical contact and interactions provide useful diagnostic information. The analysis of automated judges also shows why a judge that reproduces broad model rankings may still be unreliable for evaluating a particular clip.
Critical weaknesses
  1. The added contribution over related benchmarks is not sufficiently established. AnimationBench studies animation identity and behavior, A2Bench considers composition from multiple references, T2V-CompBench evaluates action binding, and StoryEval examines event execution. MultiRef-Compass is also a close comparison because it covers entity fidelity and temporal order. ANIMACT’s detailed human evaluation may provide additional diagnostic value, but the paper needs to show which failures its framework reveals that these existing evaluations would miss. This requires more than adding references. Applying both evaluation schemes to a shared set of clips would provide more direct evidence.
  2. Some conclusions depend on measurement choices that are not yet clear. OR3 records whether an error occurs anywhere in the cast, whereas OR1 is assessed per entity. A larger cast therefore has more chances to trigger an OR3 failure even if the error rate per character stays constant. Controlling prompt length does not remove this effect. The rankings in Section 3.1 and Table 7 also differ from Figure 7 for some model pairs. This may follow from different weighting or subsets, but the paper should explain it. Finally, the reported 6.4% success rate needs one consistent definition of the included overall and capability-specific criteria, including how partial and conditional outcomes are treated.
Questions for authors
  1. On the same clips, what failures does ANIMACT identify that the closest existing benchmarks do not? A direct comparison with MultiRef-Compass or another closely related framework would help establish the additional diagnostic value.
  2. Does the effect of cast size remain when errors are measured per character and the analysis accounts for props and generation mode? Please distinguish this from the probability that at least one character fails.
  3. Could you provide executable scoring code and the full annotations, explain the aggregation behind each reported ranking, and specify exactly how a clip qualifies for the 6.4% fully successful group?
5 comments

We thank the reviewer for their constructive and insightful feedback.

  1. Ranking Difference and success rate. The ranking differences arise from how scores are aggregated, not from the evaluation subset. Section 3.1 and Table 7 report the mean clip-level score over all 1,902 clips, weighting every scored item equally, whereas Fig. 7 takes a question- then axis-weighted mean over the 1,876 clips carrying both a human and an AI-judge label. Under the clip-level mean MiniMax-H3 leads HappyHorse 1.1 by 0.017; under axis weighting the two reverse and sit 0.015 apart. Recomputing the axis-weighted aggregation on the 1,902-clip corpus gives the same reversal (0.004), so it follows from the weighting rather than from the smaller judge-annotated subset. Seedance-2.5 is first under both aggregations. We clarified these definitions in Sections 3.1 and 4, and we refer to Fig. 7 consistently as the judge-comparison ranking.

6.4% success rate: A clip is fully successful only if it fails none of its applicable axes i.e., the applicable overall criteria (OR1–OR14) plus the single capability probe its story was written for, which is 14-15 axes per clip. An axis fails if any of its scored items is not a clean Yes, and a Partial counts as a failure. The exception is OR9, graded at the beat level, which passes when every story beat has at least half of its actions marked Yes. Conditional items that do not apply carry no score and are excluded rather than failed, for example OR13 for actions that did not occur and OR14 for stories without props. Thus, 121 of 1,902 clips (6.4%) are fully successful. We have added this definition to Section 3.4 and provide the aggregation and success-rate code in our anonymous repository (https://anonymous.4open.science/r/ANIMACT-4526/).

  1. Added contribution over related works: We address three points: (a) benchmark coverage, based on papers (Table 15); (b) ANIMACT vs. MultiRef-Compass (MRC; Zhang et al., 2026); and (c) what each detects on the same clips. We will add these comparisons to §5.1.

Table 15. Capabilities scored by related benchmarks. ✅: separately reported metric for the column's failure. ◐: named only inside a pooled score or a narrower metric, which the cell names. —: not scored. Per-edge contact checks occurrence and direction (M3.1, 3.2), touch per clip (M3.3). MSVBench's Physical Interaction Accuracy scores object reactions (such as deformation), not character contact. BC and AQ: MultiRef-Compass's Binding Correctness and Anatomical Quality checklists. Judged from the papers.

BenchmarkMulti-ref.AnimatedRef. fidelityAction bindingConfusable identityOcclusion recoveryPer-edge contactProp lifecycleEvent order
T2V-CompBench———✅——◐ Whole-prompt interaction; no touch check◐ One state change—
StoryEval———◐ Same subject across events——◐ Event interaction; no touch check◐ Same object across events—
A2-Bench✅—✅——◐ Unnatural occlusion (user-study rating, not recovery)———
AnimationBench—✅✅ Canonical IP (character) description——————
MRC✅◐ 30% cartoon refs✅◐ AC1 (BC)◐ Distinguishable identities (BC)◐ Occlusion/depth coherence (AQ)◐ AC2 direction; omissions partial (BC)◐ Attached to intended owner (BC)✅
MSVBench✅◐ Mixed real/animated◐ Character consistency; anchor unspecified✅———◐ State shift/persistence; not prop-specific—
ANIMACT✅ 1–4 refs✅✅ OR10, M2.2✅ M5.2, M6.3✅ M5, M6✅ M4.2, M4.4✅ M3.1, M3.3, M7.1, M7.3✅ M8.2—

(a) Two differences emerge. First, confusable identities, occlusion recovery, per-edge contact and prop lifecycle appear in prior benchmarks at most inside pooled scores or narrower metrics (◐), whereas ANIMACT scores each as its own question (M3 to M8). Second, ANIMACT builds these conditions into its stories, 35 to 41 per mode, while the compared benchmarks measure them only when they happen to occur. MRC is an audio-video benchmark (lip sync, speech, sound), outside ANIMACT’s video-only scope;it scores event order; ANIMACT scores each action’s occurrence and staging (OR9, OR13), not order. Mapping ANIMACT questions to these metrics, 74.8% of failing clips contain at least one failure that none of the six benchmarks scores as its own quantity, and no single benchmark targets more than 52.0% of the 10,203 failed items.

(b) Comparison with MultiRef-Compass. Its relevant items (I3, AC1, AC2, B3 and the occlusion item; Table 15) are single entries within one 1 to 5 checklist score, whereas ANIMACT asks each failure directly: does each scripted contact happen (M3.1), does the target rather than the look-alike or twin act (M5.2, M6.3), does the same character return after occlusion (M4.2), and does the prop complete its arc (M8.2).

Thank you for the additional explanations. I have read the rebuttal carefully and remain with my original recommendation and score.

AuthorsOpenReview ↗

Comparison with MultiRef-Compass.

Continuing 2: 2(c) : For a clearer head-on comparison, we ran MultiRef-Compass’s toolkit at https://github.com/zxhhh0201/MultiRef-Compass.git, and with its own Gemini judges (4 fps, temperature 0; their proposed setting) on the 600 clips of our 100 public stories (6 video models). For each capability in Table 15, we paired the MRC item that targets the same failure with the corresponding ANIMACT question and scored both against the ANIMACT human labels of the same clips: AUC for the MRC item score, and balanced accuracy for a judge answering the ANIMACT question, first the same Gemini model MRC uses, so only the questions differ, then Qwen 3.8 Max, the judge ANIMACT uses (Table 17).

With the judge held fixed, ANIMACT’s questions separate human failures where MRC’s items stay at chance: on fine details (OR10), MRC’s Detail Preservation items reach 0.51 to 0.56 against 0.66 for ANIMACT, on prop fidelity (M2.2) 0.48 to 0.49 against 0.57, and on action coverage (OR9) 0.59 against 0.75. With Qwen 3.8 Max, the judge ANIMACT uses, these rise to 0.71, 0.68 and 0.78. Per-edge contact behaves differently: MRC’s AC2 item reaches 0.66, which the same Gemini judge does not match on M3.1 (0.50) and Qwen matches (0.67). MRC is ahead on shape changes (OR3) and attachment defects (OR6/OR7), and on action binding, which rests on 3 human failures; confusable identity, with 4 failures, is similar under all three (0.62 to 0.73). These results suggest that ANIMACT adds most where a failure concerns specific referenced details or whether each scripted action occurs, while for interaction the choice of judge matters as much as the question, and MRC’s anatomy metrics remain complementary.


CapabilityANIMACT questionHuman failures / clipsMRC metric · itemMRCANIMACT, same judgeANIMACT, Qwen
Ref. fidelityOR10 Fine details157 / 600Detail Preservation · H1 (hair, headwear, scars, badges, body structure)0.510.660.71
Detail Preservation · H3 (local details: patterns, logos, jewelry, handheld items)0.56
Ref. fidelityM2.2 Prop matches reference21 / 90Detail Preservation · O1 (shape, silhouette)0.480.570.68
Detail Preservation · O2 (colors, materials, textures)0.49
Detail Preservation · O3 (distinctive details)0.49
Action bindingM5.2, M6.33 / 78Binding Correctness · AC1 (actions attributed to the intended subject)0.950.660.83
Confusable identityM5.4 Never swapped, M6.2 Distinguishing detail kept4 / 102Binding Correctness · I3 (each subject clearly distinguished)0.620.660.73
Per-edge contactM3.152 / 78Binding Correctness · AC2 (directed interactions, "A gives to B")0.660.500.67
ControlOR9 Action coverage414 / 600Visual Task Following0.590.750.78
ControlOR11 Physics & deformation90 / 600Anatomical Quality · dimension 1, Biological / Structural Consistency0.600.560.69
ControlOR3 Shape or proportions change53 / 600Anatomical Quality · dimension 10.610.550.52
ControlOR6/OR7 attachment items261 / 600Anatomical Quality · dimension 3, Physical Attachment & Spatial Plausibility0.650.600.60

Table 17. MultiRef-Compass (MRC) items vs ANIMACT questions on the same 600 public clips, scored against ANIMACT's human labels (0.5 = chance). MRC: item AUC with MRC's own Gemini judge. ANIMACT: balanced accuracy of the same Gemini judge, so only the questions differ, and of Qwen 3.8 Max, ANIMACT's judge. Only pairs scoring the same failure are shown; occlusion recovery and prop lifecycle are omitted because MRC's closest items score occlusion plausibility and prop ownership. Multi-question rows count a clip as failed if any question failed.

Reviewer_tFsrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces ANIMACT, a human-evaluated benchmark for multi-reference animated video generation, aiming to assess whether video generation models can preserve character identities and visual consistency while correctly executing story-driven actions and interactions. ANIMACT consists of 317 human-verified stories involving 64 characters and 13 props, and evaluates generated videos using 14 overall criteria and eight targeted capability probes covering reference fidelity, action execution, character interactions, identity preservation, and temporal, spatial, and physical consistency. The authors evaluate six video generation models on 1,902 generated clips. The results show that Seedance 2.5 achieves the strongest overall performance, but substantial limitations remain across all models. In particular, 40% of scripted character interactions are missing, appearance consistency deteriorates as the number of references increases, and only 6.4% of clips satisfy all applicable evaluation criteria. The paper further investigates the reliability of AI-based judges and finds that they can reasonably reproduce aggregate model rankings but struggle to match fine-grained human judgments, highlighting the continued reliance on human evaluation.

Critical strengths

None

Critical weaknesses

Heavy reliance on human evaluation limits scalability, reusability, and reproducibility. Each clip needs about 52 human judgments and takes 20–30 minutes to annotate (Sec. 2.3), and the authors conclude that AI judges cannot replace humans (Sec. 4). This means that evaluating any new model requires another large round of human annotation, which limits ANIMACT's value as a reusable benchmark. The reliability evidence is also weak. The triple-annotated subset covers only 176 clips and excludes two of the six models. Its high agreement (AC1 = 0.917) is likely inflated by ceiling effects. In addition, human–human agreement (ρc = 0.724) is barely higher than Qwen–human agreement (ρc = 0.693). The authors should provide a scalable evaluation path, such as specialized verifiers or a hybrid AI-plus-human protocol validated against the existing labels, or at least release the full human annotations.

Questions for authors

Q1. Given the substantial human annotation cost (20–30 minutes per clip), have the authors considered developing a more scalable evaluation protocol, such as specialized AI verifiers or a hybrid AI–human evaluation framework? In particular, could AI judges handle high-confidence cases while human annotators focus on ambiguous or challenging samples? How much human annotation could potentially be reduced without significantly compromising evaluation reliability?

Q2. ANIMACT identifies detailed failure modes across video generation models, but it remains unclear whether these diagnostics can effectively guide improvements in generation quality. Have the authors considered using model-specific failure patterns to design capability-aware prompt optimization strategies? For example, could explicitly emphasizing actor–action bindings or decomposing complex interactions improve performance on the corresponding capability probes? Evaluating such strategies on held-out stories would help demonstrate the practical value of ANIMACT beyond model comparison.

4 comments
AuthorsOpenReview ↗

Rebuttal response for reliability evidence, human-reliance and capability-prompting

We thank the reviewer for their feedback and address their concerns with additional analyses.

  1. Heavy reliance on human evaluation. Our results should not be interpreted as showing that AI judges cannot replace humans at all. For fine-grained and professional-level video evaluation, our analysis shows that AI judges reproduce the human ranking of the six models, overall and within several individual rubrics. Our results showed strong agreement on aggregate model rankings (ρ = 0.886; 78.3% question-level ordering agreement), but weaker agreement on individual clips. This distinction was stated in the Abstract (L028–031), Introduction (L102–106), and Section 4 (L410–420), but was less clear in the Section 4 Takeaway (L422–423), which we have revised.

We also conducted a split-half analysis over 314 stories (400 random splits). Human rankings across disjoint halves achieve median ρ = 0.943 (95% interval: 0.771–1.000), compared with ρ = 0.886 (0.714–0.943) for AI-judge rankings against human rankings on disjoint halves (Appendix C). Thus, a new model can be compared against the existing six using AI-judge scores for both, without new human annotation (Appendix C.1). Human labels remain necessary for reliable clip-level diagnosis.

  1. Reliability Evidence. We clarify that the reported human–human ρc = 0.724 and Qwen–human ρc = 0.693 are not directly comparable: the former measures question-centred agreement over individual annotation items (Appendix A.10), whereas the latter measures agreement across question–model aggregate scores (Table 1). We will distinguish these as item-level and model-level question-centred correlations.

To strengthen reliability evidence, we expanded triple annotation to cover all six models, yielding 270 clips across 76 stories (46,177 rater-items), with 40–49 clips per model and 30 stories covering all six. Of these, 259 clips across 71 stories qualify for question-centred correlation analysis, as some question-level items lack triple annotations. Story-level resampling (200 draws per size) shows that the 95% interval for the human–human versus AI–human agreement gap excludes zero from 173 clips onward, reaching 0.062 at 259 clips (Table 14). This supports higher human agreement at the same evaluation granularity. We agree the skew inflates AC1, so A.7 now leads with question-centred ρc\rho_{c}, which a modal-label-only rater scores near zero on. Human annotations, rubrics, agrregations,code, and reference assets are available in our (https://anonymous.4open.science/r/ANIMACT-4526/).

new Table 14. Human–human vs. AI–human agreement (ρc\rho_{c}) across subset sizes. Medians and 95% intervals from 200 story-level resamples; gap denotes paired difference. Final row uses all 259 clips.

ClipsHuman–Human ρc\rho_{c}AI–Human ρc\rho_{c}Gap
520.656 [0.548, 0.726]0.638 [0.524, 0.728]0.011 [-0.063, 0.112]
1020.677 [0.624, 0.728]0.654 [0.590, 0.705]0.021 [-0.024, 0.087]
1420.696 [0.656, 0.728]0.668 [0.630, 0.704]0.029 [-0.005, 0.056]
1730.710 [0.671, 0.734]0.669 [0.642, 0.696]0.030 [0.007, 0.056]
2020.699 [0.681, 0.719]0.658 [0.642, 0.680]0.041 [0.021, 0.060]
2410.703 [0.694, 0.717]0.645 [0.633, 0.658]0.059 [0.049, 0.066]
2590.6990.6380.062
  1. Q:2 Capability aware prompting: We designed capability-aware prompting rules from ANIMACT's diagnosed failures: i) actor binding explicitly identifies and consistently names referenced characters; ii) action decomposition specifies each action, its participants, and visible outcome within a timed window; and iii) each story receives guidance for its targeted capability probe (M1–M8). Rules were developed on development stories only. We evaluated one rewritten prompt per story, shared across all six models, on 50 held-out stories (300 new generated clips), using the same human evaluation rubric and comparing against the original story–model pairs.

Table 12 (compact). Change in human scores with capability-aware prompting (ppt; 95% story-bootstrap confidence intervals).

Evaluated capabilityChange (95% CI)
i) Actor binding (OR1, OR4)+2.4 [+1.2, +3.6]
ii) Action decomposition (OR9, OR13)+1.9 [−3.4, +5.5]
iii) Story-specific probe (M1–M8)+4.5 [+2.1, +7.0]
Untargeted rubrics−0.1 [−1.9, +1.7]
Improvements concentrate on diagnosed weaknesses. Grok Imagine, weakest on cast/prop presence in the original benchmark, gains +9.9 points in actor binding, whereas the strongest model, Seedance 2.5, shows little change. Action decomposition improves +6.1 points [+3.7, +8.5] on 26 stories with 9–12 actions, versus −2.6 on 24 shorter stories. These results demonstrate that ANIMACT diagnostics can inform targeted prompt improvements, although gains vary by model and story complexity. Full results appear in Appendix Tables 12–13.
AuthorsOpenReview ↗

(added full table versions)

Table 12 (full). ANIMACT human scores (%) without and with capability-aware prompting, by capability.

Capability probedANIMACT questionsFull benchmark (317 stories)WithoutWithΔ points (95% CI)Stories better / same / worse
Actor bindingOR1, OR496.296.198.4+2.4 (+1.2 to +3.6)23 / 22 / 5
Action decompositionOR9, OR1382.486.087.9+1.9 (−3.4 to +5.5)33 / 2 / 15
Story's own probeM1 to M889.390.194.6+4.5 (+2.1 to +7.0)31 / 10 / 9
Appearance (indirect)OR3, OR1087.891.192.8+1.8 (−1.0 to +4.7)29 / 1 / 20
Not targeted by any prompt rule
Motion smoothnessOR287.091.791.0−0.7 (−5.2 to +3.8)14 / 22 / 14
Physics & deformationOR1188.790.593.3+2.8 (−0.8 to +6.5)20 / 21 / 9
PacingOR1295.897.394.7−2.7 (−5.7 to +0.2)9 / 27 / 14
Setting stabilityOR598.998.798.3−0.3 (−2.7 to +1.7)4 / 42 / 4
Placement & spatial senseOR686.793.192.6−0.6 (−3.1 to +1.9)19 / 8 / 23
Within-shot stabilityOR784.693.193.3+0.2 (−2.3 to +2.9)17 / 18 / 15
Cross-cut continuityOR889.192.895.8+3.0 (−0.2 to +6.2)16 / 29 / 5
Prop appearance stabilityOR1491.995.594.7−0.8 (−3.3 to +2.0)5 / 4 / 8
Average of above untargeted88.493.793.6−0.1 (−1.9 to +1.7)23 / 2 / 25
All questionsAll of the above88.692.493.5+1.1 (−0.5 to +2.7)31 / 0 / 19

Each clip's score averages the row's questions; each row averages clips with equal story weighting. Full-benchmark scores use all 317 stories, while Without and With use the same 50 held-out stories. Changes are computed between Without and With. Confidence intervals use 2,000 story-level bootstrap draws. Bold changes have intervals excluding zero. Better / same / worse counts refer to story-level changes. Untargeted rubrics are grouped separately. OR14 applies only to the 17 held-out stories containing props.



Table 13. Change in ANIMACT human scores (percentage points) with capability-aware prompting, per model.

Capability probedQuestionsGrok ImagineKling O3MiniMax-H3Wan 2.7HappyHorse 1.1Seedance 2.5All
Actor bindingOR1, OR4+9.9 (+5.0, +15.2)+2.0 (−0.8, +5.3)+1.9 (−0.5, +4.6)−0.5 (−1.5, 0.0)+1.3 (−1.0, +4.8)−0.5 (−1.5, 0.0)+2.4 (+1.2, +3.6)
Action decompositionOR9, OR13+3.5 (−2.4, +9.0)−0.1 (−7.2, +6.6)+4.6 (−1.8, +10.9)+3.3 (−3.2, +8.6)+0.3 (−5.1, +4.2)−0.3 (−6.0, +4.3)+1.9 (−3.4, +5.5)
Story's own probeM1 to M8+8.8 (+2.8, +14.9)+4.4 (−0.7, +10.1)+3.7 (−0.2, +7.9)+3.9 (−0.3, +8.5)+5.0 (+1.1, +8.7)+1.5 (−0.6, +3.5)+4.5 (+2.1, +7.0)
Appearance (indirect)OR3, OR10+7.4 (+2.1, +13.2)−0.1 (−4.1, +3.7)−0.3 (−4.5, +4.3)+1.2 (−3.0, +5.6)+3.5 (−0.6, +7.8)−1.0 (−3.7, +1.7)+1.8 (−1.0, +4.7)
Not targeted by any prompt rule
Motion qualityOR2, OR11, OR12+0.3 (−5.3, +6.3)+1.3 (−3.3, +6.0)+4.0 (+1.7, +6.3)−3.7 (−9.3, +1.7)−3.0 (−8.3, +2.0)0.0 (−3.0, +2.7)−0.2 (−2.9, +2.5)
Scene & placementOR5, OR6−1.1 (−5.7, +3.7)−1.4 (−4.6, +1.7)−0.3 (−3.4, +2.9)−3.7 (−7.7, +0.3)+2.6 (−0.6, +5.7)+0.9 (−2.0, +3.7)−0.5 (−2.7, +1.6)
Stability & continuityOR7, OR8, OR14+0.5 (−4.0, +5.2)+1.8 (−2.2, +5.6)−1.5 (−4.2, +1.6)−0.6 (−4.1, +3.0)+2.3 (−2.3, +6.8)+0.1 (−2.7, +2.7)+0.4 (−1.7, +2.4)
All questionsAll of the above+3.3 (+0.4, +6.4)+0.8 (−1.5, +3.1)+0.7 (−1.1, +2.9)−0.7 (−3.0, +1.6)+2.2 (−0.2, +4.7)+0.2 (−1.3, +1.7)+1.1 (−0.5, +2.7)
Decomposition: 9–12 actions (26 stories)OR9, OR13+9.1 (+3.7, +15.2)+4.3 (−3.0, +12.8)+10.1 (+6.0, +14.7)+6.5 (+1.7, +11.1)+3.1 (+0.3, +6.2)+3.3 (−0.6, +7.2)+6.1 (+3.7, +8.5)
Decomposition: 1–5 actions (24 stories)OR9, OR13−2.6 (−13.8, +5.1)−4.8 (−16.7, +4.2)−1.3 (−13.0, +10.2)−0.1 (−12.3, +9.3)−2.8 (−13.2, +4.1)−4.1 (−14.5, +4.2)−2.6 (−12.6, +3.8)

Changes are computed on the same 50 held-out stories. Each cell reports the percentage-point change and 95% story-bootstrap interval (2,000 draws). Bold indicates an interval excluding zero. The All column averages six models. The final two rows split stories by scripted action count. Untargeted rubrics are grouped separately.



Revised Section 4 Finding: “AI judges recover model rankings for comparison but not the clip-level judgments needed to diagnose failures.”

Thanks for sharing these additional analyses so close to the end of the rebuttal period. However, my main concerns remain:

  1. AI judges cannot reliably identify fine-grained failures, so diagnosing model weaknesses still requires human review, making the benchmark difficult for the community to directly reuse.

  2. The substantial annotation burden raises concerns about annotator fatigue and consistency.

Overall, I remain unconvinced that the work offers a sufficiently strong methodological contribution for ICLR. In my view, it may be a better fit for venues focusing on human-centered AI. I therefore maintain my original score.

AuthorsOpenReview ↗

Follow-up to the concern.

We thank the reviewer for a quick follow-up. Our first response addressed judge-based model ranking and the hybrid protocol. Two new analyses address whether ANIMACT contributes beyond human annotation.

ANIMACT scores failures that related benchmarks do not (Table 15). We reviewed six related benchmarks and draw a comparison: None scores confusable identities, occlusion recovery, per-edge contact or prop lifecycle as its own question. The closest, MultiRef-Compass, names some of them in its Binding Correctness and Anatomical Quality checklists, but each checklist returns a single 1 to 5 score. None of the six builds these conditions into its prompts; ANIMACT has 35 to 41 stories per mode.

ANIMACT’s fine-grained questions improve failure detection over MultiRef-Compass on shared clips (Table 17). We ran MultiRef-Compass’s released toolkit with its Gemini judges on our 600 public clips. We matched MRC items to ANIMACT questions for the same failure, evaluating both against human labels. With the Gemini judge held fixed, ANIMACT better detects failures that MRC struggles to discriminate, including fine-detail loss (0.66 vs. 0.51/0.56) and incomplete action coverage (0.75 vs. 0.59). With Qwen, ANIMACT also performs comparably on per-edge contact (0.67 vs. 0.66). MRC performs better on action binding and some structural defects, reflecting its targeted metrics. While MRC scores are AUC and ANIMACT scores are balanced accuracy, these results suggest that fine-grained, failure-specific questions can improve automatic evaluation beyond human annotation alone.

The released labels address the reviewer’s concern directly. Testing fine-grained AI judges requires item-level human labels. We release 34,548 human judgments on these 600 clips (18,711 on 300 now), each tied to a named character, scripted action or contact edge. Future judges and metrics can be tested against them.


Table 15. Capabilities scored by related benchmarks. ✅: separately scored; ◐: partial or pooled coverage (specified in cell); —: not scored. BC/AQ: MultiRef-Compass Binding Correctness / Anatomical Quality. Contact refers to occurrence, direction, and touch, not generic physical interactions.

BenchmarkMulti-refAnimatedRef. fidelityAction bindingConfusable identityOcclusion recoveryPer-edge contactProp lifecycleEvent order
T2V-CompBench———✅——◐ Prompt-level interaction; no touch◐ Single state change—
StoryEval———◐ Subject continuity——◐ Event interaction; no touch◐ Object continuity—
A2-Bench✅—✅——◐ Occlusion artifacts; not recovery———
AnimationBench—✅✅ Character description——————
MRC✅◐ 30% cartoon✅◐ BC: AC1◐ BC: identity distinction◐ AQ: depth/occlusion◐ BC: AC2 direction; omissions partial◐ BC: object ownership✅
MSVBench✅◐ Mixed styles◐ Consistency; anchor unspecified✅———◐ State persistence; not prop-specific—
ANIMACT✅ 1–4 refs✅✅ OR10, M2.2✅ M5.2, M6.3✅ M5, M6✅ M4.2, M4.4✅ M3.1, M3.3, M7.1, M7.3✅ M8.2—

Table 17. MRC vs. ANIMACT on 600 shared clips. Scores use ANIMACT human labels (0.5 = chance). MRC: item AUC using its Gemini judge; ANIMACT: balanced accuracy with Gemini or Qwen 3.8 Max. Only matched failures shown. Multi-question rows fail if any question fails. MRC occlusion plausibility and prop ownership differ from ANIMACT's occlusion recovery and prop lifecycle.

CapabilityANIMACT questionFail / NMRC item (metric)MRCSame judgeQwen
Ref. fidelityOR10 Fine details157/600H1: hair, headwear, scars, badges, body structure (0.51); H3: patterns, logos, jewelry, handheld items (0.56) [Detail Preservation]0.51 / 0.560.660.71
Ref. fidelityM2.2 Prop matches ref.21/90O1: shape, silhouette (0.48); O2: color, material, texture (0.49); O3: distinctive details (0.49) [Detail Preservation]0.48 / 0.49 / 0.490.570.68
Action bindingM5.2, M6.33/78AC1: action assigned to intended subject [Binding Correctness]0.950.660.83
Confusable identityM5.4 No swaps; M6.2 Distinguishing detail4/102I3: subjects clearly distinguished [Binding Correctness]0.620.660.73
Per-edge contactM3.152/78AC2: directed interaction (A gives to B) [Binding Correctness]0.660.500.67
ControlOR9 Action coverage414/600Visual Task Following0.590.750.78
ControlOR11 Physics/deformation90/600AQ1: biological/structural consistency0.600.560.69
ControlOR3 Shape/proportions53/600AQ1: biological/structural consistency0.610.550.52
ControlOR6/OR7 Attachment261/600AQ3: physical attachment/spatial plausibility0.650.600.60

Comments on the paper

2 comments
AuthorsOpenReview ↗

Shared Response on Scalable Evaluation.

Shared Response-1

-- Scalable Evaluation Path:

We thank the reviewers for this suggestion. We built the hybrid protocol and calibrated it on our human labels: 4.0% of clips need no human at all, 34.6% of items leave the human queue, and 89.9% of recorded defects still reach a human. We explain the protocol in full here.

1 · How delegation is decided. A panel of three AI judges (Qwen 3.8 Max, Gemini 3.1 Pro, Muse Spark 1.2) scores every item first. Two gates then run in order, each fitted on 67% of stories and measured on the held-out third over 20 random story splits; all figures are medians. We use a panel not for better labels but for the agreement count a single judge cannot give: under the same rule Qwen delegates 7.1% of items, against 31.7%.

Item delegation alone retires no clips; ten axes have no delegable group and every clip carries one, so a human opens every video and reads 37 of its 54 items. Stage 1 exists for that.

Stage 1 — retire a clip. For each question we take its modal human answer on the fit stories; a clip carries that question’s signal when, on some item of it, all three judges depart from that answer. A signal is trusted only when, on the fit stories: at least 20 clips carry it; the lower 95% bound on the chance such a clip fails at least one axis clears 0.90; and it retires no clip that passes every axis.

At most half the clips carrying a trusted signal are retired unseen; the rest stay in the queue as an audit sample, which keeps every axis above the Stage 2 defect floor. Half is the largest share that holds: retiring every confident clip starves the three qualifying axes and breaches the floor in all 20 splits, and still breaks in 15 splits at 0.7 and 6 at 0.6.

Stage 2 — delegate an item. On every remaining clip we group items by question, agreement count and agreed label. A group is answered by the panel only when: the lower 95% bound on its agreement with the human label clears 0.97; and every rubric axis still sends at least half its recorded defects to a human.

Table 16 sweeps each stage in turn: the upper block varies Stage 1 with Stage 2 at 0.97, the lower varies Stage 2 with Stage 1 off. 0.97 is the lowest Stage 2 threshold no split breaches; above it coverage collapses and the saving turns negative.

2 · What it saves, and what it costs. 34.6% of items leave the human queue, against 31.7% for Stage 2 alone: 219–329 annotator-hours, $3,961–6,152, or 31–32% of a human-only $12,680–19,020, after the $421 the panel costs. This assumes annotation time scales with item count, and a human still opens 96% of clips, so realised savings are lower. The price is five points of defect visibility, 89.9% against 94.9% without Stage 1. The scores in the paper are human throughout and use neither stage.

We also observe that fine details and cast count carry 78% of what Stage 2 delegates, which agrees with Table 11 showing the judge weakest on fine details: those items are 95.9% clean, so answering the usual label is 98.1% accurate while still missing 41% of real defects, and the per-axis floor is what keeps 59% of them with a human. The protocol buys throughput on near-saturated axes, not detection; Fig-Usecase in our repo plots delegable share per job, 49% to 4%.

Release. Our repository (https://anonymous.4open.science/r/ANIMACT-4526/) includes annotations, gating code, and threshold-sweep tools; all 100 stories with full annotations will be released at camera-ready.


Table 16. Medians over 20 held-out splits (628 clips each). Upper block: Stage 2 at 0.97, at most half of each signal's clips retired. Lower: Stage 1 off. Costs in USD: net saving is annotator time at 20/hour over 1,902 clips less 421 judge spend, against a human-only 12,680–19,020. In the upper block below 0.90, one split leaves cast count at 0.494 recall against the 0.50 floor; we take the conservative side.

ThresholdClips no human opensItems answered by AISplits where an axis kept <50% of its defectsNet savingDefects still reaching a human
Stage 1 swept, Stage 2 at 0.97
0.7025/628 = 4.0%34.8%1/20$3,989–6,19489.7%
0.8025/628 = 4.0%34.8%1/20$3,989–6,19489.7%
0.9025/628 = 4.0%34.6%0/20$3,961–6,15289.9%
0.9220/628 = 3.3%34.3%0/20$3,930–6,10690.4%
0.949/628 = 1.4%32.5%0/20$3,705–5,76892.3%
0.960/628 = 0.0%31.7%0/20$3,597–5,60694.9%
0.98 (Stage 1 off)0/628 = 0.0%31.7%0/20$3,594–5,60294.9%
Stage 2 swept, Stage 1 off
0.90047.8%10/20$5,643–8,67588.8%
0.95036.6%6/20$4,213–6,53093.9%
0.96035.5%5/20$4,078–6,32894.1%
0.97031.7%0/20$3,594–5,60294.9%
0.9801.9%0/20net loss $63–18399.9%

AuthorsOpenReview ↗

Summary of revisions.

Summary of revisions. We thank the reviewers for their feedback. In the revised manuscript, all changes to the main text are highlighted in blue. In the appendix, revisions to the existing sections are highlighted in blue, and newly added subsections, tables and figures are marked “added” in the summary below. All results reported in the submission are unchanged; the new analyses are additions.

#WhereWhat changed
1§3.1, §4, Table 1, Fig. 7 (revised)Separated model-level from clip-level judge agreement, revised the Section 4 takeaway, and clarified how the §3.1 and Fig. 7 rankings are aggregated.
2§3.4 (revised), Appendix B (revised)Clarified the clip-level success rate (6.4%) and added per-entity analyses of presence (OR1) and appearance (OR3).
3Appendix A.7 (revised), A.8 (added), Table 14Extended triple annotation to all six models (259 clips) and distinguished item-level from model-level ρc\rho_c.
4Appendix C.1 (revised)Included a split-half reference point for the model-ranking correlation.
5Appendix C.2 (added), Tables 16, 18, Fig. 14Added a two-stage hybrid AI–human annotation protocol, its details and per-job guidance.
6Appendix E.1 (added), Tables 12–13Evaluated capability-aware prompting on 50 held-out stories (300 human-rated clips).
7§5.1 (revised), Table 15; Appendix E.2 (added), Table 17Compared ANIMACT with related benchmarks, including MultiRef-Compass on 600 shared clips.

We have also made the annotations, aggregation code, and gating code available in our anonymous repository, linked in Appendix A, to support reproducibility and further inspection.

We would be grateful for the opportunity to provide any further clarifications or additional information that may be helpful in addressing your remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.