- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MotionGraphicsBench, a benchmark of 100 tasks in which models animate a provided static SVG according to a natural-language motion-editing instruction. Each task is paired with a deterministic verification program that checks geometric, appearance, and temporal requirements. The verification programs are constructed through an iterative pipeline involving a program writer, a test-case synthesizer, and human-labeled examples.
Critical strengths
- The benchmark addresses a useful evaluation problem with interpretable feedback. The task-specific reports could support both model evaluation and debugging.
- The paper provides substantial evaluation effort and an explicitly held-out validation set. The paper covers many model configurations, and the separation between verifier-construction examples and held-out animations is a positive design choice.
Critical weaknesses
- The contribution beyond MoVer[1] is insufficiently isolated, and the claimed scope is broader than the evaluated task.
- The validation does not yet establish that the verifiers are sufficiently robust to support the reported rankings and repeated-sampling conclusions.
[1] Mover: Motion verification for motion graphics animations[J]. ACM Transactions on Graphics (TOG), 2025, 44(4): 1-17.
Questions for authors
- Contribution relative to MoVer. Which benchmark requirements can be expressed using MoVer’s existing predicates, and which require additional capabilities? Could you report coverage across the 100 tasks and compare the two approaches on a representative subset of jointly supported tasks, using the same independently labeled outputs? Please distinguish unsupported requirements from errors in generating verification programs.
- Verifier-construction ablations. How much do human-labeled construction examples, synthesized tests, and iterative revision each contribute? A practical experiment would compare the full pipeline against versions removing each component on approximately 30 stratified tasks, with the same base model, tools, and construction-budget ceiling. Agreement, false-approval and false-rejection rates, construction success, and actual cost would be informative.
- Independent reliability and ranking validation. Can you provide additional validation using annotators who did not participate in verifier construction, preserving their initial independent decisions and withholding the source animation during the primary assessment?
- Dataset provenance and overlap. A preliminary comparison suggests that some publicly released source animations have counterparts in MMLottie-2M and on upstream platforms such as IconScout. These observations do not establish how the assets were acquired: shared upstream sources and dataset-to-dataset reuse remain possible explanations. The original LottieAnimation-660K records were not directly examined in this comparison.
5 comments
Response to Question 2: Verifier construction ablations and iterative revisions
Thank you for suggesting these ablations. We have conducted these ablations with all 100 benchmark instances rather than a subset of 30. We use the same settings as our paper's construction run, except for the components disabled in each ablation. We evaluate the verification programs on the same balanced set of 180 human-labeled animations.
Human-labeled examples and synthesized test cases. We compare our full construction procedure with two conditions: without human-labeled examples, and without either human-labeled examples or synthesized test cases. To account for variation in LLM outputs, we repeat each condition three times and report the mean results (the Full condition includes our paper's construction run).
| Construction condition | Mean agreement with humans | Mean approval recall | Mean rejection recall |
|---|---|---|---|
| Without human-labeled examples or synthesized test cases | 78.15% | 63.70% | 92.59% |
| Without human-labeled examples | 82.41% | 73.33% | 91.48% |
| Full construction | 93.33% | 90.00% | 96.67% |
Without human-labeled examples or synthesized test cases, our verification program writer still has access to the source Lottie animation as an example that matches the motion edit prompt. The resulting programs tend to be overly strict, with 63.70% approval recall compared with 92.59% rejection recall. Adding synthesized test cases alone substantially improves approval recall to 73.33%, a 9.63-percentage-point gain. Further adding human-labeled examples improves both measures, reaching 90.00% approval recall and 96.67% rejection recall. Together, these results show that our synthesized test cases improve approval recall, while human-labeled examples improve both approval and rejection recall.
Iterative revisions. To examine the effect of iterative revisions, we evaluate the initial verification programs and each revised version from the Full condition on the same set of 180 human-labeled animations. Different benchmark instances can require different numbers of iterations. For instances that finish earlier, their final results remain unchanged in the later rows of the table. We report the mean results across three repetitions of this condition.
| Program version | Mean agreement with humans |
|---|---|
| Iteration 1 | 61.79% |
| Iteration 2 | 88.32% |
| Iteration 3 | 91.28% |
| Iteration 4 | 92.02% |
| Iterations 5-7 | 93.32% |
| Final verification programs | 93.33% |
We observe that mean agreement rises from 61.79% to 93.33%, a 31.54-percentage-point improvement. The largest increase follows the first revision, and agreement stabilizes by iteration 5. These results show that iterative revisions help our verification programs better match human judgments.
In our revised paper, Section 5.2 summarizes these construction studies, and Appendix C.4 reports the repeated-run and iteration-by-iteration results.
Response to Question 3: Independent annotation and source-animation access
Clarification of the annotation procedure. We would like to first clarify our annotation process, as noted in our common response. Two annotators first independently judge whether each generated animation satisfies its motion edit prompt. All annotators need to first finish judging their assigned animations before entering the disagreement resolution phase. We recovered their initial independent judgments from preserved annotation edit histories and report initial raw inter-annotator agreement of 88.88%, with Krippendorff's and Fleiss' . Please see "Annotation procedure and initial inter-annotator agreement" in our common response above for more details.
Independent annotation and validation. Each annotator could see only their own annotations, except only when they are resolving disagreements, and we always hide from the annotators which model generated each animation. We collected these annotations before constructing any verification programs, and the annotators did not inspect, write, or revise the programs. They did not know which animations would be used for verification program construction and which would be held out. Our 180 held-out animations and their labels were not used in verification program construction.
Role of the source animation. Before the main annotation collection, we conducted a pilot annotation round with 15 benchmark instances and seven generated animations per instance, totaling 105 animations. We used the pilot only for annotator training and calibration: its annotation results were excluded from our final results, and none of its generated animations was reused in the main annotation collection. During the pilot, annotators reported that access to the source Lottie animation was necessary to help them interpret the motion edit prompts. To prevent annotators from judging against the source Lottie animation rather than the motion edit prompt, our annotation interface always hides the source Lottie animation initially for a new animation, and our annotation guide explicitly states:
The reference animation is one valid answer, not the only answer; watch it only when you cannot picture what the prompt means, not to compare against. Just because the generated animation does not look exactly like the reference animation does not make the generated one wrong.
The guide also requires annotators to identify the prompt requirement underlying a rejection:
When it is a non-match, select exactly the parts of the prompt that are not satisfied and explain why.
Our annotation protocol therefore asks the annotators to focus on judging if an animation satisfied its motion edit prompt rather than reproducing the source Lottie animation; Appendix C.1 documents this procedure and the annotation interface.
Response to Weakness 1 and Question 1: Requirement coverage beyond MoVer and task scope
Task scope. We would like to clarify that our claimed scope matches our evaluated task: animating an input static SVG according to a motion edit prompt, as defined in our abstract and task formulation.
Contribution relative to MoVer. We compared all of our 100 motion edit prompts with MoVer's existing predicates and identified that MoVer cannot fully express the requirements of 97 instances (the other three: Sailing Yacht, Skull, and Male Photographer Editing Photos). MoVer lacks the ability to check easing, changes in appearance and visibility, progressive reveal, and changes to an object's shape. Its color and shape predicates do not verify changes over time (Ma and Agrawala, 2025). MoVer also lacks the ability to check the shape of a motion path, such as a circle or spiral (Ma et al., 2026). MoVer could therefore produce false positives on these instances by vacuously approving animations that violate requirements it cannot check. A comparison restricted to these three instances would provide limited statistical evidence and would not assess the additional capabilities required by the other 97 instances.
References