acceptodds
Withdrawn from ICLR 2027

MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models can generate motion graphics from static images and natural language requests, but it remains difficult to evaluate how well models can follow requests that combine spatial, appearance, and temporal requirements. We introduce MotionGraphicsBench, an expert-curated benchmark of 100 instances for animating an input static SVG with a motion edit prompt. By supplying all required visual elements, our editing task reflects a designer-like workflow and isolates motion generation from graphic generation. Derived from real-world Lottie motion designs, our benchmark contains diverse combinations of spatio-temporal animation behaviors. Each benchmark instance pairs its model input with a deterministic, human-readable verification program that accepts different valid implementations and reports which prompt requirements pass or fail, rather than comparing against a reference animation or using a VLM judge. We construct these programs with a multi-agent pipeline using human-labeled animations and synthesized test cases. On a balanced test set of 180 animations, our programs achieve 93.3% raw agreement with human judgments, compared with 83.3% for a GPT-6 Astra VLM baseline, and outperform the baseline in both approval and rejection recall. Across 47 configurations from 32 LLMs and 12 providers, pass@1 ranges from 6.0% to 67.7%. Across 15 models evaluated at their highest available reasoning effort with 32 attempts per instance, additional attempts improve every model, but pass@32 still ranges from 38.0% to 95.0%. Our analysis of verification programs reveal that generated animations often fail to match reference poses, follow requested motion trajectories, or coordinate related events. Our benchmark instances can be viewed at https://motion-graphics-bench.github.io/.

Primary area:
datasets and benchmarks
Keywords:
motion graphics, animation, benchmark, programmatic verification, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GVXCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MotionGraphicsBench, a benchmark of 100 tasks in which models animate a provided static SVG according to a natural-language motion-editing instruction. Each task is paired with a deterministic verification program that checks geometric, appearance, and temporal requirements. The verification programs are constructed through an iterative pipeline involving a program writer, a test-case synthesizer, and human-labeled examples.

Critical strengths
  1. The benchmark addresses a useful evaluation problem with interpretable feedback. The task-specific reports could support both model evaluation and debugging.
  2. The paper provides substantial evaluation effort and an explicitly held-out validation set. The paper covers many model configurations, and the separation between verifier-construction examples and held-out animations is a positive design choice.
Critical weaknesses
  1. The contribution beyond MoVer[1] is insufficiently isolated, and the claimed scope is broader than the evaluated task.
  2. The validation does not yet establish that the verifiers are sufficiently robust to support the reported rankings and repeated-sampling conclusions.

[1] Mover: Motion verification for motion graphics animations[J]. ACM Transactions on Graphics (TOG), 2025, 44(4): 1-17.

Questions for authors
  1. Contribution relative to MoVer. Which benchmark requirements can be expressed using MoVer’s existing predicates, and which require additional capabilities? Could you report coverage across the 100 tasks and compare the two approaches on a representative subset of jointly supported tasks, using the same independently labeled outputs? Please distinguish unsupported requirements from errors in generating verification programs.
  2. Verifier-construction ablations. How much do human-labeled construction examples, synthesized tests, and iterative revision each contribute? A practical experiment would compare the full pipeline against versions removing each component on approximately 30 stratified tasks, with the same base model, tools, and construction-budget ceiling. Agreement, false-approval and false-rejection rates, construction success, and actual cost would be informative.
  3. Independent reliability and ranking validation. Can you provide additional validation using annotators who did not participate in verifier construction, preserving their initial independent decisions and withholding the source animation during the primary assessment?
  4. Dataset provenance and overlap. A preliminary comparison suggests that some publicly released source animations have counterparts in MMLottie-2M and on upstream platforms such as IconScout. These observations do not establish how the assets were acquired: shared upstream sources and dataset-to-dataset reuse remain possible explanations. The original LottieAnimation-660K records were not directly examined in this comparison.
5 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1: Requirement coverage beyond MoVer and task scope

Task scope. We would like to clarify that our claimed scope matches our evaluated task: animating an input static SVG according to a motion edit prompt, as defined in our abstract and task formulation.

Contribution relative to MoVer. We compared all of our 100 motion edit prompts with MoVer's existing predicates and identified that MoVer cannot fully express the requirements of 97 instances (the other three: Sailing Yacht, Skull, and Male Photographer Editing Photos). MoVer lacks the ability to check easing, changes in appearance and visibility, progressive reveal, and changes to an object's shape. Its color and shape predicates do not verify changes over time (Ma and Agrawala, 2025). MoVer also lacks the ability to check the shape of a motion path, such as a circle or spiral (Ma et al., 2026). MoVer could therefore produce false positives on these instances by vacuously approving animations that violate requirements it cannot check. A comparison restricted to these three instances would provide limited statistical evidence and would not assess the additional capabilities required by the other 97 instances.

References

AuthorsOpenReview ↗

Response to Question 2: Verifier construction ablations and iterative revisions

Thank you for suggesting these ablations. We have conducted these ablations with all 100 benchmark instances rather than a subset of 30. We use the same settings as our paper's construction run, except for the components disabled in each ablation. We evaluate the verification programs on the same balanced set of 180 human-labeled animations.

Human-labeled examples and synthesized test cases. We compare our full construction procedure with two conditions: without human-labeled examples, and without either human-labeled examples or synthesized test cases. To account for variation in LLM outputs, we repeat each condition three times and report the mean results (the Full condition includes our paper's construction run).

Construction conditionMean agreement with humansMean approval recallMean rejection recall
Without human-labeled examples or synthesized test cases78.15%63.70%92.59%
Without human-labeled examples82.41%73.33%91.48%
Full construction93.33%90.00%96.67%

Without human-labeled examples or synthesized test cases, our verification program writer still has access to the source Lottie animation as an example that matches the motion edit prompt. The resulting programs tend to be overly strict, with 63.70% approval recall compared with 92.59% rejection recall. Adding synthesized test cases alone substantially improves approval recall to 73.33%, a 9.63-percentage-point gain. Further adding human-labeled examples improves both measures, reaching 90.00% approval recall and 96.67% rejection recall. Together, these results show that our synthesized test cases improve approval recall, while human-labeled examples improve both approval and rejection recall.

Iterative revisions. To examine the effect of iterative revisions, we evaluate the initial verification programs and each revised version from the Full condition on the same set of 180 human-labeled animations. Different benchmark instances can require different numbers of iterations. For instances that finish earlier, their final results remain unchanged in the later rows of the table. We report the mean results across three repetitions of this condition.

Program versionMean agreement with humans
Iteration 161.79%
Iteration 288.32%
Iteration 391.28%
Iteration 492.02%
Iterations 5-793.32%
Final verification programs93.33%

We observe that mean agreement rises from 61.79% to 93.33%, a 31.54-percentage-point improvement. The largest increase follows the first revision, and agreement stabilizes by iteration 5. These results show that iterative revisions help our verification programs better match human judgments.

In our revised paper, Section 5.2 summarizes these construction studies, and Appendix C.4 reports the repeated-run and iteration-by-iteration results.

AuthorsOpenReview ↗

Response to Question 3: Independent annotation and source-animation access

Clarification of the annotation procedure. We would like to first clarify our annotation process, as noted in our common response. Two annotators first independently judge whether each generated animation satisfies its motion edit prompt. All annotators need to first finish judging their assigned animations before entering the disagreement resolution phase. We recovered their initial independent judgments from preserved annotation edit histories and report initial raw inter-annotator agreement of 88.88%, with Krippendorff's α=0.718\alpha = 0.718 and Fleiss' κ=0.718\kappa = 0.718. Please see "Annotation procedure and initial inter-annotator agreement" in our common response above for more details.

Independent annotation and validation. Each annotator could see only their own annotations, except only when they are resolving disagreements, and we always hide from the annotators which model generated each animation. We collected these annotations before constructing any verification programs, and the annotators did not inspect, write, or revise the programs. They did not know which animations would be used for verification program construction and which would be held out. Our 180 held-out animations and their labels were not used in verification program construction.

Role of the source animation. Before the main annotation collection, we conducted a pilot annotation round with 15 benchmark instances and seven generated animations per instance, totaling 105 animations. We used the pilot only for annotator training and calibration: its annotation results were excluded from our final results, and none of its generated animations was reused in the main annotation collection. During the pilot, annotators reported that access to the source Lottie animation was necessary to help them interpret the motion edit prompts. To prevent annotators from judging against the source Lottie animation rather than the motion edit prompt, our annotation interface always hides the source Lottie animation initially for a new animation, and our annotation guide explicitly states:

The reference animation is one valid answer, not the only answer; watch it only when you cannot picture what the prompt means, not to compare against. Just because the generated animation does not look exactly like the reference animation does not make the generated one wrong.

The guide also requires annotators to identify the prompt requirement underlying a rejection:

When it is a non-match, select exactly the parts of the prompt that are not satisfied and explain why.

Our annotation protocol therefore asks the annotators to focus on judging if an animation satisfied its motion edit prompt rather than reproducing the source Lottie animation; Appendix C.1 documents this procedure and the annotation interface.

Reviewer_TbaeOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MotionGraphicsBench, a benchmark for evaluating motion graphics generation. It uses deterministic verification programs to assess whether generated animations satisfy requirements of the input text prompt. The authors evaluate 32 LLMs and show that the proposed verification achieves higher agreement with human judgments than a VLM-based judge (GPT-6 Astra).

Critical strengths
  1. The paper introduces a new benchmark for evaluating motion graphics generation, and 32 LLMs are evaluated.

  2. The proposed evaluation pipline show strong alignment with human judgments, achieving 93.3% agreement, better performance than the VLM baseline (GPT-6 Astra).

Critical weaknesses
  1. The benchmark includes only 100 instances, which may be insufficient to cover the diversity of motion graphics and animation use cases.

  2. No inter-annotator agreement is reported, prompt construction seems to depend on a single human reviewer.

  3. Only one VLM judge (GPT-6 Astra) is used for comparison. Adding more VLM baselines would make the results more convincing.

  4. The supp website presents animations without their corresponding prompts. Including outputs from different models, verification results, and failure cases of the VLM judge would be helpful.

Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Weakness 1: Benchmark size and diversity

Benchmark size. Our 100 instances are comparable in scale to established benchmarks that emphasize carefully specified tasks and explicit evaluation. Terminal-Bench 2.0 contains 89 tasks (ICLR 2026), MatPlotBench contains 100 visualization tasks (Findings of ACL 2024), and ScienceAgentBench contains 102 scientific tasks (ICLR 2025). In vector animation, Vector Prism (CVPR 2026) evaluates 114 SVG-instruction pairs over 57 distinct SVGs, whereas our benchmark uses 100 distinct input SVGs.

Diversity of motion graphics and animation use cases. Our benchmark covers the three fundamental components of motion graphics: spatial structure, temporal sequencing, and appearance and visibility. We organize 17 animation requirement types into these families: eight spatial, five temporal, and four appearance/visibility types (Table 1; definitions in Appendix A.2, Table 3). Each motion edit prompt combines 7-14 types; all 100 instances combine spatial and temporal requirements, and 65 also include appearance or visibility. Models must therefore satisfy several interacting behaviors within each instance. We ground this coverage in real-world motion designs by selecting 100 distinct animations from LottieAnimation-660K, a collection of 660K real-world vector animations (Chen et al., 2026), after manually reviewing 1,000 candidates and removing duplicate graphics and motions. Our evaluation identifies 16 failure modes, with reference-pose mismatch, motion trajectory mismatch, and event coordination mismatch as the three leading patterns, showing that these instances reveal varied recurring shortcomings in generated animations.

References

AuthorsOpenReview ↗

Response to Weakness 2: Annotation agreement and prompt construction

Inter-annotator agreement. We recovered the initial independent judgments from preserved annotation edit histories and now report initial raw inter-annotator agreement of 88.88%, with Krippendorff's α=0.718\alpha = 0.718 and Fleiss' κ=0.718\kappa = 0.718. Please see "Annotation procedure and initial inter-annotator agreement" in our common response above for the procedure, supporting comparisons, and appendix correction.

Motion edit prompt construction. We would like to clarify that two human reviewers participated in our LLM-assisted motion edit prompt construction process. An LLM drafts each prompt from the input static SVG and source animation, and human reviewers compare the draft with this evidence, then approve it, edit it, or request a targeted revision. All four authors additionally reviewed a randomly sampled subset of 15 motion edit prompts. Our approach of multi-person review of benchmark tasks and instructions is also used in BigCodeBench (Zhuo et al., 2025), ScienceAgentBench (Chen et al., 2025), and DS-1000 (Lai et al., 2023). In our revised paper, Section 3.2 states the two-reviewer process and the additional four-author review.

References

AuthorsOpenReview ↗

Response to Weaknesses 3 and 4: Additional VLM judges and failure cases

Thank you for suggesting additional VLM baselines. We have added Opus 5, Opus 5.5, and GPT-6.1 Sol alongside our original GPT-6 Astra baseline. We evaluate all four judges on the same balanced set of 180 human-labeled animations.

EvaluatorAgreement with humansApproval recallRejection recall
Opus 5145/180 (80.6%)90/90 (100.0%)55/90 (61.1%)
Opus 5.5160/180 (88.9%)87/90 (96.7%)73/90 (81.1%)
GPT-6 Astra150/180 (83.3%)74/90 (82.2%)76/90 (84.4%)
GPT-6.1 Sol149/180 (82.8%)75/90 (83.3%)74/90 (82.2%)
Our verification programs168/180 (93.3%)81/90 (90.0%)87/90 (96.7%)

The VLM judges based on Opus models tend to approve animations too readily: Opus 5 and Opus 5.5 recover 100.0% and 96.7% of human approvals, but only 61.1% and 81.1% of human rejections, respectively. The GPT judges have more balanced approval and rejection recall, but both measures remain between 82.2% and 84.4%. Our verification programs achieve 93.3% agreement, an absolute gain of 4.4-12.8% over the four tested VLM judges, while maintaining balanced performance with high recall for both human approvals (90.0%) and rejections (96.7%).

In our revised paper, Section 5.1 includes the comparison table for all four VLM judges and our verification programs.

VLM judge failure cases. To complement the expanded comparison above and address your request for "failure cases of the VLM judge," we examined all 116 cases where a VLM judge failed to match the human judgment, covering 53 distinct animations. We identify five recurring failure patterns: misjudging part connections, grouping, and pivots (43 judgments); misjudging motion direction, amount, or count (35); misjudging shape and appearance (34); misjudging timing and coordination (29); and misjudging spatial placement and reference poses (26). We have added the Shoulder raise, Puzzle, and Van examples to their corresponding instance pages. In our revised paper, Section 5.1 summarizes this analysis, and Appendix C.2 reports the five failure patterns and uses Shoulder raise and Puzzle to illustrate a false approval and a false rejection, respectively.

Here we discuss a few specific examples. For the misjudged pivot failure case in the Shoulder raise instance (https://motion-graphics-bench.github.io/instances/095-woman-doing-barbell-incline-front-raise-exercise-for-shoulders/#vlm-judge-failure-cases), all four judges approve an output and describe a shoulder-centered rotation, but the arm rotates around the wrong shoulder pivot. For the false rejection in the Puzzle instance (https://motion-graphics-bench.github.io/instances/061-product-solution-puzzle/), GPT-6 Astra and GPT-6.1 Sol reject a human-approved Sonnet 5 output because their reports say that the notches disappear when the pieces separate. However, the puzzle pieces keep their shapes as they move. Our verification programs agree with the human rejection of Shoulder raise and approval of Puzzle. For the misjudged timing failure case in the Van instance (https://motion-graphics-bench.github.io/instances/088-van/#vlm-judge-failure-cases), Astra and Opus 5.5 reject a human-approved output because their reports infer that the wheels stop before the body settles. However, the body settles while the wheels are still moving. This remaining wheel motion is subtle in the sampled frames shown to the judges.

AuthorsOpenReview ↗

Response to Weakness 4: Prompts, model outputs, and verification results

We would like to clarify that the motion edit prompts for all 100 benchmark instances have been available on our supplementary website since the paper submission deadline. Clicking any instance on the homepage opens its detail page, where the prompt appears under "Motion edit prompt." For example, instance 030 includes its full prompt, a human-approved GPT-5.6 Luna animation, a human-rejected Qwen3.6 27B animation, and their check-level verification reports (https://motion-graphics-bench.github.io/instances/030-five-squares-moving-loader/). Our appendix documents the prompts for benchmark input construction, verification program construction, VLM judging, and animation generation; our released code also includes the animation generation prompts. To address your request for broader outputs and verification results, we release all 100 benchmark instances and 53,180 LLM-generated output records from different models, with verification-program decisions and human labels where available, in an anonymous code and data archive in the Common response: code and data release and inter-annotator agreement section above

Reviewer_kmirOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MotionGraphicsBench, a benchmark of 100 motion-editing tasks derived from real Lottie animations. Each task provides a static SVG, a natural-language motion prompt, and an instance-specific verification program. The programs are constructed using human-labeled animations and synthesized tests. On 180 held-out animations, they achieve 93.3% agreement with human judgments. The benchmark evaluates 47 configurations from 32 LLMs and analyzes common motion-generation failures.

Critical strengths
  1. Well-designed task formulation. Supplying all visual elements in the static SVG isolates motion generation from graphic creation. The 17 requirement types cover spatial, appearance, and temporal behaviors, making the benchmark consequential for evaluating instruction-following.

  2. Interpretable verification. The deterministic programs accept multiple valid implementations and provide check-level failure reports. Their 93.3% agreement with human judgments exceeds the reported 83.3% VLM baseline and enables detailed failure analysis.

Critical weaknesses
  1. Limited validation of instance-specific verifiers. Only 180 held-out animations validate 100 different verification programs, averaging fewer than two examples per program. The aggregate 93.3% agreement therefore may not establish the reliability of each verifier. Moreover, initial annotator agreement cannot be reconstructed.

  2. Unequal precision in the model leaderboard. Fifteen configurations use 32 attempts per instance, while the remaining configurations use only one. Although all are reported as pass@1, their statistical uncertainty differs substantially, and no confidence intervals are provided. This weakens fine-grained rankings and the proprietary-versus-open-weight comparison.

Questions for authors
  1. What are the minimum, median, and maximum numbers of held-out animations per verification program? Are per-program agreement statistics available?

  2. How is human-label reliability assessed when initial independent annotator agreement is unavailable?

  3. Can the authors provide uncertainty estimates for Figure 3 and separate configurations evaluated with one attempt from those evaluated with 32 attempts?

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1: Per-program validation

Expanded validation. We designed our original held-out evaluation to assess benchmark-wide agreement with human judgments across all 100 instances. Our original 180-animation study is comparable in sample size to the human-validation studies in FollowBench (Jiang et al., 2024; 100 responses) and ToolEmu (Ruan et al., 2024; 200 trajectories across 100 cases). Because our benchmark is challenging for the models, we have more human-rejected animations than human-approved animations. We had 90 human-approved and 274 human-rejected animations from model generation configurations not used in our verifier construction set. To keep our original held-out evaluation balanced, we retained 90 of each and excluded the remaining 184 human-rejected animations. To strengthen per-program validation, we have now evaluated these 184 animations with the unchanged verification programs and human labels.

Evaluation setHeld-out animationsMin / median / max examples per programPrograms with at least 3 examplesAgreementApproval recallRejection recall
Original1801 / 1 / 427168/180 (93.3%)81/90 (90.0%)87/90 (96.7%)
Expanded3641 / 4 / 497334/364 (91.8%)81/90 (90.0%)253/274 (92.3%)

We report that our programs agree with human judgments on 334/364 animations (91.8%). The expanded set covers all 100 verification programs, with 97 now having at least three held-out examples, compared with 27 originally; the median increases from one to four. On the additional 184 human-rejected animations, our programs correctly reject 166 (90.2%) and falsely approve 18. The expanded set contains 90 human-approved and 274 human-rejected animations, with 21 false approvals and nine false rejections. In our revised paper, Section 5.1 describes this expanded evaluation, and Appendix C.2 reports the results.

RL application. Our verification programs also provide rewards that improve pass@1 from 30.31% to 52.19% on ten instances held out from RL training. Please see our common response, "Applying our verification programs to reinforcement learning," for the setup and results.

References

AuthorsOpenReview ↗

Response to Weakness 1 and Question 2: Initial inter-annotator agreement

We recovered the initial independent judgments from preserved annotation edit histories and now report initial raw inter-annotator agreement of 88.88%, with Krippendorff's α=0.718\alpha = 0.718 and Fleiss' κ=0.718\kappa = 0.718. Please see "Annotation procedure and initial inter-annotator agreement" in our common response above for the procedure, supporting comparisons, and appendix correction.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 3: Sampling budgets and uncertainty

We would like to clarify that we first generated one response per benchmark instance for all 47 configurations, then collected 31 additional independent responses per instance for 15 configurations to evaluate pass@k. However, we would like to acknowledge that the original Figure 3 computed pass@1 using 32 responses per instance for 15 configurations, but only one response per instance for the remaining 32 configurations. To make the comparison use the same sampling budget, we have recomputed pass@1 using only the first recorded response per instance for every configuration, over the same 100 benchmark instances.

The original and recomputed rankings across all 47 configurations have a Spearman rank correlation of 0.995, where 1.0 indicates identical rankings. This shows that the overall ordering remains closely aligned under the equal-sampling-budget comparison.

To quantify uncertainty, we computed 95% confidence intervals for pass@1 and pass@32 using 2,000 resamples of our 100 benchmark instances with replacement (seed 921), taking the 2.5th and 97.5th percentiles of the resampled scores. For pass@32, each resampled instance retains its full 32-response set. For the proprietary/open-weight comparison, we retain one highest-effort configuration per model: the 16 proprietary models average 38.9% pass@1, while the 16 open-weight models average 21.3%. Their difference is 17.7 percentage points, with a 95% confidence interval of 14.4-21.0 points, using the same resampled instances for both groups. This supports our group-level finding without treating small differences between individual configurations as definitive.

In our revised paper, Figure 3 and the related pass@1 statistics use one response per instance for all 47 configurations. Appendix D.2 reports the pass@1 and pass@32 confidence intervals, while Section 6.1 separately presents the pass@k analysis using all 32 responses per instance for its 15 configurations.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 3: Sampling budgets and uncertainty (cont.)

Updated pass@1: one response per instance for all 47 configurations.

All scores and confidence bounds in both tables are percentages.

ConfigurationPass@1 [95% CI]ConfigurationPass@1 [95% CI]
Fable 5.1 (max)68.0 [59.0, 76.0]GPT-6 Astra (max)67.0 [58.0, 76.0]
GPT-6 Astra (high)60.0 [50.0, 69.0]GPT-5.6 Sol (max)52.0 [42.0, 62.0]
Opus 5 (max)49.0 [39.0, 58.0]GPT-5.6 Sol (xhigh)48.0 [38.0, 58.0]
GPT-5.6 Terra (max)46.0 [36.0, 56.0]Opus 5 (xhigh)45.0 [36.0, 54.0]
GPT-5.5 (xhigh)43.0 [33.0, 53.0]Muse Spark 1.3 (max)41.0 [32.0, 50.0]
Opus 4.8 (xhigh)41.0 [31.0, 50.0]Grok 4.6 (xhigh)41.0 [31.0, 51.0]
Qwen 3.8 (xhigh)39.0 [30.0, 49.0]GLM 5.3 (max)36.0 [26.0, 45.0]
Opus 5 (high)36.0 [27.0, 46.0]Gemini 3.8 Flash (high)36.0 [27.0, 45.0]
GPT-5.6 Sol (high)35.0 [26.0, 44.0]Qwen 3.8 Flash-Next (xhigh)34.0 [25.0, 44.0]
Gemini 3.1 Pro (high)33.0 [24.0, 42.0]DeepSeek V4.1 Flash (max)33.0 [24.0, 42.0]
GPT-5.6 Luna (xhigh)32.0 [23.0, 41.0]GPT-5.6 Luna (max)32.0 [23.0, 41.0]
DeepSeek V4 Flash (high)31.0 [22.0, 41.0]Kimi K3 (max)30.0 [21.0, 39.0]
GPT-5.6 Terra (xhigh)29.0 [20.0, 38.0]Gemini 3.8 Flash (medium)29.0 [20.0, 38.0]
DeepSeek V4 Flash (max)29.0 [20.0, 38.0]GLM 5.3 Flash (max)28.0 [19.0, 37.0]
Sonnet 5 (xhigh)26.0 [18.0, 35.0]DeepSeek V4 Pro (max)25.0 [17.0, 34.0]
GPT-5.6 Luna (high)25.0 [17.0, 34.0]GLM 5.2 (max)25.0 [17.0, 33.0]
GPT-5.6 Terra (high)21.0 [13.0, 29.0]Kimi K3 (high)21.0 [13.0, 29.0]
Gemini 3 Flash (high)18.0 [11.0, 26.0]GLM 5.2 (high)18.0 [11.0, 26.0]
Gemini 3.5 Flash (high)16.0 [9.0, 23.0]Gemini 3.6 Flash (high)14.0 [8.0, 21.0]
DeepSeek V4 Flash (low)13.0 [7.0, 20.0]Inkling (xhigh)13.0 [7.0, 20.0]
Kimi K3 (low)12.0 [6.0, 18.0]Inkling Small (xhigh)12.0 [6.0, 19.0]
Nemotron 3 Ultra (full thinking)9.0 [4.0, 15.0]Muse Glimmer 30B (xhigh)9.0 [4.0, 15.0]
Gemma 4 31B IT (default)7.0 [3.0, 12.0]Qwen 3.6 27B (thinking)6.0 [2.0, 11.0]
MiniMax M3 (max)5.0 [1.0, 10.0]

Pass@32: 32 responses per instance for 15 configurations.

The pass@32 scores are unchanged; we add their confidence intervals. An instance succeeds if at least one of its 32 responses passes. Rows follow the updated pass@1 order.

ConfigurationPass@32 [95% CI]ConfigurationPass@32 [95% CI]
Fable 5.1 (max)95.0 [90.0, 99.0]GPT-6 Astra (max)94.0 [89.0, 98.0]
GPT-5.6 Sol (max)86.0 [79.0, 92.0]Opus 5 (max)86.0 [79.0, 92.0]
Muse Spark 1.3 (max)80.0 [72.0, 87.0]Grok 4.6 (xhigh)75.0 [66.0, 83.0]
Qwen 3.8 (xhigh)82.0 [74.0, 89.0]GLM 5.3 (max)83.0 [76.0, 90.0]
Gemini 3.8 Flash (high)68.0 [59.0, 77.0]DeepSeek V4.1 Flash (max)80.0 [72.0, 87.0]
Kimi K3 (max)76.0 [67.0, 84.0]DeepSeek V4 Pro (max)66.0 [57.0, 75.0]
Inkling (xhigh)51.0 [41.0, 61.0]Nemotron 3 Ultra (full thinking)46.0 [36.0, 56.0]
MiniMax M3 (max)38.0 [28.0, 47.0]

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common response: summary

We thank all reviewers for their thoughtful feedback.

Our paper introduces MotionGraphicsBench, an expert-curated benchmark of 100 instances grounded in real-world motion designs, where models animate an input static SVG according to a motion edit prompt. We provide deterministic evaluation through human-readable verification programs, which accept different valid implementations and report which verification checks pass or fail. We also evaluate 47 configurations from 32 LLMs, identifying recurring failures to match reference poses, follow requested motion trajectories, and coordinate related events.

We appreciate that the reviewers recognized several strengths of our work:

  • A useful evaluation problem (GVXC), with a benchmark design that isolates motion generation from graphic creation (kmir).
  • Benchmark coverage spanning 17 animation requirement types across spatial, appearance, and temporal behaviors, supporting the evaluation of instruction-following (kmir).
  • Interpretable, instance-specific verification reports that can support both model evaluation and debugging (kmir, GVXC).
  • An explicitly held-out validation set, separate from the animations used to construct the verification programs (GVXC).
  • Aligning with human judgments better than the VLM baseline (kmir, Tbae).
  • Substantial evaluation effort across many models and configurations (Tbae, GVXC).

We answer the questions commonly asked by reviewers below, and address the questions of each reviewer in separate responses.

AuthorsOpenReview ↗

Common response: applying our verification programs to reinforcement learning

In response to Reviewer kmir's questions about per-program validation and Reviewer GVXC's concerns about the reliability of our rankings and repeated-sampling results, we test whether our verification programs can also improve animation generation by providing rewards for reinforcement learning (RL). This experiment follows prior uses of automatic evaluation as a learning signal, including test-based rewards in HardTestGen (ICLR 2026) and GenEval-based rewards in Flow-GRPO (NeurIPS 2025).

We fine-tuned Qwen3.8-27B using group relative policy optimization (GRPO) with small trainable adapters (LoRA) on 90 of our 100 benchmark instances, holding out the remaining ten from RL training. GRPO compares rewards among outputs generated for the same instance; our reward is the fraction of verification checks that pass, with invalid outputs receiving zero. Each policy update uses 32 groups of 16 generation attempts (512 attempts).

The model receives the motion edit prompt, input static SVG, and timing constraints, but not the verification-program code or source animations. We evaluate the base and trained models with 32 independent generation attempts per held-out instance (320 per model), using medium reasoning and temperature 1.

After 41 policy updates, pass@1 increases by 21.88 percentage points:

PolicyMean fraction of verification checks passedPass@1Pass@32
Base model67.05%97/320 (30.31%)6/10 (60%)
After 41 updates84.22%167/320 (52.19%)10/10 (100%)

These results show that rewards from our verification programs improve pass@1 and pass@32 on held-out instances, demonstrating their usefulness beyond benchmark scoring. In our revised paper, Section 5.3 presents this experiment, with training and evaluation details in Appendix C.5.

References

AuthorsOpenReview ↗

A gentle reminder to check our responses

Dear Reviewers,

Thank you so much again for your thoughtful feedback. We have responded to all comments and updated the manuscript PDF to incorporate all changes described in our rebuttal.

We would appreciate it if you could let us know whether our responses and revised manuscript address your concerns, or if you have any remaining questions. We would be happy to clarify or discuss any point further.

Sincerely, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.