- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces AnimateBanana, a framework for transforming static, process-oriented scientific diagrams into animated and narrated presentations. Given a diagram and a user-specified animation style, the method first constructs an editable, animation-aware programmatic representation, then uses specialized multimodal agents to plan the presentation sequence and narration and generate the final animation. Critic-mediated feedback is used to refine intermediate outputs. The resulting narratives can be exported to formats including SVG, GIF, MP4, PPTX, and animated PDF.
The paper also introduces AnimateBench, comprising 5,000 curated diagrams with verified animated narratives and 500 human-authored presentations extracted from related conference talks, together with task-specific metrics for animation validity and narrative quality. The method is evaluated through MLLM-based automatic evaluation, comparisons with direct-generation and alternative-backbone baselines, component ablations, and a user study. The authors report substantial improvements over the evaluated baselines.
Critical strengths
-
Novel and practically relevant task formulation. While prior work has explored paper-to-slide generation, narrated figure walkthroughs, and SVG animation, this paper focuses on converting process-oriented scientific diagrams into editable, narrated animations with controllable styles and multiple export formats. The task addresses a real and labor-intensive scientific communication workflow.
-
Structured and multi-format generation. The method explicitly represents diagram elements, presentation order, and step-level narration before rendering the final animation. This design supports controllable animation styles and export to multiple presentation formats rather than only producing a raster video.
Critical weaknesses
1. Questionable benchmark independence and insufficient annotation transparency. AnimateBench is bootstrapped using AnimateBanana itself, which may introduce method-dependent biases into the reference narratives and task formulation. The paper also provides almost no information about the human verification process, including the number and expertise of verifiers, the number of verifiers per sample, cross-checking or adjudication, modification rates, and annotation workload. Given the claimed scale of 5,000 samples, these omissions make the quality and independence of the benchmark difficult to assess.
2. Insufficiently controlled and reproducible evaluation. The large performance gap between AnimateBanana and strong closed-source models is difficult to interpret. The paper provides almost no details about how the closed-source direct-generation baselines were implemented. Missing information includes, but is not limited to, the prompts, model and API versions, provided inputs and context, expected output format, and, where applicable, tool access, interaction procedure, rendering process, and treatment of failed generations. Moreover, the complete MLLM-judge prompts and implementation are unavailable, while the human quality evaluation covers only 25 diagrams per system. Together, these issues substantially weaken the reliability of the main empirical conclusions.
Questions for authors
-
How was the human verification of AnimateBench conducted? In particular, how many verifiers were involved, what was their expertise, how many verifiers examined each sample, was any cross-checking or adjudication performed, and what fraction of the automatically generated narratives required modification?
-
What role do the verified animated narratives in AnimateBench play in the reported evaluation? Are they used as reference annotations by any of the main metrics, and if not, what is their purpose in the benchmark?
-
What exact protocol was used for the closed-source direct-generation baselines? Please provide the exact prompts, model/API versions, and generation and rendering code used for the closed-source direct-generation baselines. These details are necessary to assess and reproduce the comparison. Please also provide complete outputs for several representative end-to-end baseline examples, rather than only extracted frames and judge results.
-
What was the complete MLLM-judge protocol, including the prompts, model version and settings, frame-sampling procedure, presentation-step extraction, and input format?
-
How should the reported frame/step inconsistencies be interpreted?In data/evaluation_runs.json, the sample Claude Code with Opus 5/CVPR_2025_arch00012 has 15 extracted frames but 20 judged steps, with the last five frames set to null while still contributing to the score. Please clarify this inconsistency and whether it affects the reported results.
-
The MLLM-judge validation covers only 50 outputs for cross-judge agreement and 75 human-matched outputs. How were these sample sizes and samples determined, and what evidence shows that the reported agreement is reliable and representative?
-
The human study compares AnimateBanana, AnimateBanana with a Qwen backbone, and Gemini 3.1 Pro direct generation. What is the rationale for mixing backbone and generation-method comparisons, and what conclusion is this design intended to support?
Closing Statement and Summary
We sincerely thank all the reviewers for their time and thoughtful feedback. We are encouraged that the reviews have recognised the novelty and practical relevance of our work and found our evaluation idea to be useful for future work.
During the discussion, we addressed all the reviewers' concerns through additional analyses, experiments, and clarifications. For ease of reference, we summarise the key points below.
We have incorporated additional requested analyses into the supplementary material and companion website, with representative visual examples to make the results easier to inspect.
We hope that our responses and the additional evidence address the reviewers' concerns and provide a clearer picture of the contributions of our work. We respectfully request that these be taken into consideration in the final assessment of our submission.
Sincerely,
Authors of Submission 4633