- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VTR-Bench, a benchmark for visual text rendering in video generation. Evaluations on 11 models show text rendering is a common weakness. A keyframe-guided agentic framework iteratively refines generation with visual feedback, reducing word error rate by 32.5 percent relatively.
Critical strengths
The benchmark fills a genuine gap and the failure analysis is instructive. The 1202 text blocks bound to concrete carriers give WER practical semantics, and the systematic evaluation of 11 models with failure classification identifies concrete sources of capability differences.
Critical weaknesses
- Video scoring and transcription use the same VLM family (Wen), whose recognition biases may systematically affect results, with no cross-evaluator experiments.
- The benchmark scale is limited, only concentrated on static scene text and no dynamic text such as scrolling captions.
Questions for authors
- Can the agentic framework be validated on other models, and what is the end-to-end extra cost?
- How consistent are the results when other VLMs serve as the evaluator?
4 comments
Rebuttal for Reviewer qpp8
Question 1. Other generation models and additional cost
We evaluated an additional configuration using GLM-5.3-Flash as the Director, GLM-Image for image generation, and LTX-2.5 for video generation. The table below compares its T2V, I2V and Agentic results with those of the MiniMax H3 configuration.
| Video generator | Evaluator | T2V: WER / Score | I2V: WER / Score | Agentic: WER / Score |
|---|---|---|---|---|
| Minimax H3 | Qwen3.8-27B | 0.4468 / 0.7562 | 0.3328 / 0.8132 | 0.3015 / 0.8315 |
| Minimax H3 | GLM-5.3-Flash | 0.4836 / 0.7910 | 0.3681 / 0.8503 | 0.3198 / 0.8623 |
| LTX-2.5 | Qwen3.8-27B | 0.9917 / 0.6162 | 0.7752 / 0.6262 | 0.7131 / 0.6278 |
| LTX-2.5 | Qwen3.6-27B | 0.9880 / 0.6898 | 0.7728 / 0.6787 | 0.7086 / 0.6747 |
We can see that Agentic continues to improve text accuracy over I2V after replacing the Director, image generator and video generator.
We also report the cost of the agentic framework in the table below, measured by the average number of tool calls, API calls and video generations per sample.
| Metric | MiniMax-H3 Agentic | LTX-2.5 Agentic |
|---|---|---|
| Average tool calls | 8.63 | 10.66 |
| Average API calls | 16.39 | 20.72 |
| Average video generations | 1.81 | 2.00 |
Weakness 2: Benchmark scale and temporal content
We appreciate the reviewer’s suggestion to broaden the benchmark. Each additional prompt requires video generation and evaluation across multiple models, with proprietary APIs adding to the cost. The current scale reflects a balance between this cost and coverage: VTR-Bench contains 300 prompts spanning five application scenarios and 1,202 text blocks, as described in Section 3.2. The suite also includes temporal requirements involving moving carriers and changing text. We provide concrete examples in the table below.
| Example | Temporal requirement |
|---|---|
| AD-0019 | A printed label moves with a rotating coffee tin. |
| UI-0002 | A vote tally changes from 41 to 42 and a user badge updates. |
| LIFE-0001 | A schedule card is updated after inspecting wet camping gear. |
These examples demonstrate coverage of moving text carriers, changing numerical values and updates to textual content. We also agree that scrolling captions would be a valuable addition. They would allow us to assess whether text remains legible and accurate as it moves across the frame and new words enter view. We will consider dedicated scrolling-caption cases in future versions of VTR-Bench, alongside broader coverage of dynamic text.
Dear Reviewer qpp8,
Thank you for your thoughtful review and constructive suggestions. Following our rebuttal, we have incorporated the additional experiments and clarifications into the revised manuscript:
- Consistency across evaluator families: To address your concern about reliance on the Qwen family, Appendix C.1 now includes GLM-5.3-Flash, reporting its overall and per-scenario results together with model rankings across all three evaluators.
- Human alignment: To further validate evaluation reliability, Section 4.5 and Appendix C.2 expand human validation to 220 videos, with 20 from each of the 11 generation models. Appendix C.2 reports human alignment for Qwen3.8-27B, Qwen3.6-27B, and GLM-5.3-Flash.
- Generalization to other models: Appendix C.5.1 adds a configuration using GLM-5.3-Flash as the Director agent, GLM-Image for image generation, and LTX-2.5 for video generation. The results show that Agentic generation continues to improve text accuracy over I2V after replacing all three models.
- Additional computational cost: Alongside these results, Appendix C.5.2 reports the generation budgets and average tool calls, API calls, and video generations.
As the discussion period is drawing to a close, we would greatly appreciate your feedback on these responses and revisions. We hope our rebuttal has addressed your concerns and would be happy to clarify any remaining questions.
Best regards,
The Authors
Dear Reviewer qpp8,
We would like to further clarify the benchmark size and model coverage of VTR-Bench.
VTR-Bench contains 300 prompts specifying 1,202 text blocks across five application scenarios. Evaluating each prompt across multiple video generators entails substantial generation and evaluation costs, including calls to commercial APIs. Our design therefore combines broad scenario coverage with detailed, carrier-specific text requirements and human-reviewed prompts. For reference, the table below summarizes the evaluation suites and model coverage of several published benchmarks.
| Benchmark | Venue | Evaluation prompts / cases | Open-source | Closed-source | Total models / systems |
|---|---|---|---|---|---|
| VideoPhy [1] | ICLR 2025 | 344 | 7 | 5 | 12 |
| Video-Bench [2] | CVPR 2025 | 419 | 4 | 3 | 7 |
| PhyGenBench [3] | ICML 2025 | 160 | 8 | 6 | 14 |
| TC-Bench [4] — T2V | Findings of ACL 2025 | 150 | 10 | 4 | 14 |
| TC-Bench [4] — I2V | Findings of ACL 2025 | 120 | 6 | 0 | 6 |
| VideoPhy-2 [5] | ICLR 2026 | 591 | 7 | 2 | 9 |
| AVGen-Bench [6] | ICML 2026 | 235 | 6 | 6 | 12 |
| AV-Phys Bench [7] | NeurIPS 2026 | 268 | 4 | 3 | 7 |
| VTR-Bench | ours | 300 | 7 | 5 | 12 |
During the rebuttal period, we additionally evaluated LTX-2.5 on all 300 prompts, expanding our coverage from 11 to 12 video generation models: seven open-source and five closed-source models. These additional results, including baseline, I2V, and Agentic generation, are reported in Appendix C.5.1. VTR-Bench combines a prompt-suite size comparable to several published benchmarks with coverage of 12 generation models and 1,202 individually specified text blocks. We hope this comparison helps clarify the scale of the evaluation and the granularity of its textual requirements, and we would appreciate your thoughts on whether it addresses your concern.
References:
[1] Bansal, Hritik, et al. "Videophy: Evaluating physical commonsense for video generation." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Han, Hui, et al. "Video-bench: Human-aligned video generation benchmark." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
[3] Meng, Fanqing, et al. "Towards world simulator: Crafting physical commonsense-based benchmark for video generation." arXiv preprint arXiv:2410.05363 (2024).
[4] Feng, Weixi, et al. "TC-bench: Benchmarking temporal compositionality in conditional video generation." Findings of the Association for Computational Linguistics: ACL 2025. 2025.
[5] Bansal, Hritik, et al. "Videophy-2: A challenging action-centric physical commonsense evaluation in video generation." International Conference on Learning Representations. Vol. 2026. 2026.
[6] Zhou, Ziwei, et al. "AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation." Forty-third International Conference on Machine Learning. 2026.
[7] Cui, Zijun, et al. "Do Joint Audio-Video Generation Models Understand Physics?." arXiv preprint arXiv:2605.07061 (2026).
Rebuttal for Reviewer qpp8
We thank the reviewer for recognizing the value of our benchmark and failure analysis.
Weakness 1 and Question 2: Consistency with another evaluator family
To assess consistency across evaluator families, we evaluated all 11 baseline models with GLM-5.3-Flash. The table below compares its overall WER results with those of Qwen3.8 and Qwen3.6.
The corresponding Video Scores are reported in the table below.
To compare model rankings, we report Spearman correlations across the 11 models in the table below.
All three evaluators produce the same WER ranking, while GLM and Qwen yield highly consistent Video Score rankings, supporting the benchmark findings across evaluator families.
We also compare all three evaluators with human judgments on 220 videos, with 20 per generation model, covering 4,400 CoQ judgments and 883 text blocks. Following the human-alignment protocol in Section 4.5, we report the results in the table below.
GLM also agrees closely with human video-level WER, with Pearson correlation of 0.9744 and MAE of 0.0336.
The table below compares Qwen3.8-27B and GLM-5.3-Flash evaluations of the MiniMax-H3 generation methods.
GLM confirms that Agentic improves both text accuracy and Video Score over I2V, reproducing the framework’s gains with an evaluator outside the Qwen family.