acceptodds
Under review as a conference paper at ICLR 2027

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Abstract

Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce VTR-Bench, a systematic benchmark for evaluating the Visual Text Rendering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a Keyframe-Guided Agentic Framework in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement.

Primary area:
datasets and benchmarks
Keywords:
Visual Text Rendering, Video Generation, Dataset and Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qpp8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VTR-Bench, a benchmark for visual text rendering in video generation. Evaluations on 11 models show text rendering is a common weakness. A keyframe-guided agentic framework iteratively refines generation with visual feedback, reducing word error rate by 32.5 percent relatively.

Critical strengths

The benchmark fills a genuine gap and the failure analysis is instructive. The 1202 text blocks bound to concrete carriers give WER practical semantics, and the systematic evaluation of 11 models with failure classification identifies concrete sources of capability differences.

Critical weaknesses
  1. Video scoring and transcription use the same VLM family (Wen), whose recognition biases may systematically affect results, with no cross-evaluator experiments.
  2. The benchmark scale is limited, only concentrated on static scene text and no dynamic text such as scrolling captions.
Questions for authors
  1. Can the agentic framework be validated on other models, and what is the end-to-end extra cost?
  2. How consistent are the results when other VLMs serve as the evaluator?
4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer qpp8

We thank the reviewer for recognizing the value of our benchmark and failure analysis.

Weakness 1 and Question 2: Consistency with another evaluator family

To assess consistency across evaluator families, we evaluated all 11 baseline models with GLM-5.3-Flash. The table below compares its overall WER results with those of Qwen3.8 and Qwen3.6.

Video generatorQwen3.8-27BQwen3.6-27BGLM-5.3-Flash
Wan2.2-5B0.99560.99310.9946
HunyuanVideo-1.50.96950.96430.9650
LTX-2.30.99560.99200.9941
Lingbot-Video-Dense0.98850.98320.9849
Lingbot-Video-MoE0.89560.88150.8816
Minimax H30.44680.45040.4836
ViduQ30.95000.94680.9517
HappyHorse1.10.87990.86270.8753
Kling v3.00.97900.97660.9668
Seedance2.50.64080.63180.6327
Wan3.00.25010.24840.2746

The corresponding Video Scores are reported in the table below.

Video generatorQwen3.8-27BQwen3.6-27BGLM-5.3-Flash
Wan2.2-5B0.4360.5070.5047
HunyuanVideo-1.50.6010.6630.6568
LTX-2.30.6070.6870.6698
Lingbot-Video-Dense0.5340.6010.6012
Lingbot-Video-MoE0.5490.6290.6160
Minimax H30.7560.8090.7910
ViduQ30.7860.8310.8129
HappyHorse1.10.8310.8690.8523
Kling v3.00.7930.8370.8183
Seedance2.50.7910.8360.8227
Wan3.00.8490.8860.8855

To compare model rankings, we report Spearman correlations across the 11 models in the table below.

Evaluator comparisonOverall WER rank correlation ↑Overall Video Score rank correlation ↑
GLM-5.3-Flash vs. Qwen3.8-27B1.0000.991
GLM-5.3-Flash vs. Qwen3.6-27B1.0000.991
Qwen3.8-27B vs. Qwen3.6-27B1.0001.000

All three evaluators produce the same WER ranking, while GLM and Qwen yield highly consistent Video Score rankings, supporting the benchmark findings across evaluator families.

We also compare all three evaluators with human judgments on 220 videos, with 20 per generation model, covering 4,400 CoQ judgments and 883 text blocks. Following the human-alignment protocol in Section 4.5, we report the results in the table below.

EvaluatorEvaluationAgreement ↑Pearson ↑Spearman ↑Lin’s CCC ↑MAE ↓
Qwen3.8-27BCoQ judgments91.98%————
Qwen3.8-27BText-block WER—0.96100.87000.96040.0404
Qwen3.8-27BVideo-level WER—0.98600.92210.98510.0290
Qwen3.6-27BCoQ judgments91.50%————
Qwen3.6-27BText-block WER—0.95930.88320.95810.0418
Qwen3.6-27BVideo-level WER—0.98330.92640.98300.0272
GLM-5.3-FlashCoQ judgments89.18%————
GLM-5.3-FlashText-block WER—0.93890.87020.93860.0494
GLM-5.3-FlashVideo-level WER—0.97440.93290.97410.0336

GLM also agrees closely with human video-level WER, with Pearson correlation of 0.9744 and MAE of 0.0336.

The table below compares Qwen3.8-27B and GLM-5.3-Flash evaluations of the MiniMax-H3 generation methods.

MethodQwen3.8-27B WER ↓Qwen3.8-27B Video Score ↑GLM-5.3-Flash WER ↓GLM-5.3-Flash Video Score ↑
T2V0.44680.75620.48360.7910
I2V0.33280.81320.36810.8503
Agentic0.30150.83150.31980.8623

GLM confirms that Agentic improves both text accuracy and Video Score over I2V, reproducing the framework’s gains with an evaluator outside the Qwen family.

AuthorsOpenReview ↗

Rebuttal for Reviewer qpp8

Question 1. Other generation models and additional cost

We evaluated an additional configuration using GLM-5.3-Flash as the Director, GLM-Image for image generation, and LTX-2.5 for video generation. The table below compares its T2V, I2V and Agentic results with those of the MiniMax H3 configuration.

Video generatorEvaluatorT2V: WER / ScoreI2V: WER / ScoreAgentic: WER / Score
Minimax H3Qwen3.8-27B0.4468 / 0.75620.3328 / 0.81320.3015 / 0.8315
Minimax H3GLM-5.3-Flash0.4836 / 0.79100.3681 / 0.85030.3198 / 0.8623
LTX-2.5Qwen3.8-27B0.9917 / 0.61620.7752 / 0.62620.7131 / 0.6278
LTX-2.5Qwen3.6-27B0.9880 / 0.68980.7728 / 0.67870.7086 / 0.6747

We can see that Agentic continues to improve text accuracy over I2V after replacing the Director, image generator and video generator.

We also report the cost of the agentic framework in the table below, measured by the average number of tool calls, API calls and video generations per sample.

MetricMiniMax-H3 AgenticLTX-2.5 Agentic
Average tool calls8.6310.66
Average API calls16.3920.72
Average video generations1.812.00

Weakness 2: Benchmark scale and temporal content

We appreciate the reviewer’s suggestion to broaden the benchmark. Each additional prompt requires video generation and evaluation across multiple models, with proprietary APIs adding to the cost. The current scale reflects a balance between this cost and coverage: VTR-Bench contains 300 prompts spanning five application scenarios and 1,202 text blocks, as described in Section 3.2. The suite also includes temporal requirements involving moving carriers and changing text. We provide concrete examples in the table below.

ExampleTemporal requirement
AD-0019A printed label moves with a rotating coffee tin.
UI-0002A vote tally changes from 41 to 42 and a user badge updates.
LIFE-0001A schedule card is updated after inspecting wet camping gear.

These examples demonstrate coverage of moving text carriers, changing numerical values and updates to textual content. We also agree that scrolling captions would be a valuable addition. They would allow us to assess whether text remains legible and accurate as it moves across the frame and new words enter view. We will consider dedicated scrolling-caption cases in future versions of VTR-Bench, alongside broader coverage of dynamic text.

Dear Reviewer qpp8,

Thank you for your thoughtful review and constructive suggestions. Following our rebuttal, we have incorporated the additional experiments and clarifications into the revised manuscript:

  • Consistency across evaluator families: To address your concern about reliance on the Qwen family, Appendix C.1 now includes GLM-5.3-Flash, reporting its overall and per-scenario results together with model rankings across all three evaluators.
  • Human alignment: To further validate evaluation reliability, Section 4.5 and Appendix C.2 expand human validation to 220 videos, with 20 from each of the 11 generation models. Appendix C.2 reports human alignment for Qwen3.8-27B, Qwen3.6-27B, and GLM-5.3-Flash.
  • Generalization to other models: Appendix C.5.1 adds a configuration using GLM-5.3-Flash as the Director agent, GLM-Image for image generation, and LTX-2.5 for video generation. The results show that Agentic generation continues to improve text accuracy over I2V after replacing all three models.
  • Additional computational cost: Alongside these results, Appendix C.5.2 reports the generation budgets and average tool calls, API calls, and video generations.

As the discussion period is drawing to a close, we would greatly appreciate your feedback on these responses and revisions. We hope our rebuttal has addressed your concerns and would be happy to clarify any remaining questions.

Best regards,

The Authors

Dear Reviewer qpp8,

We would like to further clarify the benchmark size and model coverage of VTR-Bench.

VTR-Bench contains 300 prompts specifying 1,202 text blocks across five application scenarios. Evaluating each prompt across multiple video generators entails substantial generation and evaluation costs, including calls to commercial APIs. Our design therefore combines broad scenario coverage with detailed, carrier-specific text requirements and human-reviewed prompts. For reference, the table below summarizes the evaluation suites and model coverage of several published benchmarks.

BenchmarkVenueEvaluation prompts / casesOpen-sourceClosed-sourceTotal models / systems
VideoPhy [1]ICLR 20253447512
Video-Bench [2]CVPR 2025419437
PhyGenBench [3]ICML 20251608614
TC-Bench [4] — T2VFindings of ACL 202515010414
TC-Bench [4] — I2VFindings of ACL 2025120606
VideoPhy-2 [5]ICLR 2026591729
AVGen-Bench [6]ICML 20262356612
AV-Phys Bench [7]NeurIPS 2026268437
VTR-Benchours3007512

During the rebuttal period, we additionally evaluated LTX-2.5 on all 300 prompts, expanding our coverage from 11 to 12 video generation models: seven open-source and five closed-source models. These additional results, including baseline, I2V, and Agentic generation, are reported in Appendix C.5.1. VTR-Bench combines a prompt-suite size comparable to several published benchmarks with coverage of 12 generation models and 1,202 individually specified text blocks. We hope this comparison helps clarify the scale of the evaluation and the granularity of its textual requirements, and we would appreciate your thoughts on whether it addresses your concern.

References:

[1] Bansal, Hritik, et al. "Videophy: Evaluating physical commonsense for video generation." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Han, Hui, et al. "Video-bench: Human-aligned video generation benchmark." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[3] Meng, Fanqing, et al. "Towards world simulator: Crafting physical commonsense-based benchmark for video generation." arXiv preprint arXiv:2410.05363 (2024).

[4] Feng, Weixi, et al. "TC-bench: Benchmarking temporal compositionality in conditional video generation." Findings of the Association for Computational Linguistics: ACL 2025. 2025.

[5] Bansal, Hritik, et al. "Videophy-2: A challenging action-centric physical commonsense evaluation in video generation." International Conference on Learning Representations. Vol. 2026. 2026.

[6] Zhou, Ziwei, et al. "AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation." Forty-third International Conference on Machine Learning. 2026.

[7] Cui, Zijun, et al. "Do Joint Audio-Video Generation Models Understand Physics?." arXiv preprint arXiv:2605.07061 (2026).

Reviewer_GYAoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VTR-Bench evaluates text rendering in generated videos using 300 prompts and 1,202 text blocks across five scenario categories. It separately measures textual accuracy through carrier-specific transcription and WER, and scene and motion requirements through question-based evaluation. The authors compare 11 models, validate the evaluation against human judgments, and demonstrate a keyframe-guided agentic framework using MiniMax H3.

Critical strengths

1.The problem has practical relevance. Evaluating textual accuracy separately from scene and motion requirements reveals text errors that may persist despite good overall visual quality. Multiple carriers and longer passages also make the benchmark useful.

2.Construction and validation are reasonably thorough. The authors combine iterative reference-image checks with human review and report validation by three annotators on 100 videos, an alternative evaluator, and WER-parameter sensitivity analyses.

Critical weaknesses

1.WER provides limited coverage of textual changes over time (§3.3, §4.4, Appendix C.4). Transcribing each carrier from its clearest occurrence reduces the influence of blur or occlusion, but does not establish whether its text remains correct throughout the video. The frame-selection protocol is insufficiently specified, and whether Temporal Consistency explicitly checks sustained textual correctness remains unclear. Although the first generated frame is removed, subsequent early frames may retain text from the input image. Reporting input-image accuracy and accuracy in the early, middle, and late portions of a representative video subset would help distinguish initial image quality from subsequent text preservation.

2.The framework’s gains and dependence on Qwen-Image need clearer interpretation (§3.4, §4.4, Appendix B.2). The I2V baseline is useful, but the full framework’s gains may reflect first-frame quality, visual feedback, and additional candidate selection. Generation budgets and stopping and selection rules would help clarify these contributions. Qwen-Image also assists with prompt-feasibility checks. The authors state that these reference images are not video-conditioning inputs, spelling errors do not trigger revisions, and final human review is performed; there is therefore insufficient evidence to conclude that the benchmark is biased. Nevertheless, reporting the main types and frequency of prompt revisions would clarify the construction pipeline’s dependence on this image generator.

Questions for authors

1.How are frames sampled and each carrier’s clearest occurrence selected for transcription? Does Temporal Consistency explicitly assess sustained textual correctness? Please provide example questions and assessment criteria.

2.Could a representative subset be evaluated for input-image accuracy and early, middle, and late video accuracy under I2V and Agentic, including preservation of initially correct text while distinguishing expected occlusion or movement out of view from content degradation?

3.What are the generation budgets and candidate-selection rules for Agentic and I2V? Which prompt attributes are typically revised during feasibility validation, and how many prompts are affected?

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer GYAo

We thank the reviewer for recognizing the practical relevance and careful validation of our benchmark.

Weakness 1 and Question 1: Frame sampling and Temporal Consistency

Following the official recipes for Qwen3.8 and Qwen3.6, we sample videos at 2 FPS. The VLM receives the video, carrier descriptions and a scene description with target strings masked, and transcribes each carrier from its clearest visible occurrence. The VLM makes this choice itself; target strings are used only afterward to compute WER.

In our CoQ evaluation, Temporal Consistency assesses whether events and states follow the temporal requirements of the prompt, such as whether an action occurs before another or a resulting state persists. The table below lists example questions and their assessment criteria.

TypeExample questionAssessment criterion
State change“Does the liquid change from colorless to pale pink?”The specified transition is visible.
Event ordering“Is the stopcock opened before the titrant drop lands?”The two events occur in the requested order.
Synchronization“Does the titrant drop land while the flask is swirling?”The specified events overlap in time.
Persistence“Does the pale pink color persist after the drop lands?”The resulting state persists after the event.

Text accuracy over time is assessed separately through the early-, middle-, and late-segment WER and word-retention analyses reported below.

Weakness 1and Question 2. Input-image accuracy and early, middle, and late text accuracy

To examine how text accuracy evolves over time, we evaluated all 300 I2V/Agentic pairs in each generation configuration. We split each video into equal early, middle and late segments and evaluated each independently with Qwen3.8. We report the segment-level results in the table below.

Video generatorMethodEarly WER ↓Middle WER ↓Late WER ↓
MiniMax H3I2V0.42230.50040.5420
MiniMax H3Agentic0.34910.41260.4396
LTX-2.5I2V0.76680.82980.8820
LTX-2.5Agentic0.70360.76300.8183

Agentic achieves lower WER in all three segments under both generation configurations, showing that its text-accuracy advantage persists through the late portion of the video.

To address the reviewer’s question about initial image quality, we also evaluated the conditioning images used by I2V and Agentic with LTX-2.5. The table below reports input-image and full-video WER under both Qwen evaluators.

EvaluatorI2V input-image WER ↓Agentic input-image WER ↓I2V video WER ↓Agentic video WER ↓
Qwen3.8-27B0.76630.69460.77520.7131
Qwen3.6-27B0.76320.69980.77280.7086

Both evaluators find that Agentic produces more accurate text in the conditioning images and the resulting videos.

To assess preservation from a common starting point, we then track the same 4,763 word occurrences that are correct in both methods’ input images, spanning 617 text blocks and 250 paired LTX-2.5 videos. Each segment’s retention rate is the proportion of these words still correctly transcribed in that segment. Repeated words are counted by their positions in the required text. The table below reports retention for this shared set, including the proportions correct in at least one segment and in all three.

MethodShared initially correct word occurrencesEarlyMiddleLateCorrect in ≥1 segmentCorrect in all three segments
I2V4,76393.7%67.0%47.0%96.0%44.4%
Agentic4,76395.9%73.1%58.8%97.0%56.4%

On these same initially correct words, Agentic achieves higher retention in every segment: late-segment retention is 58.8% versus 47.0% for I2V, and retention across all three segments is 56.4% versus 44.4%.

We also report retention for all initially correct words within each method, using 6,826 word occurrences for I2V and 8,614 for Agentic. The results are shown in the table below.

MethodInitially correct word occurrencesEarlyMiddleLateCorrect in ≥1 segmentCorrect in all three segments
I2V6,82692.2%65.5%45.8%94.7%42.9%
Agentic8,61493.2%72.1%55.4%94.9%52.4%

Agentic starts with more correctly rendered words and retains a larger proportion of them through the video, consistent with the comparison on the shared word set.

AuthorsOpenReview ↗

Rebuttal for Reviewer GYAo

Weakness 1 and Question2: Carrier visibility and unreadability

We report transcription states in the early, middle and late segments for both generation configurations in the table below, distinguishing missing carriers from carriers with unreadable text. Each row covers 1,202 text targets.

Video generatorMethodSegmentCarrier not foundCarrier found but text unreadableTranscribedTranscribed rate
MiniMax H3I2VEarly3620296480.2%
MiniMax H3AgenticEarly23164101584.4%
MiniMax H3I2VMiddle12317590475.2%
MiniMax H3AgenticMiddle5915199282.5%
MiniMax H3I2VLate20614784970.6%
MiniMax H3AgenticLate9714096580.3%
LTX-2.5I2VEarly13925780667.1%
LTX-2.5AgenticEarly14215091075.7%
LTX-2.5I2VMiddle17629673060.7%
LTX-2.5AgenticMiddle15621583169.1%
LTX-2.5I2VLate25632761951.5%
LTX-2.5AgenticLate22922275162.5%

Agentic achieves higher transcribability in every segment under both configurations, with more text targets remaining readable through the middle and late portions of the video.

For MiniMax H3, we compare text accuracy on two fixed sets of text targets: (1) Carrier found: the carrier is found in the early, middle and late segments under both I2V and Agentic. (2) Text transcribed: the text can be transcribed in every segment under both methods, though the transcribed content may contain errors.

The same text targets are used for both methods and all three segments within each subset. We report their WER in the table below.

Video generatorSubsetTargets / paired videosMethodEarly WER ↓Middle WER ↓Late WER ↓
MiniMax H3Carrier found883 / 295I2V0.41230.44220.4787
MiniMax H3Carrier found883 / 295Agentic0.34110.38780.3953
MiniMax H3Text transcribed622 / 232I2V0.22970.26040.3075
MiniMax H3Text transcribed622 / 232Agentic0.17910.22120.2350

Agentic achieves lower WER in every segment on both subsets, including when the same text targets can be transcribed under both methods in all three segments.

Weakness 2 and Question 3: Generation budgets and candidate selection

The table below lists the budgets of the Agentic framework.

BudgetLimit per sample
Tool actions20
API-call credits30
Video-generation attempts3

As described in Section 4.4, the I2V baseline uses the original prompt and the first image generated by the agentic framework to generate video. As for the agentic generation, The Director selects candidates using the prompt and accumulated visual feedback on text, carriers, motion and scene fidelity. It invokes inspection/comparison as needed and stops by finalizing a candidate with a rationale; WER and Video Score are not selection inputs.

We also report the cost of the agentic framework in the table below, measured by the average number of tool calls, API calls and video generations per sample.

MetricMiniMax-H3 AgenticLTX-2.5 Agentic
Average tool calls8.6310.66
Average API calls16.3920.72
Average video generations1.812.00

Weakness 2 and Question 3: Prompt attributes revised during feasibility validation

Across the 300 prompts, 91 (30.3%) were revised during feasibility validation. Revisions mainly adjusted text-carrier placement, layout and text organization, and instructions for legibility. The required text remained unchanged throughout validation, consistent with the constraints described in Appendix B.2.

Thank you for the clarifications and additional experiments, which address my main concern.

Dear Reviewer GYAo,

Thank you for your thoughtful review and for taking the time to consider our clarifications and additional experiments. We appreciate your constructive feedback, which has helped us strengthen the paper. Following our discussion, we have incorporated the corresponding updates into the revised manuscript:

  • Temporal evaluation: Section 4.4 and Appendix C.5.1 now report text accuracy across early, middle, and late video stages. Appendix C.5.1 also reports input-image accuracy.
  • Transcription and candidate selection: Section 3.3 clarifies how the VLM selects each carrier’s clearest occurrence, while Section 3.4 explains how the Director agent selects candidates and decides when to stop generation.
  • Generation budgets and prompt revisions: Appendix C.5.2 specifies the generation budgets and computational cost. In addition, Appendix B.2 reports how many prompts were revised and the main attributes affected.

Thank you again for your constructive feedback and engagement throughout the discussion.

Best regards,

The Authors

Reviewer_iidqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VTR-Bench tests prescribed text rendering in generated videos using 300 prompts across five scenarios. It evaluates text with carrier-specific transcription and WER, and scene quality with a separate checklist. Eight of 11 models have near-ceiling WER. A keyframe-guided agentic framework improves Minimax H3’s text accuracy and Video Score.

Critical strengths
  1. The benchmark tests multiple text blocks and long passages, extending beyond the short labels used in existing benchmarks.

  2. Separating text accuracy from scene quality is useful. Masking reference text during transcription also helps prevent the evaluator from copying the prompt.

  3. Reporting missing carriers, unreadable text, and successful transcriptions makes the results easier to interpret than WER alone.

Critical weaknesses
  1. Human validation focuses on models with readable text, leaving the other eight models’ failures unvalidated. Their high WER may reflect either generation or transcription errors, and capping WER hides differences. High Video Scores despite poor text are also expected because the checklist excludes text accuracy.

  2. The framework is tested on one model in one run, without component ablations or cost reporting. I2V alone provides most of the gain. Using the evaluator’s model family for feedback also raises concerns about generalization.

Questions for authors

Please see the weakness section.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer iidq

We thank the reviewer for recognizing the value of our benchmark and evaluation design.

Weakness 1: Human validation of the other eight models

We added 120 human-annotated videos from the eight models in question, bringing the total to 220 videos, with 20 per model. Each video has three annotations. The combined set contains 4,400 CoQ judgments and 883 text blocks.

Following the same setting described in Section 4.5, we compare CoQ answers with the majority human vote and evaluator WER with the mean WER of the three human transcriptions. We report the alignment results in the table below.

EvaluatorEvaluationAgreement ↑Pearson ↑Spearman ↑Lin’s CCC ↑MAE ↓
Qwen3.8-27BCoQ judgments91.98%————
Qwen3.8-27BText-block WER—0.96100.87000.96040.0404
Qwen3.8-27BVideo-level WER—0.98600.92210.98510.0290
Qwen3.6-27BCoQ judgments91.50%————
Qwen3.6-27BText-block WER—0.95930.88320.95810.0418
Qwen3.6-27BVideo-level WER—0.98330.92640.98300.0272
GLM-5.3-FlashCoQ judgments89.18%————
GLM-5.3-FlashText-block WER—0.93890.87020.93860.0494
GLM-5.3-FlashVideo-level WER—0.97440.93290.97410.0336

New 120-video subset: 15 additional videos per model, 2,400 CoQ judgments and 479 text blocks. The table below reports results for this subset separately.

EvaluatorEvaluationAgreement ↑Pearson ↑Spearman ↑Lin’s CCC ↑MAE ↓
Qwen3.8-27BCoQ judgments91.83%————
Qwen3.8-27BText-block WER—0.93220.71680.92970.0285
Qwen3.8-27BVideo-level WER—0.90950.81560.89440.0192
Qwen3.6-27BCoQ judgments91.42%————
Qwen3.6-27BText-block WER—0.92770.75470.91990.0279
Qwen3.6-27BVideo-level WER—0.93930.81080.92730.0166
GLM-5.3-FlashCoQ judgments90.17%————
GLM-5.3-FlashText-block WER—0.89720.77250.89560.0313
GLM-5.3-FlashVideo-level WER—0.89950.84520.89770.0192

The additional annotations show close agreement with human WER on the eight high-error models, extending validation beyond models with readily readable text.

Weakness 1: WER capping

We report both R+5 and full-transcription WER in the table below, each with and without the cap.

Video generatorQwen3.8 R+5, cap 1Qwen3.8 R+5, uncappedQwen3.8 full, cap 1Qwen3.8 full, uncappedQwen3.6 R+5, cap 1Qwen3.6 R+5, uncappedQwen3.6 full, cap 1Qwen3.6 full, uncapped
Wan2.2-5B0.99561.00310.99581.03680.99311.00020.99341.0461
HunyuanVideo-1.50.96950.97990.97521.04590.96430.97140.96981.0083
LTX-2.30.99561.01040.99581.07270.99201.00200.99271.0575
Lingbot-Video-Dense0.98851.01540.99171.14680.98321.00350.98671.0910
Lingbot-Video-MoE0.89560.90290.93751.09430.88150.88650.91711.0439
Minimax H30.44680.44680.47990.49580.45040.45110.47670.4926
ViduQ30.95000.96170.95561.04440.94680.95580.95121.0159
HappyHorse1.10.87990.88510.89930.95970.86270.86670.87500.9106
Kling v3.00.97901.00500.98681.21570.97660.99650.98391.1536
Seedance2.50.64080.64320.67920.71870.63180.63380.65740.6875
Wan3.00.25010.25010.27460.27530.24840.24840.26780.2698

The top three models remain consistent across all four settings under both evaluators, and removing truncation while retaining the cap preserves the full model ranking. The settings mainly affect the scores and relative ordering of high-error models, reflecting how excess transcription tokens are penalized. While uncapped WER distinguishes insertion-heavy outputs, values above 1 offer limited additional insight into how faithfully the required text is rendered. We therefore cap per-video WER at 1 to emphasize text fidelity on a bounded scale, while reporting uncapped results for completeness.

AuthorsOpenReview ↗

Rebuttal for Reviewer iidq

Weakness2. Framework gains, generalization, and cost

To assess generalization, we additionally evaluated the framework with GLM-5.3-Flash as the Director, GLM-Image for image generation, and LTX-2.5 for video generation. This replaces all three components of the original configuration, which uses Qwen3.7-plus, Qwen-Image-3.0, and MiniMax H3. The table below compares both configurations against T2V and I2V.

Video generatorEvaluatorT2V: WER / ScoreI2V: WER / ScoreAgentic: WER / Score
Minimax H3Qwen3.8-27B0.4468 / 0.75620.3328 / 0.81320.3015 / 0.8315
Minimax H3GLM-5.3-Flash0.4836 / 0.79100.3681 / 0.85030.3198 / 0.8623
LTX-2.5Qwen3.8-27B0.9917 / 0.61620.7752 / 0.62620.7131 / 0.6278
LTX-2.5Qwen3.6-27B0.9880 / 0.68980.7728 / 0.67870.7086 / 0.6747

Although I2V already brings a substantial improvement, the Agentic framework still delivers considerable additional gains. On MiniMax H3, it reduces WER from 0.3328 to 0.3015 under Qwen3.8 and from 0.3681 to 0.3198 under GLM, corresponding to relative reductions of 9.4% and 13.1%, respectively. After replacing all three generation components, Agentic also reduces LTX-2.5 WER from 0.7752 to 0.7131 under Qwen3.8 and from 0.7728 to 0.7086 under Qwen3.6, giving further relative reductions of 8.0% and 8.3%.

We also report the cost of the agentic framework in the table below, measured by the average number of tool and API calls per sample.

MetricMiniMax-H3 AgenticLTX-2.5 Agentic
Average tool calls8.6310.66
Average API calls16.3920.72

Dear Reviewer iidq,

Thank you again for your thoughtful comments. Following our rebuttal, we have incorporated the additional experiments and clarifications into the revised manuscript, with the relevant changes summarized below:

  • Human validation: To address your concern about the eight high-error models, Section 4.5 and Appendix C.2 now report human alignment on 220 videos, with 20 from each of the 11 models.
  • WER capping: To clarify its effect on model comparisons, Appendix C.3 presents results with and without transcription truncation and capping, alongside an explanation of the bounded WER design.
  • Agentic gains and generalization: Regarding the additional gains over I2V, Section 4.4 now compares text accuracy across temporal stages. Furthermore, Appendix C.5.1 shows that Agentic continues to improve over I2V after replacing the Director, image generator, and video generator. To complement this evidence, Appendix C.1 adds evaluation with GLM-5.3-Flash.
  • Computational cost: Finally, Appendix C.5.2 reports the generation budgets and average tool calls, API calls, and video generations.

As the discussion period is drawing to a close, we would greatly appreciate your feedback on these responses and revisions. We hope our rebuttal has addressed your concerns and would be happy to clarify any remaining questions.

Best regards,
The Authors

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your careful reading and constructive feedback. We appreciate your recognition of the practical importance of visual text rendering and the value of VTR-Bench. Your comments have helped us strengthen the evidence supporting the benchmark and explain its scope and evaluation more clearly.

The central goal of this work is to make visual text rendering a systematic and actionable dimension of video-generation evaluation. Text carries information that viewers rely on, from scientific explanations to advertisements and interfaces. VTR-Bench places these textual requirements within concrete application scenarios and evaluates their accuracy alongside fulfillment of scene and motion requirements. Together, the prompt suite, automated evaluation with human alignments, and analysis of 11 video generation models provide a basis for understanding where current models succeed and where text rendering needs further development.

Our responses provide additional evidence on three questions raised by the reviews. First, evaluation reliability is supported across generation models and evaluator families: human validation now covers 220 videos with 20 per model, and the independent GLM evaluator reproduces the WER ranking obtained with both Qwen evaluators. Second, the temporal analyses examine text accuracy in the early, middle and late portions of videos, as well as the preservation of initially correct words. Agentic improves text retention even when both methods are compared on the same words that are initially correct. Third, the framework’s gains generalize across model configurations: after replacing the Director, image generator and video generator, Agentic continues to improve text accuracy over baseline and I2V.

The agentic framework complements the benchmark by demonstrating a practical way to improve the capabilities it measures. Its role is to connect evaluation findings with inference-time refinement through visual feedback. The additional comparisons and cost measurements help characterize these improvements and the computation they require.

We address each concern in the individual responses below. We also appreciate the suggestions for broadening dynamic-text coverage, including scrolling captions, which provide useful directions for future versions of VTR-Bench. Thank you again for helping us improve the clarity and empirical grounding of this work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.