acceptodds
Under review as a conference paper at ICLR 2027

Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation

Abstract

A video can exhibit convincing motion and photorealism yet still fail immediately when visual text collapses. Unlike generic scene content, visual text is exceptionally unforgiving in video generation, where minor stroke corruption, temporal instability, or editing errors instantly break legibility and realism. Existing benchmarks overlook this challenge by treating text as incidental scene content or relying on static OCR metrics that ignore temporal dynamics. We introduce **VidScribe**, a unified diagnostic benchmark for visual text across four core generation regimes: writing from language (T2V), transferring text identity from reference (R2V), sustaining consistency under dynamics (I2V), and editing localized text within video (V2V). VidScribe provides 803 human-verified samples instantiated over a 12-axis conditionally orthogonal factor space spanning Intrinsic Text Properties, Physical Imaging Conditions, and Temporal Behavior. To enable reliable evaluation, we build a track-grounded, gated suite with 11 shared metrics and 2 task-specific probes that enforce strict measurability conditions. Extensive benchmarking across 11 representative commercial and open-source systems reveals that video text capability is non-monolithic, with a clear decoupling between content recognition and stroke-level glyph correctness. Performance is highly task-asymmetric, with I2V sustaining text most reliably and V2V editing emerging as the primary bottleneck. Counter-intuitively, performance degradation is more concentrated on a small subset of text-centric structural and temporal factors than on adverse imaging conditions. Further diagnostic probes show that visual reference improves glyph and typographic fidelity rather than content accuracy, while localized editing struggles to isolate target text without corrupting undeclared source text. Beyond evaluation, VidScribe also provides an actionable training signal, where benchmark-aligned preference optimization measurably improves visual text generation. We release the benchmark at https://huggingface.co/datasets/anon-submission-7k2/vidscribe-benchmark.

Primary area:
datasets and benchmarks
Keywords:
video generation, visual text, text rendering, text editing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Cxu6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces VidScribe, a benchmark for evaluating visual text generation and editing in video across four regimes: text-to-video (T2V), reference-to-video (R2V), image-to-video (I2V), and video-to-video editing (V2V). It contains 803 human-verified samples organized around 12 conditionally orthogonal factor axes and uses 11 shared evaluation metrics plus two V2V-specific probes. The authors evaluate seven commercial video generators and several specialized open-source models, reporting three main findings: lexical correctness does not guarantee glyph fidelity, text preservation is easier than localized editing, and structural and temporal challenges can be more damaging than adverse imaging conditions.

Critical strengths
  1. The work is well motivated and provides valuable failure cases; The four-regime formulation is a useful conceptual contribution. Separating lexical accuracy from glyph structure, temporal stability, and localized editing quality is more informative than relying on OCR accuracy or generic video-quality scores alone.
  2. The benchmark combines deterministic vision tools, VLM-based rubric evaluation, dedicated typography classifiers, and human assessment. The reported metric-level correlations with human judgments and cross-evaluator analysis are encouraging.
Critical weaknesses
  1. Technical concerns: The factor axes are described as conditionally orthogonal, but this does not establish statistical independence or causal isolation. The evaluation suite also combines heterogeneous metrics whose validity and sensitivity to nuisance factors are not fully established.
  2. Experimental gaps: The 803-sample benchmark is relatively small for drawing broad conclusions about commercial video generators. The factor-sensitivity analysis relies on an assigned primary metric for each factor, making the conclusions dependent on the chosen mapping. Uncertainty estimates for the main model rankings and factor effects are insufficiently prominent.
Questions for authors
  1. The (a)(b)(c) in Figure 1 are too small and hard to distinguish at first glance;
  2. How do you empirically verify conditional orthogonality among the 12 factors after feasibility filtering? Can you report dependencies and interaction effects?
  3. Are all commercial systems evaluated with matched prompts, resolution, duration, and generation settings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Cxu6

[Q1]: The (a), (b), and (c) panels in Figure 1 are too small and difficult to distinguish at first glance.

[A1]: Thank you for the suggestion. We have enlarged the text, reference images, and key visual regions in Figure 1 of the revised manuscript to improve readability.

[Q2]: How do you empirically verify conditional orthogonality among the 12 factors after feasibility filtering? Can you report dependencies and interaction effects?

[A2]: “Conditional orthogonality” is a design principle used during dataset construction to distinguish different challenges and compare model performance within feasible combinations. Prior relationships in real-world scenes and feasibility constraints restrict factor combinations, so this principle does not imply that the final samples are entirely free of associations between factors. We use Cramér’s V to measure the strength of association between the values of two factors after filtering. It ranges from 0 to 1, with larger values indicating stronger associations; representative results are shown in the table below. After simultaneously controlling for task and the other 11 factors, seven of the eight factors ranked as most damaging in Table 3 retain significantly negative effects (as shown in Table 12 of the revised manuscript, p_FDR < 0.05). This further supports the principal damaging factors identified in Table 3 and indicates that the main conclusions about degradation are not driven solely by factor co-occurrence.

Factor pairPooled V
F7 Carrier Type × F12 Content Evolution0.441
F7 Carrier Type × F10 Motion Pattern0.308
F1 Text Amount × F2 Layout Hierarchy0.288
F8 Illumination × F9 Text-Background Separability0.286
F4 Typographic Style × F12 Content Evolution0.266
F3 String Type × F7 Carrier Type0.209
F10 Motion Pattern × F11 Foreground Occlusion0.193

[Q3]: Are all commercial systems evaluated with matched prompts, resolution, duration, and generation settings?

[A3]: Within the same task and scene, all commercial systems use the same prompt and task-required inputs, with 5 seconds as the target duration. For resolution, MiniMax H3 supports only 768p, while all other systems use 720p.

Regarding W2, “The 803-sample benchmark is relatively small for drawing broad conclusions about commercial video generators,” VidScribe provides 803 human-verified scenes covering four tasks, expanding the evaluation scale and task coverage of existing video text benchmarks. A brief comparison with prior work is provided in the table below. The final evaluation set was determined after feasibility checks, expert review, and validation of text and temporal conditions (see Appendices A.3–A.4). The entire construction and evaluation process cost approximately $13,000.

BenchmarkNumber of evaluation prompts/scenesTasks
T2VTextBench73T2V
VTE-Bench200V2V
VidScribe803T2V, R2V, I2V, V2V

Most of my concerns have been addressed, and I appreciate the authors' clarifications. I therefore maintain my original rating.

Thank you once again for the time and effort you have dedicated to reviewing our manuscript.

Best regards,

The VidScribe Authors

Reviewer_HUizOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VidScribe is a benchmark for visual text in video across four regimes: writing from a prompt, transferring from a reference image, sustaining text under motion, and editing text in a video. It has 803 human-verified samples over 12 factor axes, scored by 11 metrics and 2 editing probes built on tracked OCR and VLM arbitration. Eleven generators are evaluated, and the metrics also drive DPO.

Critical strengths
  1. The decoupling between content and glyph quality is a well-supported and non-obvious result.

  2. The evaluation is validated rather than asserted. Eight typography and video experts rated 950 videos with at least five raters each, giving Spearman correlations of 0.72–0.96 across all 11 metrics, and swapping the VLM arbiter to Gemini gives nearly identical values.

Critical weaknesses
  1. Finding 3 depends on how hard the authors made each factor, not only on which factors matter. Regular and Hard are author-assigned per axis with no cross-axis calibration of severity, so degradation magnitudes are not comparable across factors. The ordering also flips by metric choice: on assigned metrics, Carrier Type, an imaging factor, drops 18.9 points, against its last-place Overall rank of 0.7.

  2. The DPO result partly optimizes the metric it is scored on. Preference pairs are selected by VidScribe Overall and then evaluated by VidScribe metrics, so the +2.5 gain is in part expected. The user study helps, but win rates are 51.6–54.8% on three of four dimensions, near chance. The base model is also near floor (A1 16.8, B1 14.5), so transfer to stronger models is unclear.

  3. Although the evaluation was systematic, the author did not seem to offer any possible solutions or ideas behind this series of failures, nor did they provide any relevant experimental support. This limits the contribution of this paper.

Questions for authors
  1. How were Hard levels chosen per axis, and is severity comparable across axes? Would a severity-matched design, or reporting effect sizes only within an axis, change the ordering in Table 3 and Figure 1(d)?

  2. What is inter-rater agreement among the 8 experts? Correlation with their consensus does not establish that the consensus itself is reliable.

  3. Does the DPO model improve on an independent text-video metric not used to build the preference pairs?

  4. F3 String Type has a significantly positive effect on its assigned metric A2. Does this indicate a problem with the glyph inspector on non-semantic strings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer HUiz (Part 1/2)

[Q1]: How were Hard levels chosen per axis, and is severity comparable across axes? Would a severity-matched design, or reporting effect sizes only within an axis, change the ordering in Table 3 and Figure 1(d)?

[A1]: Hard is not a score assigned on a unified difficulty scale. Instead, it denotes stress conditions fixed for each axis before data construction, rather than inferred retrospectively from model scores. Our Hard levels are inspired by the text conditions examined in prior work and are set consistently by senior experts in writing systems: long text [1], hierarchical layouts [2], non-semantic strings [3], 3D geometry and illumination [4], dynamic glyphs [5], dense/small text in videos [6], and typing/scrolling [7]. A sample is labeled Hard whenever it exhibits the challenging condition targeted by that axis. Hard has meaning only within its own axis, with no unified scale across axes, so severity cannot be compared directly across different axes.

Table 3 and Figure 1(d) both show “how much worse Hard performs than Regular within the same axis,” rather than “which axis has a more severe Hard condition.” Reporting only within-axis effect sizes is already our current approach, so the ordering in Table 3 and Figure 1(d) remains unchanged. Under a severity-matched design, both the conditions being compared and the calculation method would change. The ordering would need to be recomputed and could not be guaranteed to remain the same.

References

[1] STRICT: Stress-Test of Rendering Image Containing Text. EMNLP 2025.

[2] DocHieNet: A Large and Diverse Dataset for Document Hierarchy Parsing. EMNLP 2024.

[3] Revisiting Scene Text Recognition: A Data Perspective. ICCV 2023.

[4] Synthetic Data for Text Localisation in Natural Images. CVPR 2016.

[5] Dynamic Typography: Bringing Text to Life via Video Diffusion Prior. ICCV 2025.

[6] DSText V2: A comprehensive video text spotting dataset for dense and small text. Pattern Recognition, 2024.

[7] T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Models. arXiv:2505.04946, 2025.

[Q2]: What is inter-rater agreement among the eight experts? Correlation with their consensus does not establish that the consensus itself is reliable.

[A2]: Both studies involved eight domain experts, and each video received at least five independent ratings, with model identities and other experts’ judgments hidden during rating.

Human studyNumber of videosKrippendorff’s αICC(2,k)
Main-text Table 5 (original Table 4)9500.810.86
Appendix Table 10 (original Table 7)5600.790.84

These results support the consistency of expert judgments. We will report inter-rater agreement separately from the correlations between metrics and expert consensus, and describe the inputs to each statistic, the treatment of categorical and ordinal scales, and the rater assignment and aggregation rules.

[Q3]: Does the DPO model improve on an independent text-video metric not used to build the preference pairs?

[A3]: We added an independent OCR evaluation based on PPOCR-v6, sampling frames from the baseline and DPO test videos using a fixed protocol and scoring them against the predefined target text, temporal phases, and matching rules. This evaluation was not used for preference construction, training, or model selection. The results show that the backbone after DPO training achieves a 3.1 pp decrease in character error rate (CER) and a 3.7 pp increase in full-string accuracy.

The mean VBench Imaging Quality and Overall Consistency scores improve by 1.4 and 0.6 pp, respectively, providing complementary evaluations of overall visual quality and text consistency.

MetricBaselineDPO
CER ↓57.1%54.0%
Full-string accuracy ↑19.5%23.2%
Imaging Quality ↑57.4%58.8%
Overall Consistency36.6%37.2%

In the human study, DPO achieves a 68.8% win rate in Text Fidelity comparisons; the full results across the four dimensions are as follows.

Evaluation dimensionDPO winsTiesBaseline winsDPO win rate
A: Text Fidelity172126668.8%
B: Temporal Stability137298454.8%
C: Instruction Compliance135239254.0%
D: Non-Text Preservation129319051.6%
AuthorsOpenReview ↗

Response to Reviewer HUiz (Part 2/2)

[Q4]: The DPO result partly optimizes the metric it is scored on (+2.5 gain), user study win rates are near chance (51.6–54.8%), and transfer from a weak base model to stronger models is unclear.

[A4]: The modest +2.5 gain and mild human preference win rates in the initial experiment were constrained by offline DPO sampling on a weak base model rather than limitations of our metric. To evaluate stronger optimization regimes, cross-metric generalization, and backbone scalability, we conducted three new experiments using DiffusionNFT with group_size=32 and batch_size=256 for 40 epochs on the VidScribe T2V subset, as shown in the table below:

ModelRewardΔA1ΔA2ΔB1ΔB2ΔC1ΔC2ΔC3ΔC4ΔC5ΔC6ΔD1Mean gain
Wan2.2-TI2V-5BAll+23.3+21.3+24.4+30.0+7.8+3.9+0.9−3.9−6.7+6.8+6.1+10.4
A1+B1+B2+C2+C3+7.5+8.1+7.0+12.5+9.3+2.9+5.4+1.3-0.7+5.0+2.2+5.5
Wan2.1-T2V-14BAll+19.5+24.9+19.1+13.1+6.5+6.5+17.0−2.4+4.1+0.5+3.6+10.2

Switching from offline DPO to online DiffusionNFT with the full reward on **Wan2.2-TI2V-5B** elevates the Mean Gain to +10.4, driving substantial improvements in previously near-floor dimensions with Δ\DeltaA1 at +23.3, Δ\DeltaB1 at +24.4, and Δ\DeltaB2 at +30.0. Furthermore, to verify that the model is not merely overfitting or hacking the optimized metrics, we trained Wan2.2-TI2V-5B using only a partial subset of metrics, specifically A1+B1+B2+C2+C3, as the reward signal. Remarkably, the unoptimized held-out metrics improved simultaneously, with Δ\DeltaA2 rising by +8.1, Δ\DeltaC1 by +9.3, Δ\DeltaC6 by +5.0, Δ\DeltaC4 by +1.3, and non-text preservation Δ\DeltaD1 by +2.2, achieving an overall Mean Gain of +5.5 and proving genuine cross-metric generalization rather than reward hacking. Finally, scaling the full reward to the much stronger **Wan2.1-T2V-14B** backbone yields a massive Mean Gain of +10.2, with consistent surges in Δ\DeltaA1 by +19.5, Δ\DeltaA2 by +24.9, Δ\DeltaB1 by +19.1, Δ\DeltaC3 by +17.0, and Δ\DeltaD1 by +3.6. Together, these results confirm the strong generalizability and effectiveness of VidScribe metrics as training signals across different optimization methods, subset objectives, and large-scale backbones, and we will open-source our training code upon acceptance.

[Q5]: F3 String Type has a significantly positive effect on its assigned metric A2. Does this indicate a problem with the glyph inspector on non-semantic strings?

[A5]: A2 examines glyph structure rather than string semantics or matching to the target content. Therefore, even incorrectly rendered non-semantic strings may consist of complete, valid glyphs and receive high A2 scores. The positive association in Figure 10 (the heatmap) of the current manuscript is consistent with the paper’s conclusion that “Content Accuracy and glyph quality are decoupled” and does not, by itself, constitute evidence that the glyph inspector fails.

In the existing validation, A2 has a Spearman correlation of 0.84 with human judgments and scores 100% on ideal synthetic videos. These results support its overall measurement performance, while misclassification rates across different categories remain a separate validation question. We will describe this finding as an association between String Type and Glyph Correctness observed in the current sample and discuss the possible influence of character-set and font composition.

Reviewer_asgAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VidScribe is a benchmark for visual text in video generation. It covers four regimes: T2V (write), R2V (transfer from a reference), I2V (sustain from a first frame), and V2V (edit). Its 803 human-verified samples span a 12-axis factor schema. A VLM-based OCR tracking pipeline with measurability gates scores 11 shared metrics and 2 V2V probes (editing residue and source-text preservation). Across 11 generators, the authors report three findings: lexical accuracy and glyph correctness decouple, I2V is easiest and V2V hardest, and temporal and structural factors hurt more than imaging conditions. DPO on Wan2.2-TI2V-5B with VidScribe-derived preference pairs adds 2.5 Overall points.

Critical strengths
  1. One factor schema across four regimes. The shared schema makes the regimes directly comparable. The V2V probes capture failures the shared metrics miss. PixVerse V6 and Wan 3.0 have similar residue (18.6 vs. 19.5), yet Wan preserves twice as much source text (69.5 vs. 33.1, Table 2).

  2. Broad evaluator validation. The paper reports human correlation on 950 videos (Spearman 0.72–0.96, Table 4), a synthetic sanity check (97.9 Overall, Table 8), a discriminator comparison justifying C2/C3 (Fig. 9), and a cross-backbone rerun.

  3. Careful factor statistics. The analysis uses task-balanced Cliff’s δ with bootstrap CIs, HC3 OLS adjustment for co-occurring factors, and BH-FDR. Null results are reported (F6 after adjustment; no threshold effect in C.5).

Critical weaknesses
  1. Some metrics penalize requested dynamics, which may inflate Finding 3. For T2V/R2V/I2V, D1 is the mean pairwise CLIP similarity across output frames (Eq. 12), so a correctly rendered passer-by or camera move lowers the score. F11 and F10 have their largest effects on D1 (−0.35 and −0.26, Fig. 10). Because D1 is the only Non-Text metric, it carries a quarter of Overall. B1 has a similar problem: it anchors on the modal reading (Eq. 4), so a correct content switch, typing reveal, or scroll (F12 Hard) produces nonzero error by design. As a result, the temporal-vs-imaging ranking cannot yet separate model failure from metric artifact.

  2. Comparative claims are confounded and lack uncertainty. Task means are computed over different sample sets. High Hard-axis counts concentrate in T2V and R2V (C.5), and I2V includes pre-screened real images, so the task asymmetry may partly reflect sample difficulty. In the matched comparison, the R2V prompt omits the target string while the T2V prompt keeps it (A.2). The “no spelling gain from a reference” result therefore compares text-in-prompt with text-in-image only. The top four systems span 1.4 points in Table 1 and 1.1 points in the human study (Table 7), and no confidence intervals are reported.

  3. The evaluator depends on one model family, and the DPO evidence is thin. Qwen3.7-Plus grounds OCR and judges most metrics. It comes from the same organization as Wan, which leads A1, B1, and B2. The cross-backbone check re-tuned prompts for each backbone and reports only radar plots. The DPO experiment selects and evaluates with the same pipeline on prompts from the same generator. It is a single run, and C1 and C3 decrease.

Questions for authors
  1. How do A1/B1 handle gt_text_switched and partial strings? Does the factor ranking (Tables 3/9) hold with time-indexed B1/B2 and without D1?
  2. Why drop the target string from the auxiliary R2V prompt? Please compare full R2V (text + reference) against matched T2V with paired CIs.
  3. How many generations per sample were drawn, and how were failed or refused API calls scored? Please add bootstrap CIs to Tables 1, 2, and 7.
  4. Can you provide a numeric Gemini-backbone table with identical prompts, plus inter-rater agreement for the human studies?
  5. Do the DPO gains hold under cross-backbone scoring, on free-form prompts, and across seeds? How many ties were there in Table 11?
4 comments
AuthorsOpenReview ↗

Response to Reviewer asgA (Part 1/4)

[Q1]: How do A1/B1 handle **gt_text_switched** and partial strings? Does the factor ranking (Tables 3/9) hold with time-indexed B1/B2 and without D1?

[A1]: A1, B1, and B2 already use time-indexed evaluation, with tracks partitioned according to predefined phases during grounding. For content_switch, the default transition occurs at the midpoint of the video, consistent with the “switch mid-clip” instruction in Figure 14. For progressive modes such as typing and scroll, prefix or visible-substring matching is used to evaluate partially displayed text. Appendix B.1 clarifies this procedure.

For Finding 3, we repeated the task-balanced and adjusted analyses after removing D1. Compared with the original Table 9 (Table 12 in the revised manuscript), the set of the Top 4 factors remains unchanged, with Content Evolution and Foreground Occlusion still ranked first and second. The Cliff’s δ estimates and their 95% confidence intervals for these two factors, as well as Layout Hierarchy and Motion Pattern, are all below zero. Under the predefined Hard/Regular levels, performance degradation remains concentrated in these structural and temporal factors.

Rank without D1FactorOriginal δδ without D1 [95% CI]
1F12 Content Evolution−0.347−0.330 [−0.416, −0.239]
2F11 Foreground Occlusion−0.245−0.200 [−0.338, −0.057]
4F2 Layout Hierarchy−0.208−0.191 [−0.308, −0.070]
3F10 Motion Pattern−0.193−0.198 [−0.286, −0.108]
5F1 Text Amount−0.154−0.158 [−0.257, −0.059]
6F6 Text Scale−0.139−0.113 [−0.229, −0.001]
7F5 Stroke Weight−0.119−0.092 [−0.175, −0.008]
9F4 Typographic Style−0.096−0.064 [−0.163, +0.035]
10F9 Text-Background Separability−0.071−0.035 [−0.142, +0.078]
8F8 Illumination−0.034−0.066 [−0.175, +0.045]
11F3 String Type−0.028−0.017 [−0.159, +0.140]
12F7 Carrier Type−0.028−0.010 [−0.101, +0.079]

[Q2]: Why drop the target string from the auxiliary R2V prompt? Please compare full R2V (text + reference) against matched T2V with paired confidence intervals.

[A2]: The main R2V setting provides text information through both the prompt and the reference image, making it impossible to determine whether the text in the output comes from “reading the prompt” or “reading the image.” Auxiliary R2V removes the target string from the prompt so that the text can only come from the reference image. It is then paired with matched T2V, which provides text only through language. The sole variable is thus the medium conveying the text information (visual versus linguistic), enabling the conclusion that “reference images improve glyph and typographic fidelity but not Content Accuracy.”

We added two paired comparisons on the same 113 scenes and three models, using 5,000 scene-level bootstrap resamples to compute 95% confidence intervals for the mean differences.

Consistent with Figure 4, auxiliary R2V shows no clear improvement in Content Accuracy (A1) over T2V: the difference is −0.1 percentage points (pp), with a 95% CI of [−5.1, +4.9]. Glyph Correctness (A2) on non-semantic strings, Stroke Weight Agreement (C2), and Typographic Style Matching (C3) improve by 13.4, 10.8, and 9.5 pp, respectively, with all three confidence intervals above zero. Full R2V improves A1 by 4.8 pp [+0.4, +9.1], while A2, C2, and C3 improve by 5.4, 11.2, and 11.3 pp, respectively. Both settings improve the fidelity of glyphs, stroke weight, and typographic style.

MetricT2VAuxiliary R2VFull R2VAuxiliary R2V − T2V [95% CI]Full R2V − T2V [95% CI]
A1 Content Accuracy79.279.184.0−0.1 [−5.1, +4.9]+4.8 [+0.4, +9.1]
A2 Glyph Correctness (All)58.165.663.5+7.6 [+3.4, +11.7]+5.4 [+1.0, +9.8]
A2 Glyph Correctness (Non-semantic subset)54.367.764.5+13.4 [+6.6, +20.4]+10.3 [+2.4, +18.3]
C2 Stroke Weight Agreement73.784.584.9+10.8 [+7.0, +14.5]+11.2 [+7.3, +15.0]
C3 Typographic Style Matching76.485.987.7+9.5 [+4.6, +14.6]+11.3 [+6.7, +16.2]
AuthorsOpenReview ↗

Response to Reviewer asgA (Part 2/4)

[Q3]: How many generations were drawn per sample, and how were failed or refused API calls scored? Please add bootstrap confidence intervals to Tables 1, 2, and 7.

[A3]: To ensure a strictly fair comparison, we filtered out any prompts that caused API failures or refusals on any evaluated commercial system during benchmark construction. Consequently, the final 803 samples contain zero failed calls, with one successful generation drawn per model-sample pair. Because a single run across the seven commercial systems already cost ~$9,000, we assessed multi-generation stability on a randomly selected 80-sample subset (matching the four-task distribution of the full set) using the top-3 performing models: MiniMax H3, Wan 3.0, and Seedance 2.5. For each model and sample combination, we generated two additional outputs and evaluated them together with the corresponding subset of initial VidScribe generations (denoted round1). Across generation rounds, the Overall ranking of these three models still tended to stabilize. The additional videos generated for this subset have been uploaded to https://huggingface.co/datasets/anon-submission-7k2/vidscribe-benchmark/tree/main/subset_multiround_results.

ModelRoundOverallText FidelityTemporal StabilityInstruction ComplianceNon-Text Preservation
MiniMax H3round180.370.581.877.191.8
round281.271.682.678.092.7
round379.369.081.175.991.2
Seedance 2.5round179.872.779.875.591.1
round279.673.478.574.592.0
round379.171.878.774.391.6
Wan 3.0round179.365.784.577.789.5
round279.466.985.177.088.4
round378.864.785.876.388.5

As requested, we have added the 95% bootstrap confidence intervals to Tables 1, 2, and 7 (presented below). These intervals were computed using 5,000 task-stratified, scene-level bootstrap resamples, which strictly preserve the original aggregation weights and the cross-model pairing within each scene.

Video generatorOverall95% CI
MiniMax H379.2[78.2, 80.3]
Wan 3.078.5[77.5, 79.4]
Seedance 2.578.1[76.9, 79.2]
Gemini Omni 1.1 Flash77.8[76.8, 78.9]
HappyHorse75.0[73.9, 76.1]
PixVerse V673.2[72.1, 74.3]
Kling3.0-Omni70.5[69.3, 71.8]
Video generatorT2VR2VI2VV2VE1E2
MiniMax H380.2 [78.5, 81.8]81.9 [79.8, 83.9]84.2 [82.5, 85.8]68.7 [66.3, 71.2]29.2 [21.2, 37.2]49.6 [43.8, 55.3]
Wan 3.079.6 [78.4, 80.8]80.4 [78.4, 82.3]81.8 [80.0, 83.5]70.3 [68.1, 72.6]19.5 [12.4, 26.5]69.5 [64.9, 73.8]
Seedance 2.580.2 [78.3, 82.0]80.9 [78.9, 83.0]83.5 [81.6, 85.2]64.9 [62.4, 67.5]34.5 [25.7, 43.4]62.7 [57.7, 67.5]
Gemini Omni 1.1 Flash80.9 [79.5, 82.3]76.3 [73.7, 78.9]81.6 [79.8, 83.4]69.3 [67.0, 71.7]28.3 [20.4, 37.2]68.3 [64.2, 72.5]
HappyHorse76.6 [74.8, 78.2]78.3 [76.1, 80.6]80.4 [78.6, 82.0]62.3 [60.1, 64.6]29.2 [20.4, 37.2]38.5 [33.0, 43.7]
PixVerse V671.0 [69.1, 72.8]75.4 [73.4, 77.5]80.5 [78.5, 82.3]64.6 [62.2, 67.1]18.6 [11.5, 25.7]33.1 [28.1, 38.6]
Kling3.0-Omni65.9 [64.1, 67.7]71.1 [68.8, 73.5]83.5 [81.9, 85.2]58.9 [56.4, 61.3]39.8 [31.0, 48.7]55.0 [49.5, 60.2]
Video generatorHuman Overall95% CI
Wan 3.080.4[76.4, 84.0]
MiniMax H380.1[76.2, 83.7]
Seedance 2.579.5[76.0, 82.7]
Gemini Omni 1.1 Flash79.3[76.8, 81.8]
HappyHorse77.6[73.1, 81.8]
PixVerse V674.3[70.8, 77.6]
Kling3.0-Omni73.9[69.2, 78.0]
AuthorsOpenReview ↗

Response to Reviewer asgA (Part 3/4)

[Q4]: Can you provide a numeric Gemini-backbone table with identical prompts, plus inter-rater agreement for the human studies?

[A4]: Evaluation with Gemini-3.8-Flash keeps the metric definitions, score mappings, aggregation weights, and expert vision modules unchanged. Prompt adaptation involves only adjustments to formatting and option presentation required by Gemini’s JSON output and instruction format. The scoring rubrics and thresholds are identical.

Video generatorOverallA1A2B1B2C1C2C3C4C5C6D1
MiniMax H378.080.153.873.183.279.974.175.077.870.680.590.6
Wan 3.077.280.842.874.985.580.775.379.179.873.380.988.6
Gemini Omni 1.1 Flash76.777.547.072.384.079.272.976.175.773.780.689.9
Seedance 2.576.675.454.870.682.880.572.770.177.367.377.690.5
HappyHorse73.670.145.664.882.080.172.569.776.067.479.189.0
PixVerse V671.869.544.960.977.879.373.774.871.861.979.887.1
Kling3.0-Omni69.255.241.052.280.177.673.566.875.165.879.789.4

The Overall rankings under Gemini-3.8-Flash and Qwen3.7-Plus have a Spearman correlation of ρ = +0.964, with only the third- and fourth-ranked systems exchanging positions and a very small score gap between them. Regarding the concern in W3 that Qwen and Wan are from the same provider, Wan 3.0 still ranks first on A1, B1, and B2 under Gemini evaluation.

Both human studies involved eight domain experts, and each video received at least five independent ratings. The agreement results below support the consistency of expert judgments used for metric-level validation and overall system comparison.

Human studyNumber of videosKrippendorff’s αICC(2,k)
Main-text Table 5 (original Table 4)9500.810.86
Appendix Table 10 (original Table 7)5600.790.84

[Q5]: Concerns regarding single-family evaluator dependency and potential same-organization bias (since Qwen3.7-Plus and the top-performing Wan 3.0 are from the same provider).

[A5]: We selected Qwen3.7-Plus as our primary evaluator solely for its superior visual grounding and OCR capabilities, which are critical in practical deployment for minimizing evaluation noise from missed detections or localization errors. However, to thoroughly address the concern regarding potential intra-organization bias toward Wan 3.0, we re-evaluated the entire benchmark using an independent, third-party VLM from a different provider: DeepSeek-V4.1-Flash-1017. The full evaluation results are presented below:

Video generatorOverall ↑Text Fidelity ↑Temporal Stability ↑Instruction Compliance ↑Non-Text Preservation ↑
MiniMax H379.167.678.380.390.1
Seedance 2.577.764.876.479.390.5
Wan 3.077.660.679.482.188.4
Gemini Omni 1.1 Flash77.360.378.080.989.9
HappyHorse75.258.474.179.489.0
PixVerse V673.458.069.479.087.1
Kling3.0-Omni70.147.966.077.089.5

As shown, the evaluation results demonstrate exceptional cross-provider consistency. The Overall rankings between DeepSeek-V4.1-Flash-1017 and Qwen3.7-Plus achieve a near-perfect Spearman correlation of ρ = +0.964. Specifically, the rank correlations for Text Fidelity, Temporal Stability, Instruction Compliance, and Non-Text Preservation are +0.964, +1.000, +0.964, and +1.000, respectively. Notably, the rankings for Temporal Stability and Non-Text Preservation are 100% identical, and Wan 3.0 consistently retains its 1st place in both Temporal Stability and Instruction Compliance even under the third-party DeepSeek judge. This confirms that our core ranking conclusions are robust across different organizations and that Wan 3.0's leading performance is not an artifact of evaluator bias. We have uploaded the comparative evaluation logs for both judges to the anonymous link: https://huggingface.co/datasets/anon-submission-7k2/vidscribe-benchmark.

AuthorsOpenReview ↗

Response to Reviewer asgA (Part 4/4)

[Q6]: Do the DPO gains hold under cross-backbone scoring, on free-form prompts, and across seeds? How many ties were there in Table 11?

[A6]: Table 13 (original Table 10) reports an average Overall gain of 2.5 pp across eight generation seeds for the trained DPO model. Rescoring with Gemini-3.8-Flash gives a gain of 2.7%, with the direction of change for every metric consistent with that under Qwen3.7-Plus evaluation.

After rewriting the existing test prompts into free-form language using DeepSeek-V4.1-Flash-1017, DPO still maintains an average Overall gain of 1.8% over the baseline.

Evaluation settingOverall gain
Original test prompts, Qwen3.7-Plus evaluation, average across eight generation seeds+2.5% (±0.28)
Gemini-3.8-Flash cross-backbone evaluation+2.7% (±0.34)
Free-form rewrites of the existing test prompts+1.8% (±0.19)

After submission, we added further experiments using the metrics as training signals, extending their use to other training methods and a larger backbone. Specifically, we used DiffusionNFT to train Wan2.2-TI2V-5B and Wan2.1-T2V-14B with the Overall signal, setting group_size=32 and batch_size=256 and training for 40 epochs. The gains on the VidScribe T2V subset are shown in the table below:

ModelΔA1ΔA2ΔB1ΔB2ΔC1ΔC2ΔC3ΔC4ΔC5ΔC6ΔD1Mean gain
Wan2.2-TI2V-5B+23.3+21.3+24.4+30.0+7.8+3.9+0.9−3.9−6.7+6.8+6.1+10.4
Wan2.1-T2V-14B+19.5+24.9+19.1+13.1+6.5+6.5+17.0−2.4+4.1+0.5+3.6+10.2

The results demonstrate the general applicability of the Overall training signal across different training methods and backbones. We will release our training code as open source after the paper is accepted.

The human-study counts for Table 14 (original Table 11) are as follows:

Evaluation dimensionDPO winsTiesBaseline winsDPO win rate
A: Text Fidelity172126668.8%
B: Temporal Stability137298454.8%
C: Instruction Compliance135239254.0%
D: Non-Text Preservation129319051.6%

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all the reviewers for their positive feedback and constructive suggestions. We are encouraged that the reviewers found our four-regime formulation over a shared 12-axis factor schema well-motivated and directly comparable (asgA, Cxu6), noted that our hybrid evaluation suite and V2V-specific probes capture valuable failure modes and the non-obvious decoupling between lexical accuracy and glyph correctness (asgA, HUiz, Cxu6), and highlighted our careful factor statistics, dedicated typography classifiers, cross-backbone analysis, and 950-video expert validation as convincing empirical support (asgA, HUiz, Cxu6).

Below, we briefly summarize our responses to shared concerns and list the revisions in the updated PDF and supplementary material.

Shared concerns:

Evaluation reliability (asgA, HUiz, Cxu6). We provide numeric cross-backbone evaluations with Gemini-3.8-Flash and DeepSeek-V4.1-Flash-1017 under identical scoring rubrics, separate multi-expert inter-rater agreement for both human studies, task-stratified bootstrap confidence intervals across all main and task-wise tables, and multi-round repeated-generation results under matched commercial generation settings. These complementary checks rule out single-provider evaluator bias and confirm the stability of both system rankings and expert consensus.

Benchmark design and interpretation (asgA, HUiz, Cxu6). We clarify how time-indexed phase partitioning in A1/B1/B2 accommodates requested content switches and progressive reveals, verify that our factor sensitivity rankings hold when excluding background consistency D1, and explain why literature-grounded Regular/Hard definitions measure within-axis stress rather than cross-axis severity alongside the decoupling of glyph topology from string semantics on F3. We further quantify post-filtering pairwise factor dependencies via Cramér’s V, confirm that the principal damaging factors remain statistically significant after joint adjustment for task and co-occurring axes, contextualize our 803-scene four-regime coverage against prior benchmarks, and expand the matched T2V–R2V analysis to compare both text-free auxiliary R2V and full R2V against T2V with paired confidence intervals. Together, these analyses rule out metric and co-occurrence artifacts while making the interpretation of factor rankings, task asymmetry, and visual versus linguistic reference benefits much more precise.

Utility as a training signal (asgA, HUiz). To show that preference alignment gains extend beyond the training evaluator and a single run, we validate the DPO-tuned model using an independent third-party OCR pipeline (PPOCR-v6) and VBench metrics, cross-backbone Gemini rescoring, free-form prompt rewrites, multiple generation seeds, and complete user-study tie breakdowns. Additional DiffusionNFT experiments further extend these improvements to a second optimization paradigm across both 5B and 14B video generation backbones.

Main revisions in the uploaded PDF (changes in blue)

  • Figure 1: improved readability of the motivating examples (Cxu6).

  • Appendices A.1–A.2 and B.1, Table 7: clarified difficulty definitions and time-indexed evaluation, and added paired comparisons for auxiliary and full R2V (asgA, HUiz).

  • Section 4.3: added human inter-rater agreement and DeepSeek-V4.1-Flash-1017 validation; Appendix B.2 (Table 9): added numerical Gemini-3.8-Flash evaluation results (asgA, HUiz).

  • Section 4.2, Table 4, and Figure 6: expanded training-signal experiments with DiffusionNFT across model scales (asgA, HUiz).

Additional robustness results are reported in our detailed responses, and the repeated-generation videos are available in the supplementary dataset. We will release the complete VidScribe benchmark, evaluation pipeline, and post-training code upon acceptance, providing a comprehensive baseline to support future research in the community.

Our detailed response to each reviewer is posted under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.