- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces VidScribe, a benchmark for evaluating visual text generation and editing in video across four regimes: text-to-video (T2V), reference-to-video (R2V), image-to-video (I2V), and video-to-video editing (V2V). It contains 803 human-verified samples organized around 12 conditionally orthogonal factor axes and uses 11 shared evaluation metrics plus two V2V-specific probes. The authors evaluate seven commercial video generators and several specialized open-source models, reporting three main findings: lexical correctness does not guarantee glyph fidelity, text preservation is easier than localized editing, and structural and temporal challenges can be more damaging than adverse imaging conditions.
Critical strengths
- The work is well motivated and provides valuable failure cases; The four-regime formulation is a useful conceptual contribution. Separating lexical accuracy from glyph structure, temporal stability, and localized editing quality is more informative than relying on OCR accuracy or generic video-quality scores alone.
- The benchmark combines deterministic vision tools, VLM-based rubric evaluation, dedicated typography classifiers, and human assessment. The reported metric-level correlations with human judgments and cross-evaluator analysis are encouraging.
Critical weaknesses
- Technical concerns: The factor axes are described as conditionally orthogonal, but this does not establish statistical independence or causal isolation. The evaluation suite also combines heterogeneous metrics whose validity and sensitivity to nuisance factors are not fully established.
- Experimental gaps: The 803-sample benchmark is relatively small for drawing broad conclusions about commercial video generators. The factor-sensitivity analysis relies on an assigned primary metric for each factor, making the conclusions dependent on the chosen mapping. Uncertainty estimates for the main model rankings and factor effects are insufficiently prominent.
Questions for authors
- The (a)(b)(c) in Figure 1 are too small and hard to distinguish at first glance;
- How do you empirically verify conditional orthogonality among the 12 factors after feasibility filtering? Can you report dependencies and interaction effects?
- Are all commercial systems evaluated with matched prompts, resolution, duration, and generation settings?
3 comments
Most of my concerns have been addressed, and I appreciate the authors' clarifications. I therefore maintain my original rating.
Thank you once again for the time and effort you have dedicated to reviewing our manuscript.
Best regards,
The VidScribe Authors
Response to Reviewer Cxu6
[Q1]: The (a), (b), and (c) panels in Figure 1 are too small and difficult to distinguish at first glance.
[A1]: Thank you for the suggestion. We have enlarged the text, reference images, and key visual regions in Figure 1 of the revised manuscript to improve readability.
[Q2]: How do you empirically verify conditional orthogonality among the 12 factors after feasibility filtering? Can you report dependencies and interaction effects?
[A2]: “Conditional orthogonality” is a design principle used during dataset construction to distinguish different challenges and compare model performance within feasible combinations. Prior relationships in real-world scenes and feasibility constraints restrict factor combinations, so this principle does not imply that the final samples are entirely free of associations between factors. We use Cramér’s V to measure the strength of association between the values of two factors after filtering. It ranges from 0 to 1, with larger values indicating stronger associations; representative results are shown in the table below. After simultaneously controlling for task and the other 11 factors, seven of the eight factors ranked as most damaging in Table 3 retain significantly negative effects (as shown in Table 12 of the revised manuscript, p_FDR < 0.05). This further supports the principal damaging factors identified in Table 3 and indicates that the main conclusions about degradation are not driven solely by factor co-occurrence.
[Q3]: Are all commercial systems evaluated with matched prompts, resolution, duration, and generation settings?
[A3]: Within the same task and scene, all commercial systems use the same prompt and task-required inputs, with 5 seconds as the target duration. For resolution, MiniMax H3 supports only 768p, while all other systems use 720p.
Regarding W2, “The 803-sample benchmark is relatively small for drawing broad conclusions about commercial video generators,” VidScribe provides 803 human-verified scenes covering four tasks, expanding the evaluation scale and task coverage of existing video text benchmarks. A brief comparison with prior work is provided in the table below. The final evaluation set was determined after feasibility checks, expert review, and validation of text and temporal conditions (see Appendices A.3–A.4). The entire construction and evaluation process cost approximately $13,000.