- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark for video text understanding that evaluates whether models can interpret on-screen text together with its visual context, beyond plain text extraction. It contains about 16K multiple-choice and paired true/false items across diverse scenes. The items are generated by an LLM pipeline from structured evidence records, then filtered for text-only shortcuts and corrected by humans. Experiments on 26 open-weight models show substantial remaining errors, a clear bias toward accepting statements over rejecting counterfactuals, and notable performance variation across scenes.
Critical strengths
-
Well-motivated problem. Existing video text benchmarks are largely answerable from a single frame or by text extraction alone. Targeting the joint understanding of on-screen text and visual context is a relevant direction for evaluating video LLMs.
-
Useful paired judgment design. Each supported statement is paired with a minimally edited counterfactual, and a pair counts as correct only when both are judged correctly. This exposes a strong bias toward accepting statements that a single aggregate accuracy would hide.
-
Careful and transparent construction. Models involved in data construction are excluded from evaluation, text-only baselines are reported for every model, and confidence intervals are clustered by source video. Each item also keeps traceable evidence and provenance records, which makes the benchmark easy to audit.
Critical weaknesses
-
The motivation is not empirically established. The claim that existing video text benchmarks mainly reward single-frame reading or text extraction rests on a prior audit of two older datasets (M4-ViteVQA and NewsVideoQA). The paper does not test more recent benchmarks such as VidText, MME-VideoOCR, or ViTexQA, nor does it compare them with the proposed benchmark under the same models and protocol, as Appendix H acknowledges. It is therefore unclear what this benchmark measures that existing ones do not.
-
Text-only shortcuts remain large and grow with model scale. Filtering relies on only three small text-only selectors, yet strong models still reach around 60% on four-option MCQA without video (e.g., InternVL3-78B at 61.69%). Text-only accuracy also correlates with parameter scale, so much of the benchmark may become solvable without visual input as models improve.
-
The “beyond text extraction” claim is not directly verified. Key baselines are missing, including OCR transcripts fed to an LLM, single-frame input, and caption-only input. Many counterfactuals change only a single number or word, which mainly tests precise reading rather than joint text–visual reasoning.
-
The evaluation setup is incomplete. There is no human baseline or inter-annotator agreement, and no proprietary frontier models are evaluated, so both headroom and label quality remain unclear. Each model also uses its own frame sampling and resolution, so the leaderboard mixes input budget with model capability.
-
Limited novelty and diagnostic depth. Most source questions are re-annotations of existing datasets, mainly VidText, and the core MCQA set has only about 2K items with an imbalanced scene distribution. Results are not broken down by the proposed ability taxonomy, and the frame-intervention study shows only small effects, which limits the benchmark’s diagnostic value.
Questions for authors
-
Can the authors evaluate existing video text benchmarks (e.g., VidText, MME-VideoOCR) with the same models and protocol, reporting text-only and single-frame accuracy, to show empirically that they suffer from the claimed issues and that the proposed benchmark addresses them?
-
How does an OCR-transcript + LLM baseline perform? If it solves a large fraction of items, how do the authors justify that the benchmark goes beyond text extraction?
-
What is the text-only accuracy when stronger text-only models (e.g., 70B+ LLMs) are used? How many items remain unsolved by all text-only models, and how do video models perform on this subset?
-
Can the authors report human accuracy and inter-annotator agreement on a random sample, along with results for proprietary models such as GPT or Gemini?
-
How do the results break down across the F1–F6 ability categories, and do model rankings stay stable under a unified frame-sampling and resolution setting?