acceptodds
Withdrawn from ICLR 2027

Video Text Understanding Across Diverse Scenes: An Evidence-Guided Benchmark

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video text understanding involves interpreting on-screen text together with its visual context. Audits of existing benchmarks have identified questions answerable from a single frame or by text extraction alone, motivating broader assessment of this joint understanding. We introduce an evidence-guided benchmark for video text understanding across diverse scenes. Its 16,165 evaluation items comprise multiple-choice questions and true/false judgments over 2,629 videos, organized into 10 primary scene categories and 34 fine-grained subcategories. The construction workflow plans structured evidence before generating questions, followed by automated verification, text-only filtering, and human correction. We evaluate 26 open-weight multimodal checkpoints spanning 1B–235B parameters. On the matched multiple-choice roster, mean accuracy is 55.15% with video versus 41.49% with questions and options alone. Paired judgments reveal difficulty both accepting supported statements and rejecting their counterfactual counterparts, while performance varies across scenes. We further manually review a subset of 500 base questions and their associated variants for controlled evidence diagnostics: at equal frame counts, sampling distinct evidence windows outperforms repeating one window center on average. The benchmark provides a testbed for evaluating joint text–visual understanding and diagnosing remaining model weaknesses.

Primary area:
datasets and benchmarks
Keywords:
Video Text Understanding, Multimodal Large Language Models, Video Question Answering, Evidence-Guided Benchmark, Temporal Evidence

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_KwmMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for video text understanding that evaluates whether models can interpret on-screen text together with its visual context, beyond plain text extraction. It contains about 16K multiple-choice and paired true/false items across diverse scenes. The items are generated by an LLM pipeline from structured evidence records, then filtered for text-only shortcuts and corrected by humans. Experiments on 26 open-weight models show substantial remaining errors, a clear bias toward accepting statements over rejecting counterfactuals, and notable performance variation across scenes.

Critical strengths
  1. Well-motivated problem. Existing video text benchmarks are largely answerable from a single frame or by text extraction alone. Targeting the joint understanding of on-screen text and visual context is a relevant direction for evaluating video LLMs.

  2. Useful paired judgment design. Each supported statement is paired with a minimally edited counterfactual, and a pair counts as correct only when both are judged correctly. This exposes a strong bias toward accepting statements that a single aggregate accuracy would hide.

  3. Careful and transparent construction. Models involved in data construction are excluded from evaluation, text-only baselines are reported for every model, and confidence intervals are clustered by source video. Each item also keeps traceable evidence and provenance records, which makes the benchmark easy to audit.

Critical weaknesses
  1. The motivation is not empirically established. The claim that existing video text benchmarks mainly reward single-frame reading or text extraction rests on a prior audit of two older datasets (M4-ViteVQA and NewsVideoQA). The paper does not test more recent benchmarks such as VidText, MME-VideoOCR, or ViTexQA, nor does it compare them with the proposed benchmark under the same models and protocol, as Appendix H acknowledges. It is therefore unclear what this benchmark measures that existing ones do not.

  2. Text-only shortcuts remain large and grow with model scale. Filtering relies on only three small text-only selectors, yet strong models still reach around 60% on four-option MCQA without video (e.g., InternVL3-78B at 61.69%). Text-only accuracy also correlates with parameter scale, so much of the benchmark may become solvable without visual input as models improve.

  3. The “beyond text extraction” claim is not directly verified. Key baselines are missing, including OCR transcripts fed to an LLM, single-frame input, and caption-only input. Many counterfactuals change only a single number or word, which mainly tests precise reading rather than joint text–visual reasoning.

  4. The evaluation setup is incomplete. There is no human baseline or inter-annotator agreement, and no proprietary frontier models are evaluated, so both headroom and label quality remain unclear. Each model also uses its own frame sampling and resolution, so the leaderboard mixes input budget with model capability.

  5. Limited novelty and diagnostic depth. Most source questions are re-annotations of existing datasets, mainly VidText, and the core MCQA set has only about 2K items with an imbalanced scene distribution. Results are not broken down by the proposed ability taxonomy, and the frame-intervention study shows only small effects, which limits the benchmark’s diagnostic value.

Questions for authors
  1. Can the authors evaluate existing video text benchmarks (e.g., VidText, MME-VideoOCR) with the same models and protocol, reporting text-only and single-frame accuracy, to show empirically that they suffer from the claimed issues and that the proposed benchmark addresses them?

  2. How does an OCR-transcript + LLM baseline perform? If it solves a large fraction of items, how do the authors justify that the benchmark goes beyond text extraction?

  3. What is the text-only accuracy when stronger text-only models (e.g., 70B+ LLMs) are used? How many items remain unsolved by all text-only models, and how do video models perform on this subset?

  4. Can the authors report human accuracy and inter-annotator agreement on a random sample, along with results for proprietary models such as GPT or Gemini?

  5. How do the results break down across the F1–F6 ability categories, and do model rankings stay stable under a unified frame-sampling and resolution setting?

Reviewer_GFsGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a benchmark for interpreting on-screen text together with its video context. It has 2,023 four-option multiple-choice questions and 7,071 pairs of supported and minimally altered statements, 16,165 items in total, derived from 7,757 source questions over 2,629 videos in 10 scene categories. Construction plans structured evidence (visible text, events, windows, answer slots), generates questions with Qwen3-VL, verifies with Qwen3.6-27B, filters with three small text-only models, and applies human correction. The evaluation covers 26 open-weight checkpoints with and without video, scene breakdowns, and a Pair metric that requires accepting the supported statement and rejecting its counterpart. A 500-source subset that was fully human-reviewed supports four input interventions on 11 pipeline-external models.

Critical strengths
  1. Linking a multiple-choice question to independently presented supported and altered statements exposes behavior that a single accuracy hides. Sections 3.3, 4.4 and 5.2 show that under video input the 23-model average is 78.96% on supported statements but 40.05% on their altered counterparts, and that video raises item accuracy from 50.72% to 59.50% while Pair rises from 3.91% to 20.65%. The text-only Pair of 3.91% against an item accuracy near 50% shows concretely how item averages hide accept-everything behavior. The shared source ID and the examples in Figures 7, 12 and 13 make this analysis inspectable.

  2. The 500-source study separates what it can claim from what it cannot. Section 4.3 keeps the frame count fixed for the distinct-versus-repeated contrast (C1, +1.9 MCQA) and states that the complete-versus-selected and complete-versus-drop contrasts (C3, C4) change the frame budget. The reversal contrast (C2, +0.3 / +0.1 / +0.2) is reported as showing no order sensitivity rather than explained away, and the F labels are described as design intent, not certified evidence requirements (Table 1). This calibration is why the empirical statements in the paper can be taken at face value.

Critical weaknesses
  1. The benchmark is positioned as testing joint text-visual understanding, but no control separates that from reading and aggregating on-screen strings. The introduction motivates the work with an audit showing that earlier benchmarks were 92%/95% single-frame answerable and 81%/98% extractive (Jahagirdar et al., 2023), yet the paper reports neither statistic for its own questions. The only non-video control (Section 5.1) gives question and options without any video text, so the 13.66-point video gain shows that video content helps, not that non-text scene content is needed. The altered statements shown in Figure 13 and Appendix K change a string that is rendered on screen (10 → 12 reps; 3rd → 4th quarter), and judgments are 14,142 of 16,165 items. The 500-source results point the same way for questions drawn from the multi-window cross-frame pool: giving a model only the single selected two-frame window comes within 3.0 MCQA points and 4.0 Pair points of the complete evidence input (C3), although these sources have two to eight evidence windows. The authors acknowledge that C3 changes the frame budget and that the F labels do not certify necessity, which is why this remains an open question and not a contradiction. A direct control on the reviewed 500 sources would resolve this: run an off-the-shelf OCR over all frames that the evaluated models receive, give a text-only model the resulting strings with their timestamps but no images, and report its accuracy. The OCR should cover the full input rather than only the evidence windows, since restricting it to the evidence windows would also tell the model where the relevant text is. The paper could also report the fraction of items answerable from a single frame or by text extraction, the same audit it cites against prior work. High accuracy under this control would indicate that the items mainly test locating and reading on-screen text, rather than the "joint interpretation with visual entities, states, and events" the introduction states as the focus.

  2. Label validity for the full release is documented as a procedure, not as outcomes, and the text-only filter may preferentially retain items whose labels are less clear. Section 3.5 states that no human-performance baseline and no inter-annotator agreement are reported, and Table 8 lists the types of checks without the number of sources inspected, variants corrected or records excluded for the 7,757-source release. Only the 500-source subset is described as completely reviewed. The leaderboard and the Pair findings are computed on the larger bank, where an altered statement that is accidentally still true, or a multiple-choice item with two defensible options, directly changes Pair. Section 3.3 retains only items that at most one (MCQA) or two (judgment) of three small selectors answer correctly, keeping 7,071 of the 10,598 valid pairs (66.72%). Adversarial filtering of this kind is known to over-select examples with low annotator agreement and to make model rankings depend on the choice of adversary (Phang et al., "Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair", 2022). Please report the review and correction counts separately for the release and for the diagnostic subset, and compare the human-flagged defect rate for items retained with zero versus one selector vote on a random sample. I am not assuming that unreviewed records are defective.

Questions for authors
  1. On the reviewed 500 sources, what accuracy does a text-only model reach when it receives OCR output, with timestamps, of all frames the evaluated models see, but no images? What fraction of sources are answerable from a single frame or from extracted text alone?
  2. How many of the 7,757 released source questions and their variants were inspected by a human, how many were corrected, and how many were removed? Does Table 8 describe exhaustive review of the release or a procedure whose fully reviewed instance is the 500-source subset?
Reviewer_HHhWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a video-text QA benchmark built through evidence-first question generation and human review. It evaluates 26 models using multiple-choice questions, paired true/false statements, and video-frame controls. 2027 - Video Text Understanding…

Critical strengths
  1. Testing both a correct statement and a slightly altered incorrect statement exposes models that accept both. Ordinary accuracy can hide this problem.
  2. The evaluation covers 26 models and reports performance across different scenes rather than only an overall score.
Critical weaknesses
  1. The advantage over existing benchmarks is not shown. MME-VideoOCR and ViTexQA already test cross-frame video-text understanding. The paper describes a different way to build questions, but does not show why the resulting benchmark is a better or meaningfully different test.
  2. Reading the text alone may be enough. The paper does not test whether models can answer using only text extracted from the video. Its “text-only” baseline sees only questions and options, so it does not check this.
  3. A single frame may still be enough. The paper does not report how many questions actually need multiple frames. Testing one selected frame is insufficient: another single frame may already contain the answer.
  4. The benefit of evidence-first generation is not tested. There is no comparison with direct question generation using the same filtering and human review. We therefore do not know whether planning evidence first improves the questions.
  5. Question and label quality is not measured.The paper says humans checked the questions, but reports no human accuracy, annotator agreement, or remaining error rate. Describing the checks does not show how reliable the final dataset is.
Questions for authors

1.What fraction of questions can be answered from extracted text alone or a single frame? 2.Does evidence-first generation produce better questions than direct generation under the same checks? 3. And see above.

Reviewer_W7MfOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a novel dataset for video-text understanding with linked multiple choice and judgement (true/false) items. It does this via an "evidence-structured" workflow on existing datasets (primarily VidText). The workflow consists of evidence structured generation (using Qwen3-VL). The initial set of open-ended questions is filtered for accuracy, and then used to create MCQ-style and true/false style statements. During the generation stage questions are also separated into 7 "ability target" categories, which guide the question generation. Following this questions are filtered using 3 text-only selectors. If more than 1 selectors correctly answer a question using text-only then the question is dropped. If all 3 selectors get the true/false statements correct it is dropped. These questions are then reviewed by humans and a 500-source or questions (this is not clear from the paper) set is created to evaluate evidence diagnostics. The paper also provides evaluation metrics Pair and AllClosed. Lastly 26 open-weight VL models across scales are used to evaluate performance on the dataset.

Critical strengths
  1. The paper has a clear focus on correctness of evidence for the QA pairs as well as a focus on producing a benchmark that is not easily solved or trivially answerable without the original video sources. Thought is also given to the construction of evaluation metrics to ensure models cannot trivially answer questions based on text only. Additionally the human evaluation strengthens the claims of correctness for the dataset.

  2. The paper conducts evaluation across many existing open-weight VLMs. Performance by VLMs on each question modality is well quantified, and the text-only scores help evaluate the utility of the new dataset.

Critical weaknesses
  1. In its current form the paper is extremely difficult to follow and vague/confusing in several places. This is especially true of Section 3 which makes it difficult to understand how exactly the dataset was generated. It is not clear how the source QA were used by the generator to "structure video evidence". Several terms such as "format variants", "semantic slot" and "evidence-diagnostic sources" are not clearly defined, or defined only using examples/in text in figures. Similarly the importance of scene and ability categories is not clear to me. The abstract mentions "500 base questions" but the paper later mentions "500-source subset", "500 source evidence diagnostic", "500 evidence-diagnostic sources". It is not clear to me if there was human intervention in the actual benchmark released and if it is the same as the 500 mentioned throughout the paper.

  2. A key motivation of the paper is that existing datasets are answerable from single frames or text extraction alone as mentioned in the abstract. However the paper does not provide extensive comparisons of the performance of the 26 VLMs on existing datasets. It appears the text-only MCQA accuracy on the dataset is always >25% (and average roughly 40%) which would be the expected value if it was just chance, therefore it is unclear if the original goal was achieved. Moreover it seems logical that an MCQ or simple true/false question is more likely to be susceptible to exactly this failure mode, and the choice of these QA modalities is not sufficiently qualified.

Questions for authors
  1. The role of the 500 subset is hard to determine. It is called the "experimental benchmark" and shown in Figure 2 as the final step, but it appears to only be used in Section 4.2? If this is the case then how many QA pairs in the actual benchmark were human reviewed and excluded?

  2. How do these 26 models perform on existing QA benchmarks? It would be helpful to have some sort of direct comparison to justify the creation of the new benchmark from existing sources.

  3. Although text-only accuracy is lower than video accuracy for MCQA on all models it is >25% (and average performance is roughly 40%) which suggests the core problem is not well resolved. It seems intuitive that models would have lower text-only accuracy for open ended questions rather than MCQs or true/false statements. It would be useful to compare performance of the MCQ/TF questions versus the open-ended questions they were generated from.

  4. Section 4.3 is very unclear to me, would the authors please clarify its relevance to the robustness of the benchmark? Why were 11 models used for this section instead of the full 26? Doesn't the small drop in accuracy across conditions suggest that most questions only need a few windows?

  5. It would be useful to have some further analysis of failure modes in the main paper (rather than in the supplementary material), especially judgement pairs. Some explanation of why existing models have low scores on this particular metric would enhance the paper. Per category (for F0-6) would also help since they are presented as an important part of dataset construction.

  6. It would be helpful to know how the generator/filtering models were prompted for the dataset construction stage.

  7. The exact set of models used for evaluation is unclear. The abstract says 26 models are used but the paper reveals: "MCQA covers 26 video models; its text–video comparison uses the same 24 matched models and 2,023 items. Judgment uses 23 models with complete text/video predictions on 7,071 pairs". Please clarify the final number of models used for evaluation and also why some models are left out on the judgement modality?

My Clear Rejection recommendation is mainly due to poor presentation and lack of clarity about the effectiveness of the proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.