- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AVTextBench, a benchmark for evaluating whether audio-visual large language models can ground on-screen text in its surrounding audio-visual context. The benchmark organizes 2,541 video-question pairs into three proposed relationship types: (i) corresponding, which tests subtitle-speech alignment, mismatch type, and mismatch localization; (ii) complementary, which tests bidirectional grounding between transient speaker-name or step-marker text and audio-visual events; and (iii) independent, which tests whether a statement originates from spoken audio, scrolling text, both, or neither. The dataset is constructed from Shot2Story, TalkVid, YouCook2, and MultiVENT2.0 using controlled subtitle modification, clip concatenation and name overlays, step markers, and unrelated ticker text. Seven model configurations are evaluated in zero-shot settings. The reported results show near-chance performance for several open-source models on correspondence tasks and substantially stronger performance for Gemini 3.8 Flash. Additional analyses examine OCR/ASR behavior and the effect of supplying transcript-like textual inputs.
Critical strengths
-
A clear and practically relevant problem formulation. The paper identifies a capability that is not well captured by either vision-only video-OCR benchmarks or general audio-visual benchmarks: understanding how on-screen text relates to, complements, or conflicts with the surrounding stream. This distinction is consequential because provenance errors can change the interpretation of news, instructional, and conversational videos.
-
A structured diagnostic benchmark with diverse relationship types. The three-way taxonomy and six tasks move beyond plain text recognition and include alignment verification, temporal localization, bidirectional association, and source attribution. The balanced answer options and inclusion of both open and commercial systems provide a useful first empirical map of the problem.
Critical weaknesses
-
The benchmark construction is insufficiently validated, so synthetic artifacts and ambiguous labels may drive part of the reported gap. The paper relies on Whisper transcripts, GPT-5-generated semantic substitutions, randomly permuted subtitles, randomly assigned names, and unrelated text transferred across videos, but reports no human annotation protocol, number of annotators, inter-annotator agreement, acceptance/rejection rate, or audit of ambiguity and artifact leakage. The overlapping localization options in Figure 2 provide a concrete example of possible ambiguity. Moreover, strong Gemini performance does not itself establish dataset validity or absence of exploitable artifacts. This weakness is consequential because AVTextBench is the paper’s primary contribution; without a representative human-validated sample and artifact analysis, it is unclear whether the benchmark consistently measures the intended construct.
-
The experiments do not adequately isolate “cross-modal grounding,” and the model comparison is under-specified. The paper does not report decisive modality controls such as audio-only, video-only, OCR-text-only, temporally shuffled audio/text, or text-overlay-masked inputs. Table 3 examines recognition behavior for only one model and uses Whisper output as a pseudo-reference, while Table 4 also appears limited to one model and does not fully describe how the supplied text was produced. In addition, frame sampling, image resolution, audio preprocessing, context truncation, decoding parameters, random seeds/repeated runs, exact commercial API snapshot, and confidence intervals are not reported. Thus, failures may reflect OCR/ASR quality, temporal sampling, or unequal model interfaces rather than the proposed relational capability, and the large open-versus-closed model claim is not yet supported by a controlled comparison.
Questions for authors
-
Can the authors provide a human validation study for each task—covering annotator instructions, number of annotators, agreement, invalid/ambiguous sample rate, and stratified results—and explain how overlapping or otherwise non-exclusive temporal answer options are detected and removed? The answer would directly affect my confidence that the benchmark measures the intended construct.
-
How do results change under modality and counterfactual controls: audio only, visual stream with text masked, OCR text only, temporally shuffled audio, temporally shifted/removed overlays, and full input? Please report these controls by task and model, since they are needed to distinguish relational grounding from unimodal recognition and dataset shortcuts.
-
What exact preprocessing and inference configuration was used for every model (frame rate/count, resolution, audio sampling and duration, context truncation, prompt formatting, decoding parameters, number of runs, and the dated/versioned commercial API endpoint)? Are the main pairwise gaps robust under uncertainty estimates or bootstrap confidence intervals?
-
What are the licenses and redistribution conditions of the source videos and derived clips, and what review was performed for privacy, personally identifiable information, and the use of gender-consistent sampled names? This information affects whether the proposed dataset release is ethically and legally viable.
The statement of AI documentation
I have filled out the AI documentation. The review was independently assessed by me. An LLM was only used for language polishing; all technical evaluation and the final score are my own judgment. There is no mismatch between my human assessment and the AI-assisted drafting. I will actively join the discussion with authors and the AC during the discussion period.