acceptodds
Withdrawn from ICLR 2027

AVTextBench: Benchmarking Cross-modal On-Screen Text Grounding in Audio-Visual Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-screen text has become a widely applied tool to convey video information more effectively, such as subtitles. Despite the success of audio-visual large language models (LLMs), it remains unclear whether they can ground on-screen text in the surrounding audio-visual context. In this work, we show that audio-visual LLMs perform poorly in understanding the relationships between on-screen text and audio-visual streams. To systematically study this problem, we introduce a benchmark for probing these capabilities. Specifically, we introduce a controlled framework covering three fundamental relationships: corresponding, complementary, and independent. This framework provides a comprehensive evaluation of whether audio-visual LLMs can correctly match, integrate, and segregate on-screen textual and audio-visual information in real-world videos. We evaluate our benchmark on state-of-the-art audio-visual LLMs, including both open-source models and commercial models. Experimental results show that existing open-source audio-visual LLMs fail to ground on-screen text in context, with performance near random guessing in multiple tasks. On the other hand, the Gemini-3.8-Flash model achieves strong accuracy, highlighting a large gap between open-source and commercial models. Our work uncovers an unexplored limitation in current audio-visual LLMs and provides new insights for future improvement.

Primary area:
datasets and benchmarks
Keywords:
Audio-visual LLMs, Benchmark, Cross-modal grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_V4MkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AVTextBench, a benchmark for evaluating whether audio-visual large language models can ground on-screen text in its surrounding audio-visual context. The benchmark organizes 2,541 video-question pairs into three proposed relationship types: (i) corresponding, which tests subtitle-speech alignment, mismatch type, and mismatch localization; (ii) complementary, which tests bidirectional grounding between transient speaker-name or step-marker text and audio-visual events; and (iii) independent, which tests whether a statement originates from spoken audio, scrolling text, both, or neither. The dataset is constructed from Shot2Story, TalkVid, YouCook2, and MultiVENT2.0 using controlled subtitle modification, clip concatenation and name overlays, step markers, and unrelated ticker text. Seven model configurations are evaluated in zero-shot settings. The reported results show near-chance performance for several open-source models on correspondence tasks and substantially stronger performance for Gemini 3.8 Flash. Additional analyses examine OCR/ASR behavior and the effect of supplying transcript-like textual inputs.

Critical strengths
  1. A clear and practically relevant problem formulation. The paper identifies a capability that is not well captured by either vision-only video-OCR benchmarks or general audio-visual benchmarks: understanding how on-screen text relates to, complements, or conflicts with the surrounding stream. This distinction is consequential because provenance errors can change the interpretation of news, instructional, and conversational videos.

  2. A structured diagnostic benchmark with diverse relationship types. The three-way taxonomy and six tasks move beyond plain text recognition and include alignment verification, temporal localization, bidirectional association, and source attribution. The balanced answer options and inclusion of both open and commercial systems provide a useful first empirical map of the problem.

Critical weaknesses
  1. The benchmark construction is insufficiently validated, so synthetic artifacts and ambiguous labels may drive part of the reported gap. The paper relies on Whisper transcripts, GPT-5-generated semantic substitutions, randomly permuted subtitles, randomly assigned names, and unrelated text transferred across videos, but reports no human annotation protocol, number of annotators, inter-annotator agreement, acceptance/rejection rate, or audit of ambiguity and artifact leakage. The overlapping localization options in Figure 2 provide a concrete example of possible ambiguity. Moreover, strong Gemini performance does not itself establish dataset validity or absence of exploitable artifacts. This weakness is consequential because AVTextBench is the paper’s primary contribution; without a representative human-validated sample and artifact analysis, it is unclear whether the benchmark consistently measures the intended construct.

  2. The experiments do not adequately isolate “cross-modal grounding,” and the model comparison is under-specified. The paper does not report decisive modality controls such as audio-only, video-only, OCR-text-only, temporally shuffled audio/text, or text-overlay-masked inputs. Table 3 examines recognition behavior for only one model and uses Whisper output as a pseudo-reference, while Table 4 also appears limited to one model and does not fully describe how the supplied text was produced. In addition, frame sampling, image resolution, audio preprocessing, context truncation, decoding parameters, random seeds/repeated runs, exact commercial API snapshot, and confidence intervals are not reported. Thus, failures may reflect OCR/ASR quality, temporal sampling, or unequal model interfaces rather than the proposed relational capability, and the large open-versus-closed model claim is not yet supported by a controlled comparison.

Questions for authors
  1. Can the authors provide a human validation study for each task—covering annotator instructions, number of annotators, agreement, invalid/ambiguous sample rate, and stratified results—and explain how overlapping or otherwise non-exclusive temporal answer options are detected and removed? The answer would directly affect my confidence that the benchmark measures the intended construct.

  2. How do results change under modality and counterfactual controls: audio only, visual stream with text masked, OCR text only, temporally shuffled audio, temporally shifted/removed overlays, and full input? Please report these controls by task and model, since they are needed to distinguish relational grounding from unimodal recognition and dataset shortcuts.

  3. What exact preprocessing and inference configuration was used for every model (frame rate/count, resolution, audio sampling and duration, context truncation, prompt formatting, decoding parameters, number of runs, and the dated/versioned commercial API endpoint)? Are the main pairwise gaps robust under uncertainty estimates or bootstrap confidence intervals?

  4. What are the licenses and redistribution conditions of the source videos and derived clips, and what review was performed for privacy, personally identifiable information, and the use of gender-consistent sampled names? This information affects whether the proposed dataset release is ethically and legally viable.

1 comment
Reviewer_V4MkOpenReview ↗

The statement of AI documentation

I have filled out the AI documentation. The review was independently assessed by me. An LLM was only used for language polishing; all technical evaluation and the final score are my own judgment. There is no mismatch between my human assessment and the AI-assisted drafting. I will actively join the discussion with authors and the AC during the discussion period.

Reviewer_s6nAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AVTextBench to evaluate how audio-visual LLMs connect on-screen text with the surrounding audio and video. It contains 2,541 video-question pairs across three categories: correspondence, tested through subtitle alignment verification, mismatch classification, and temporal localization; complementarity, tested by matching speaker names and step markers to speech in both directions; and independence, tested through source attribution. The benchmark is built by adding or modifying text in existing video datasets. The evaluated open models perform near chance on correspondence tasks, achieve better results on several complementary tasks, and vary considerably on source attribution. Gemini 3.8 Flash performs substantially better across the reported tasks.

Critical strengths
  1. The benchmark focuses on a worthwhile problem. Text and speech can serve different purposes in a video. A name label identifies the speaker, while a news ticker may discuss something unrelated to the main report. Mixing these sources can lead to misleading answers even if the model recognizes their contents correctly.

  2. The three task categories offer a clear way to organize the problem: checking whether information agrees, combining information that belongs together, and keeping unrelated information separate. The subtitle tasks also ask what went wrong and when, which provides more detail than a simple correct/incorrect judgment.

Critical weaknesses
  1. The constructed videos simplify the situations encountered in practice. The authors swap subtitle sentences, assign random names to concatenated speaker clips, and turn another video’s speech into a news ticker. Remembering a name shown in a separate speaker clip is not the same as following that person through a conversation with interruptions and camera cuts. Similarly, detecting an inserted subtitle error does not establish how well a model handles naturally occurring subtitle problems. Although the source footage is real, the paper does not evaluate these tasks on videos with naturally occurring text. There is no evidence that models scoring well on the constructed examples also perform well on the same information needs in ordinary videos.

  2. The contribution of each modality is unclear. Some questions ask models to match speech to a speaker. On-screen text may help, but audio and video without that text, audio alone, or video with text but no audio may already provide enough clues to answer some of these questions. The paper lacks comparisons using videos with the added text removed, as well as audio-only and video-only baselines. Without these comparisons, it is difficult to tell which tasks actually require the model to combine text, audio, and visual information.

  3. The dataset relies on model-generated content without sufficient reported human verification. Whisper large-v3 provides speech transcripts and utterance-level timestamps for the correspondence tasks, and its transcripts are also used to construct the independent-task examples. GPT-5 modifies subtitle meanings to create substitution errors. Errors in transcription, timing, or the intended semantic change could therefore affect the benchmark labels. The paper does not report a human checking procedure or an estimate of annotation quality. For a benchmark, this leaves a substantial concern about whether the reference answers are reliable.

  4. The analysis does not clearly distinguish why models fail. A model may misread the text, mishear the speech, or recognize both correctly but confuse their relationship. These failures have different implications and call for different improvements. For example, speech recognition is worse on the independent tasks, but those videos also come from a different dataset. This does not establish that unrelated on-screen text caused the decline. Table 4 adds only one text stream at a time, so it does not answer whether models would still fail if both streams were provided accurately. As it stands, the main conclusion is a fairly unsurprising one: some tested open models score below the tested commercial model. The paper offers limited deeper explanation.

Questions for authors
  1. What exact evaluation settings were used for each model, including video sampling rate (FPS) and whether thinking mode was enabled where available? How did you ensure a fair comparison?
Reviewer_PhYAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes AVTextBench which is a benchmark for evaluating whether AVLLMs can understand the relationship between on screen text and surrounding audio visual content. The benchmark contains video QA pairs organized into 3 categories: correspondence, which evaluates whether subtitles match spoken content, complementary grounding, which tests whether models can associate textual cues with audio visual events and independent source attribution, which examines whether models can distinguish information presented through text and speech. Experiments on several open source and commercial models reveal that open source models often perform close to random guessing, particularly on correspondence tasks, while a closed source model Gemini 3.8 Flash achieves substantially higher accuracy. The authors also analyze text recognition, speech recognition and the effects of providing additional textual information.

Critical strengths
  • The paper explores a relevant and relatively underexplored task/capability of AVLLMs, especially open source ones.

  • Also conducting broad model evaluation revealed meaningful performance gaps between open and closed source AVLLMs.

Critical weaknesses
  • This work does not provide substantial investigation of potential mitigation strategies. Even though the paper identifies significant failures in cross modal grounding of on screen text, it provides limited insight into how these failures could be mitigated. The authors investigate whether supplying speech transcripts or on screen text as additional textual inputs improves performance, but Table 4 shows that these interventions provide little or inconsistent benefit. Beyond this preliminary analysis, the paper does not systematically explore potential remedies or identify which model capabilities need to be improved to address the observed failures. Particularly, given that the authors identify substantial text and speech recognition errors, it would be useful to investigate whether improving these perceptual components leads to better grounding performance. Ideally, proposing some newer training strategies which would strengthen the performance of open-source AVLLMs in such tasks (see AVTrustBench ICCV 2026 for example, a work this paper cites). Hence, the lack of a more comprehensive diagnostic or mitigation analysis limits the actionable insights provided by the benchmark.

  • I think the experiments do not sufficiently distinguish failures in modality perception from failures in cross modal grounding. The paper attributes the poor performance of open source models primarily to their inability to understand relationships between on screen text and audio visual information. However, the authors own analysis reveals substantial weaknesses in basic modality recognition. For example, Qwen2.5 Omni 7B achieves only 66.98% accuracy in speaker name recognition and a word error rate of 338.12% for scrolling text. These perception failures could directly explain poor performance on the corresponding grounding tasks, without necessarily indicating a failure of cross modal reasoning. Even though the authors investigate speech recognition and provide additional textual inputs, the analysis is limited and does not systematically separate perception errors from reasoning errors. Further, Table 4 shows that providing textualized modality information does not consistently improve performance, but this alone does not establish whether the underlying bottleneck is perception, temporal alignment or multimodal reasoning. More controlled experiments using verified transcripts and on screen text annotations are needed to support the paper's central conclusions.

Questions for authors
  • What sort of post-training paradigm or fine-tuning paradigm can be adopted for mitigation of these issues or improving current capabilities of the open source AVLLMs? The paper does not currently provide any substantial novel mitigation strategy.

  • Since several tasks are constructed using synthetic subtitle modifications, concatenated speaker clips and text from unrelated videos, have the authors evaluated whether these transformations introduce unintended artifacts? Could the authors provide a human validation study or results on naturally occurring examples to establish that the benchmark reflects realistic cross modal grounding challenges?

  • The analysis shows substantial text recognition errors, particularly for scrolling text and speaker names. Could the authors evaluate grounding performance when models are provided with verified ground truth transcripts and on screen text, rather than model generated or automatically transcribed text? This would help determine whether poor performance primarily reflects perception failures or an inability to reason across modalities.

  • Gemini 3.8 Flash achieves substantially higher performance than the evaluated open source models. Could the authors clarify whether all models receive comparable temporal coverage, video resolution and audio information? Differences in input processing or frame sampling could contribute to the observed performance gap. In fact, how valid is this comparison (of open and closed weight models) from the perspective of model sizes?

  • Have the authors conducted any human evaluation of the generated questions and answers, particularly for subtitle substitutions and temporal localization? It is hard to understand if the ground truth answers are unambiguous and whether the reported model failures reflect genuine grounding errors.

  • The misalignment localization task divides videos into 3 temporal regions. Could the authors analyze how performance varies with video duration, subtitle density and the length of the mismatched segment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.