acceptodds
Withdrawn from ICLR 2027

OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Current benchmarks for graphical user interface (GUI) agents predominantly rely on static screenshots. However, real-world smartphone interaction routinely requires agents to process transient audio cues and temporal video dynamics that are tightly coupled with the moment of action. To bridge this gap, we introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments. OmniGUI provides continuous, interleaved multimodal inputs—comprising static images, synchronous audio, and video clips—at every action step. The dataset encompasses 706 expert-demonstrated episodes (2,616 action steps) across 29 applications, systematically annotated with objective multimodal dependency levels. Because dedicated omni-modal GUI agent frameworks are currently in their nascent stage, we select foundational omni-modal models capable of natively processing interleaved inputs to serve as agent proxies for our initial baselines. Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals. Furthermore, ablation studies isolate specific operational bottlenecks, notably cross-modal interference when processing task-irrelevant environmental noise. The complete dataset, evaluation pipeline, and baseline prompts are provided in the supplementary material for peer review. The full dataset will be publicly released upon publication.

Primary area:
datasets and benchmarks
Keywords:
GUI agent, benchmark, dataset, Multimodal llm, OMNI llm

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GipqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces OmniGUI, a benchmark for smartphone GUI action prediction using screenshots, synchronized audio, video clips, and action histories. It contains 709 expert-demonstrated episodes across 29 applications, organized into five task dimensions and three multimodal dependency levels. The authors evaluate eight foundation models using an offline, teacher-forced protocol and conduct modality-removal and text-to-speech ablations.

Critical strengths
  • Collecting synchronized GUI observations and expert action annotations across 29 applications is a useful effort toward evaluating richer smartphone interactions.

  • The distinction between AV-Critical, AV-Supportive, and AV-Present tasks provides a useful framework for analyzing modality dependence.

Critical weaknesses
  • The motivation for native audio processing is insufficiently established. The paper does not clearly separate tasks requiring speech recognition from those requiring non-speech acoustic understanding or precise audiovisual synchronization. For spoken instructions, converting text to synthesized speech changes the input format but does not necessarily introduce a new reasoning capability: an ASR module could recover the instruction for an existing GUI agent. Although the dataset includes non-speech and timing-sensitive examples, the paper does not quantify their prevalence or establish which remain difficult after transcription. The text-versus-TTS experiment in §4.3 measures sensitivity to instruction format, while removing audio discards its information entirely. Neither comparison determines whether native audio processing offers an advantage over an ASR-based pipeline.

  • The experimental coverage is too limited to support the broader conclusions. All four proprietary models in §4.1 belong to the Gemini family. Consequently, the reported proprietary-versus-open-source gap cannot establish a general difference between these categories. More importantly, the evaluation omits specialized GUI agents equipped with ASR. Comparing general-purpose omni-modal models against GUI-specialized models supplied with transcribed speech is essential to determine whether the observed failures arise from audio processing, GUI grounding, or their integration. The API compatibility explanation for excluding GPT-4o does not resolve this missing comparison; modular baselines can be evaluated separately under clearly specified input conditions.

  • The organization and presentation require substantial revision. The writing repeatedly uses emphatic claims and dash-separated asides where concise explanations would be clearer. Page 3 contains excessive whitespace around headings, while the related-work discussion could be shortened and focused more directly on the closest comparisons. The labels and annotations in Figure 2 are too small to read comfortably at normal viewing size. Cross-references are also awkwardly presented, such as “(Appendix D, Table 8)” in the introduction. These issues make the manuscript harder to follow and suggest insufficient editorial revision.

Questions for authors
  1. Which tasks require native audio processing beyond transcription, how prevalent are they, and does native audio outperform an ASR-based pipeline?

  2. Can the authors broaden the evaluation beyond Gemini to include ASR-equipped GUI agents and improve the manuscript’s concision, layout, and figure readability?

  3. Inconsistent numerical results (Table 2, p. 8) Gemini 2.5 Flash’s overall EM is 37.8% but its EM across the three exhaustive dependency groups is 39.7%, 41.6%, and 40.8%. Under consistent aggregation, the overall score cannot be lower than every subgroup score. The reported step counts imply approximately 40.7%.

  4. Incorrect description of Multimodal-Mind2Web (§2.1; Table 8). The paper claims that this dataset adds audio transcriptions and marks it as supporting audio. Its official documentation describes pairing HTML documents with webpage screenshots, not adding audio transcriptions. Official source

  5. Wrong OmniEval reference (§2.2). Wang et al. (2025) is cited as a tri-modal understanding benchmark, but the bibliography points to a financial RAG evaluation benchmark. The relevant audiovisual OmniEval is a different paper. Cited financial benchmark, audiovisual benchmark

Reviewer_TYM7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces OmniGUI, a step-level benchmark for smartphone GUI agents in which every action step has a screenshot, a video clip, and a synchronous audio track. The video and audio cover the interval since the previous action. It contains 709 expert-demonstrated episodes across 29 Chinese and English apps. Eight omni-modal foundation models are evaluated. Modality ablations show that removing audio and video hurts AV-Critical episodes more than AV-Present ones. Two further findings are reported: (i) cross-modal interference, where irrelevant audio and video lower accuracy on AV-Present tasks, and (ii) degradation when the instruction is delivered as TTS audio alongside environmental audio.

Critical strengths
  1. The benchmark targets a real gap. Prior multimodal GUI benchmarks such as VideoWebArena, GUI-World, and VideoGUI mostly treat audio and video as pre-task reference material. OmniGUI instead ties audio and video to the moment of action at every step. This enables evaluation of transient signals, such as auditory pauses and changing playback states, that static screenshots cannot adequately capture.
  2. The experiments provide detailed and informative diagnostics. The paper evaluates eight proprietary and open-source models across five task dimensions and three multimodal dependency levels. Beyond overall performance, it systematically removes audio, video, or both, and compares textual instructions with spoken instructions. These experiments help distinguish reliance on task-relevant multimodal information from interference caused by irrelevant signals. Moreover, the paper is clearly written and well organized. The presentation progresses naturally from the motivating gap to benchmark construction, evaluation, and diagnostic analysis.
Critical weaknesses

Action-aligned segmentation may undermine the benchmark’s temporal and instant-response claims. Each video clip ends when the human demonstrator initiates the corresponding action, potentially revealing the decisive moment through the clip boundary itself. Models therefore recognize events in pre-segmented clips without demonstrating continuous event detection or timely responses. This concern extends beyond the absence of interactive rollouts: even within the offline evaluation, performance may reflect segmentation cues rather than the intended temporal capabilities.

Questions for authors
  1. Can you test Temporal Reasoning and Instant Response with clip boundaries selected independently of human action timestamps, while excluding visible expert actions and their consequences? This would clarify whether performance depends on action-aligned segmentation.
  2. Can you compare a screenshot-only GUI specialist against a cascaded ASR/video-captioning-plus-GUI-agent baseline, reporting EM separately on steps that require audio or video? This would help distinguish failures in multimodal perception from failures in GUI grounding.
Reviewer_5shxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OmniGUI, a benchmark for smartphone GUI agents that integrate image, audio, and video inputs. It includes expert demonstrations and action steps across diverse applications, covering five task dimensions and three levels of multimodal dependency. Eight models are evaluated using expert action histories under four metrics. The main evaluation results demonstrate substantial limitations in temporal and auditory grounding, while modality ablations show that irrelevant audio and video inputs can degrade performance.

Critical strengths

The benchmark addresses a significant gap in existing GUI evaluation benchmarks by introducing audio and temporal video at action steps. Its diverse applications and explicit modality dependency analysis provide a promising foundation for studying multimodal interaction.

Critical weaknesses

1. Evaluation of real time interaction.

The benchmark relies on prerecorded observations and expert action histories rather than actual environment interaction. Moreover, observation timestamps are determined by expert demonstrations, so agents do not need to decide when to observe or act. This is particularly limiting for tasks involving temporal reasoning and instant responses, where the timing of actions is essential. Therefore, the benchmark cannot evaluate reaction latency, error recovery, or the ability to handle evolving environments, which limits its scope and effectiveness in assessing interactive GUI agents.

2. Multimodal reasoning capabilities.

The benchmark does not clearly separate multimodal reasoning from general GUI operation capabilities. In particular, modality dependency is defined at the episode level, where an AV-Critical episode may contain only one step that actually requires audio or video. Evaluating performance across the entire episode thus mixes multimodal decisions with common visual grounding, which makes it difficult to determine whether model failures are from multimodal understanding or general action prediction. More fine grained evaluation of modality dependent steps would make the benchmark more informative.

Questions for authors

1.Since observations are sampled at timestamps determined by expert demonstrations, how would the results change if agents had to decide when to observe and act? This seems particularly important for evaluating temporal reasoning and instant response capabilities.

2.Since AV-Critical is defined at the episode level, how can we distinguish failures in multimodal understanding from failures in general GUI operation? Would evaluating the specific steps that require audio or video lead to different conclusions about model capabilities?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.