- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces OmniGUI, a benchmark for smartphone GUI action prediction using screenshots, synchronized audio, video clips, and action histories. It contains 709 expert-demonstrated episodes across 29 applications, organized into five task dimensions and three multimodal dependency levels. The authors evaluate eight foundation models using an offline, teacher-forced protocol and conduct modality-removal and text-to-speech ablations.
Critical strengths
-
Collecting synchronized GUI observations and expert action annotations across 29 applications is a useful effort toward evaluating richer smartphone interactions.
-
The distinction between AV-Critical, AV-Supportive, and AV-Present tasks provides a useful framework for analyzing modality dependence.
Critical weaknesses
-
The motivation for native audio processing is insufficiently established. The paper does not clearly separate tasks requiring speech recognition from those requiring non-speech acoustic understanding or precise audiovisual synchronization. For spoken instructions, converting text to synthesized speech changes the input format but does not necessarily introduce a new reasoning capability: an ASR module could recover the instruction for an existing GUI agent. Although the dataset includes non-speech and timing-sensitive examples, the paper does not quantify their prevalence or establish which remain difficult after transcription. The text-versus-TTS experiment in §4.3 measures sensitivity to instruction format, while removing audio discards its information entirely. Neither comparison determines whether native audio processing offers an advantage over an ASR-based pipeline.
-
The experimental coverage is too limited to support the broader conclusions. All four proprietary models in §4.1 belong to the Gemini family. Consequently, the reported proprietary-versus-open-source gap cannot establish a general difference between these categories. More importantly, the evaluation omits specialized GUI agents equipped with ASR. Comparing general-purpose omni-modal models against GUI-specialized models supplied with transcribed speech is essential to determine whether the observed failures arise from audio processing, GUI grounding, or their integration. The API compatibility explanation for excluding GPT-4o does not resolve this missing comparison; modular baselines can be evaluated separately under clearly specified input conditions.
-
The organization and presentation require substantial revision. The writing repeatedly uses emphatic claims and dash-separated asides where concise explanations would be clearer. Page 3 contains excessive whitespace around headings, while the related-work discussion could be shortened and focused more directly on the closest comparisons. The labels and annotations in Figure 2 are too small to read comfortably at normal viewing size. Cross-references are also awkwardly presented, such as “(Appendix D, Table 8)” in the introduction. These issues make the manuscript harder to follow and suggest insufficient editorial revision.
Questions for authors
-
Which tasks require native audio processing beyond transcription, how prevalent are they, and does native audio outperform an ASR-based pipeline?
-
Can the authors broaden the evaluation beyond Gemini to include ASR-equipped GUI agents and improve the manuscript’s concision, layout, and figure readability?
-
Inconsistent numerical results (Table 2, p. 8) Gemini 2.5 Flash’s overall EM is 37.8% but its EM across the three exhaustive dependency groups is 39.7%, 41.6%, and 40.8%. Under consistent aggregation, the overall score cannot be lower than every subgroup score. The reported step counts imply approximately 40.7%.
-
Incorrect description of Multimodal-Mind2Web (§2.1; Table 8). The paper claims that this dataset adds audio transcriptions and marks it as supporting audio. Its official documentation describes pairing HTML documents with webpage screenshots, not adding audio transcriptions. Official source
-
Wrong OmniEval reference (§2.2). Wang et al. (2025) is cited as a tri-modal understanding benchmark, but the bibliography points to a financial RAG evaluation benchmark. The relevant audiovisual OmniEval is a different paper. Cited financial benchmark, audiovisual benchmark