acceptodds
Withdrawn from ICLR 2027

GUI-ACE: ACTION-CONDITIONED EXPLANATION OF GUI INTERACTIONS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

To serve as practical personal assistants, graphical user interface (GUI) agents need to execute tasks reliably and provide personalized services. Interaction records can support large-scale agent training and user behavior analysis, and their reuse benefits from understanding recorded actions' targets and functions. General vision-language models (VLMs) can describe these actions, but large models are costly at scale. They also struggle to align numerical action coordinates with visual user interface (UI) elements. Even with accurate point localization, they may misidentify the complete functional unit receiving the action. Element boxes or an additional screenshot can aid target identification but may be unavailable or add overhead. We therefore introduce ActionScope, the smallest complete UI region corresponding to the action-receiving functional unit. This region specifies which functional unit the explanation should describe. Building on ActionScope, we propose Action-Conditioned Explanation (ACE), a compact VLM that takes a single screenshot, action parameters, and available previous-step textual context. It generates target and purpose descriptions conditioned on its predicted ActionScope bounding box field. Our ACE-Curator pipeline generates ActionScope and semantic annotations through region guidance and quality control. We use these annotations to train 1.3B and 8B variants. On an evaluation set with human-verified reference annotations, ACE-1.3B and ACE-8B achieve 92.50% and 94.54% joint semantic accuracy, respectively. Single-screenshot ACE-8B outperforms Qwen3.8-2.4T (Max API) and Claude Opus 5 with an additional screenshot in overall joint semantic accuracy. Ablations show that ActionScope supervision improves joint semantic accuracy. Replacing predicted boxes with incorrect ones reduces subsequent explanation accuracy. Together, these findings support ActionScope as an intermediate representation connecting spatial evidence to semantic explanation. For offline GUI navigation and grounding, we train models with instructions reconstructed from ACE descriptions. These models outperform those trained with original instructions on several benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Interaction Understanding, Vision-Language Models, Automated Data Annotation, GUI Agents, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mavVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GUI-ACE for explaining recorded GUI interactions from a screenshot, action parameters, and previous-step textual context. The authors introduce ActionScope as an intermediate representation of the UI region receiving an action, build the ACE-Curator pipeline to generate training annotations, and train ACE-1.3B and ACE-8B to predict ActionScope together with action target and purpose descriptions. The paper also evaluates the generated semantics on downstream GUI navigation and grounding tasks.

Critical strengths
  1. The overall pipeline is complete, covering ActionScope annotation, quality filtering, model training, and downstream use of the generated action semantics.
  2. The experimental evaluation is comprehensive, including comparisons with general VLMs, ActionScope ablations, bounding-box interventions, different visual inputs, and downstream navigation and grounding experiments.
Critical weaknesses
  1. Unclear practical necessity of a dedicated action-explanation model. The paper shows that ACE outperforms direct prompting of general VLMs, but does not compare against a natural alternative of using a strong VLM followed by verification or filtering. This baseline is especially relevant because ACE-Curator itself shows that generation with quality filtering can already achieve high annotation accuracy.
  2. Insufficient direct validation of ActionScope. ActionScope is the central representation in the paper, but the main evaluation metric about ActionScope focuses on semantic correctness and explicitly excludes bounding-box geometry. The ablation study shows that region supervision helps semantic explanation, but do not directly measure whether ACE predicts accurate ActionScope regions.
Questions for authors
  1. How does ACE compare with a strong VLM plus verification/filtering pipeline in terms of accuracy, cost, and inference efficiency?
  2. Can the authors provide the localization quality and consistency of predicted ActionScope regions?
5 comments
AuthorsOpenReview ↗

Application requirements and the motivation for ACE

To explain why a dedicated model is needed, we first clarify the application requirements that motivate ACE. We then discuss whether a strong VLM with verification and filtering can meet these requirements.

Building training data for graphical user interface (GUI) agents and analyzing user behavior both rely on semantic explanations of recorded actions. For training data construction, the goal is to lower annotation costs at scale while maintaining quality. For user behavior analysis, interpreting everyday interactions can support preference extraction and proactive assistance. Privacy concerns motivate local processing of interaction records, while proactive assistance calls for low end-to-end latency. We therefore study small models for action explanation to reduce annotation costs and end-to-end latency and support future on-device deployment.

When given action coordinates and a single screenshot, general vision-language models (VLMs) may struggle to identify the complete functional unit receiving the action. This can cause them to describe the wrong UI element. To address this problem, existing methods provide VLMs with external element boxes, screenshots taken before and after the action, or both. The models use these inputs together with action parameters to generate semantic descriptions. Element boxes can be obtained from a webpage’s Document Object Model (DOM) or an application’s accessibility tree. Detection or segmentation models can also generate them from screenshots. These boxes provide candidate locations, while the screenshot pairs provide clues about interface changes.

However, both types of auxiliary input have limitations in availability and usefulness. For element boxes, some applications or environments do not allow access to the DOM or accessibility tree. Detection or segmentation models may also fail to produce a candidate box that matches the action location. Even when a box is available, it may not match the complete functional unit receiving the action. It may cover only part of that unit or include unrelated controls. For screenshot pairs, the second image may not accurately reflect the response to the current action if captured too early or too late. Even with appropriate timing, an action may produce no visible feedback, or different controls may produce similar feedback. In either case, the screenshots may provide little guidance for action explanation. Moreover, obtaining and processing both types of auxiliary input adds overhead. Together with the inference cost of large models, this overhead limits annotation at scale and use on local devices.

We therefore aim to train small models to identify the functional unit receiving an action and understand its function on the current page. We define ActionScope as the smallest complete UI region corresponding to this unit. Its bounding box (bbox) provides supervision for learning to identify the region receiving the action. Semantic supervision teaches the model to interpret the unit’s function using information from the current page. Using both forms of supervision, we train Action-Conditioned Explanation (ACE), a small VLM. Within a single autoregressive sequence, ACE first generates the ActionScope bbox field. It then generates descriptions of the recipient and the action’s function, conditioned on that field. At inference, ACE needs only a single screenshot, action parameters, and available textual context from the previous step. It requires no external element boxes or second screenshot.

AuthorsOpenReview ↗

ACE-Curator as an alternative to ACE

Why ACE-Curator does not meet the deployment requirements (W1, Q1)

First, our current ACE-Curator pipeline calls cloud-hosted large VLMs. It therefore does not meet the requirement for on-device inference that keeps personal interaction records entirely local.

Second, the reported annotation configuration uses screenshots before and after actions. It therefore does not meet the single-screenshot input requirement targeted by ACE.

Third, localization, semantic generation, and verification involve multiple large-model calls. Repeating this process for each new action retains high inference costs, which conflicts with our goal of low-cost, repeated action explanation.

Finally, filtering rejects explanations that fail verification. Actions that are harder for the model to interpret reliably are therefore more likely to be excluded. For training data construction, retaining these cases at lower automatic coverage requires more human annotation, increasing labor costs. For continuous user behavior analysis, excluding them leaves gaps in the semantic stream. Such selective coverage does not meet our requirements for low-cost annotation and continuous interpretation across routine and difficult scenarios.

We therefore use ACE-Curator to construct training annotations offline and manually annotate the difficult cases flagged by the pipeline. We then train ACE as a student model with both region and semantic supervision, so it learns to identify the region receiving an action and explain the action more accurately.

Additional experiments with ACE-Curator (W1, Q1)

To address the reviewer's question, we additionally compare ACE with ACE-Curator in accuracy and inference efficiency.

Accuracy at full coverage. We evaluate all 1,666 actions without filtering, so difficult cases remain in the comparison. We use Qwen3.5-35B-A3B for the first three Curator stages: candidate localization, visual correction, and separate screen and action annotation. These stages propose and refine a target region, then generate screen and action descriptions in separate passes. We omit Stage 4 quality filtering. When no usable region is obtained, the pipeline falls back to direct annotation from the pre- and post-action screenshots. We use the same Qwen judge protocol as the main evaluation. The table reports joint semantic accuracy (%); Point includes clicks and long presses.

MethodOverallPointTypeSwipeStandardHard
Qwen3.5-35B-A3B Curator92.3892.2794.4891.1895.7888.14
ACE-1.3B92.5093.2992.2787.7596.1088.01
ACE-8B94.5495.0895.0390.6997.2991.11

Curator correctly explains 1,539 of 1,666 actions. With a single screenshot, ACE-1.3B achieves comparable overall accuracy, exceeding Curator by 0.12 percentage points. ACE-8B improves overall accuracy by 2.16 percentage points.

Inference efficiency. We measure all methods on the same 100 actions using one RTX PRO 6000 96GB GPU, BF16, batch size 1, and serial inference. We report the fastest working configuration tested for each method. All use Transformers, SDPA, and four CPU threads.

MethodMean latency (s/action)P95 latency (s)Peak GPU memory (GiB)GPU energy (J/action)
ACE-1.3B0.7050.8455.33134.0
ACE-8B2.3522.83320.22717.0
Qwen3.5-35B-A3B Curator25.61945.45469.163652.6

Relative to Curator, ACE-1.3B and ACE-8B achieve 36.3× and 10.9× speedups in mean latency, respectively, while using less GPU memory and energy. These results show the practical benefit of training ACE: it provides comparable or higher overall accuracy with single-image input and substantially lower inference costs on the same hardware.

AuthorsOpenReview ↗

Direct evaluation of ActionScope localization quality

Direct evaluation of ActionScope localization (W2, Q2)

We added a direct evaluation of the predicted ActionScope boxes on all 1,666 actions for both single-image ACE variants. We assess whether each box identifies the intended action recipient and measure its geometric overlap with the reference region.

Evaluation protocol. Human reviewers judge target correctness. Oversized or undersized boxes count as correct if they still unambiguously identify the intended recipient. Boxes pointing to another element or leaving the target unclear count as incorrect. We separately report intersection over union (IoU) to measure geometric agreement with the reference boxes. Both annotators reviewed all 1,666 predictions for each model. Their binary target-correctness judgments differed on only four cases (0.24%) for ACE-1.3B and eight (0.48%) for ACE-8B. Given this close agreement, we use the second annotator's judgments in the reported results.

Localization quality. The table reports human-assessed target correctness and geometric overlap. The percentage of boxes with IoU ≥ 0.5 is computed over all 1,666 actions.

ModelTarget correctnessMean IoUMedian IoUIoU ≥ 0.5
ACE-1.3B97.18% (1,619/1,666)0.74330.802287.09% (1,451/1,666)
ACE-8B97.24% (1,620/1,666)0.77260.839188.36% (1,472/1,666)

Both models identify the intended recipient in over 97% of cases. Their geometric overlap with the reference boxes also shows that the predictions generally capture the relevant region, beyond identifying the target alone.

Relationship to explanation accuracy. We further group predictions by whether their target and purpose descriptions are jointly correct, using the original semantic evaluation protocol. Percentages in each row use that group's sample count as the denominator.

ModelSemantic judgmentSamplesTarget correctnessMean IoUIoU ≥ 0.5
ACE-1.3BCorrect1,54198.70%0.758489.16%
ACE-1.3BIncorrect12578.40%0.557361.60%
ACE-8BCorrect1,57598.73%0.784189.84%
ACE-8BIncorrect9171.43%0.572762.64%

For both models, correct explanations are associated with more accurate target identification and higher IoU. These results complement the semantic evaluation by directly measuring localization quality and showing its relationship to explanation correctness.

Please also see the supplementary experiment in our response to Reviewer 1, which fixes Qwen3.8 and the prompt while changing only the box source. ACE-8B supplies its predicted ActionScope boxes without semantic descriptions. On all 1,281 point actions, joint semantic accuracy reaches 94.30% with ActionScope, compared with 80.02% for SAM2 and 84.86% for OmniParser. Cases without a usable box fall back to the clean screenshot and remain in the evaluation. Together with the human review and IoU results above, this provides further evidence that learned ActionScope regions help identify the action recipient and support more accurate action explanations.

Reviewer_uPaMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ActionScope, a representation of the smallest complete functional region that receives a GUI action. Given a screenshot, an action, and limited historical context, the model predicts the action-related screen information, the ActionScope region, the target, and the action purpose. The authors construct a large training set through candidate generation, visual correction, rule-based filtering, model-based verification, and human review. They then train models based on MiniCPM-V-1.3B and Qwen3-VL-8B and evaluate them on action interpretation as well as downstream navigation and grounding tasks.

Critical strengths

S1: The problem framing is genuinely fresh. The gap between "a click coordinate" and "which functional unit actually received it" (local content vs. an independent child control vs. the parent container) is real, and ActionScope is a clean way to model it.

S2: A meaningful controlled ablation. Table 3a compares a four-field model without the ActionScope field against a five-field model with ActionScope, while keeping the backbone, data, semantic labels, and training configuration fixed. Adding the region field improves overall accuracy from 88.06% to 92.50% for ACE-1.3B and from 91.60% to 94.54% for ACE-8B. The gains are larger on the Hard subset. This provides direct evidence that region supervision contributes beyond the other semantic fields.

S3:Relatively careful data construction and quality control. The paper reports region correction, cropping, automatic filtering, risk validation, and human review, together with relevant ablations. The appendix provides useful details about the annotation pipeline and its coverage–quality trade-off.

Critical weaknesses

W1: The core eval is self-built, auto-judged, and drawn from the same distribution as training. Evaluation is further based largely on model judges whose families overlap with the models used in annotation and verification.

W2: The generalization evidence is limited. The evaluation appears to rely primarily on action-level holdout and dataset-provided historical context. The paper does not sufficiently test unseen applications, history-free inputs, or histories generated recursively by ACE itself. Thus, the current results support action interpretation under held-out actions with available historical context, but they do not yet establish robust generalization to unfamiliar applications or long online trajectories.

W3: The downstream gains are not fully attributable to ActionScope. The navigation experiments combine ActionScope with language rewriting and trajectory-level information. Consequently, the results demonstrate the value of the complete ACE supervision pipeline, but do not isolate the contribution of the ActionScope representation itself.

W4: Efficiency claims are not sufficiently supported. The motivation keeps hammering "compact VLM" and "low overhead," and they even pick MiniCPM-V-4.6-1.3B for mobile deployment — yet there is no latency, throughput, or memory number anywhere. If efficiency is a selling point against big VLMs, report ACE-1.3B's per-step cost versus a Qwen3.8 API call and a local 8B model.

W5: Inconsistent coordinate conventions. The paper normalizes bounding-box coordinates (b) to ([0,1000]), whereas the reverse-grounding prompt in Appendix F.4 requests coordinates in ([0,1]), and Figure 1 also includes examples in the latter format.

Questions for authors
  1. Could you clarify how training–test overlap is controlled beyond identical actions, and provide an independently human-evaluated subset to verify whether the reported accuracy and model rankings hold without relying on the annotation-related model judges?

  2. What happens without historical context, or when the history is generated by ACE itself rather than provided by the dataset?

  3. How much of the downstream improvement comes from ActionScope, language rewriting, and trajectory-level information, respectively?

4 comments
AuthorsOpenReview ↗

Evaluation reliability, generalization, and history dependence

Evaluation reliability and judge dependence (W1, Q1)

We thank the reviewer for raising this concern. Our judging protocol uses tailored prompts for multiple stages, with reference-based text checks followed by visual review of uncertain cases. Our evaluation includes human cross-checks, as reported in Section 4.1. We sampled 100 actions, stratified by action type and difficulty, and assessed 300 responses from three models. Human and automated judgments agreed on 298 responses (99.33%). Separately, the evaluation reference annotations were verified and corrected by humans.

To assess potential bias from using Qwen models for annotation, verification, and evaluation, we replaced the Qwen visual judge with MiniMax M3. MiniMax M3 was not used for annotation or verification. We reevaluated all 11 models on the same 1,666 actions. The table reports overall joint semantic accuracy; changes are in percentage points.

ModelOriginal judge (%)M3 visual judge (%)Change (pp)
Qwen3.8-2.4T68.1968.01−0.18
Claude Opus 567.2367.29+0.06
GPT-5.6-Sol55.4055.400.00
MiniMax M353.3653.24−0.12
Kimi K2.655.4655.64+0.18
AutoGUI-Qwen10.2010.56+0.36
Ferret-UI-8B26.5926.83+0.24
MiniCPM-V-1.3B1.321.44+0.12
Qwen3-VL-8B50.1849.82−0.36
ACE-1.3B92.5092.08−0.42
ACE-8B94.5494.12−0.42

Overall accuracy changes by at most 0.42 percentage points, and the ranking of all 11 models remains unchanged. ACE-8B and ACE-1.3B remain the top two models. These results show that ACE's advantage remains stable after replacing the Qwen visual judge with MiniMax M3. Together with the human cross-checks, they support the reliability of our evaluation.

Generalization and reliance on history (W2, Q1, Q2)

Thank you for highlighting the need to test generalization to unseen apps and robustness to different history settings.

Design of the original test set. Our original test set evaluates action explanation across diverse apps, screens, and functions. It contains 1,666 actions from at least 221 apps, with 924 Standard and 742 Hard samples. Training and test sets do not share identical screenshots or action instances. Sampling individual actions allows us to cover varied controls and functions across different workflows. The difficulty split also tests cases involving similar controls, parent-child ambiguity, and subtle state differences. This design provides broad coverage for evaluating how models interpret new screenshots and actions.

Training with missing or mismatched history. We deliberately include training records with missing or mismatched history so ACE learns to interpret actions without reliable historical context. Of the 298,503 training records, 76,785 (25.72%) have no previous-step history, and 6,778 (2.27%) have deliberately mismatched history. The target and purpose labels still describe the current action. This encourages ACE to rely primarily on the current screenshot and action parameters. The prompts reinforce this design by treating history as supplementary context.

Additional evaluation on unseen apps. The original split did not exclude apps seen during training. Despite the training design above, we had not directly tested ACE's action explanations with recursively generated history or without history. We therefore constructed an additional evaluation covering 110 apps that had never appeared in ACE's training data. The randomly sampled AndroidControl data comprise 359 complete trajectories with 888 scored actions. We compare three history settings. Annotated history provides the annotated context from the preceding step. Recursive history uses each model's own previous descriptions along the recorded trajectory. No history omits these previous-step descriptions. All three models use a single screenshot per action.

Overall joint semantic accuracy (%) is reported below. Both ACE variants use the five-field output format.

ModelAnnotated historyRecursive historyNo history
ACE-1.3B93.6993.4791.55
ACE-8B95.7295.6194.14
Qwen3.8-2.4T (Max API)79.0577.7076.80

These results show that ACE maintains high accuracy on apps unseen during training. Replacing annotated history with recursive history reduces accuracy by only 0.23 and 0.11 percentage points for ACE-1.3B and ACE-8B, respectively. Omitting history causes larger drops of 2.14 and 1.58 percentage points relative to annotated history, indicating that historical context provides useful information for action explanation. Without history, both models still correctly explain more than 90% of the actions. Their accuracy remains well above Qwen3.8-Max's 76.80% under the same no-history setting.

AuthorsOpenReview ↗

Purpose and interpretation of the downstream experiments

Downstream utility and attribution (W3, Q3)

We first clarify why we did not consider a downstream ablation necessary in the original study. The dedicated action-explanation ablation provides a more direct and conclusive test of ActionScope's contribution, which downstream scores reflect only indirectly.

The downstream experiments evaluate the practical value of ACE annotations. A text LLM reconstructs user instructions from ACE's descriptions of action targets and functions. These instructions train GUI navigation and grounding models. The original supervision includes human annotations and annotations from large models such as GPT-4o and Gemini 1.5 Pro [1–4]. Compared with this supervision, ACE-reconstructed instructions yield comparable or better training outcomes on several benchmarks. These results indirectly support the quality of reconstructed user instructions and their potential for understanding user behavior.

We understand the reviewer's interest in isolating ActionScope's contribution to downstream performance. However, downstream accuracy measures a separate model's action or coordinate predictions, so it is an indirect measure of ActionScope's effect. ActionScope first influences ACE's explanations of action targets and functions. These descriptions then undergo instruction reconstruction and are used to train the downstream model. The resulting gain depends on how that model learns and uses the information. We therefore evaluate ActionScope's effect on action explanation directly through a dedicated semantic ablation. Under matched training settings, ActionScope supervision improves joint semantic accuracy by 4.44 percentage points for ACE-1.3B and 2.94 for ACE-8B.

References

[1] Quanfeng Lu et al. GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22404–22414, 2025.

[2] Yang Li et al. Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5495–5510, 2020.

[3] Hongxin Li et al. AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10323–10358, 2025.

[4] Yuxiang Chai et al. AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 2138–2156, 2025.

AuthorsOpenReview ↗

ActionScope, language rewriting, and trajectory-level information

With this clarification, we address the three factors and their experimental comparisons below.

ActionScope supervision

We compared downstream training with four-field annotators without ActionScope supervision and five-field annotators with it. Instruction reconstruction and downstream training settings were held fixed. The tables report accuracy (%), with changes in percentage points.

Offline GUI navigation with reference history

SettingFour fields (%)Five fields (%)Change (pp)
B: reconstruct step instructions52.3953.43+1.04
C: reconstruct step instructions and overall trajectory goals51.1351.09−0.04
Mean across settings51.7652.26+0.50

GUI grounding

BenchmarkFour fields (%)Five fields (%)Change (pp)
ScreenSpot-v2 (mobile)22.5524.95+2.40
MoTIF (click subset)28.1527.61−0.54
AndroidControl click50027.2026.80−0.40
MMBench-GUI (L2, mobile)11.3711.96+0.59
UI-I2E (mobile)13.0513.90+0.85
Mean across benchmarks20.4621.04+0.58

On average, ActionScope supervision slightly improves downstream navigation and grounding performance.

Language rewriting

Language rewriting adds no new information. It converts ACE's output into the instructions required for downstream training. All compared settings use the same procedure. Removing this step would change the supervision format, so we keep it fixed and do not ablate it separately.

Trajectory-level information

Here, trajectory-level information refers to the overall trajectory goal. B/C compares downstream training with original and reconstructed goals. B uses GUIOdyssey's original goals, developed with human input [1]. C has no access to these goals. A text LLM reconstructs them from ACE's step descriptions and recorded actions. Both groups use their respective goals for navigation training and combine them with the complete trajectories to generate swipe instructions.

In the five-field experiment, C achieves 51.09%, compared with 53.43% for B and 48.26% for original supervision. Reconstructed overall trajectory goals support effective training, although the original goals lead to better results. Section 4.4 discusses a possible explanation: reconstructed goals emphasize concrete operation sequences, while AndroidControl High includes abstract goals requiring planning. This difference may favor B.

References

[1] Quanfeng Lu et al. GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22404–22414, 2025.

AuthorsOpenReview ↗

Inference cost estimates and coordinate conventions

Inference efficiency and cost (W4)

Accuracy under matched inputs. With a single screenshot, ACE-1.3B and ACE-8B achieve 92.50% and 94.54% joint semantic accuracy on 1,666 actions, compared with 68.19% for Qwen3.8-Max (Qwen3.8-2.4T in the paper), the strongest general VLM in this setting. Both ACE models also outperform Qwen with pre- and post-action screenshots plus external boxes on the same 1,281 point actions. Accuracy is 93.29% for ACE-1.3B and 95.08% for ACE-8B, versus 90.63% with SAM2 boxes and 92.43% with OmniParser boxes. These auxiliary inputs are not consistently available in practice.

Cost under the same single-screenshot input condition. We compare costs for 100 action explanations. ACE uses a single fully utilized RTX 4090, with costs based on AutoDL rental pricing [5]. Qwen3.8-Max costs use its standard API rates [6].

ModelEstimated cost per 100 actions (USD)
Qwen3.8-Max API$0.249–0.252
ACE-1.3B$0.0031–0.0034
ACE-8B$0.0088

Under these conditions, Qwen3.8-Max costs approximately 72–79 times as much as ACE-1.3B and 28–29 times as much as ACE-8B. ACE therefore offers higher action-explanation accuracy at substantially lower estimated inference cost on a single GPU.

Coordinate conventions and conversion (W5)

We define ACE's ActionScope coordinates on the [0,1000] scale. Reverse grounding is performed by a separate model, POINTS-GUI-G, whose [0,1] coordinate convention comes from its original paper [7]. Appendix F.4 already identifies this model and provides its prompt. We retain its original convention and multiply its predicted coordinates by 1,000 before checking containment in an ActionScope box. The two conventions therefore do not conflict, and geometric comparisons use a common scale. Figure 1 shows raw action records in [0,1]; for example, (0.427, 0.632) corresponds to (427, 632) on the [0,1000] scale.

References

[5] AutoDL. GPU rental pricing. Official website, n.d. URL https://autodl.com/home. Accessed: 2026-11-12.

[6] Alibaba Cloud. Qwen3.8-Max model information and pricing. Official documentation, n.d. URL https://help.aliyun.com/en/model-studio/qwen3-8-max. Accessed: 2026-11-12.

[7] Zhongyin Zhao et al. POINTS-GUI-G: GUI-Grounding Journey. arXiv preprint arXiv:2602.06391, 2026.

Reviewer_yQrDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies GUI action explanation: identifying the target and local purpose of a recorded action from a single screenshot, action parameters, and available previous-step textual context. It introduces ActionScope, the smallest complete region corresponding to the functional recipient, and ACE, compact VLMs trained to predict this region before generating semantic descriptions. An annotation pipeline, ACE-Curator, supplies region and semantic supervision. Validation includes annotation-quality evaluations, model comparisons, matched box-supervision ablations, incorrect-box interventions, and downstream experiments using reconstructed instructions for offline GUI navigation and grounding.

Critical strengths
  • The paper identifies a meaningful distinction between point localization and functional-target identification. Correctly locating an action does not necessarily identify the control receiving it, particularly in nested interfaces. This highlights a concrete source of semantic annotation errors worth addressing.

  • The core ablation is well controlled. The four-field versus five-field comparison holds the backbone, data, and training settings fixed and shows improvements at both model scales, providing credible evidence for the benefit of explicit spatial supervision.

Critical weaknesses
  • ActionScope’s motivation is unclear. The abstract and introduction shift between personalization, annotation cost, and target identification without establishing a central bottleneck. They do not convincingly explain why conventional element localization is insufficient or why ActionScope is necessary, leaving its conceptual contribution unclear.

  • The experiments do not isolate ActionScope’s advantage. The ablation shows that box supervision helps, but not that ActionScope outperforms conventional element boxes. Incorrect-box interventions demonstrate sensitivity, not faithful grounding. These results therefore fall short of substantiating the claimed representational contribution.

Questions for authors

Does it make sense to organize the motivation around a single progression: reusable interaction semantics are needed -> functional-target misidentification limits annotation accuracy -> ActionScope explicitly supervises the receiving region -> compact models enable scalable annotation -> downstream experiments validate the resulting supervision.

2 comments
AuthorsOpenReview ↗

Limitations of conventional element boxes and the role of ActionScope

We thank the reviewer for their comments. We discussed personalization and annotation cost to explain the need for action explanation. Our goal is accurate, low-cost action explanation from readily available inputs. This supports large-scale annotation and future local analysis of user behavior. We encountered limitations of external element boxes. First, usable boxes may not be obtained reliably. Second, their extent may not match the complete functional unit receiving the action. These limitations motivate ActionScope, which teaches ACE to infer this region directly. For a more detailed explanation of our motivation, please see our first response to Reviewer 3. We will carefully consider your suggestions and revise the Introduction to make this motivation clearer.

Existing work obtains element boxes through three main approaches.

DOM and accessibility trees. These methods depend on environmental interfaces and application-provided structure. Mobile accessibility research and official documentation show that custom controls may not expose each interactive region separately. These regions then lack corresponding element boxes [1, 2]. Even available nodes may not match the visible interface. They may refer to occluded or non-interactable elements or have misaligned boundaries [3, 4].

OmniParser. It generates candidate boxes through element detection and OCR. The original report identifies incorrect box granularity as a common failure. One example shows an OCR box containing clickable text and surrounding text. Its center falls outside the clickable region [5]. Detecting relevant content therefore does not necessarily establish the region receiving the action.

SAM2. SAM2 uses point coordinates as prompts to segment image regions. The SAM2 paper notes that a single point can correspond to multiple valid masks [6]. These may cover a part or a larger whole. A visually plausible segmentation therefore may not match the functional unit receiving the action.

Beyond these availability and granularity issues, matching an action point to a box also has limitations for typing and swiping in our setting. Our typing records contain no coordinates for identifying the input field through spatial matching. Swipes may act on a scrollable region containing multiple elements. The element at the start point may therefore not represent the full region receiving the gesture.

We therefore train ACE to infer the region receiving each action directly from the screenshot, action parameters, and available previous-step textual context. We use ActionScope, the smallest complete UI region receiving the action, as the supervision target. ACE first predicts the corresponding bounding-box field and then conditions its action descriptions on that field, removing the need for externally supplied boxes at inference.

References

[1] Xiaoyi Zhang et al. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–15, 2021.

[2] Android Developers. Make custom views more accessible (Views). Official documentation, n.d. URL https://developer.android.com/guide/topics/ui/accessibility/views/custom-views. Accessed: 2026-11-11.

[3] Gang Li et al. Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale. In CHI Conference on Human Factors in Computing Systems, pp. 1–13, 2022.

[4] Yuxiang Chai et al. AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 2138–2156, 2025.

[5] Yadong Lu et al. OmniParser for Pure Vision Based GUI Agent, 2024. URL https://arxiv.org/abs/2408.00203.

[6] Nikhila Ravi et al. SAM 2: Segment Anything in Images and Videos. In International Conference on Learning Representations, volume 2025, pp. 28085–28128, 2025.

AuthorsOpenReview ↗

Comparing ActionScope with conventional element boxes

To test whether ActionScope offers an advantage over conventional element boxes, we compare their effects on explanation accuracy with Qwen3.8 held fixed. We also measure localization quality directly.

We evaluated all 1,281 click and long-press actions. We compared box sources using the same Qwen3.8 model, prompt, and scoring procedure. All conditions use the pre-action screenshot. With a box, the model receives the marked screenshot and a region crop. Without one, it falls back to the clean screenshot. Every action remains in the evaluation. ACE-8B supplies only its predicted box, without semantic descriptions.

Box sourceCorrect / totalJoint semantic accuracyActions without a usable box
ACE-8B ActionScope1,208 / 1,28194.30%0 (0.00%)
SAM21,025 / 1,28180.02%49 (3.83%)
OmniParser1,087 / 1,28184.86%217 (16.94%)

With Qwen3.8 held fixed, ACE-8B's ActionScope boxes yield higher semantic accuracy than SAM2 or OmniParser boxes. This indicates that the learned regions provide more useful guidance for identifying the action recipient and explaining its function.

We also evaluate localization using intersection over union (IoU) on the same actions. Each reference box was manually annotated and checked, then reviewed and corrected in two rounds by different reviewers.

Mean IoU is computed only over samples with available boxes. The IoU threshold rates use all 1,281 actions as the denominator; samples without a box do not meet either threshold.

MethodSamples with a boxMean IoU (available boxes)IoU ≥ 0.50IoU ≥ 0.75
SAM21,232 (96.17%)0.277820.30%11.87%
OmniParser1,064 (83.06%)0.572555.19%26.54%
ACE-1.3B1,281 (100%)0.726386.65%55.35%
ACE-8B1,281 (100%)0.750486.96%62.14%

Both ACE variants provide boxes for all evaluated actions and achieve higher mean IoU with the human reference regions than SAM2 or OmniParser. These findings are consistent with ActionScope's definition as the smallest complete UI region receiving the action. Its supervision encourages ACE to cover the functional recipient as a whole while limiting unrelated surrounding content.

Comments on the paper

1 comment
Area_Chair_kV1uOpenReview ↗

AC Note: Rebuttal Received, Please Engage in Discussion

Dear authors and reviewers,

Thank you to the authors for their detailed rebuttal and for the time and care they put into addressing the reviewers' comments. I appreciate the additional clarifications and results.

Reviewers: please read the authors' responses carefully and reply to them before the end of the discussion period. Even a short acknowledgment is valuable. It is especially helpful to say whether your concerns have been addressed and, if not, what remains unresolved. If the rebuttal changes your assessment, please update your score and review accordingly, whether up or down. If your score stays the same, a brief explanation of why helps me and the senior area chairs.

Authors: I will follow the discussion closely and take your rebuttal fully into account when forming my recommendation.

Thank you all for your time and effort.

Best regards, Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.