- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents GUI-ACE for explaining recorded GUI interactions from a screenshot, action parameters, and previous-step textual context. The authors introduce ActionScope as an intermediate representation of the UI region receiving an action, build the ACE-Curator pipeline to generate training annotations, and train ACE-1.3B and ACE-8B to predict ActionScope together with action target and purpose descriptions. The paper also evaluates the generated semantics on downstream GUI navigation and grounding tasks.
Critical strengths
- The overall pipeline is complete, covering ActionScope annotation, quality filtering, model training, and downstream use of the generated action semantics.
- The experimental evaluation is comprehensive, including comparisons with general VLMs, ActionScope ablations, bounding-box interventions, different visual inputs, and downstream navigation and grounding experiments.
Critical weaknesses
- Unclear practical necessity of a dedicated action-explanation model. The paper shows that ACE outperforms direct prompting of general VLMs, but does not compare against a natural alternative of using a strong VLM followed by verification or filtering. This baseline is especially relevant because ACE-Curator itself shows that generation with quality filtering can already achieve high annotation accuracy.
- Insufficient direct validation of ActionScope. ActionScope is the central representation in the paper, but the main evaluation metric about ActionScope focuses on semantic correctness and explicitly excludes bounding-box geometry. The ablation study shows that region supervision helps semantic explanation, but do not directly measure whether ACE predicts accurate ActionScope regions.
Questions for authors
- How does ACE compare with a strong VLM plus verification/filtering pipeline in terms of accuracy, cost, and inference efficiency?
- Can the authors provide the localization quality and consistency of predicted ActionScope regions?
5 comments
ACE-Curator as an alternative to ACE
Why ACE-Curator does not meet the deployment requirements (W1, Q1)
First, our current ACE-Curator pipeline calls cloud-hosted large VLMs. It therefore does not meet the requirement for on-device inference that keeps personal interaction records entirely local.
Second, the reported annotation configuration uses screenshots before and after actions. It therefore does not meet the single-screenshot input requirement targeted by ACE.
Third, localization, semantic generation, and verification involve multiple large-model calls. Repeating this process for each new action retains high inference costs, which conflicts with our goal of low-cost, repeated action explanation.
Finally, filtering rejects explanations that fail verification. Actions that are harder for the model to interpret reliably are therefore more likely to be excluded. For training data construction, retaining these cases at lower automatic coverage requires more human annotation, increasing labor costs. For continuous user behavior analysis, excluding them leaves gaps in the semantic stream. Such selective coverage does not meet our requirements for low-cost annotation and continuous interpretation across routine and difficult scenarios.
We therefore use ACE-Curator to construct training annotations offline and manually annotate the difficult cases flagged by the pipeline. We then train ACE as a student model with both region and semantic supervision, so it learns to identify the region receiving an action and explain the action more accurately.
Additional experiments with ACE-Curator (W1, Q1)
To address the reviewer's question, we additionally compare ACE with ACE-Curator in accuracy and inference efficiency.
Accuracy at full coverage. We evaluate all 1,666 actions without filtering, so difficult cases remain in the comparison. We use Qwen3.5-35B-A3B for the first three Curator stages: candidate localization, visual correction, and separate screen and action annotation. These stages propose and refine a target region, then generate screen and action descriptions in separate passes. We omit Stage 4 quality filtering. When no usable region is obtained, the pipeline falls back to direct annotation from the pre- and post-action screenshots. We use the same Qwen judge protocol as the main evaluation. The table reports joint semantic accuracy (%); Point includes clicks and long presses.
| Method | Overall | Point | Type | Swipe | Standard | Hard |
|---|---|---|---|---|---|---|
| Qwen3.5-35B-A3B Curator | 92.38 | 92.27 | 94.48 | 91.18 | 95.78 | 88.14 |
| ACE-1.3B | 92.50 | 93.29 | 92.27 | 87.75 | 96.10 | 88.01 |
| ACE-8B | 94.54 | 95.08 | 95.03 | 90.69 | 97.29 | 91.11 |
Curator correctly explains 1,539 of 1,666 actions. With a single screenshot, ACE-1.3B achieves comparable overall accuracy, exceeding Curator by 0.12 percentage points. ACE-8B improves overall accuracy by 2.16 percentage points.
Inference efficiency. We measure all methods on the same 100 actions using one RTX PRO 6000 96GB GPU, BF16, batch size 1, and serial inference. We report the fastest working configuration tested for each method. All use Transformers, SDPA, and four CPU threads.
| Method | Mean latency (s/action) | P95 latency (s) | Peak GPU memory (GiB) | GPU energy (J/action) |
|---|---|---|---|---|
| ACE-1.3B | 0.705 | 0.845 | 5.33 | 134.0 |
| ACE-8B | 2.352 | 2.833 | 20.22 | 717.0 |
| Qwen3.5-35B-A3B Curator | 25.619 | 45.454 | 69.16 | 3652.6 |
Relative to Curator, ACE-1.3B and ACE-8B achieve 36.3× and 10.9× speedups in mean latency, respectively, while using less GPU memory and energy. These results show the practical benefit of training ACE: it provides comparable or higher overall accuracy with single-image input and substantially lower inference costs on the same hardware.
Direct evaluation of ActionScope localization quality
Direct evaluation of ActionScope localization (W2, Q2)
We added a direct evaluation of the predicted ActionScope boxes on all 1,666 actions for both single-image ACE variants. We assess whether each box identifies the intended action recipient and measure its geometric overlap with the reference region.
Evaluation protocol. Human reviewers judge target correctness. Oversized or undersized boxes count as correct if they still unambiguously identify the intended recipient. Boxes pointing to another element or leaving the target unclear count as incorrect. We separately report intersection over union (IoU) to measure geometric agreement with the reference boxes. Both annotators reviewed all 1,666 predictions for each model. Their binary target-correctness judgments differed on only four cases (0.24%) for ACE-1.3B and eight (0.48%) for ACE-8B. Given this close agreement, we use the second annotator's judgments in the reported results.
Localization quality. The table reports human-assessed target correctness and geometric overlap. The percentage of boxes with IoU ≥ 0.5 is computed over all 1,666 actions.
| Model | Target correctness | Mean IoU | Median IoU | IoU ≥ 0.5 |
|---|---|---|---|---|
| ACE-1.3B | 97.18% (1,619/1,666) | 0.7433 | 0.8022 | 87.09% (1,451/1,666) |
| ACE-8B | 97.24% (1,620/1,666) | 0.7726 | 0.8391 | 88.36% (1,472/1,666) |
Both models identify the intended recipient in over 97% of cases. Their geometric overlap with the reference boxes also shows that the predictions generally capture the relevant region, beyond identifying the target alone.
Relationship to explanation accuracy. We further group predictions by whether their target and purpose descriptions are jointly correct, using the original semantic evaluation protocol. Percentages in each row use that group's sample count as the denominator.
| Model | Semantic judgment | Samples | Target correctness | Mean IoU | IoU ≥ 0.5 |
|---|---|---|---|---|---|
| ACE-1.3B | Correct | 1,541 | 98.70% | 0.7584 | 89.16% |
| ACE-1.3B | Incorrect | 125 | 78.40% | 0.5573 | 61.60% |
| ACE-8B | Correct | 1,575 | 98.73% | 0.7841 | 89.84% |
| ACE-8B | Incorrect | 91 | 71.43% | 0.5727 | 62.64% |
For both models, correct explanations are associated with more accurate target identification and higher IoU. These results complement the semantic evaluation by directly measuring localization quality and showing its relationship to explanation correctness.
Please also see the supplementary experiment in our response to Reviewer 1, which fixes Qwen3.8 and the prompt while changing only the box source. ACE-8B supplies its predicted ActionScope boxes without semantic descriptions. On all 1,281 point actions, joint semantic accuracy reaches 94.30% with ActionScope, compared with 80.02% for SAM2 and 84.86% for OmniParser. Cases without a usable box fall back to the clean screenshot and remain in the evaluation. Together with the human review and IoU results above, this provides further evidence that learned ActionScope regions help identify the action recipient and support more accurate action explanations.
Application requirements and the motivation for ACE
To explain why a dedicated model is needed, we first clarify the application requirements that motivate ACE. We then discuss whether a strong VLM with verification and filtering can meet these requirements.
Building training data for graphical user interface (GUI) agents and analyzing user behavior both rely on semantic explanations of recorded actions. For training data construction, the goal is to lower annotation costs at scale while maintaining quality. For user behavior analysis, interpreting everyday interactions can support preference extraction and proactive assistance. Privacy concerns motivate local processing of interaction records, while proactive assistance calls for low end-to-end latency. We therefore study small models for action explanation to reduce annotation costs and end-to-end latency and support future on-device deployment.
When given action coordinates and a single screenshot, general vision-language models (VLMs) may struggle to identify the complete functional unit receiving the action. This can cause them to describe the wrong UI element. To address this problem, existing methods provide VLMs with external element boxes, screenshots taken before and after the action, or both. The models use these inputs together with action parameters to generate semantic descriptions. Element boxes can be obtained from a webpage’s Document Object Model (DOM) or an application’s accessibility tree. Detection or segmentation models can also generate them from screenshots. These boxes provide candidate locations, while the screenshot pairs provide clues about interface changes.
However, both types of auxiliary input have limitations in availability and usefulness. For element boxes, some applications or environments do not allow access to the DOM or accessibility tree. Detection or segmentation models may also fail to produce a candidate box that matches the action location. Even when a box is available, it may not match the complete functional unit receiving the action. It may cover only part of that unit or include unrelated controls. For screenshot pairs, the second image may not accurately reflect the response to the current action if captured too early or too late. Even with appropriate timing, an action may produce no visible feedback, or different controls may produce similar feedback. In either case, the screenshots may provide little guidance for action explanation. Moreover, obtaining and processing both types of auxiliary input adds overhead. Together with the inference cost of large models, this overhead limits annotation at scale and use on local devices.
We therefore aim to train small models to identify the functional unit receiving an action and understand its function on the current page. We define ActionScope as the smallest complete UI region corresponding to this unit. Its bounding box (bbox) provides supervision for learning to identify the region receiving the action. Semantic supervision teaches the model to interpret the unit’s function using information from the current page. Using both forms of supervision, we train Action-Conditioned Explanation (ACE), a small VLM. Within a single autoregressive sequence, ACE first generates the ActionScope bbox field. It then generates descriptions of the recipient and the action’s function, conditioned on that field. At inference, ACE needs only a single screenshot, action parameters, and available textual context from the previous step. It requires no external element boxes or second screenshot.