acceptodds
Withdrawn from ICLR 2027

Element-Conditioned Target Realization with Frozen GUI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

GUI execution couples two decisions that need not be learned by the same model: which element the instruction refers to and where to act on that element. We introduce an element-conditioned composition method that assigns target selection to an element selector while preserving the frozen receiver’s coordinate generation. Our element selector, FPIR, selects a candidate element and supplies its recorded text, role, and bounds as a hint. On 120 held-out states across four Android applications, FPIR element hints improve frozen receivers’ coordinate realization, raising native checker success from 90/120 to 98/120 for MAI-UI-8B, 78/120 to 98/120 for UI-Venus-1.5-8B, and 36/120 to 85/120 for AgentCPM-GUI-8B. A separate supervision audit shows that assigning conflicting action labels to a shared element score distorts the element-learning target. A same-state ablation finds that text and role without bounds retain most of the native execution gains, while the value of spatial bounds depends on the receiver.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI agents, visual grounding, factorized supervision, modular composition, vision-language models, human-computer interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BmkDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies GUI target realization by separating element selection from coordinate generation. It introduces FPIR, an element selector that matches the instruction against candidate UI elements and provides the selected element’s information to a frozen GUI agent. The receiver then predicts its own execution coordinate. The evaluation covers 120 Android states across 4 apps and 3 GUI models.

Critical strengths
  1. Well-motivated problem. Separating target selection from coordinate generation is a reasonable approach to GUI action execution.
  2. Useful empirical observations. I appreciate the paired execution design, which uses independent resets and programmatic checkers. The hint ablation is also interesting.
Critical weaknesses
  1. The connection between the supervision analysis and FPIR is unclear. The paper motivates element-level supervision by showing that conflicting action labels distort a shared element score. However, Table 1 shows that mechanism-interaction action BCE achieves the same element Top-1 accuracy as element BCE. This suggests the issue is specific to shared-score parameterization. Also, the native FPIR selector uses a separately trained scoring model. I’m unclear how the supervision analysis contributes to the proposed method or explains its empirical gains. It makes the paper difficult to follow.
  2. Lack of strong baselines. The main experiments only compare frozen receivers with and without FPIR element hints. My concern is the lack of direct comparisons with existing GUI grounding methods like GUI-Actor, Aria-UI, and ReGA.
  3. The experiments do not isolate the contribution of FPIR. Appendix A.3 gives that the conditioned prompt includes not only the selected element’s text, role, and bounds, but also an explicit instruction to use that element without switching targets. The no-hint baseline does not receive this additional instruction. I would like to see more ablations about prompts.
Questions for authors
  1. What is the specific advantage of element-level supervision compared to mechanism-interaction action BCE?
  2. How does the theoretical analysis inform the design of the native FPIR selector?
  3. Could the authors compare FPIR with existing GUI grounding methods if time permits? Or could the authors clarify what improvement FPIR can bring beyond existing methods?
  4. I would like to see more ablations about prompts, especially for the additional instruction.
Reviewer_11uhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FPIR, an element-conditioned framework that addresses the coupling between target element selection and action coordinate generation in GUI agents. An independent element selector matches user intent to candidate element attributes, after which the selected element’s text, role, and bounding box are provided as prompting context to a frozen GUI agent for coordinate generation, without retraining the agent. Experiments on 120 single-step interaction states across four Android apps improve the execution success rates of three frozen GUI agents from 75.0%, 65.0%, and 30.0% to 81.7%, 81.7%, and 70.8%, respectively. The main limitation is that the evaluation covers only tap and long-press operations in four apps, with action intents and candidate elements provided in advance; generalization to complex, multistep real-world GUI tasks remains unverified.

Critical strengths
  1. The modular design is simple and practical, requiring no modifications to the base GUI agents, and its effectiveness is demonstrated across three different models.

  2. The paired experiments and ablation studies are clearly designed. The BCE derivation is mathematically sound, and the experiments help distinguish the contributions of semantic element hints and coordinate information.

Critical weaknesses
  1. The proposed framework mainly combines an independent element selector with frozen GUI agents through prompting. The underlying idea of separating decision-making from execution overlaps with existing modular GUI grounding approaches, while its distinction from these methods is not sufficiently demonstrated.

  2. Experiments are restricted to 120 single-step states across four Android applications, with predefined intents, candidate elements, and fixed action types. This controlled setting does not adequately demonstrate generalization to realistic, multi-step GUI tasks.

  3. The paper lacks comparisons with simpler alternatives, such as direct prompting with element descriptions or bounding boxes, making it difficult to isolate the contribution of the proposed selector.

Questions for authors
  1. What is the fundamental methodological difference between FPIR and existing modular grounding or decision–execution separation approaches?

  2. How does FPIR perform in open-ended, multi-step GUI tasks where candidate elements and action types are not predefined?

  3. Could the authors compare FPIR against simpler semantic matching or prompting baselines using the same element information?

  4. Why are action types fixed using MAI-UI predictions for all three receivers, rather than allowing each model to generate its own complete action? How would this affect the reported improvements?

Reviewer_iEZXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Rather than fine-tuning a GUI agent, the paper uses a selector, FPIR, to match the instruction against supplied candidate elements. It combines frozen MiniLM similarities for title and team with a learned status classifier, then passes the selected element's text, role, and bounds as a prompt hint to a frozen GUI agent. Experiments on 120 controlled Android states show execution gains across three GUI models.

Critical strengths
  • The selector improves execution across three GUI models without fine-tuning them.
Critical weaknesses
  1. The benefit of using a GUI model after element selection is unclear. The selected box is already available, and the main experiment fixes the action type. Why not execute at the box center? This would show whether model-generated coordinates justify their cost over a simple rule.

  2. The no-hint agent sees the screenshot and instruction, while FPIR uses structured candidates. The paper does not test giving the same candidate texts, roles, and bounds directly to the GUI agent. That baseline would help distinguish the value of FPIR from the benefit of providing structured information.

  3. Section 4.3 explicitly separates the audit scorer from the deployed selector. The former studies element-level BCE; the latter uses similarities and a learned status classifier. This makes the connection between the two studies indirect. A matched supervision comparison within the execution pipeline would better establish the audit's practical relevance.

  4. The paper describes the components and training, but I still could not follow how an instruction becomes the title, team, and requested status fields in Equation 4. Are these parsed automatically or supplied by the task setup? A worked example showing the inputs, preprocessing, and final hint would help.

Questions for authors
  1. How does receiver-generated execution compare with simply acting at the selected element's box center, in both success rate and latency?
  2. Does FPIR still help compared with giving the same candidate information directly to the GUI agent?
  3. How are the instruction fields and candidate attributes in Equation 4 obtained?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.