- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies GUI target realization by separating element selection from coordinate generation. It introduces FPIR, an element selector that matches the instruction against candidate UI elements and provides the selected element’s information to a frozen GUI agent. The receiver then predicts its own execution coordinate. The evaluation covers 120 Android states across 4 apps and 3 GUI models.
Critical strengths
- Well-motivated problem. Separating target selection from coordinate generation is a reasonable approach to GUI action execution.
- Useful empirical observations. I appreciate the paired execution design, which uses independent resets and programmatic checkers. The hint ablation is also interesting.
Critical weaknesses
- The connection between the supervision analysis and FPIR is unclear. The paper motivates element-level supervision by showing that conflicting action labels distort a shared element score. However, Table 1 shows that mechanism-interaction action BCE achieves the same element Top-1 accuracy as element BCE. This suggests the issue is specific to shared-score parameterization. Also, the native FPIR selector uses a separately trained scoring model. I’m unclear how the supervision analysis contributes to the proposed method or explains its empirical gains. It makes the paper difficult to follow.
- Lack of strong baselines. The main experiments only compare frozen receivers with and without FPIR element hints. My concern is the lack of direct comparisons with existing GUI grounding methods like GUI-Actor, Aria-UI, and ReGA.
- The experiments do not isolate the contribution of FPIR. Appendix A.3 gives that the conditioned prompt includes not only the selected element’s text, role, and bounds, but also an explicit instruction to use that element without switching targets. The no-hint baseline does not receive this additional instruction. I would like to see more ablations about prompts.
Questions for authors
- What is the specific advantage of element-level supervision compared to mechanism-interaction action BCE?
- How does the theoretical analysis inform the design of the native FPIR selector?
- Could the authors compare FPIR with existing GUI grounding methods if time permits? Or could the authors clarify what improvement FPIR can bring beyond existing methods?
- I would like to see more ablations about prompts, especially for the additional instruction.