acceptodds
Withdrawn from ICLR 2027

From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mobile GUI agents complete tasks using GUI actions like taps and swipes. These actions are broadly applicable across applications, but reaching a task-relevant screen can require many intermediate navigation steps, lengthening trajectories and creating more opportunities for error. Fortunately, many Android applications expose deeplinks—external entry points to specific in-app screens—that provide a higher-level navigation interface. A single deeplink call can replace a sequence of screen-by-screen GUI actions. We therefore introduce hybrid interaction, using deeplinks for direct navigation and GUI actions for other on-screen operations and fallback. To enable this, we discover candidate deeplinks through static analysis, validate them on real devices, and describe their observed landing screens. This process creates a verified and grounded deeplink catalog that pairs each working deeplink with a description of its landing screen. Using this catalog, we introduce GUI-Hopper, a deeplink-augmented GUI agent that learns when to use a deeplink and when to rely on GUI actions. On MobileWorld, GUI-Hopper achieves higher task success with fewer steps than its GUI-only counterpart; its gains persist even without deeplinks at inference. On MobileWorld-Real, it also improves task success in commercial applications on real devices, further demonstrating the benefits of hybrid interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mobile GUI Agents, Deeplinks, Hybrid Interaction, Tool Use, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wzZPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GUI-Hopper, a mobile agent that combines standard GUI actions with app-native deeplinks for more efficient navigation. The authors build a verified deeplink catalog through static analysis and on-device validation, then train the agent to select between deeplink and GUI actions using hybrid trajectories and online RL.

Critical strengths

The idea of augmenting GUI-only mobile agents with app-native deeplinks is reasonable and interesting. Deeplinks provide a natural high-level navigation mechanism that can bypass unnecessary intermediate screens while leaving fine-grained interaction to standard GUI actions.

Critical weaknesses

1.The evaluation is limited to the MobileWorld benchmark family. Results on another established mobile benchmark, such as AndroidWorld, would help establish whether the gains generalize across different task and environment distributions.

2.The paper reports 1,251 verified deeplinks across 26 commercial apps, but it does not quantify how often useful deeplinks are available for realistic mobile tasks. This makes the practical coverage of the proposed interaction paradigm unclear.

Questions for authors

Table 2 does not report GUI-Hopper trained with hybrid actions but evaluated without deeplinks. Could the authors provide this setting on MobileWorld-Real to separate the effect of hybrid training from the benefit of inference-time deeplink access?

Reviewer_szzCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GUI-Hopper, a mobile agent that uses app-native deeplinks alongside ordinary GUI actions. The authors mine deeplink candidates, test them on devices, and describe their landing screens to build a catalog the agent can call. They then train the agent to choose between deeplinks and GUI actions, with a harness that retrieves relevant entries and checks the landing Activity. On MobileWorld and a subset of MobileWorld-Real, the method achieves higher task success and shorter trajectories than GUI-only training.

Critical strengths

S1. I appreciate that the paper tackles the practical details of using deeplinks, rather than simply providing a list of shortcuts in the prompt. The on-device checks, landing-screen descriptions, parameter definitions, and GUI fallback make this a more complete system. Building a catalog of 1,251 verified entries across 26 commercial applications is also a substantial implementation effort (Section 3.2 and Appendix A).

S2. The gains are fairly consistent across model sizes. Compared with GUI-only training, hybrid execution improves success by 6.3–9.1 percentage points on MobileWorld and 9.5–13.8 points on MobileWorld-Real (Tables 1–2). The real-device experiments also show lower elapsed time (Table 8), although the time savings are more modest than the reduction in interaction steps.

Critical weaknesses

W1. I could not determine how the training tasks relate to the evaluation tasks. Appendix C says that each online-RL iteration samples 117 tasks, while Section 4.1 evaluates all 117 MobileWorld tasks. The matching number does not necessarily mean they are the same tasks, but the paper does not clarify the relationship. I also found the source and composition of the SFT trajectories difficult to establish. Without an explicit account of task overlap, it is hard to judge whether the reported gains reflect performance on unseen tasks or further optimization on familiar ones.

W2. I am less convinced by the reliability of the landing verification and recovery mechanism. The harness checks whether the expected Activity is reached, but that does not tell us whether the deeplink filled in the correct task-specific values. Figure 7 illustrates the problem: the agent opens a Calendar form with the wrong date, saves the event, and later creates the requested event through GUI actions. The authors acknowledge this case, and the figure does not show the wrong-date event being removed. So although the agent eventually completes the requested action, it may leave an unintended change behind. I would like to see how common this is and whether the evaluation catches such cases.

Questions for authors

Q1. Could the authors clarify the sources of the SFT and RL tasks and their overlap, if any, with the evaluation sets? In particular, are the 117 tasks sampled per RL iteration distinct from the 117 MobileWorld evaluation tasks? If there is overlap, are results available on held-out tasks?

Q2. Are the GUI-only and hybrid models compared with the same training budget, including online RL? Section 4.1 describes matched training configurations, but I could not tell whether the GUI-only baselines also receive RL updates. It would also help to identify the checkpoints used for each trained-model row in Tables 1–2. In particular, Section 4.2.1 calls the no-deeplink results “hybrid-SFT checkpoints,” while Table 3 reports a final 4B result after online RL.

Q3. Do the task-success checks penalize unintended changes, such as the extra calendar event in Figure 7? How often does a deeplink reach the expected Activity while showing incorrect task-relevant content, and how often does recovery leave such changes behind?

Reviewer_jVYPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GUI-Hopper, which allows mobile agents to use both regular GUI operations and native deeplinks. The authors construct a deeplink catalog through static analysis and runtime validation, retrieve candidate entry points for tasks, and train the agent using supervised fine-tuning and reinforcement learning. Experiments evaluate task success rate and operational costs on MobileWorld and MobileWorld-Real.

Critical strengths

The method addresses the actual efficiency bottlenecks of mobile agents. By utilizing the application's native entry point to skip repetitive navigation while retaining GUI back navigation, it represents a valuable design approach.

The consistent results across multiple model sizes and evaluation scenarios demonstrate the method's generalization ability. For example, the 9B model improved the success rate on MobileWorld from 54.4% to 63.5% on GUI-only, and from 77.1% to 86.6% on MobileWorld-Real.

Critical weaknesses

The paper doesn't clearly state whether the training and testing sets use the same set of questions. The main text states that 117 tasks from MobileWorld were tested, while the appendix mentions that 117 tasks are used per round of reinforcement learning. Do these two sets of questions overlap? If the test questions were already used for training, the increased success rate might simply be because the model practiced with those questions, not necessarily indicating it's better at handling new tasks.

While the task was ultimately completed correctly, were any errors made in the middle cleaned up? In the calendar example in the paper, the agent first created an event with an incorrect date, then created the correct one, but it doesn't explain whether the incorrect date was deleted. If two dates—one correct and one incorrect—are left at the end, I don't consider it a complete success. It needs to be clarified whether the testing checks for these additional errors, not just whether the target date was created.

Questions for authors

How much manual work is required to prepare these deeplinks? Which steps can be automated, and which require human review or modification? Have these preparation costs been factored into the efficiency comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.