- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SAGE combines autonomous mobile-app exploration with training for selective use of imperfect trajectory priors. Its explorer builds a deduplicated UI transition graph and synthesizes task-aligned semantic paths. Five drift-inspired corruption operators, plus an identity condition, generate supervision for FOLLOW, PARTIAL, and IGNORE decisions, which precede action generation in a shared autoregressive policy. The experiments measure exploration efficiency and subtask-interface coverage, controlled prior-induced step-accuracy gains across five backbones, online execution on 144 exploration-covered tasks, cross-dataset transfer, and a matched ablation of explicit reliability-label supervision. The evidence supports improved utilization of supplied priors in the evaluated conditions; the main results leave the absolute and population-wide deployment benefit less settled.
Critical strengths
A concrete and empirically motivated reliability problem
The introduction compares completion trajectories for 176 tasks across old and new versions of 20 apps and reports that only 44% remain unchanged (Figure 1; PDF p. 2, lines 054–064). Sections 3.3–3.4 translate recurring drift modes into explicit reliability and trusted-edge supervision, while keeping executable action targets tied to recorded trajectories (PDF p. 5, lines 216–269). This gives the method a clear operational motivation and an inspectable training mechanism. Explicit partial use is a useful design choice for priors containing both valid and stale segments, and deriving actions from recorded trajectories avoids introducing additional teacher-generated action errors.
Exploration is evaluated for functional relevance as well as state count
The acquisition pipeline combines semantic-region budgets, state deduplication, and recovery-aware graph repair (Section 3.2). Figure 3 reports approximately 148 deduplicated screens and 28 activities within 600 steps, while LLM-Explorer reaches approximately 101 screens and 10 activities at 1,500 steps. Under the stated common exploration budget, overall coverage of 428 atomic subtask target interfaces rises from 57.2% to 74.1%, with gains on each benchmark (PDF pp. 6–7, lines 302–327). The LLM explorer comparisons share a backbone and initial conditions. The interface-coverage measure connects exploration to downstream task requirements and makes the acquisition result more useful than a raw count of visited screens. The results provide concrete evidence that the graph can supply a broader pool of relevant candidate priors under an interaction budget.
The reliability-label contribution receives a meaningful matched control
Section 4.5 states that Part-CoT and SAGE share inputs, Action/Tool targets, training, and evaluation, with explicit reliability-label supervision as the difference (PDF p. 9, lines 461–466). Table 5 shows higher mean prior-induced gains for SAGE in 12 of 15 backbone–dataset settings and an overall increase from 5.59 to 7.87 percentage points (lines 473–478). Table 2 also shows improvements in prior-induced gains over direct SFT prior injection across all five evaluated backbones and six controlled conditions. The Part-CoT comparison addresses the important alternative explanation that ordinary rationale supervision alone accounts for the result. Its consistency across backbones and datasets strengthens the evidence for explicit reliability supervision, within the reported gain metric.
Critical weaknesses
Larger prior-induced gains do not establish better absolute execution
over each method’s own no-prior baseline
Source: Table 3 caption, Section 4.3; PDF p. 8; manuscript lines 378–379.
Concern. The gain metric validly measures how adding a prior changes a particular trained policy, but each variant has a different no-prior baseline. A larger gain can coexist with lower final accuracy or success, including in the matched Part-CoT ablation. Consequently, the green improvements in the tables cannot by themselves establish that SAGE is the better-performing policy in absolute terms. This is consequential online: Table 3 reports SAGE gains of −2.60 and −1.70 percentage points on AndroidWorld, so using the supplied priors still reduces success there; AndroidLab gains of 0.31 and 1.26 points also warrant uncertainty estimates. Equal averaging over prior categories further leaves the expected benefit under a realistic retrieval mixture uncertain. The paper acknowledges the negative effects, which is a useful qualification, but the practical utility remains unresolved by the reported main tables.
Suggested clarification or experiment. Report absolute no-prior and prior-conditioned accuracy/success for each compared variant alongside the gains, using the same evaluation instances. For the online study, give per-condition paired differences with uncertainty and show how the result changes under plausible prior-quality mixtures. Evaluate a concrete retrieval-bypass or no-prior fallback on the same tasks to determine whether the remaining AndroidWorld degradation can be avoided without forfeiting useful-prior gains.
The online result is conditional on coverage and task-specific prior supply
144 exploration-covered tasks
Source: Section 4.3, Setup; PDF p. 7; manuscript lines 376–377.
Concern. The online study supplies task-specific priors constructed from exploration graphs for a selected covered subset. Figure 3 measures a different quantity: coverage of 428 atomic subtask target interfaces derived from 289 benchmark tasks. Visiting one such interface does not establish that a complete usable task path is available, and the 74.1% subtask statistic cannot be used as the coverage denominator for the 144 online tasks. The main description also does not establish how an operational retriever selects a prior from the full graph for an incoming task, despite introducing a retriever in Section 3.1. These selection steps affect both which tasks receive help and the distribution of errors encountered. The reported experiment therefore provides conditional evidence about supplied-prior use, with an incomplete link from the measured exploration improvement to overall task success.
Suggested clarification or experiment. Provide benchmark-specific counts and criteria from eligible tasks through subtask coverage, complete usable paths, and selection of the 144 online cases. Specify and evaluate the actual prior-selection procedure over the full graph. With the existing policies, report success on the full eligible task set, including a defined no-prior fallback for unavailable or unsuitable paths, and separate covered and uncovered outcomes. A paired comparison using SAGE and LLM-Explorer graphs with the same retriever and executor would directly test whether the coverage improvement translates into end-to-end benefit.
Transfer across datasets does not isolate robustness beyond the corruption construction
retaining the six-way perturbation taxonomy
Source: Section 4.4, Setup; PDF p. 9; manuscript lines 435–436.
Concern. Training reliability labels are determined by the generating operator, and the teacher receives the fixed reliability and action targets when writing its rationale (Sections 3.3–3.4). The controlled tests and cross-dataset transfers retain the same six corruption conditions. Thus, transfer changes the trajectory domain while preserving the label-generating construction, leaving open whether part of the effect comes from recognizable corruption cues rather than assessing applicability in the current execution state. Natural drift can combine changes and alter action feasibility in ways that a single synthetic operator does not capture. Section 4.2 explicitly reports a held-out real cross-version evaluation with class-wise F1 and Macro-F1 in Appendix O; that relevant validation should be credited. The unresolved scientific issue is whether such reliability generalization yields correct actions and successful tasks under natural drift, since classification performance and the current controlled gain experiments answer different questions.
Suggested clarification or experiment. Evaluate the frozen policy on genuinely historical priors executed in held-out newer app versions, reporting both reliability decisions and absolute task success relative to no prior. A held-out subset of the paired-version task pool already introduced in Figure 1 could make this practical. Include a modest control with unseen compositions or a held-out perturbation generator to check whether improvements persist when corruption cues change, and audit reliability judgments against current-state applicability as well as the operator label.
Questions for authors
-
What are the absolute no-prior and prior-conditioned online success rates for SFT and SAGE, and for the Part-CoT ablation offline? Do the policy rankings agree with the rankings of prior-induced gains?
-
How are the 144 online tasks selected from the eligible benchmark tasks, what constitutes a usable full-task path, and how is a task-specific prior retrieved from the complete graph without task-specific selection assistance?
-
On naturally drifted, held-out app versions, how do reliability predictions relate to executable action correctness and task success? Do the gains persist for unseen combinations or generators of drift, and with a practical no-prior fallback?
4 comments
Response to Reviewer 7XV4 (2/2)
Q2 & W2: Task selection and online retrieval.
- Selection. We start from the 289 tasks of the coverage study (MW/AL/AW = 129/85/75), split each task into subtasks by the validator's checks and identify their target interfaces. A usable full-task path contains all screenshots and executed actions from the home screen to the destination interface. After removing cross-app tasks and, for MobileWorld, tasks requiring the MCP tool protocol, we keep all tasks with a usable full-task path: 144 tasks (33/64/47). The 74.1% is subtask coverage over the full 289-task pool and is not the denominator for these 144 tasks.
- Retrieval. The main online study injects matched priors, because its focus is how erroneous knowledge affects the model. To test retrieval, we used a fixed retriever and executor to obtain priors from both the SAGE and the LLM-Explorer graphs. On the same 144 tasks, we generated priors for LLM-Explorer with the same prior-generation method as ours. We adapted the Manager-RAG retrieval component of Mobile-Agent-RAG (AAAI 2026): the task instruction retrieves the top-1 passage from the complete benchmark-specific prior bank, with no app-level filtering and no manual prior assignment at inference, and only the retrieved path text is given to the executor. Both banks use the same retrieval settings and the same executor (Ours on Qwen3-VL-4B), with matched task initializations and execution budgets. We report the task success rate and the retrieval accuracy (the share of tasks whose retrieved prior matches the task) for each exploration graph.
Task success rate
| Benchmark | SAGE Graph | LLM-Explorer Graph |
|---|---|---|
| AndroidWorld | 25.53% | 17.02% |
| AndroidLab | 35.94% | 29.69% |
| MobileWorld | 15.15% | 9.09% |
| Overall | 27.78% | 20.83% |
Retrieval accuracy
| Benchmark | SAGE Prior-Matched | LLM-Explorer Prior-Matched |
|---|---|---|
| AndroidWorld | 87.23% | 38.30% |
| AndroidLab | 95.31% | 71.88% |
| MobileWorld | 78.79% | 51.52% |
| Overall | 88.89% | 56.25% |
Q3 & W3: Natural drift, unseen drift generators and fallback.
- Natural drift. We ran frozen SFT and Ours on held-out tasks in newer app versions: 176 paired tasks across 20 apps (1,137 old-version and 1,023 new-version steps), excluded from training. Each new-version step is evaluated with no prior, a same-version prior and a cross-version prior from the old-version trajectory. Step accuracy:
| Backbone | No prior: SFT / Ours | Same-ver.: SFT / Ours | Cross-ver.: SFT / Ours |
|---|---|---|---|
| Qwen3-VL-2B | 61.28 / 66.96 | 64.19 / 73.62 | 62.25 / 72.27 |
| Qwen3-VL-4B | 75.75 / 79.42 | 81.64 / 83.09 | 77.85 / 82.13 |
| Qwen3-VL-8B | 69.64 / 73.72 | 76.69 / 80.39 | 74.57 / 80.10 |
| MAI-UI-2B | 64.18 / 66.73 | 65.13 / 76.65 | 61.42 / 73.06 |
| GELab-Zero-4B | 79.08 / 78.24 | 81.42 / 82.70 | 79.33 / 80.99 |
With cross-version priors, Ours is more accurate than SFT on all five backbones, and its gain over no prior is positive and larger than SFT's on all five. Labels on this set are assigned against the recorded new-version trajectory, i.e. by applicability in the current state. 2. Unseen drift generators. This drift comes from real version updates, and no perturbation operator generated it. It therefore serves as a held-out drift generator, and the gains persist under it. 3. No-prior fallback. We simulated a hard fallback offline: whenever Ours outputs IGNORE, its action is replaced by its own no-prior prediction. This does not improve accuracy, so the IGNORE decision already acts as an internal fallback.
Revised PDF
Thank you for your careful review and constructive feedback. We have submitted a revised PDF incorporating your suggestions. Note that, for readability, we marked the revisions to Tables 3, 5, and 6 in their captions and kept the numerical entries in black.
- Result reporting. As requested, Table 3 now reports absolute online task success rates for the no-prior baseline and each of the six prior conditions. Table 6 reports absolute offline step accuracies with and without priors for the Part-CoT ablation. Both tables retain mean Δ for reference.
- New experiments. We compared prior banks built from SAGE and LLM-Explorer graphs on the same 144 tasks with a fixed retriever and executor (Appendix Q, Table 14). The SAGE bank yielded higher top-1 retrieval accuracy (88.89% vs. 56.25%) and task success (27.78% vs. 20.83%). We also tested generalization to real version drift on 176 held-out task pairs across 20 apps (Sec. 4.4, Table 4).
- Evaluation protocol. Section 4.3 clarifies how we selected the 144 online tasks, including the requirement for a complete recorded path from the home screen to the target interface.
We hope these revisions address your concerns, and we look forward to continuing the discussion with you.
Follow-up on Our Response and Revised Manuscript
We understand this is a busy period and sincerely appreciate your time. As the discussion period is drawing to a close, we would like to kindly follow up on our response.
In particular, the revised PDF now reports absolute online and offline results (Tables 3 and 6), a retrieval-based comparison of SAGE and LLM-Explorer prior banks (Appendix Q), and results under real app-version drift (Sec. 4.4). We would be grateful to know whether these address your concerns, and we are happy to clarify anything further.
Response to Reviewer 7XV4 (1/2)
We thank the reviewer. Since you and the other reviewers raised concerns about the uncertainty of our results, we re-ran the offline and online experiments several times; all numbers below are from the latest runs, and the main conclusions remain unchanged.
Q1 & W1: Absolute results and ranking consistency.
With priors, Ours has a higher mean SR than SFT in 6/6 benchmark–backbone pairs, a higher SR in 35/42 cells and a larger prior-induced gain in 35/36. Ours' own gain pooled over benchmarks is +6.53 pp [+0.43, +12.57] for Qwen3-VL-4B and +3.93 pp [−1.52, +9.22] for GUI-Owl-1.5-4B (paired bootstrap over tasks, B = 5,000).
Do the rankings agree? For the matched ablation, mostly yes. The Δ ranking and the absolute ranking of Ours vs. Part-CoT agree in 13/15 in-domain and 25/30 transfer settings. In the 2 in-domain disagreements, Ours has the higher absolute accuracy despite a smaller Δ; in transfer, Ours has a larger Δ but a lower absolute accuracy in only 2 settings, by 0.52 and 0.16 pp, both with CIs including 0.
Against SFT they agree less often, because the methods start from different no-prior baselines. Offline, Ours with prior still exceeds the better of SFT's no-prior and with-prior accuracy in 22/30 transfer settings (+1.84 [+1.09, +2.44]) and 10/15 in-domain settings (+0.66 [−0.37, +1.52]). Online, it does so on all three benchmarks for Qwen3-VL-4B, but for GUI-Owl only on MW.