acceptodds
Under review as a conference paper at ICLR 2027

Learning Reliable GUI Agents under Imperfect Priors

Abstract

GUI agents built on large language and vision-language models still struggle on unseen applications and complex multi-step tasks, as completing real GUI tasks depends on app-specific, temporally volatile operational knowledge that is scarce in pretraining corpora. Retrieval-augmented execution offers a natural remedy but faces two coupled bottlenecks: knowledge at scale is hard to acquire, and self-collected priors inevitably drift from the live environment due to version updates, promotions, ads, A/B tests, and personalization. We therefore argue that GUI agents should not pursue perfect knowledge but learn to act correctly under imperfect priors, and propose our framework that couples knowledge acquisition with noise-robust utilization: a structured exploration strategy traverses interactive elements, builds a UI state-transition graph, and synthesizes (task, trajectory) pairs via a VLM without human annotation; a noise-aware training strategy, grounded in a taxonomy of real GUI drift patterns, injects five types of realistic errors into self-explored trajectories to teach the agent to assess prior reliability before acting. Experiments on physical devices and online emulator benchmarks show that our method discovers more unique screens, covers more benchmark tasks, and more effectively rejects erroneous priors while leveraging correct ones, with accuracy gains that transfer across datasets.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
GUI Agents, Imperfect Priors, Reliability-Aware Learning, GUI Exploration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7XV4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SAGE combines autonomous mobile-app exploration with training for selective use of imperfect trajectory priors. Its explorer builds a deduplicated UI transition graph and synthesizes task-aligned semantic paths. Five drift-inspired corruption operators, plus an identity condition, generate supervision for FOLLOW, PARTIAL, and IGNORE decisions, which precede action generation in a shared autoregressive policy. The experiments measure exploration efficiency and subtask-interface coverage, controlled prior-induced step-accuracy gains across five backbones, online execution on 144 exploration-covered tasks, cross-dataset transfer, and a matched ablation of explicit reliability-label supervision. The evidence supports improved utilization of supplied priors in the evaluated conditions; the main results leave the absolute and population-wide deployment benefit less settled.

Critical strengths

A concrete and empirically motivated reliability problem

The introduction compares completion trajectories for 176 tasks across old and new versions of 20 apps and reports that only 44% remain unchanged (Figure 1; PDF p. 2, lines 054–064). Sections 3.3–3.4 translate recurring drift modes into explicit reliability and trusted-edge supervision, while keeping executable action targets tied to recorded trajectories (PDF p. 5, lines 216–269). This gives the method a clear operational motivation and an inspectable training mechanism. Explicit partial use is a useful design choice for priors containing both valid and stale segments, and deriving actions from recorded trajectories avoids introducing additional teacher-generated action errors.

Exploration is evaluated for functional relevance as well as state count

The acquisition pipeline combines semantic-region budgets, state deduplication, and recovery-aware graph repair (Section 3.2). Figure 3 reports approximately 148 deduplicated screens and 28 activities within 600 steps, while LLM-Explorer reaches approximately 101 screens and 10 activities at 1,500 steps. Under the stated common exploration budget, overall coverage of 428 atomic subtask target interfaces rises from 57.2% to 74.1%, with gains on each benchmark (PDF pp. 6–7, lines 302–327). The LLM explorer comparisons share a backbone and initial conditions. The interface-coverage measure connects exploration to downstream task requirements and makes the acquisition result more useful than a raw count of visited screens. The results provide concrete evidence that the graph can supply a broader pool of relevant candidate priors under an interaction budget.

The reliability-label contribution receives a meaningful matched control

Section 4.5 states that Part-CoT and SAGE share inputs, Action/Tool targets, training, and evaluation, with explicit reliability-label supervision as the difference (PDF p. 9, lines 461–466). Table 5 shows higher mean prior-induced gains for SAGE in 12 of 15 backbone–dataset settings and an overall increase from 5.59 to 7.87 percentage points (lines 473–478). Table 2 also shows improvements in prior-induced gains over direct SFT prior injection across all five evaluated backbones and six controlled conditions. The Part-CoT comparison addresses the important alternative explanation that ordinary rationale supervision alone accounts for the result. Its consistency across backbones and datasets strengthens the evidence for explicit reliability supervision, within the reported gain metric.

Critical weaknesses

Larger prior-induced gains do not establish better absolute execution

over each method’s own no-prior baseline

Source: Table 3 caption, Section 4.3; PDF p. 8; manuscript lines 378–379.

Concern. The gain metric validly measures how adding a prior changes a particular trained policy, but each variant has a different no-prior baseline. A larger gain can coexist with lower final accuracy or success, including in the matched Part-CoT ablation. Consequently, the green improvements in the tables cannot by themselves establish that SAGE is the better-performing policy in absolute terms. This is consequential online: Table 3 reports SAGE gains of −2.60 and −1.70 percentage points on AndroidWorld, so using the supplied priors still reduces success there; AndroidLab gains of 0.31 and 1.26 points also warrant uncertainty estimates. Equal averaging over prior categories further leaves the expected benefit under a realistic retrieval mixture uncertain. The paper acknowledges the negative effects, which is a useful qualification, but the practical utility remains unresolved by the reported main tables.

Suggested clarification or experiment. Report absolute no-prior and prior-conditioned accuracy/success for each compared variant alongside the gains, using the same evaluation instances. For the online study, give per-condition paired differences with uncertainty and show how the result changes under plausible prior-quality mixtures. Evaluate a concrete retrieval-bypass or no-prior fallback on the same tasks to determine whether the remaining AndroidWorld degradation can be avoided without forfeiting useful-prior gains.

The online result is conditional on coverage and task-specific prior supply

144 exploration-covered tasks

Source: Section 4.3, Setup; PDF p. 7; manuscript lines 376–377.

Concern. The online study supplies task-specific priors constructed from exploration graphs for a selected covered subset. Figure 3 measures a different quantity: coverage of 428 atomic subtask target interfaces derived from 289 benchmark tasks. Visiting one such interface does not establish that a complete usable task path is available, and the 74.1% subtask statistic cannot be used as the coverage denominator for the 144 online tasks. The main description also does not establish how an operational retriever selects a prior from the full graph for an incoming task, despite introducing a retriever in Section 3.1. These selection steps affect both which tasks receive help and the distribution of errors encountered. The reported experiment therefore provides conditional evidence about supplied-prior use, with an incomplete link from the measured exploration improvement to overall task success.

Suggested clarification or experiment. Provide benchmark-specific counts and criteria from eligible tasks through subtask coverage, complete usable paths, and selection of the 144 online cases. Specify and evaluate the actual prior-selection procedure over the full graph. With the existing policies, report success on the full eligible task set, including a defined no-prior fallback for unavailable or unsuitable paths, and separate covered and uncovered outcomes. A paired comparison using SAGE and LLM-Explorer graphs with the same retriever and executor would directly test whether the coverage improvement translates into end-to-end benefit.

Transfer across datasets does not isolate robustness beyond the corruption construction

retaining the six-way perturbation taxonomy

Source: Section 4.4, Setup; PDF p. 9; manuscript lines 435–436.

Concern. Training reliability labels are determined by the generating operator, and the teacher receives the fixed reliability and action targets when writing its rationale (Sections 3.3–3.4). The controlled tests and cross-dataset transfers retain the same six corruption conditions. Thus, transfer changes the trajectory domain while preserving the label-generating construction, leaving open whether part of the effect comes from recognizable corruption cues rather than assessing applicability in the current execution state. Natural drift can combine changes and alter action feasibility in ways that a single synthetic operator does not capture. Section 4.2 explicitly reports a held-out real cross-version evaluation with class-wise F1 and Macro-F1 in Appendix O; that relevant validation should be credited. The unresolved scientific issue is whether such reliability generalization yields correct actions and successful tasks under natural drift, since classification performance and the current controlled gain experiments answer different questions.

Suggested clarification or experiment. Evaluate the frozen policy on genuinely historical priors executed in held-out newer app versions, reporting both reliability decisions and absolute task success relative to no prior. A held-out subset of the paired-version task pool already introduced in Figure 1 could make this practical. Include a modest control with unseen compositions or a held-out perturbation generator to check whether improvements persist when corruption cues change, and audit reliability judgments against current-state applicability as well as the operator label.

Questions for authors
  1. What are the absolute no-prior and prior-conditioned online success rates for SFT and SAGE, and for the Part-CoT ablation offline? Do the policy rankings agree with the rankings of prior-induced gains?

  2. How are the 144 online tasks selected from the eligible benchmark tasks, what constitutes a usable full-task path, and how is a task-specific prior retrieved from the complete graph without task-specific selection assistance?

  3. On naturally drifted, held-out app versions, how do reliability predictions relate to executable action correctness and task success? Do the gains persist for unseen combinations or generators of drift, and with a practical no-prior fallback?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7XV4 (1/2)

We thank the reviewer. Since you and the other reviewers raised concerns about the uncertainty of our results, we re-ran the offline and online experiments several times; all numbers below are from the latest runs, and the main conclusions remain unchanged.

Q1 & W1: Absolute results and ranking consistency.

  1. Online. Absolute success rate (SR, %) on 144 tasks (AndroidWorld, AW: 47; AndroidLab, AL: 64; MobileWorld, MW: 33), averaged over repeated runs. SR follows each benchmark's official protocol, which excludes tasks that return no result because of environment errors from the denominator.
Bench.BackboneMethodNo priorCorrectLocalNoiseOmitRewriteOtherAvg.
AWQwen3-VL-4BSFT17.2312.3520.0023.6324.7019.825.0017.58
Ours19.1324.4428.5126.8338.4132.1617.0727.90
GUI-Owl-4BSFT26.4220.887.3221.0432.1625.917.3219.11
Ours21.7224.2515.1529.2729.2725.9115.0023.14
ALQwen3-VL-4BSFT30.9336.5130.9731.9829.2735.9930.6432.56
Ours31.6338.2835.9436.7238.2840.1534.3837.29
GUI-Owl-4BSFT35.4131.9730.1129.6932.2534.0031.8731.65
Ours25.7832.0833.5433.2330.0830.4134.2632.26
MWQwen3-VL-4BSFT17.3113.948.6114.1615.5216.8210.7813.31
Ours15.3823.8819.6221.6217.7719.2321.1520.54
GUI-Owl-4BSFT18.009.5513.5515.8311.5917.5418.0414.35
Ours27.1834.5933.4237.5327.5636.0617.1731.05

With priors, Ours has a higher mean SR than SFT in 6/6 benchmark–backbone pairs, a higher SR in 35/42 cells and a larger prior-induced gain in 35/36. Ours' own gain pooled over benchmarks is +6.53 pp [+0.43, +12.57] for Qwen3-VL-4B and +3.93 pp [−1.52, +9.22] for GUI-Owl-1.5-4B (paired bootstrap over tasks, B = 5,000).

  1. Offline Part-CoT ablation. Absolute step accuracy averaged over backbones, with a paired bootstrap over decision states (B = 10,000, 95% CI):
SettingNo prior: Part-CoT / OursPrior (mean of 6): Part-CoT / OursOurs − Part-CoT with prior
In-domain (15 settings)73.3 / 73.475.7 / 77.9+2.22 [+1.55, +2.89]
Transfer (30 settings)60.1 / 61.362.3 / 66.1+3.78 [+3.31, +4.28]
  1. Do the rankings agree? For the matched ablation, mostly yes. The Δ ranking and the absolute ranking of Ours vs. Part-CoT agree in 13/15 in-domain and 25/30 transfer settings. In the 2 in-domain disagreements, Ours has the higher absolute accuracy despite a smaller Δ; in transfer, Ours has a larger Δ but a lower absolute accuracy in only 2 settings, by 0.52 and 0.16 pp, both with CIs including 0.

    Against SFT they agree less often, because the methods start from different no-prior baselines. Offline, Ours with prior still exceeds the better of SFT's no-prior and with-prior accuracy in 22/30 transfer settings (+1.84 [+1.09, +2.44]) and 10/15 in-domain settings (+0.66 [−0.37, +1.52]). Online, it does so on all three benchmarks for Qwen3-VL-4B, but for GUI-Owl only on MW.

AuthorsOpenReview ↗

Response to Reviewer 7XV4 (2/2)

Q2 & W2: Task selection and online retrieval.

  1. Selection. We start from the 289 tasks of the coverage study (MW/AL/AW = 129/85/75), split each task into subtasks by the validator's checks and identify their target interfaces. A usable full-task path contains all screenshots and executed actions from the home screen to the destination interface. After removing cross-app tasks and, for MobileWorld, tasks requiring the MCP tool protocol, we keep all tasks with a usable full-task path: 144 tasks (33/64/47). The 74.1% is subtask coverage over the full 289-task pool and is not the denominator for these 144 tasks.
  2. Retrieval. The main online study injects matched priors, because its focus is how erroneous knowledge affects the model. To test retrieval, we used a fixed retriever and executor to obtain priors from both the SAGE and the LLM-Explorer graphs. On the same 144 tasks, we generated priors for LLM-Explorer with the same prior-generation method as ours. We adapted the Manager-RAG retrieval component of Mobile-Agent-RAG (AAAI 2026): the task instruction retrieves the top-1 passage from the complete benchmark-specific prior bank, with no app-level filtering and no manual prior assignment at inference, and only the retrieved path text is given to the executor. Both banks use the same retrieval settings and the same executor (Ours on Qwen3-VL-4B), with matched task initializations and execution budgets. We report the task success rate and the retrieval accuracy (the share of tasks whose retrieved prior matches the task) for each exploration graph.

Task success rate

BenchmarkSAGE GraphLLM-Explorer Graph
AndroidWorld25.53%17.02%
AndroidLab35.94%29.69%
MobileWorld15.15%9.09%
Overall27.78%20.83%

Retrieval accuracy

BenchmarkSAGE Prior-MatchedLLM-Explorer Prior-Matched
AndroidWorld87.23%38.30%
AndroidLab95.31%71.88%
MobileWorld78.79%51.52%
Overall88.89%56.25%

Q3 & W3: Natural drift, unseen drift generators and fallback.

  1. Natural drift. We ran frozen SFT and Ours on held-out tasks in newer app versions: 176 paired tasks across 20 apps (1,137 old-version and 1,023 new-version steps), excluded from training. Each new-version step is evaluated with no prior, a same-version prior and a cross-version prior from the old-version trajectory. Step accuracy:
BackboneNo prior: SFT / OursSame-ver.: SFT / OursCross-ver.: SFT / Ours
Qwen3-VL-2B61.28 / 66.9664.19 / 73.6262.25 / 72.27
Qwen3-VL-4B75.75 / 79.4281.64 / 83.0977.85 / 82.13
Qwen3-VL-8B69.64 / 73.7276.69 / 80.3974.57 / 80.10
MAI-UI-2B64.18 / 66.7365.13 / 76.6561.42 / 73.06
GELab-Zero-4B79.08 / 78.2481.42 / 82.7079.33 / 80.99

With cross-version priors, Ours is more accurate than SFT on all five backbones, and its gain over no prior is positive and larger than SFT's on all five. Labels on this set are assigned against the recorded new-version trajectory, i.e. by applicability in the current state. 2. Unseen drift generators. This drift comes from real version updates, and no perturbation operator generated it. It therefore serves as a held-out drift generator, and the gains persist under it. 3. No-prior fallback. We simulated a hard fallback offline: whenever Ours outputs IGNORE, its action is replaced by its own no-prior prediction. This does not improve accuracy, so the IGNORE decision already acts as an internal fallback.

AuthorsOpenReview ↗

Revised PDF

Thank you for your careful review and constructive feedback. We have submitted a revised PDF incorporating your suggestions. Note that, for readability, we marked the revisions to Tables 3, 5, and 6 in their captions and kept the numerical entries in black.

  • Result reporting. As requested, Table 3 now reports absolute online task success rates for the no-prior baseline and each of the six prior conditions. Table 6 reports absolute offline step accuracies with and without priors for the Part-CoT ablation. Both tables retain mean Δ for reference.
  • New experiments. We compared prior banks built from SAGE and LLM-Explorer graphs on the same 144 tasks with a fixed retriever and executor (Appendix Q, Table 14). The SAGE bank yielded higher top-1 retrieval accuracy (88.89% vs. 56.25%) and task success (27.78% vs. 20.83%). We also tested generalization to real version drift on 176 held-out task pairs across 20 apps (Sec. 4.4, Table 4).
  • Evaluation protocol. Section 4.3 clarifies how we selected the 144 online tasks, including the requirement for a complete recorded path from the home screen to the target interface.

We hope these revisions address your concerns, and we look forward to continuing the discussion with you.

AuthorsOpenReview ↗

Follow-up on Our Response and Revised Manuscript

We understand this is a busy period and sincerely appreciate your time. As the discussion period is drawing to a close, we would like to kindly follow up on our response.

In particular, the revised PDF now reports absolute online and offline results (Tables 3 and 6), a retrieval-based comparison of SAGE and LLM-Explorer prior banks (Appendix Q), and results under real app-version drift (Sec. 4.4). We would be grateful to know whether these address your concerns, and we are happy to clarify anything further.

Reviewer_eKv9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies retrieval-augmented GUI agents whose retrieved priors (past task trajectories) can be wrong because apps change. SAGE has three parts. A real-device explorer builds a deduplicated UI graph, and a VLM turns graph paths into (task, trajectory) priors without human labels. Six perturbation operators, motivated by a study of 176 tasks across app versions, turn clean priors into noisy ones with FOLLOW, PARTIAL or IGNORE labels. LoRA fine-tuning then teaches the model to state a reliability decision before it acts. At 600 steps, the explorer finds 148 unique screens on average, against 54 for LLM-Explorer (Table 7). Across five backbones, priors raise the trained model's step accuracy by 7.1-11.8 points on average over six conditions, against 1.6-8.4 points for SFT (Table 2). Similar advantages appear across datasets (Table 4) and on 144 online tasks (Table 3).

Critical strengths

S1. A useful framing, backed by data. Treating retrieved GUI knowledge as an imperfect prior, not as ground truth, fits how apps really change. In Figure 1, only 44% of 176 tasks keep their trajectory after an update, and three models lose 13-15 points even with correct old-version trajectories.

S2. A strong explorer, tested on real devices. With the same backbone and deduplication rule as the LLM-based baselines (L308-310), it finds the most unique screens in 19 of 20 apps (Table 7). It covers 74.1% of 428 benchmark subtasks, against 57.2% for LLM-Explorer (Figure 3c).

S3. Broad and consistent learning results at low cost. Ours beats SFT in all 30 cells of Table 2 and in 28 of 30 transfer settings (Table 4), and the gain holds on three new splits (Appendix G). Teacher rationales never set the action labels (L258-262), and a 4B model trains on one RTX 4090 (Appendix H).

Critical weaknesses

W1. The headline metric may favour Ours, and absolute accuracy is never shown. Tables 2-5 report only Δ, the gain over each model's own no-prior run. Ours is trained only on samples that carry a prior (Table 6; L677-683), so its no-prior run uses an input it never saw in training. SFT is the reverse: it is trained without priors and tested with them (L680-682). A weak no-prior run would raise Ours' Δ in every condition. There is one warning sign. In Table 2, Ours gains 0.41-8.98 points from Other priors, which point to a different goal and should be ignored. Early steps shared with other paths in the same graph (Table 1 caption) could also explain this, but without absolute numbers the reader cannot tell. Every learning claim rests on Δ, so this point decides how much the results show.

W2. The motivating setting is tested only weakly. The paper is motivated by real version drift and by retrieval (L161). However, the main tests use synthetic edits of a prior built for each test task (L377), and no retriever is run. The only real-drift test, Table 9 in Appendix O, reports label F1 with no baseline, set size or class balance. No experiment checks whether SAGE shrinks the 13-15 point drop in Figure 1(b). Online, priors still lower Ours' success on AndroidWorld (−2.60 and −1.70 points) and barely help on AndroidLab (0.31 and 1.26) in Table 3. So the claim that online evaluation "extends this advantage to end-to-end execution" (L484-485) is narrower than stated. Ours is hurt less than SFT, but priors do not yet help on two of the three benchmarks.

W3. The new reliability label is only weakly separated from known noisy-retrieval training. Training a generator to judge retrieved text before using it is established in RAG, for example in Self-RAG (ICLR 2024), RAAT (Fang et al., ACL 2024) and Chain-of-Note (EMNLP 2024, not cited). Section 2 (L141-143) does not state the concept-level difference. The label's own effect rests on Table 5: Ours beats the matched Part-CoT in 12 of 15 settings, and the mean rises from 5.59 to 7.87 points. These are single runs, and the gap for Qwen3-VL-8B is only 0.37 points. The test sets are small, so one decision state moves a cell by 0.34-0.78 points (Table 6). Appendix G gives intervals only for Ours versus SFT. An action-only model trained on the same perturbed priors is also missing.

Questions for authors

Q1. Please give the absolute step accuracy of Ori., SFT, Part-CoT and Ours with no prior and under each condition, and say which prompt Ours uses with no prior. If Ours with priors beats SFT's best absolute accuracy, W1 is resolved and I would raise my rating. (W1)

Q2. Could you rerun the Figure 1(b) protocol with SFT and Ours, using old-version trajectories as priors? Could you also add baselines and the set size to Table 9? Evidence that SAGE closes part of the drop would raise my rating. (W2)

Q3. What is the Appendix G bootstrap interval for Ours minus Part-CoT, and how does an action-only model trained on the same perturbed priors do? A clearly positive interval would change my view of W3. (W3)

4 comments
AuthorsOpenReview ↗

Response to Reviewer eKv9 (1/2)

We thank the reviewer. Since you and the other reviewers raised concerns about the uncertainty of our results, we re-ran the offline and online experiments several times; all numbers below are from the latest runs, and the main conclusions remain unchanged.

W1 & Q1: Absolute accuracy.

No-prior input. In our latest experiments, Ours is also trained on one no-prior sample per decision state (7 samples per state instead of 6). With no prior, Ours receives its usual input with the [Prior Knowledge] field set to None, exactly as in training, so its no-prior evaluation is in-distribution.

Absolute step accuracy (%) on the same instances, averaged per backbone over 3 in-domain datasets (15 settings) and 6 transfer directions (30 settings). Δ is always computed against the same model's own no-prior run; "prior" is the mean over the six conditions. Cells: SFT / Part-CoT / Ours.

BackboneIn-domain, no priorIn-domain, priorTransfer, no priorTransfer, prior
Qwen3-VL-2B73.1/70.2/67.268.2/72.1/74.660.5/56.1/58.859.5/58.1/64.0
Qwen3-VL-4B76.8/75.6/75.778.8/77.9/79.065.0/62.9/62.865.5/65.1/67.0
Qwen3-VL-8B77.1/75.3/76.780.5/77.7/79.765.2/61.7/62.466.1/63.8/66.7
MAI-UI-2B73.3/67.5/69.464.2/70.9/73.456.8/54.2/53.952.3/56.7/59.0
GELab-Zero-4B79.9/78.0/77.878.4/79.8/82.967.9/65.8/68.869.0/68.0/74.0
Mean76.0/73.3/73.474.0/75.7/77.963.1/60.1/61.362.5/62.3/66.1
  1. Your criterion: Ours with prior vs. SFT's best accuracy, i.e. the better of SFT's no-prior and with-prior accuracy, re-selected in every resample (paired bootstrap over decision states, B = 10,000, 95% CI). In transfer, Ours is higher in 22/30 settings, +1.84 pp [+1.09, +2.44], with the CI above 0 in 12. In-domain, Ours is higher in 10/15 settings, +0.66 pp [−0.37, +1.52], so it is on par with SFT's best.

  2. A weak no-prior run does not explain the gain from the label. Part-CoT and Ours have the same no-prior accuracy (73.3 vs. 73.4 in-domain), and Ours is higher with priors in 15/15 in-domain settings.

  3. Per condition (in-domain means; no prior: 76.0 / 73.3 / 73.4):

    ConditionSFT / Part-CoT / Ours
    Correct73.2 / 77.6 / 82.3
    Noise71.7 / 77.2 / 80.2
    Rewrite77.4 / 75.0 / 78.7
    Local72.4 / 75.1 / 75.7
    Omit75.5 / 75.6 / 76.7
    Other73.9 / 73.6 / 73.8

    Adding priors lowers SFT's accuracy in 11/15 in-domain settings, whereas Ours improves over its own no-prior run in 15/15 in-domain and 30/30 transfer settings.

  4. Other priors. Ours' accuracy under Other (73.8) essentially equals its no-prior accuracy (73.4): its mean Δ is +0.42 pp in-domain (range −2.97 to +4.69) and +0.15 pp in transfer, so it largely ignores priors pointing to a different goal. SFT instead loses 2.08 pp (negative in 13/15 settings). The few positive cells are consistent with early steps shared with the target path, as you suggested.

AuthorsOpenReview ↗

Response to Reviewer eKv9 (2/2)

W2 & Q2: Real drift.

We agree that label F1 alone does not establish better action prediction. To test whether Ours keeps its advantage when priors are outdated by real app-version changes, we evaluate SFT and Ours on the same new-version task states under three conditions: no prior, a same-version prior (from the new-version trajectory) and a cross-version prior (from the old-version trajectory). The held-out set has 176 paired tasks across 20 apps, with 1,137 old-version and 1,023 new-version steps, all excluded from training.

BackboneMethodNo priorSame-ver.Cross-ver.Cross − sameCross − no prior
Qwen3-VL-2BSFT61.2864.1962.25−1.94+0.97
Ours66.9673.6272.27−1.35+5.31
Qwen3-VL-4BSFT75.7581.6477.85−3.79+2.10
Ours79.4283.0982.13−0.96+2.71
Qwen3-VL-8BSFT69.6476.6974.57−2.12+4.93
Ours73.7280.3980.10−0.29+6.38
MAI-UI-2BSFT64.1865.1361.42−3.71−2.76
Ours66.7376.6573.06−3.59+6.33
GELab-Zero-4BSFT79.0881.4279.33−2.09+0.25
Ours78.2482.7080.99−1.71+2.75

With cross-version priors, Ours is more accurate than SFT on all five backbones, its gain over no prior is positive and larger than SFT's on all five, and its drop relative to same-version priors is smaller on all five.

Online: Priors now raise Ours’ own success rate on all three benchmarks for both backbones, including AndroidWorld (+8.77 pp on Qwen3-VL-4B, +1.42 pp on GUI-Owl-1.5-4B) and AndroidLab (+5.66 / +6.48 pp). Pooled over benchmarks, the gain is +6.53 pp [+0.43, +12.57] for Qwen3-VL-4B and +3.93 pp [−1.52, +9.22] for GUI-Owl-1.5-4B (paired bootstrap over tasks, B = 5,000).

W3 & Q3: Ours − Part-CoT intervals and an action-only model.

Paired bootstrap over decision states (each no-prior input resampled with its six conditions, B = 10,000, 95% CI). Each cell gives the pooled difference and the number of settings where Ours is higher.

ComparisonSettingsAccuracy with prior (pp)Δ (pp)
Ours − Part-CoT, in-domain15+2.22 [+1.55, +2.89], 15/15+2.19 [+1.24, +3.13], 13/15
Ours − Part-CoT, transfer30+3.78 [+3.31, +4.28], 28/30+2.57 [+1.92, +3.22], 27/30
Ours − action-only, GELab-Zero-4B, in-domain3−2.11 [−3.72, −0.46], 0/3+1.17 [−0.71, +3.08], 3/3

Both Ours − Part-CoT intervals are clearly positive, and no single setting has a CI below 0.

Action-only is trained on the same perturbed priors and Action/Tool targets, without Thought. On GELab-Zero-4B it reaches higher absolute accuracy than Ours, while Ours gains more from priors. We therefore do not claim that explicit reasoning raises absolute accuracy over action-only training. Our claim is the matched one: with the rationale held fixed, adding the reliability label improves accuracy (Ours − Part-CoT), and it makes each decision auditable.

Self-RAG / RAAT / Chain-of-Note. We will cite Chain-of-Note, and we do not claim as new either judging retrieved content before use or the three-way decision. The difference is what is judged and how negatives are built: the decision is conditioned on the current screen and execution progress, not only on the query, and the negatives are neighbouring paths from the same app's exploration graph rather than unrelated passages.

AuthorsOpenReview ↗

Revised PDF

Thank you for your careful review and constructive feedback. We have submitted a revised PDF incorporating your suggestions. Note that, for readability, we marked the revisions to Tables 3, 5, and 6 in their captions and kept the numerical entries in black.

  • Result reporting. As requested, Tables 3, 5, and 6 now present absolute scores as the main results and retain mean Δ for reference. Table 12 provides the per-condition absolute step accuracies corresponding to the gains in Table 2.
  • New experiments and analyses. Section 4.4 and Table 4 compare SFT, Part-CoT, and Ours under real app-version drift. Table 8 in Appendix G reports paired 95% bootstrap confidence intervals for the differences between Ours and Part-CoT in both absolute accuracy and prior-induced gain, for in-domain and cross-dataset evaluation.
  • Clarifications. Appendices D describe the no-prior input and explain how we evaluate each method’s no-prior baseline using its own checkpoint on the same instances. Section 2 now discusses how our use of the current screen and execution history to assess a prior relates to Self-RAG, RAAT, and Chain-of-Note.

We hope these revisions address your concerns, and we look forward to continuing the discussion with you.

AuthorsOpenReview ↗

Follow-up on Our Response and Revised Manuscript

We understand this is a busy period and sincerely appreciate your time. As the discussion period is drawing to a close, we would like to kindly follow up on our response.

In particular, the revised PDF now presents absolute scores as the main results (Tables 3, 5, 6), evaluates SFT, Part-CoT and Ours under real app-version drift (Sec. 4.4), and reports paired confidence intervals for Ours vs. Part-CoT (Appendix G). We would be grateful to know whether these address your concerns, and we are happy to clarify anything further.

Reviewer_9gDtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how GUI agents can acquire and use procedural knowledge when retrieved trajectories are incomplete, stale, or irrelevant. A study of 176 tasks across paired versions of 20 apps finds that only 44% retain identical trajectories after an update. SAGE combines semantic-region exploration and UI-graph construction with automatic task and prior synthesis. Training priors are generated using one identity operator and five perturbation operators, with corresponding FOLLOW/PARTIAL/IGNORE labels. A single autoregressive model learns to assess the prior, identify trusted edges in PARTIAL cases, and generate the next action. A teacher supplies the reasoning, while action targets are reconstructed from recorded trajectories. Evaluation covers real-device exploration, offline action prediction across five backbones and three datasets, cross-dataset transfer, 144 exploration-covered online tasks, and reliability classification on held-out real cross-version drift. The reported results show broader interface coverage and larger average gains from using priors. In the matched Part-CoT ablation, adding explicit reliability-label supervision increases the mean accuracy gain over each variant’s own no-prior baseline from 5.59 to 7.87 percentage points.

Critical strengths
  1. The exploration pipeline has useful empirical support. Figure 3(c) shows higher subtask coverage than LLM-Explorer on all three benchmarks: 74.1% versus 57.2% overall across 428 subtasks. Table 7 also compares methods at the same 600th exploration step, where SAGE reaches an average of 148.1 screens versus 54.3 for LLM-Explorer. These results support its ability to acquire broader interface coverage, though coverage alone does not establish task success.

  2. The paper goes beyond comparing the full method with ordinary SFT. The Part-CoT ablation in Section 4.5 holds inputs, Action/Tool targets, training, and evaluation fixed while changing the reliability-label supervision. Ours achieves higher mean gains from priors in 12 of 15 backbone–dataset settings. Appendix O also tests the three-way decision against human labels on held-out real drift, with a mean Macro-F1 of 70.48%. This is useful evidence beyond the synthetic labels, although Table 9's class-wise summary has an inconsistency discussed below.

Critical weaknesses
  1. The tables do not show absolute performance. Tables 2–5 report Δ, the change in accuracy or success rate when a prior is added. Tables 3 and 5 use each variant's own no-prior baseline; Tables 2 and 4 do not explicitly say whether they do the same. These gains cannot establish which trained variant performs best with priors: a larger gain can still end at a lower success rate.

The online results also vary considerably across benchmarks. In Table 3, Ours has negative mean gains on AndroidWorld (−2.60/−1.70 pp), small positive gains on AndroidLab (+0.31/+1.26 pp), and larger gains on MobileWorld (+7.70/+5.22 pp). The positive values do show improvement over the same model without priors. What remains unclear is the final success rate and which prior conditions account for the gains or failures, since each cell averages six conditions. There are no uncertainty estimates for these contrasts, and the evaluation uses only 144 exploration-covered tasks. I would like to see absolute success rates, results by prior condition, and the task selection criteria before drawing conclusions about deployment.

  1. The matched control is missing from the online and transfer experiments. On CUTG, SFT uses 1,789 Direct samples, while Ours uses 10,734 Full-CoT samples expanded from the same decision states. Reasoning supervision and evaluation prompts also differ (Appendices B and D). Section 4.2 acknowledges that this compares the full training recipes. It therefore cannot isolate what the reliability labels add.

Table 5 does provide that control: Ours improves mean Δ over Part-CoT by 2.28 pp, from 5.59 to 7.87 pp. The gain is only 0.37 pp for Qwen3-VL-8B, with negative differences on two datasets, as Section 4.5 acknowledges. Tables 3 and 4 have no Part-CoT results, and Appendix G's intervals concern Ours−SFT on additional splits, not Ours−Part-CoT. The conclusion nevertheless extends the claim about explicit prior assessment to transfer and end-to-end execution. A matched online comparison on Qwen3-VL-4B, together with representative transfer results, would test that extension directly.

I lean toward weak rejection. The exploration results and matched offline ablation are strengths, but I cannot tell whether Ours achieves better absolute online success than SFT, or whether reliability-label supervision accounts for the online and transfer gains. These are the main reasons for my rating.

Additional concerns:

  • Table 9 needs correction or an explanation of how it was aggregated. The four displayed models average to 57.90% PARTIAL F1 and 62.34% IGNORE F1, rather than the reported 58.82% and 63.57%. The mean of the three reported class averages is 71.20%, not 70.48%. The mean of the four model-level Macro-F1 scores does round to 70.48%, so that value remains consistent with the displayed rows. Appendix O gives no set size, class counts, annotation procedure beyond stating that labels were assigned manually, or Ori./Part-CoT comparison. The model is also trained to predict trusted-edge indices (Table 1, Section 3.4), but the paper does not score those predictions; the Omit columns evaluate actions instead.

  • The training mixture is unclear. The sixfold expansion in Appendix B and the six conditions in Appendix C imply a nominal FOLLOW/PARTIAL/IGNORE ratio of 2:3:1. Section 3.4 describes sampling from Cat(w), without giving the effective weights. There is also no separately named operator for the step-reorder cases in Figure 1(a), so it is unclear whether the simulated tests cover them.

Questions for authors
  1. Absolute performance. Please report absolute no-prior and with-prior results for the main online and matched-ablation comparisons, with uncertainty for the key differences. Can Table 3 be broken down by prior condition, with task counts for each benchmark? Do Tables 2 and 4 also use each variant's own no-prior baseline?

  2. Part-CoT online and in transfer. Can you add Part-CoT for Qwen3-VL-4B online and in representative transfer settings, with uncertainty for Ours−Part-CoT? Please specify which reliability-label and trusted-edge targets are supervised in each variant, and their token weights. Appendix N reports the Thought:Action:Tool weights, but does not resolve this distinction.

  3. Real drift and trusted edges. How was each column of Table 9's Average row calculated, and over which models or runs? Please reconcile it with the displayed rows and the definition of Macro-F1. What are the real-drift set size, class counts, and annotation/adjudication procedure? Ori. and Part-CoT results under a common assessment format would help establish what the training adds. Can the predicted trusted edges also be scored against the existing synthetic targets?

  4. Training mixture. Are the six conditions used uniformly, giving the nominal 2:3:1 label ratio, or reweighted/resampled according to Eq. 4? Please give the effective weights and clarify whether an existing operator covers step reorder.

  5. Online task selection. How does the subtask coverage criterion in Section 4.1 relate to selection of the 144 online tasks? How are their Correct priors checked against the initialized execution state, and how often do they match? Section 4.3 already notes possible mismatch. Also, does view-hierarchy hashing count evaluation screens while Eq. 3 governs internal graph construction?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9gDt (1/3)

We thank the reviewer. Since you and the other reviewers raised concerns about the uncertainty of our results, we re-ran the offline and online experiments several times; all numbers below are from the latest runs, and the main conclusions remain unchanged.

Q1 & W1/W2: Absolute performance.

  1. Baseline for Δ. Yes: in Tables 2, 4, 5 and 8, Δ is always computed against the same model's own no-prior run on the same instances.
  2. Online, by prior condition. Absolute success rate (SR, %) on 144 tasks (AndroidWorld, AW: 47; AndroidLab, AL: 64; MobileWorld, MW: 33), averaged over repeated runs. SR follows each benchmark's official protocol, which excludes tasks that return no result because of environment errors from the denominator.
Bench.BackboneMethodNo priorCorrectLocalNoiseOmitRewriteOtherAvg.
AWQwen3-VL-4BSFT17.2312.3520.0023.6324.7019.825.0017.58
Part-CoT16.4521.6620.0019.8217.3821.0418.7519.77
Ours19.1324.4428.5126.8338.4132.1617.0727.90
GUI-Owl-4BSFT26.4220.887.3221.0432.1625.917.3219.11
Ours21.7224.2515.1529.2729.2725.9115.0023.14
ALQwen3-VL-4BSFT30.9336.5130.9731.9829.2735.9930.6432.56
Part-CoT24.5619.5520.6916.5321.6719.0717.5819.18
Ours31.6338.2835.9436.7238.2840.1534.3837.29
GUI-Owl-4BSFT35.4131.9730.1129.6932.2534.0031.8731.65
Ours25.7832.0833.5433.2330.0830.4134.2632.26
MWQwen3-VL-4BSFT17.3113.948.6114.1615.5216.8210.7813.31
Part-CoT11.6917.319.8511.5411.5414.467.6912.06
Ours15.3823.8819.6221.6217.7719.2321.1520.54
GUI-Owl-4BSFT18.009.5513.5515.8311.5917.5418.0414.35
Ours27.1834.5933.4237.5327.5636.0617.1731.05
  • With priors, Ours has the highest mean SR in 6/6 benchmark–backbone pairs, the highest SR in 34/42 cells and the highest prior-induced gain in 34/36.
  • Ours' own mean gain is positive in 6/6 pairs; pooled over benchmarks it is +6.53 pp [+0.43, +12.57] for Qwen3-VL-4B and +3.93 pp [−1.52, +9.22] for GUI-Owl-1.5-4B (paired bootstrap over tasks, B = 5,000).
  • On Qwen3-VL-4B, Ours with priors exceeds SFT both with and without priors on all three benchmarks. On GUI-Owl, Ours' no-prior SR is lower, and on AW and AL Ours with priors stays below SFT without priors.
AuthorsOpenReview ↗

Response to Reviewer 9gDt (2/3)

  1. Offline matched ablation. Absolute step accuracy, averaged over backbones (SFT / Part-CoT / Ours):
SettingNo priorPrior (mean of 6 conditions)
In-domain (15 settings)76.0 / 73.3 / 73.474.0 / 75.7 / 77.9
Transfer (30 settings)63.1 / 60.1 / 61.362.5 / 62.3 / 66.1

In-domain, Ours − Part-CoT is +2.22 pp [+1.55, +2.89] in accuracy with prior (Ours higher in 15/15 settings) and +2.19 pp [+1.24, +3.13] in Δ (13/15), from a paired bootstrap over decision states (B = 10,000, 95% CI). The larger Δ of Ours therefore does not end in a lower final accuracy.

Q2 & W3/W4: Part-CoT online and in transfer; supervised targets.

  1. Transfer. We added Part-CoT to all 30 transfer settings. Ours − Part-CoT is +3.78 pp [+3.31, +4.28] in accuracy with prior (Ours higher in 28/30; the CI is above 0 in 18 settings and below 0 in none) and +2.57 pp [+1.92, +3.22] in Δ (27/30). For Qwen3-VL-4B in transfer, the differences are +1.85 [+0.88, +2.82] and +1.90 [+0.59, +3.19].

  2. Online, Qwen3-VL-4B (table in Part 1). With priors, Ours has a higher mean SR than Part-CoT on all three benchmarks (27.90 vs. 19.77, 37.29 vs. 19.18, 20.54 vs. 12.06). ΔOurs − ΔPart-CoT is +5.45 [−4.41, +15.59], +11.03 [−0.82, +23.92] and +4.79 [−2.01, +14.13] pp on AW / AL / MW; each per-benchmark CI includes 0 at this task count, while the equal-weight mean is +7.09 pp [+1.48, +12.99].

  3. Supervised targets. Token weights are 1:1:1 for all three variants; Appendix N is a separate sensitivity study.

    VariantSupervised targets
    SFTAction + Tool
    Part-CoTRationale + Action + Tool
    OursRationale with the explicit reliability decision + Action + Tool

    Part-CoT and Ours differ only in the explicit reliability decision.

Q4 & A2: Training mixture. The conditions are used uniformly, with no reweighting or resampling. Every task receives exactly one prior from each operator, so all operators contribute the same number of samples: each decision state is paired with 7 inputs (no prior, the clean prior and one prior per perturbation operator), each 1/7 of the training data. At the prior level this gives the nominal FOLLOW / PARTIAL / IGNORE ratio of 2:3:1, as you note. We will revise Eq. 4 to describe this uniform expansion instead of sampling from Cat(w). There is no dedicated step-reorder operator, and we will state this in the limitations.

AuthorsOpenReview ↗

Response to Reviewer 9gDt (3/3)

Q3 & A1: Real drift and trusted edges.

  1. Table 9. The PARTIAL and IGNORE averages were transcribed wrongly. Over the four displayed models they are 57.90 and 62.34; with FOLLOW at 91.21, the three class means then average to 70.48, matching the mean Macro-F1. We apologize and will correct the Average row and the text.
  2. Set and annotation. The held-out set has 176 tasks across 20 apps. We completed each task manually in the new app version and recorded the trajectory, giving paired old- and new-version trajectories of 1,137 and 1,023 steps. Same-version and cross-version priors are built from the new- and old-version trajectories. Labels are assigned per step, using the recorded new-version trajectory to decide whether each old-version prior still applies. All tasks and trajectories are held out from training.
  3. Ori. and Part-CoT. SFT and Part-CoT do not emit a reliability decision, so we compare all variants on step accuracy under real drift:
BackboneMethodNo priorSame-ver.Cross-ver.Cross − sameCross − no prior
Qwen3-VL-2BOri.59.5256.2353.24−2.99−6.28
SFT61.2864.1962.25−1.94+0.97
Part-CoT56.5060.7955.77−5.02−0.73
Ours66.9673.6272.27−1.35+5.31
Qwen3-VL-4BOri.79.2381.8481.26−0.58+2.03
SFT75.7581.6477.85−3.79+2.10
Part-CoT75.0576.2177.56+1.35+2.51
Ours79.4283.0982.13−0.96+2.71
Qwen3-VL-8BOri.75.2978.0176.52−1.49+1.23
SFT69.6476.6974.57−2.12+4.93
Part-CoT70.1877.1575.77−1.38+5.59
Ours73.7280.3980.10−0.29+6.38
MAI-UI-2BOri.63.8672.9565.80−7.15+1.94
SFT64.1865.1361.42−3.71−2.76
Part-CoT62.4759.1056.71−2.39−5.76
Ours66.7376.6573.06−3.59+6.33
GELab-Zero-4BOri.80.3581.4180.37−1.04+0.02
SFT79.0881.4279.33−2.09+0.25
Part-CoT72.1972.9570.73−2.22−1.46
Ours78.2482.7080.99−1.71+2.75

With cross-version priors, Ours is the most accurate variant on all five backbones, and its gain over no prior is positive and the largest on all five.

  1. Trusted edges. Thank you for catching this. The description of trusted edges in Table 1, Sec. 3.4 and Eq. 5 does not accurately reflect our implementation; in particular, no tokens are upweighted. We will correct these details in the revision.

Q5: Online task selection.

  1. From the 289 tasks of the coverage study (MW/AL/AW = 129/85/75), we removed cross-app tasks and, for MobileWorld, tasks that need the MCP tool protocol, and kept the tasks whose subtask target interfaces are all covered by the exploration graph. This gives 144 tasks (33/64/47). The 74.1% is subtask coverage over the full 289-task pool, a different quantity from this selection.
  2. The online Correct prior is the exploration path without further perturbation. It can mismatch the initialized task because the benchmarks generate parameters at run time and modify app data: 32/47 AndroidWorld tasks draw random parameters and 18/33 MobileWorld tasks inject fixed content, while AndroidLab tasks are essentially matched.
  3. View-hierarchy hashing is used only to count screens in the exploration comparison; our explorer builds its graph with Eq. 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.