- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies local execution for GUI agents: given an action intent from an upstream planner, how can a controller ground the target, verify the resulting interface change, and reduce large-model calls? FAC-CUA learns three factors for subgoal consistency, target grounding, and interface change from interaction logs, and combines them with bounded self-recovery. On 286 tasks with shared planner outputs, it improves success and execution time over a monolithic latent controller. The paper also evaluates grounding, failure diagnosis, and transfer on a WindowsAgentArena subset.
Critical strengths
- Dividing local GUI control into subgoal consistency, target grounding, and interface change connects representation learning directly to grounding, verification, and recovery. The research question is clear and has practical value independently of high-level planning.
- Grounding experiments and recovery ablations examine the representation and control mechanism separately, while the full controller improves both success and execution time. The system gains therefore do not depend on a single resource metric.
Critical weaknesses
- The factorized model adds specialized branches and receives 8 additional training epochs after initialization from the monolithic model, without a monolithic comparison matched for capacity and training. It remains unclear whether factorization, a stronger model, or additional training drives the improvement. This weakens the paper's central representation-learning conclusion.
- Failure diagnosis primarily uses synthetic errors constructed around the factor roles, without establishing a link between diagnoses and successful recovery during actual execution. The recovery ablation shows that recovery helps, but does not establish that factor-level diagnosis is more useful than a generic consistency score. The distinctive contribution of the representation to recovery therefore remains unclear.
Questions for authors
- after matching representation capacity and giving the monolithic model the same additional training, do the grounding and task-success gaps persist?
- which recovery actions are triggered by each factor during actual execution? Can existing traces show when factor-level diagnosis helps recovery, rather than merely detecting an execution anomaly?
5 comments
Response to Reviewer e2jJ (2/4)
W2 / Q2. Which recovery actions does each factor trigger, and does factor-level diagnosis help recovery beyond anomaly detection?
- Routing rule (Section 4.5, Appendix D.1). Passing all three thresholds continues execution. The rules apply in order: a Ground failure retries the next retained candidate; otherwise a Delta failure waits 0.6 s and re-observes the interface; any remaining inconsistency, including a Goal failure, invokes the LLM fallback. Each planner step allows one target retry and up to two waits, and the verifier-triggered fallback is limited to one call per step.
- Real execution failures (Appendix H.2). We re-executed, with both controllers, the 49 tasks that both latent controllers failed in the 286-task evaluation. Two authors annotated the first failing step of each failed run, defined as the first instruction-specific checkpoint not reached, as GROUND, DELTA, UPSTREAM_PLAN, or OTHER.
| Controller | Re-run success | GROUND | DELTA | UPSTREAM_PLAN | OTHER |
|---|---|---|---|---|---|
| Monolithic latent | 5/49 | 19 | 6 | 19 | 0 |
| FAC-CUA | 16/49 | 14 | 1 | 17 | 1 |
Paired by task, 11 tasks succeed only with FAC-CUA and none only with the monolithic controller (exact McNemar p = 0.001). DELTA failures, where the interface moved to the wrong application and the controller kept acting, drop from 6 to 1; in five of the six monolithic cases, pressing Enter in Windows Search opened a web search or left the search panel open instead of opening the target file.
- What happened in actual execution (Appendix H.3). We counted the recovery actions in the existing traces of the 286-task run reported in Table 1, over the 2,988 planner steps scored by the verifier. A step is resolved without LLM when all margins become non-negative without an LLM call.
| Recovery in the step | Steps | Resolved without LLM |
|---|---|---|
| Flagged at first scoring | 1,282 | 861 (67.2%) |
| Ground retry only | 577 | 393 (68.1%) |
| Delta wait only | 421 | 341 (81.0%) |
| Ground retry and Delta wait | 134 | 127 (94.8%) |
| No local recovery (sent to the fallback or ended) | 150 | — |
| LLM fallback, any reason | 482 | — |
| (Goal margin only negative, at first scoring or after a retry or wait / unresolved retry or wait / selector abstention) | (218 / 138 / 126) | — |
Each factor triggered a different recovery: the Ground retry and the Delta wait resolved 68.1% and 81.0% of their steps without an LLM call, and a weak Goal score was escalated to the fallback, as intended, because a subgoal inconsistency cannot be repaired locally. The remaining 65 unresolved flagged steps made no LLM call, either because their subgoal had already used its two fallback calls or because the rollout ended at that step. Because the controller continues with the remaining planner steps after an unrecovered failure, one failed task can contribute several such steps.
- Factor roles under controlled interventions (Section 7.2, Appendix E.3). Changing only the subgoal, the GUI target, or the post-action outcome moves the intended factor most in 12 of 15 randomly selected validation contexts, and the Goal, Ground, and Delta heads separate compatible transitions from mismatches on validation pairs with AUROC 0.686, 0.784, and 0.726.
Response to Reviewer e2jJ (3/4)
W2 / Q2 (continued)
-
Factor-aware vs. generic recovery (Appendix H.3). We compared FAC-CUA with a generic variant that keeps the same latent, verifier scores, thresholds, candidate pool, per-step retry budget, safety policy, and fallback, and removes only the factor-level diagnosis. The variant flags a step with a single consistency score , exactly as FAC-CUA does, but applies the same recovery regardless of which factor is weak: it retries the next retained candidate and calls the LLM fallback if the step remains inconsistent. Both policies were run on 30 tasks, using for each task the same plan, generated once by GPT-5.2 and cached. The 30 tasks were selected from the original 286-task evaluation: 15 of the 17 tasks FAC-CUA solved but its w/o self-recovery ablation failed (excluding two with anonymized instructions), plus 15 randomly sampled from the 49 tasks both FAC-CUA and the monolithic controller failed. The random sampling did not use the human failure labels in Appendix H.2. This set intentionally emphasizes recovery-sensitive and difficult tasks and is not representative of the full benchmark. Because this experiment evaluates how the verifier's diagnosis is used for recovery rather than resource cost, both policies used GPT-5.2 instead of Qwen2.5-VL-7B for every LLM call (planning, LocalPlanner, and verifier-triggered fallback), whereas Table 1 uses Qwen2.5-VL-7B for planning and the fallback; absolute success rates and call counts are therefore not comparable to Table 1. Each task was run once per policy, and a run was counted as successful when the requested result was reached on screen, judged from the execution logs.
Across all 30 tasks, no flagged step had a retained alternative, so neither policy executed a candidate retry, and the two policies differed only on Delta-flagged steps, where FAC-CUA waited and re-observed while the generic variant called the fallback. The comparison therefore tests the Delta-specific wait rather than every factor-specific recovery; the Ground retry is exercised in the 286-task traces above. FAC-CUA succeeds on 9 of the 30 tasks (30.0%) and the generic variant on 5 (16.7%). Seven tasks could not succeed under either policy because their preconditions were unavailable in our environment (an expired Office license, a missing database client, a required application or video not open, or a travel date already in the past); on the remaining 23 tasks, success is 39.1% versus 21.7%. Across all 30 tasks, FAC-CUA makes 3.0 fallback calls per task and the generic variant 3.5. All four tasks on which the policies disagree are solved only by FAC-CUA (exact McNemar p = 0.125). In two of them, the generic variant called the fallback at a Delta-flagged step where FAC-CUA waited, and the run opened the wrong application or never left Windows Search. The generic variant's additional fallback calls were also followed by corrupted address-bar text in 8 runs, against 2 for FAC-CUA. The direction is consistent across these measures, but the sample is small and the difference is not statistically significant.
-
Which recovery each factor triggered in actual execution. In the 30 FAC-CUA executions, the verifier flagged 182 of the 191 scored steps. Every step with a negative Delta margin was routed to a wait and re-observation (49 steps, no LLM call). Steps whose Delta margin was non-negative but whose Goal or Ground margin was negative were sent to the LLM fallback (91 calls), or continued once the per-subgoal fallback budget was used (42 steps); because no flagged step had a retained alternative, Ground-flagged steps also went to the fallback.
Response to Reviewer e2jJ (1/4)
We thank the reviewer for the clear summary and for recognizing that the research question is valuable independently of high-level planning. We uploaded a revised PDF, and we address the two concerns with new analyses below.
W1 / Q1. After matching capacity and training, do the grounding and task-success gaps persist?
We added three controls (Appendix H.1). Mono-16 trains the monolithic baseline (Mono-8) for 8 more epochs, matching FAC-CUA's 8 + 8 epochs. Mono-384 widens the monolithic latent to 384 dimensions. Shared-MultiHead copies the Mono-8 encoders, uses the same inputs, objective, schedule, and trainable parameter count as FAC-CUA (25,903,876 vs. 25,904,846), and removes only the role-specific input routing.
R@k is context–outcome retrieval over the 444 validation records, which are disjoint from the training episodes but also used for checkpoint selection; Top-1 is target selection on held-out test queries. All models are evaluated at their final epoch.