acceptodds
Under review as a conference paper at ICLR 2027

Fac-CUA: Factorized Transition-Latent Representations for Efficient Local GUI Control

Abstract

Computer-use agents repeatedly reason about what local action is intended, where that action should be grounded in the current interface, and whether the resulting GUI transition is consistent with task progress. Performing these decisions through large language or vision-language model inference at every interaction step can accumulate substantial latency, token usage, memory demand, and compute. We introduce FAC-CUA, a factorized transition-latent representation learned from logged GUI interactions. Rather than compressing local-control state into a single embedding, FAC-CUA separates three control-functional roles: goal consistency, target grounding, and action-conditioned interface change. These factors support selective target grounding, post-action transition verification, and bounded self- recovery before LLM fallback. We evaluate representation quality separately from system-level resource efficiency. Under frozen diagnostic probes, the evaluated factorized representation achieves 64.7% 4-way local-failure Macro-F1, compared with 39.3% for a monolithic repre- sentation trained on the same transition data. In a controlled 286-task evaluation with frozen planner outputs, FAC-CUA reaches 79.0% task success and 52.8% within-budget success (WBS), compared with 47.6% and 16.4% for the monolithic latent controller, while reducing median rollout wall-clock time from 89.55 s to 49.29 s. Without retraining or threshold calibration, the same controller further completes 17/40 tasks on a fixed WindowsAgentArena subset. These results support the evaluated control-functional factorized architecture as a lightweight local-control substrate for GUI agents with structured actionable observations, while WBS provides a complementary system-level view of whether successful execution remains feasible under shared resource constraints.

Primary area:
applications to robotics, autonomy, planning
Keywords:
computer-use agents, GUI agents, latent representations, transition modeling, factorized representations, resource-efficient agents, planner-controller architecture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e2jJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies local execution for GUI agents: given an action intent from an upstream planner, how can a controller ground the target, verify the resulting interface change, and reduce large-model calls? FAC-CUA learns three factors for subgoal consistency, target grounding, and interface change from interaction logs, and combines them with bounded self-recovery. On 286 tasks with shared planner outputs, it improves success and execution time over a monolithic latent controller. The paper also evaluates grounding, failure diagnosis, and transfer on a WindowsAgentArena subset.

Critical strengths
  1. Dividing local GUI control into subgoal consistency, target grounding, and interface change connects representation learning directly to grounding, verification, and recovery. The research question is clear and has practical value independently of high-level planning.
  2. Grounding experiments and recovery ablations examine the representation and control mechanism separately, while the full controller improves both success and execution time. The system gains therefore do not depend on a single resource metric.
Critical weaknesses
  1. The factorized model adds specialized branches and receives 8 additional training epochs after initialization from the monolithic model, without a monolithic comparison matched for capacity and training. It remains unclear whether factorization, a stronger model, or additional training drives the improvement. This weakens the paper's central representation-learning conclusion.
  2. Failure diagnosis primarily uses synthetic errors constructed around the factor roles, without establishing a link between diagnoses and successful recovery during actual execution. The recovery ablation shows that recovery helps, but does not establish that factor-level diagnosis is more useful than a generic consistency score. The distinctive contribution of the representation to recovery therefore remains unclear.
Questions for authors
  1. after matching representation capacity and giving the monolithic model the same additional training, do the grounding and task-success gaps persist?
  2. which recovery actions are triggered by each factor during actual execution? Can existing traces show when factor-level diagnosis helps recovery, rather than merely detecting an execution anomaly?
5 comments
AuthorsOpenReview ↗

Response to Reviewer e2jJ (1/4)

We thank the reviewer for the clear summary and for recognizing that the research question is valuable independently of high-level planning. We uploaded a revised PDF, and we address the two concerns with new analyses below.

W1 / Q1. After matching capacity and training, do the grounding and task-success gaps persist?

We added three controls (Appendix H.1). Mono-16 trains the monolithic baseline (Mono-8) for 8 more epochs, matching FAC-CUA's 8 + 8 epochs. Mono-384 widens the monolithic latent to 384 dimensions. Shared-MultiHead copies the Mono-8 encoders, uses the same inputs, objective, schedule, and trainable parameter count as FAC-CUA (25,903,876 vs. 25,904,846), and removes only the role-specific input routing.

ModelEpochsParamsR@1 (%)R@5 (%)Top-1 + aux (%)
Mono-8824.954M40.9959.4676.8
Mono-168 + 824.954M42.7960.5977.9
Mono-3841625.659M43.2460.8177.9
Shared-MultiHead8 + 825.904M36.0457.2166.2
FAC-CUA8 + 825.905M64.8679.5088.8

R@k is context–outcome retrieval over the 444 validation records, which are disjoint from the training episodes but also used for checkpoint selection; Top-1 is target selection on held-out test queries. All models are evaluated at their final epoch.

  • Training matched. Eight more epochs of monolithic training add only 1.8 points of R@1 (40.99 → 42.79); FAC-CUA is 22.1 points higher (95% Wilson intervals [60.3, 69.2] vs. [38.3, 47.4]).
  • Grounding. Selectors trained on Mono-16 and Mono-384 both reach 77.9% Top-1, versus 88.8% for FAC-CUA, so the grounding gap persists under matched training and a wider latent.
  • Capacity and supervision matched. Removing only the routing lowers R@1 to 36.04 and Top-1 to 66.2. Shared-MultiHead falls below Mono-8 because the monolithic models are trained with a retrieval objective, whereas Shared-MultiHead and FAC-CUA share the factorized objective; under that shared objective, the routing accounts for a 28.8-point gain.
  • Data size. FAC-CUA trained on 25% of the records (R@1 57.88) already exceeds Mono-16 trained on all records.
  • Task success. We ran Shared-MultiHead on the 286 tasks inside the full FAC-CUA controller, with its own selector and verifier heads trained by the same procedures, the same runtime thresholds (selected for FAC-CUA, not re-tuned), and all other components unchanged. Task success is 42.0% (120/286) versus 79.0% for FAC-CUA, task progress 79.1% versus 92.7%, and LLM calls per task 3.50 versus 2.10 (Appendix H.1). The task-success gap therefore persists under matched capacity, supervision, controller, and recovery.
AuthorsOpenReview ↗

Response to Reviewer e2jJ (2/4)

W2 / Q2. Which recovery actions does each factor trigger, and does factor-level diagnosis help recovery beyond anomaly detection?

  • Routing rule (Section 4.5, Appendix D.1). Passing all three thresholds continues execution. The rules apply in order: a Ground failure retries the next retained candidate; otherwise a Delta failure waits 0.6 s and re-observes the interface; any remaining inconsistency, including a Goal failure, invokes the LLM fallback. Each planner step allows one target retry and up to two waits, and the verifier-triggered fallback is limited to one call per step.
  • Real execution failures (Appendix H.2). We re-executed, with both controllers, the 49 tasks that both latent controllers failed in the 286-task evaluation. Two authors annotated the first failing step of each failed run, defined as the first instruction-specific checkpoint not reached, as GROUND, DELTA, UPSTREAM_PLAN, or OTHER.
ControllerRe-run successGROUNDDELTAUPSTREAM_PLANOTHER
Monolithic latent5/49196190
FAC-CUA16/49141171

Paired by task, 11 tasks succeed only with FAC-CUA and none only with the monolithic controller (exact McNemar p = 0.001). DELTA failures, where the interface moved to the wrong application and the controller kept acting, drop from 6 to 1; in five of the six monolithic cases, pressing Enter in Windows Search opened a web search or left the search panel open instead of opening the target file.

  • What happened in actual execution (Appendix H.3). We counted the recovery actions in the existing traces of the 286-task run reported in Table 1, over the 2,988 planner steps scored by the verifier. A step is resolved without LLM when all margins become non-negative without an LLM call.
Recovery in the stepStepsResolved without LLM
Flagged at first scoring1,282861 (67.2%)
Ground retry only577393 (68.1%)
Delta wait only421341 (81.0%)
Ground retry and Delta wait134127 (94.8%)
No local recovery (sent to the fallback or ended)150—
LLM fallback, any reason482—
(Goal margin only negative, at first scoring or after a retry or wait / unresolved retry or wait / selector abstention)(218 / 138 / 126)—

Each factor triggered a different recovery: the Ground retry and the Delta wait resolved 68.1% and 81.0% of their steps without an LLM call, and a weak Goal score was escalated to the fallback, as intended, because a subgoal inconsistency cannot be repaired locally. The remaining 65 unresolved flagged steps made no LLM call, either because their subgoal had already used its two fallback calls or because the rollout ended at that step. Because the controller continues with the remaining planner steps after an unrecovered failure, one failed task can contribute several such steps.

  • Factor roles under controlled interventions (Section 7.2, Appendix E.3). Changing only the subgoal, the GUI target, or the post-action outcome moves the intended factor most in 12 of 15 randomly selected validation contexts, and the Goal, Ground, and Delta heads separate compatible transitions from mismatches on validation pairs with AUROC 0.686, 0.784, and 0.726.
AuthorsOpenReview ↗

Response to Reviewer e2jJ (3/4)

W2 / Q2 (continued)

  • Factor-aware vs. generic recovery (Appendix H.3). We compared FAC-CUA with a generic variant that keeps the same latent, verifier scores, thresholds, candidate pool, per-step retry budget, safety policy, and fallback, and removes only the factor-level diagnosis. The variant flags a step with a single consistency score sgen=min⁡f(sf−τf)<0s_{\text{gen}} = \min_f (s_f - \tau_f) < 0, exactly as FAC-CUA does, but applies the same recovery regardless of which factor is weak: it retries the next retained candidate and calls the LLM fallback if the step remains inconsistent. Both policies were run on 30 tasks, using for each task the same plan, generated once by GPT-5.2 and cached. The 30 tasks were selected from the original 286-task evaluation: 15 of the 17 tasks FAC-CUA solved but its w/o self-recovery ablation failed (excluding two with anonymized instructions), plus 15 randomly sampled from the 49 tasks both FAC-CUA and the monolithic controller failed. The random sampling did not use the human failure labels in Appendix H.2. This set intentionally emphasizes recovery-sensitive and difficult tasks and is not representative of the full benchmark. Because this experiment evaluates how the verifier's diagnosis is used for recovery rather than resource cost, both policies used GPT-5.2 instead of Qwen2.5-VL-7B for every LLM call (planning, LocalPlanner, and verifier-triggered fallback), whereas Table 1 uses Qwen2.5-VL-7B for planning and the fallback; absolute success rates and call counts are therefore not comparable to Table 1. Each task was run once per policy, and a run was counted as successful when the requested result was reached on screen, judged from the execution logs.

    Across all 30 tasks, no flagged step had a retained alternative, so neither policy executed a candidate retry, and the two policies differed only on Delta-flagged steps, where FAC-CUA waited and re-observed while the generic variant called the fallback. The comparison therefore tests the Delta-specific wait rather than every factor-specific recovery; the Ground retry is exercised in the 286-task traces above. FAC-CUA succeeds on 9 of the 30 tasks (30.0%) and the generic variant on 5 (16.7%). Seven tasks could not succeed under either policy because their preconditions were unavailable in our environment (an expired Office license, a missing database client, a required application or video not open, or a travel date already in the past); on the remaining 23 tasks, success is 39.1% versus 21.7%. Across all 30 tasks, FAC-CUA makes 3.0 fallback calls per task and the generic variant 3.5. All four tasks on which the policies disagree are solved only by FAC-CUA (exact McNemar p = 0.125). In two of them, the generic variant called the fallback at a Delta-flagged step where FAC-CUA waited, and the run opened the wrong application or never left Windows Search. The generic variant's additional fallback calls were also followed by corrupted address-bar text in 8 runs, against 2 for FAC-CUA. The direction is consistent across these measures, but the sample is small and the difference is not statistically significant.

  • Which recovery each factor triggered in actual execution. In the 30 FAC-CUA executions, the verifier flagged 182 of the 191 scored steps. Every step with a negative Delta margin was routed to a wait and re-observation (49 steps, no LLM call). Steps whose Delta margin was non-negative but whose Goal or Ground margin was negative were sent to the LLM fallback (91 calls), or continued once the per-subgoal fallback budget was used (42 steps); because no flagged step had a retained alternative, Ground-flagged steps also went to the fallback.

Reviewer_rgCpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to reduce the cost of invoking large language models (LLMs) during the local execution of GUI tasks, given a set of high-level semantic actions. The proposed method encodes structured interface transitions into three types of representations—Goal, Ground, and Delta—for candidate UI element selection, execution verification, and limited local recovery. LLMs are invoked only when these mechanisms cannot resolve an issue. In experiments with fixed high-level planning outputs, the paper reports higher task success and lower resource consumption than an every-step Qwen controller. The paper also includes diagnostic analyses of the representations, component ablations, and cross-environment testing.

Critical strengths
  1. Reducing LLM invocation costs for repetitive local operations has practical value. Delegating routine operations such as element localization and verification to lightweight models, while reserving LLMs for complex scenarios, is a promising research direction.

  2. The evaluation helps clarify the roles of different system components. The paper evaluates task success rate, latency, the number of LLM invocations, and behavior under resource constraints. It also provides ablations of the self-recovery mechanisms and diagnostic analyses of the representations.

Critical weaknesses
  1. The applicability and limitations of the planning and recovery mechanisms require further clarification. In the main experiments, high-level semantic actions are generated in advance and remain fixed, although subsequent UI states may diverge from the initial plan. The paper does not sufficiently explain how the system distinguishes planning errors from local execution errors or handles cases requiring multi-step backtracking. Recovery is largely limited to waiting or selecting alternative candidates and cannot rectify irreversible consequences of high-confidence but incorrect selections. Although the authors limit the scope to local control and include pre-execution checks, the implications of these assumptions and safeguards for practical reliability remain insufficiently explained.

  2. The performance gap between the proposed method and the Qwen baseline is insufficiently explained. Table 1 reports a raw task success rate of 29.4% for the every-step Qwen controller with verification, compared with 79.0% for the full method and 73.1% for FAC-CUA without self-recovery. The paper should provide greater detail on the baseline’s prompting strategy, verification protocols, retry policies, and termination conditions. An analysis of the primary failure modes is also needed to determine whether the gains arise from the proposed learning approach or from differences in controller implementation.

  3. The benefit attributable specifically to the three-factor representation has not been sufficiently isolated. The single-vector model uses a 256-dimensional representation, whereas the three-factor model uses three 128-dimensional representations with additional training and a different downstream architecture. The existing ablations and diagnostic analyses do not sufficiently rule out the influence of model capacity, training compute, or controller design. The three-factor representation should be compared with a standard shared representation equipped with multiple task-specific heads, while matching total capacity, training budget, and downstream control strategies.

Questions for authors
  1. What information from the actual environment is provided to the planner when generating a fixed plan? When subsequent pages deviate from the plan, which steps can the fallback mechanism modify, and what execution history does it use?

  2. What are the primary failure modes of the Qwen baseline? Do the methods share the same executor, retry permissions, and termination criteria? Please provide the full configuration and a detailed breakdown of errors in target selection and verification.

  3. Has the single-vector model been compared with the three-factor model under matched total representation dimensionality, training budget, downstream task heads, and recovery strategies? If not, to what extent do the existing results establish the benefit of the factorized representation itself?

  4. Is risk assessed based on the action type specified in the plan or on the specific UI element and target object actually selected? For irreversible actions resulting from a high-confidence but incorrect selection, can the existing safeguards intervene before execution?

5 comments
AuthorsOpenReview ↗

Response to Reviewer rgCp (1/3)

We thank the reviewer for the precise questions and have uploaded a revised PDF.

Q1. Planner input, plan divergence, and fallback scope (Weakness 1)

  • Planner input (Appendix F). The planner (Qwen2.5-VL-7B) receives the user instruction and similar task units retrieved from labeled interaction logs as in-context examples; it receives no GUI observation. Its output is generated once for the actual instruction, cached without editing, and shared by all controllers.
  • When the page diverges from the plan. Each planner step is a semantic intent, not a target identifier, so FAC-CUA re-grounds it against the current GUI state at every step. If no candidate passes the selector thresholds, or the Goal score falls below its threshold, the step goes to fallback.
  • What fallback can modify and what history it uses. Fallback calls the LocalPlanner with the current planner step, up to two neighboring planner steps on each side, and the indexed visible UI elements. It returns an action sequence for the current step only and is instructed not to re-plan the task or invent candidates that are not visible. The prompt does not include the execution log.
  • Planning errors vs. local execution errors. At runtime, the controller does not decide whether a failure comes from the plan or from local execution. The Goal factor checks consistency with the planner-provided subgoal, not whether the plan satisfies the instruction, so upstream planning errors are outside what the verifier can detect (Section 8). The factor scores act as routing signals: a Ground failure triggers a target retry, a Delta failure triggers wait-and-re-observe, and a Goal failure escalates to fallback. To relate these signals to actual causes, we re-executed the 49 tasks that both latent controllers failed and annotated the first failing step of each failed run (Appendix H.2): 17 of FAC-CUA's 33 remaining failures were upstream planning errors, which the local controller cannot repair.
  • Multi-step backtracking. The local controller does not perform multi-step backtracking or rollback; Section 8 now states this.

Q4. Risk assessment and irreversible actions (Weakness 1)

  • What counts as irreversible. We distinguish actions with lasting side effects (delete, submit, send, publish, overwrite, purchase) from wrong clicks that destroy task state, such as closing a window or leaving a page with unsaved input, after which later steps cannot proceed.
  • Unit of risk assessment. Risk is assessed on the planned action type, not on the selected UI element. Irreversible steps such as deletion are generated only when the user's command explicitly requests them, steps that need the user's credentials or decisions are handed to the user through a confirmation dialog, and the side-effectful action types above are never executed by autonomous recovery (Appendix D.2). A retained-alternative click is allowed only when the alternative remains visible and enabled and the previous transition does not indicate an irreversible state change.
  • Checks before execution. The selector evaluates whether the selected element fits the intended action: it abstains unless p(1)≥0.60p_{(1)} \ge 0.60 and m≥0.75m \ge 0.75, and the executor checks candidate validity, visibility, enabled state, geometry, supported action type, retry budget, duplicate text input, and repeated unsafe actions. This is a grounding-confidence check, not an element-level safety assessment.
  • What the safeguards cannot prevent. We agree that the current safeguards cannot guarantee preventing the irreversible consequences of a high-confidence incorrect selection. If, during an otherwise permitted click step, the selector confidently selects a dangerous element (e.g., a delete or close button), the controller may execute it without detecting the risk. The verifier may detect such an unexpected transition after execution and trigger local recovery or the LLM fallback, but it cannot guarantee detection or restore lost task state such as unsaved input. Section 8 states this limitation; element-level risk assessment before execution is future work.
  • How often this occurred. In the 286-task evaluation, 8 FAC-CUA rollouts (2.8%) contained a wrong click that destroyed task state, mainly by closing a window or leaving a page with unsaved input. Such failures are difficult to prevent when the automation environment provides incomplete information about the current interface or available interactions, particularly in dynamic web applications. We acknowledge this limitation rather than claiming the safeguards prevent such failures.
AuthorsOpenReview ↗

Response to Reviewer rgCp (2/3)

Q2. Qwen baseline configuration and failure modes (Weakness 2)

  • Shared evaluation protocol. Both methods run the same 286 tasks with the same cached plan, GUI parser, and execution backend, and are judged against the same instruction-specific checkpoints, with partial completion counted as failure.
  • Prompting strategy (Section 5.2, Appendix F). At every target-dependent step, the baseline calls with Qwen2.5-VL-7B the same action-specific LocalPlanner prompt templates that FAC-CUA uses as its fallback. The prompt contains the current planner step with up to two neighboring steps on each side, a GUI-state summary, and the indexed component dictionary; the model returns an action sequence, an executability judgment, and an observation (temperature 0.5 by default, 0.05 for open and 0.4 for repeat, and at most 4,000 output tokens). For example, the click prompt instructs the model to "select the most semantically similar component to the object in task." The GUI-state summary is generated by Qwen once per task unit from the DOM or accessibility tree and is instructed to "only mention elements present in the provided actionable list." No screenshot is used by either method.
  • Retry, verification, and termination. When Qwen judges a step non-executable, the baseline scrolls, re-parses the interface, and queries the LocalPlanner once more. After each planner step, a Qwen verifier judges from the execution log and the DOM summary whether the step succeeded and, on a failure, diagnoses its cause: for a wrong click it triggers one LocalPlanner correction that re-executes the click, and for a wrong interface transition it re-checks the transition. This is an LLM-based verifier, whereas FAC-CUA uses its learned factor-specific verifier heads. Neither method replans; both follow the cached planner steps once in order. FAC-CUA instead allows one alternative-target retry and up to two wait-and-re-observe actions per planner step before the LLM fallback. The summary and verifier requests are included in the baseline's LLM calls, tokens, latency, and WBS.
  • Failure analysis (Section 6.1). Across the 3,449 local-control steps in the Qwen traces, including re-executions after retries and corrections, grouped by stage:
StageFailure / recovery typeStepsRate (%)
Target selectionNo matching GUI element1524.4
Interface transitionNon-executable action38311.1
ExecutionExecution error411.2
VerificationLocal-planning retry62818.2

Target-selection errors are steps in which no GUI element matching the planner object could be selected. Non-executable actions arise after a wrong interface transition: the interface reached was one in which the action generated for the step could not be executed. Execution errors are wrong clicks or wrongly opened windows. Local-planning retries are the corrections triggered by the step verifier. Categories may overlap and are logged events rather than confirmed task-level causes. Of the 202 failed tasks, 174 (86.1%) made nonzero progress. These counts show difficulties in target matching and execution despite the recovery mechanisms, but they do not fully explain the 202 failures. One plausible mechanism is the propagation of a local error into later steps: when a wrong click or an unexpected interface transition leaves the agent in a GUI state inconsistent with the plan, the LLM-based step verifier may not recognize or correct the mismatch, and the controller continues with the next planned step in an unintended interface, where that action can no longer be executed. This pattern is consistent with the 383 non-executable-action steps and 41 execution-error steps above. FAC-CUA instead re-observes the structured GUI state after each action and checks factor-specific consistency, which triggers a target retry, a wait and re-observation, or the LLM fallback before the next step is attempted; this gives the controller a chance to correct a local deviation, although it does not guarantee detection or recovery for every wrong transition.

  • Learning vs. controller implementation. The Qwen baseline uses the same action-specific LocalPlanner prompt templates and execution backend as FAC-CUA, while their verification and recovery mechanisms differ as described above. FAC-CUA without self-recovery still reaches 73.1% task success versus 29.4% for the Qwen baseline (Table 1), so FAC-CUA's self-recovery alone does not account for the gap. The matched Shared-MultiHead experiments in Q3 isolate the contribution of role-specific factorization under identical downstream control and recovery.
AuthorsOpenReview ↗

Response to Reviewer rgCp (3/3)

Q3. Matched comparison with a shared representation and multiple heads (Weakness 3)

We added Shared-MultiHead, the comparison the reviewer suggested (Appendix H.1). It copies the Mono-8 encoders and receives the same field-level inputs as FAC-CUA, but all context fields pass through one shared 256-dimensional context representation and all outcome fields through one shared 256-dimensional outcome representation; the Goal, Ground, and Delta heads read these shared representations with the same head architecture as FAC-CUA and produce the same three 128-dimensional factor outputs. It matches FAC-CUA in trainable parameters (25,903,876 vs. 25,904,846), training budget (the same Mono-8 initialization, then 8 epochs with the same objective, loss weights, and auxiliary heads), and task heads, and the same selector is trained on each representation. We also added monolithic controls with matched training (Mono-16) and a wider latent (Mono-384).

ModelEpochsParamsR@1 (%)R@5 (%)Top-1 + aux (%)
Mono-8824.954M40.9959.4676.8
Mono-168 + 824.954M42.7960.5977.9
Mono-3841625.659M43.2460.8177.9
Shared-MultiHead8 + 825.904M36.0457.2166.2
FAC-CUA8 + 825.905M64.8679.5088.8

R@k is context–outcome retrieval over the 444 validation records; Top-1 is target selection on held-out test queries. FAC-CUA is 28.8 points higher than Shared-MultiHead in R@1 and 22.6 points higher in Top-1, so under matched capacity, training budget, and supervision, the gain is attributable to the role-specific routing. With matched training (Mono-16), FAC-CUA is 22.1 points higher in R@1 and 10.9 points higher in Top-1.

Task-level comparison with matched downstream control and recovery. We also ran Shared-MultiHead on the 286 tasks inside the full FAC-CUA controller. Its selector and verifier heads are trained on the Shared-MultiHead representation with the same procedures as FAC-CUA, and the runtime thresholds (selector 0.60/0.75; verifier 0.33/0.0772/0.125), cached plans, routing and recovery rules, retry budgets, safety policy, and LocalPlanner fallback are those of FAC-CUA, so the two controllers differ only in the role-specific input routing (Appendix H.1). Because these thresholds were selected for FAC-CUA and not re-tuned for Shared-MultiHead, the comparison measures the effect of removing the routing under a fixed runtime configuration.

ControllerSuccess (%)Task progress (%)LLM calls / taskMedian wall-clock (s)Median max-step latency (s)
Shared-MultiHead (in FAC-CUA controller)42.0 (120/286)79.13.5072.507.20
FAC-CUA79.092.72.1049.295.15

With the same downstream control and recovery strategy, removing only the routing lowers task success by 37.0 points, and the controller relies more on the LLM fallback (3.50 vs. 2.10 calls per task) and runs longer. The gain from the three-factor representation therefore persists at the task level, not only at the representation and target-selection levels.

We thank the reviewer again; these questions led to a clearer separation of the method's scope and safeguards.

Reviewer_69VbOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents FAC-CUA, a lightweight local GUI controller that receives semantic actions from an upstream planner and operates on structured GUI observations. It learns Goal, Ground, and Delta representations from logged transitions and uses them for target selection, transition verification, and local recovery before LLM fallback. Evaluation shows that FAC-CUA achieves 79.0% task success and 52.8% within-budget success on 286 controlled tasks.

Critical strengths
  1. The paper identifies an important problem in GUI agents: repeated grounding and transition verification can incur substantial inference costs.
  2. The evaluation reports both task completion and execution costs, and includes an ablation of local self-recovery.
Critical weaknesses
  1. The paper fails to explain its method with the clarity and completeness required for technical assessment. Dense terminology and fragmented explanations make the connections between the learned factors, target selection, verification, and recovery difficult to follow. Essential details about how the selector and verifier are trained and how their scores are computed remain inadequately specified in the main text. Repeated defensive qualifications add substantial text without clarifying these basic mechanisms. These are fundamental deficiencies that seriously undermine the method’s assessability and reproducibility.
  2. The experiments do not establish the benefit of factorization or its practical usefulness. The representation comparisons do not control model capacity and use different diagnostic features, leaving the source of the gains unclear. Failure classification relies on synthetic errors aligned with the predefined factor roles, while the intervention study contains only 15 contexts. The main system evaluation uses frozen planner outputs, and the transfer test includes only two tasks per non-Web domain. Task construction and success evaluation are also insufficiently specified. These experiments provide weak evidence that the proposed factors diagnose real execution failures and improve GUI agents under online planning.
  3. The efficiency evaluation is incomplete and not sufficiently specified. The profiling hardware, model precision, and memory and compute accounting are unclear, although these measurements directly determine whether a task satisfies the WBS budgets. The UI-TARS comparison also excludes FAC-CUA’s upstream planning costs. The reported resource figures therefore do not establish efficiency gains for complete agents.
  4. The main text shows poor proofreading and presentation issues. For example, there is a missing citation displayed as ?. The authors include too many defensive statements and disclaimers throughout the main text. These repeated statements disrupt the flow of reading and take up valuable space that should be used for technical explanations.
Questions for authors

Please see weaknesses.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 69Vb (1/4)

We thank the reviewer for the detailed and candid assessment. We agree that the original manuscript did not present the method clearly enough for technical assessment. We uploaded a revised PDF; all results reported in the original submission are unchanged. Below we point to the exact changes.

W1. Clarity and completeness of the method

  • Factor inputs and outputs (Section 4.2, Appendix C.1). Each factor has a context-side and an outcome-side branch. Goal takes the subgoal text and state summary; Ground takes the state summary and the candidate slot set, without action text; Delta takes the state summary, semantic action, and slot set on the context side, and the slot-level transition diff and post-action slot set on the outcome side. Appendix C.1 lists every input field.
  • From factors to selection, verification, and recovery (Sections 4.4–4.5, Appendix D.1).
    1. Stage 1 keeps a high-recall candidate set (validation-selected threshold 0.01, top-5 floor, at most 30 candidates).
    2. Stage 2 ranks the retained candidates with Ground and Delta interaction features and runtime metadata. It executes the top candidate only if its softmax score p(1)≥0.60p_{(1)} \ge 0.60 and the raw-logit margin m=s(1)−s(2)≥0.75m = s_{(1)} - s_{(2)} \ge 0.75; otherwise it abstains and the step goes to the LLM fallback.
    3. sgrounds_{\text{ground}} checks, from pre-action evidence only, whether the selected pre-action target is compatible with a query built from the context-side Goal and Ground factors. After execution, sgoals_{\text{goal}} compares the context- and outcome-side Goal factors and sΔs_{\Delta} compares the predicted change z^Δ=PΔ(zΔctx)\hat{z}_{\Delta} = P_{\Delta}(z_{\Delta}^{\text{ctx}}) with the observed outcome-side Delta factor. Goal and Ground use learned sigmoid heads over the pair feature ϕ(x,y)=[x, y, x⊙y, ∣x−y∣, cos⁡(x,y)]\phi(x, y) = [x,\, y,\, x \odot y,\, \lvert x - y \rvert,\, \cos(x, y)]; the Delta head uses an augmented feature that also includes the context-side Delta factor and its cosine similarity with the observed outcome. Cosine similarity is an input feature rather than the score itself.
    4. The router computes the margins μf=sf−τf\mu_f = s_f - \tau_f (τgoal=0.33\tau_{\text{goal}} = 0.33, τground=0.0772\tau_{\text{ground}} = 0.0772, τΔ=0.125\tau_{\Delta} = 0.125) and applies the first matching rule. If all margins are non-negative, execution continues. If the Ground margin is negative and a retained alternative exists, the controller clicks the next retained candidate (once per step, click actions only, after the safety gate). Otherwise, if the Delta margin is negative, it waits 0.6 s, re-parses the interface, and re-scores the step without re-executing the action (at most twice per step). All remaining cases, including a negative Goal margin and exhausted retry budgets, invoke the LocalPlanner fallback with the post-action state, limited to one verifier-triggered call per step and two per subgoal; a step in which the selector abstains additionally uses one LocalPlanner call to choose the target. If the fallback also fails, the step ends as a failure.
  • Training of the representation (Section 4.3, Appendix C.3). Section 4.3 now describes the objective used to train the reported checkpoint: relation-conditioned context–outcome alignment, counterfactual consistency, auxiliary supervision (target-slot ranking, goal-family classification and supervised contrastive alignment, action-type and transition-diff prediction), a variance–covariance regularizer, and a one-epoch distillation warmup from the monolithic encoder. Appendix C.3 gives every term, its supervision, and its weight. The original text described the alignment term as InfoNCE, omitted the auxiliary, regularization, and distillation terms, and misstated the checkpoint-selection rule and several settings (micro-batch size, margins, precision); these now match the released code. All reported results come from this implementation.
AuthorsOpenReview ↗

Response to Reviewer 69Vb (2/4)

W1 (continued)

  • Training of the selector and verifier (Sections 4.4–4.5, Appendix D.1). All heads are trained on features of the frozen factorized encoder. Stage 1 uses weighted binary cross-entropy plus a margin ranking loss between the logged target and the negative candidates, and its keep threshold is selected on validation queries for target retention and then retained-set size. Stage 2, a two-layer candidate-set Transformer, uses listwise cross-entropy over the candidate set; its checkpoint is selected on validation queries by 0.7·Top-1 + 0.3·MRR, and its execution thresholds (0.60, 0.75) are set on the training queries. The verifier heads are trained on groups built from the 1,175 training records, each with one logged transition as the positive and negatives from the same record: for Goal, a wrong post-action state, a different subgoal, or another transition's outcome; for Ground, lexically similar and random slots from the same screen; for Delta, action counterfactuals, a no-change outcome, and a wrong post-action state. Every head uses weighted binary cross-entropy with a within-group pairwise hinge loss (Ground adds listwise cross-entropy, Delta a cosine term), AdamW, and 20 epochs (Delta: 10). Checkpoints are selected on the 444 validation records by a ranking-based score, and each τf\tau_f is the threshold that maximizes balanced accuracy on the same records (Appendix D.1; supervision in Table 11).
  • Verifier-head quality (Appendix E.3). On validation transition pairs from episodes disjoint from training, the heads rank compatible transitions above mismatches with AUROC 0.686 (Goal), 0.784 (Ground), and 0.726 (Delta); these validation records are also used for checkpoint and threshold selection.
  • Terminology (Appendix B.1). A new table defines every term and symbol used in the method and runtime sections (task unit, planner step, slot set, transition diff, context and outcome sides, the three factors, candidate and retained sets, Stage-1/Stage-2, selector abstention, verifier scores and margins, self-recovery, LLM fallback, LLM-mediated call, and WBS@Ref), and Section 4 points to it.
  • Reproducibility. All thresholds are in Appendix D.1 and all training hyperparameters and loss weights in Appendices C.3 and D.1. We will release the code, checkpoints, and the 286-task configuration on GitHub.
  • Defensive statements. We removed repeated qualifications from the main text and kept the scope statements in Section 8. The freed space holds the technical details above.

W2. Evidence for the benefit of factorization

  • Model capacity and training (Appendix H.1). We added three controls. Mono-16 trains the monolithic baseline (Mono-8) for 8 more epochs, matching FAC-CUA's 8 + 8 epochs. Mono-384 widens the monolithic latent to 384 dimensions (16 epochs). Shared-MultiHead copies the same Mono-8 encoders, receives the same field-level inputs, objective, schedule, and seed as FAC-CUA, and matches its trainable parameter count (25,903,876 vs. 25,904,846); it removes only the role-specific input routing by mapping all context fields and all outcome fields to shared 256-dimensional representations read by the Goal, Ground, and Delta heads. R@k is context–outcome retrieval over the 444 validation records (also used for checkpoint selection, so these are validation comparisons); Top-1 is target selection on held-out test queries.
ModelEpochsParamsR@1 (%)R@5 (%)Top-1 + aux (%)
Mono-8824.954M40.9959.4676.8
Mono-168 + 824.954M42.7960.5977.9
Mono-3841625.659M43.2460.8177.9
Shared-MultiHead8 + 825.904M36.0457.2166.2
FAC-CUA8 + 825.905M64.8679.5088.8

Eight more epochs raise R@1 by only 1.8 points, and a wider latent gives a similar value; FAC-CUA is 22.1 points higher in R@1 and 10.9 points higher in Top-1 than Mono-16 (R@1 95% Wilson intervals [60.3, 69.2] vs. [38.3, 47.4]). With capacity, inputs, and objective matched, removing only the routing lowers R@1 to 36.04 ([31.7, 40.6]) and Top-1 to 66.2. Shared-MultiHead falls below Mono-8 because the monolithic models are trained with a retrieval objective, whereas Shared-MultiHead and FAC-CUA share the factorized objective; under that shared objective, the routing accounts for a 28.8-point gain in R@1. Run inside the full FAC-CUA controller on the 286 tasks, with its own selector and verifier heads and identical thresholds (selected for FAC-CUA), recovery, and fallback, Shared-MultiHead reaches 42.0% task success versus 79.0% for FAC-CUA, with 3.50 versus 2.10 LLM calls per task (Appendix H.1).

AuthorsOpenReview ↗

Response to Reviewer 69Vb (3/4)

W2 (continued)

  • Synthetic failures (Appendix H.2). We re-executed, with both controllers, the 49 tasks that both latent controllers failed in the 286-task evaluation. Two authors annotated the first failing step of each failed run, defined as the first instruction-specific checkpoint not reached, as GROUND (wrong target), DELTA (expected transition did not occur and was not recovered), UPSTREAM_PLAN (the plan could not complete the task even if executed perfectly), or OTHER (execution error).
ControllerRe-run successGROUNDDELTAUPSTREAM_PLANOTHER
Monolithic latent5/49196190
FAC-CUA16/49141171

Paired by task, 11 tasks succeed only with FAC-CUA and none only with the monolithic controller (exact McNemar p = 0.001). In five of the six monolithic DELTA cases, pressing Enter in Windows Search opened a web search or left the search panel open instead of opening the target file, and the controller continued in the wrong application. Most remaining FAC-CUA GROUND failures (11 of 14) occur on web pages, many on date pickers and filter controls.

  • Different diagnostic features (Appendix E.2). Both representations are probed with the same examples, splits, selected-candidate identities, and binary-head architecture, with 513-dimensional pair features. They differ only in input construction: monolithic latents pass through trainable 256→128 adapters, and the monolithic Ground input encodes the selected candidate from its raw features. The new evidence above does not depend on these features.
  • Only 15 intervention contexts. The 15 contexts were selected at random before outcomes were inspected, five per factor, and only the failure-relevant component is changed. The intended factor changes most in 12 of 15 pairs (Section 7.2). We present this as a sanity check and rely on the matched controls and real failures above.
  • Frozen planner outputs. The cached plans are the planner's unedited outputs for the actual instructions; we did not correct, filter, or regenerate them. Because the planner receives no GUI observation and does not re-plan during execution, running it online would issue the same single planning call per task; caching only fixes when that call is made and gives all controllers the identical plan (Section 5.1). Plan errors remain in the evaluation: 17 of FAC-CUA's 33 remaining failures above are upstream planning errors. All selector and verifier thresholds were set on the training or validation data and fixed before the 286-task and transfer evaluations, and with this unchanged configuration FAC-CUA also completes 19/30 tasks from 12 domains absent from the training and evaluation domain lists (Appendix E.9). Online re-planning is outside the scope of this paper (Section 8).
  • Two tasks per non-Web domain (Section 6.2, Appendix E.8). With the same checkpoint, thresholds, recovery rules, and fallback policy, FAC-CUA completes 53 of all 154 WindowsAgentArena tasks (34.4%) under the official evaluators with a 50-step horizon: Windows System 12/24, VS Code 11/24, VLC 8/21, Windows Utilities 8/12, Web 11/30, and Office 3/43. It completes 39/86 tasks on applications seen in the training logs and 14/68 on unseen ones (e.g., VLC and Windows Clock); WindowsAgentArena instructions and trajectories are not used for training. Appendix E.8 also keeps the earlier same-environment comparison on a fixed 40-task subset, where FAC-CUA completes 17/40 tasks and Navi (with GPT-4o in place of its unavailable GPT-4V backend) completes 11/40.
  • Task construction and success evaluation (Section 5.2, Appendix A.2). The 286 tasks comprise 100 in-house tasks, 116 WebVoyager tasks, and 70 ScreenAgent tasks; the benchmark tasks are sampled at random from their full task sets. The in-house tasks were curated by the authors across Local (30), Web (26), App (15), and Crossover (29) environments. For each instruction we defined the checkpoints a successful execution must pass (e.g., open Word, write the content, save, enter the target folder, set the file name), and every method is judged against the same checkpoints; partial completion counts as failure.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript addressing the main concerns raised in the reviews.

The revision clarifies the factorized representation, selector, verifier, and recovery mechanisms; adds capacity- and training-matched baselines, including Shared-MultiHead; extends the real-execution failure analysis and WindowsAgentArena evaluation; and provides detailed resource-accounting protocols, including upstream planning costs.

We also corrected presentation issues and consolidated the scope and limitations in Section 8.

Detailed point-by-point responses are provided under each review, with references to the corresponding sections and appendices of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.