Fac-CUA: Factorized Transition-Latent Representations for Efficient Local GUI Control

Computer-use agents repeatedly reason about what local action is intended, where that action should be grounded in the current interface, and whether the resulting GUI transition is consistent with task progress. Performing these decisions through large language or vision-language model inference at every interaction step can accumulate substantial latency, token usage, memory demand, and compute. We introduce FAC-CUA, a factorized transition-latent representation learned from logged GUI interactions. Rather than compressing local-control state into a single embedding, FAC-CUA separates three control-functional roles: goal consistency, target grounding, and action-conditioned interface change. These factors support selective target grounding, post-action transition verification, and bounded self- recovery before LLM fallback. We evaluate representation quality separately from system-level resource efficiency. Under frozen diagnostic probes, the evaluated factorized representation achieves 64.7% 4-way local-failure Macro-F1, compared with 39.3% for a monolithic repre- sentation trained on the same transition data. In a controlled 286-task evaluation with frozen planner outputs, FAC-CUA reaches 79.0% task success and 52.8% within-budget success (WBS), compared with 47.6% and 16.4% for the monolithic latent controller, while reducing median rollout wall-clock time from 89.55 s to 49.29 s. Without retraining or threshold calibration, the same controller further completes 17/40 tasks on a fixed WindowsAgentArena subset. These results support the evaluated control-functional factorized architecture as a lightweight local-control substrate for GUI agents with structured actionable observations, while WBS provides a complementary system-level view of whether successful execution remains feasible under shared resource constraints.