CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation
Abstract
Graphical user interface (GUI) agents powered by vision language models (VLMs) are rapidly moving from passive assistance to autonomous operation. Yet a single unsafe action can cause financial, privacy, or social harm, and existing safeguards typically operate at heuristically chosen thresholds without explicit control over harmful execution. We propose **CORA** (**CO**nformal **R**isk-controlled GUI **A**gent), a pre-action controller that casts safeguarding as selective action execution. A non-generative, action-conditioned **Guardian** assigns a risk score to each proposed action. Conformal Risk Control then calibrates the execute/abstain threshold on this score, with a finite-sample guarantee that the probability of autonomously executing a harmful action stays within a user-specified budget. For each rejected action, a generative **Diagnostician** explains the risk and proposes a minimal intervention, yet it cannot override the rejection. A Goal-Lock input contract further separates the user's goal from untrusted interface content. We also introduce **Phone-Harm**, a mobile-agent benchmark with step-level harm labels. On a disjoint holdout, CORA autonomously executes 89.95% of proposed actions with a joint harmful-execution rate of 2.42% under a 5% budget. It also outperforms various baselines in routing accuracy on Phone-Harm and in F1 on MobileRisk. These results suggest that calibrating risk at the action-execution boundary offers a principled path towards reliable GUI agents.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.