acceptodds
Under review as a conference paper at ICLR 2027

CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation

Abstract

Graphical user interface (GUI) agents powered by vision language models (VLMs) are rapidly moving from passive assistance to autonomous operation. Yet a single unsafe action can cause financial, privacy, or social harm, and existing safeguards typically operate at heuristically chosen thresholds without explicit control over harmful execution. We propose **CORA** (**CO**nformal **R**isk-controlled GUI **A**gent), a pre-action controller that casts safeguarding as selective action execution. A non-generative, action-conditioned **Guardian** assigns a risk score to each proposed action. Conformal Risk Control then calibrates the execute/abstain threshold on this score, with a finite-sample guarantee that the probability of autonomously executing a harmful action stays within a user-specified budget. For each rejected action, a generative **Diagnostician** explains the risk and proposes a minimal intervention, yet it cannot override the rejection. A Goal-Lock input contract further separates the user's goal from untrusted interface content. We also introduce **Phone-Harm**, a mobile-agent benchmark with step-level harm labels. On a disjoint holdout, CORA autonomously executes 89.95% of proposed actions with a joint harmful-execution rate of 2.42% under a 5% budget. It also outperforms various baselines in routing accuracy on Phone-Harm and in F1 on MobileRisk. These results suggest that calibrating risk at the action-execution boundary offers a principled path towards reliable GUI agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Mobile GUI Agents, Agent Safety, Conformal Risk Control, Selective Execution, Vision-Language Models, Safety Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_piD1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on improving the safety of mobile GUI agents by controlling harmful action execution while preserving agent autonomy. Existing safety mechanisms often rely on heuristic thresholds and lack explicit guarantees on execution risks. To address this limitation, the authors propose CORA, a pre-action safety framework that formulates safeguarding as selective action execution. CORA employs an action-conditioned Guardian to estimate the risk of proposed actions and uses Conformal Risk Control (CRC) to calibrate execution thresholds under a user-specified risk budget. For rejected actions, a generative Diagnostician provides explanations and intervention suggestions without overriding the safety decision, while a Goal-Lock mechanism separates trusted user goals from untrusted interface content. The authors also introduce Phone-Harm, a benchmark containing 150 harmful tasks with step-level annotations and 150 matched benign tasks. Experiments show that CORA achieves 89.95% autonomous coverage with a 2.42% joint harmful-execution rate under a 5% risk budget on a disjoint holdout, while outperforming evaluated baselines in routing accuracy on Phone-Harm and F1 on MobileRisk.

Critical strengths
  1. The paper presents a well-defined risk-budgeted safety control framework.

The paper formulates GUI agent safety as selective action execution under an explicit risk budget. It separates the action-conditioned Guardian from the generative Diagnostician, ensuring that rejected actions cannot be autonomously re-enabled by generated explanations. The CRC calibration procedure is carefully specified, including the finite-sample correction, tied scores, and the hard-abstention case (Appendix F.3). The reported holdout results are also consistent with the underlying action counts (Table 2 and Appendix E). The design provides a clear execution criterion while separating risk estimation from intervention generation.

  1. The paper provides transparent experimental reporting and a clear discussion of its limitations.

The paper reports detailed experimental results on runtime overhead, annotation reliability, and the scope of its theoretical guarantees. It shows a relatively small average latency increase from 20.52 to 21.37 seconds per step (Table 5). The paper also validates the step-level harm judge against human annotations and reports task-clustered bootstrap confidence intervals (Appendix G.4). Furthermore, it explicitly acknowledges that the CRC guarantee is marginal, relies on exchangeability, and does not directly extend to distribution shifts or adaptive closed-loop execution.

Critical weaknesses
  1. The paper's risk-control guarantee does not fully capture the safety objectives of practical GUI agents.

The paper uses CRC to bound the joint harmful-execution probability (P(H \cap E)), rather than the probability of executing a harmful proposal (P(E \mid H)) or the risk of harm over a complete task. At the reported operating point, CORA still allows 22 of 76 harmful calibration proposals (28.9%) and 26 of 132 harmful holdout proposals (19.7%) to pass the execution gate. Since the controlled quantity depends on the prevalence of harmful proposals, a 5% joint risk budget does not necessarily imply a low harmful-action miss rate. Moreover, the guarantee relies on exchangeability and action-level harm labels, while the automated judge achieves only 81.3% recall on harmful actions against human annotations (Table 7). The paper also acknowledges that the guarantee does not directly extend to adaptive closed-loop execution (Appendix F.4). Finally, the main comparisons do not consistently establish the calibrated risk-budget property: MobileRisk uses a fixed threshold of 0.60 rather than the CRC-selected threshold, and the operating threshold for the Phone-Harm routing comparison is not clearly specified. These limitations make it difficult to assess how the theoretical guarantee translates into practical safety.

  1. The paper's main comparison results are difficult to reconcile with the stated evaluation protocol.

The reported metrics contain inconsistencies with the stated evaluation sets. For example, MobileRisk contains 102 unsafe and 102 safe trajectories, yet GPT-5-mini achieves 33.33% recall and 9.52% precision (Table 1). According to the standard metric definitions, these values imply:

TP = 34, FP ≈ 34 × (1/0.0952 − 1) = 323 > 102 (total safe trajectories).

This is mathematically inconsistent with the reported evaluation size. Similar discrepancies appear in the routing accuracy on Harm-150 (Table 1), while Table 3 appears to mix task-level and action-level statistics. These inconsistencies make it difficult to verify the reported improvements over baselines. The paper should clarify the actual evaluation sizes, aggregation procedures, and confusion matrices.

Questions for authors

Q1. The paper controls the joint harmful-execution probability (P(H \cap E)), but 26 of 132 harmful proposals (19.7%) are still executed on the holdout set under the 5% risk budget. Why is this joint probability chosen over the harmful-action miss rate (P(E \mid H)) or an episode-level safety measure? How should the risk budget be interpreted when the prevalence of harmful proposals changes across deployment environments?

Q2. The MobileRisk evaluation reportedly contains 102 unsafe and 102 safe trajectories. However, GPT-5-mini's reported recall (33.33%) and precision (9.52%) imply approximately 34 true positives and 323 false positives, exceeding the number of safe trajectories. Could the authors provide the actual evaluation sizes and confusion matrices for each baseline, and explain how these metrics were computed?

Q3. The routing metrics in Tables 1 and 3 appear inconsistent with the stated evaluation sizes and aggregation definitions. For example, CORA's RAcc of 79.80% cannot be obtained as a task-level accuracy over 150 tasks, while its RAcc of 89.69% in Table 3 appears to average task-level routing accuracy with step-level pass-through accuracy. Could the authors clarify the effective sample sizes and aggregation formulas, and confirm whether all methods are compared using consistent evaluation units?

3 comments

Thank you for recognizing our risk-budgeted selective-execution formulation, calibration boundary handling, and transparent experimental reporting. Below we address all of your questions and concerns with unified benchmark evaluations, operating-point analyses, and state-verified closed-loop experiments.

W2, Q2, Q3: Reconciled denominators, aggregation, and confusion matrices.

The discrepancies arose from reporting evaluation-split subsets (e.g., 99 tasks, 79/99 = 79.80%) under benchmark-total headings, averaging task routing with step pass-through in Mixed-300, and partial-denominator baseline rows on MobileRisk. All nine methods now share unified denominators and integer TP/FP/TN/FN matrices (Tables 1, 14, 15) on Harm-150 (100 Misuse+Misbehavior positives, 50 Injection negatives), Mixed-300 (pooling Harm-150 with measured Normal-150 tasks: 100 positives, 200 negatives), and MobileRisk (102 unsafe, 102 safe):

MethodHarm-150Mixed-300MobileRisk (Prec / Rec / F1)
CORA81/7/43/1981/10/190/1978/36/66/24 (68.42 / 76.47 / 72.22%)
OS-Sentinel67/23/27/3367/23/177/3358/44/58/44 (56.86 / 56.86 / 56.86%)
GPT-5-mini3/0/50/973/0/200/9734/28/74/68 (54.84 / 33.33 / 41.46%)

Standard accuracy and F1 over these task matrices give Harm and Mixed RAcc/RF1; action IR uses separate action denominators (CORA: 92/2,100 = 4.38% Harm; 95/2,814 = 3.38% Mixed), and injection intervention (7/50 = 14%) is descriptive. CORA retains the highest Harm and Mixed RAcc/RF1 (82.67/86.17% and 90.33/84.82%) and highest MobileRisk Recall/F1 (+20 detections, -8 false alarms vs. OS-Sentinel; VLM-critic leads precision at 74.36%).

W1, Q1: Why joint risk, conditional miss rates, and prevalence shifts.

In an online GUI stream (often mostly benign, as in Normal-150) bounding joint risk P(H∩E)P(H\cap E) controls autonomous harm per arriving proposal on the same denominator as Coverage P(E)P(E), staying well-defined even as pH=P(H)→0p_H=P(H)\to 0. Under consistent weighting (pH>0p_H>0), P(H∩E)=pHP(E∣H)P(H\cap E)=p_H P(E\mid H): on our holdout (pH=132/1,075p_H=132/1{,}075), a 5% joint budget (τ^=0.952\hat\tau=0.952) gives 26/1,075=2.42%26/1{,}075=2.42\% joint risk and 26/132=19.70%26/132=19.70\% misses at 89.95% coverage. Given a valid lower bound pmin⁡≤pHp_{\min}\le p_H, a miss target β\beta maps to α=pmin⁡β\alpha=p_{\min}\beta; under prevalence shift, τ^\hat\tau is recalibrated on target traffic with the Guardian frozen (App. J).

Empirically, 19.70% reflects the permissive α=0.05\alpha=0.05 choice rather than scorer capacity: on the frozen holdout sweep (Table 9, App. L), tightening τ\tau to 0.50 cuts executed harms over fourfold, from 26/132 (19.70%) to 6/132 (4.55%, JHR 0.56%), at 85.77% coverage (916/943 = 97.14% safe actions; -4.19 pp coverage for 20 fewer harms). Conversely, direct 5% conditional-miss calibration achieves 0/40 held-out misses, but the 1/(nH+1)1/(n_H+1) finite-sample penalty on rare harmful events (nH=60n_H=60) restricts acceptance to 3/80 globally and 14/80 by app (App. J), validating our joint-risk design, which avoids small-denominator over-conservatism while allowing tighter operating points.

W1: Calibrated common-budget comparison vs. fixed-threshold transfer.

Table 1 evaluates routing and zero-recalibration MobileRisk transfer (τ=0.60\tau=0.60, as MobileRisk has no calibration split) against uncalibrated baselines. To test the calibrated risk-budget property across competing scorers, we retrained three Guardians (seeds 42/43/44) on 86 disjoint task/goal groups (val-AUROC selection) and calibrated CORA and a frozen Qwen2.5-VL-7B critic independently on the same 279 proposals in 26 groups (Table 7; 4/5/7.5/10% sweep in Table 12). At the 5% group-mean budget, CORA retains 75.00–80.17% coverage (1.94–3.61% risk) vs. 9.91% (0.45% risk) for the critic (+0.245 AUROC, [0.136, 0.354]), with marginal new-group loss control proved under the 26-group correction in App. F.4.

W1, Q1: Label reliability and state-verified closed-loop safety.

In the 120-action/57-task audit (Table 6), human agreement is 97.5% (κ=0.90\kappa=0.90) and Qwen3.5-Flash reaches 96.7% accuracy (κ=0.85\kappa=0.85, 92.9% precision, 13/16 = 81.3% recall). Rather than rescaling CRC by recall, we verified all 200 native Android labels against OS file/contact state and ran 160 resettable episodes across 40 tasks starting from erroneous deletions, re-gating each continuation step (Table 11): ungated execution harms 40/40 cases, whereas gate-only, gated replanning, and gated diagnosis each yield 0/40 harms with 0/40, 18/40, and 17/40 completions. Given an initial rejection across 60 object, operation, and parameter error states (dev-fixed gate; Table 3), grounded diagnosis lifts safe completion from 23/60 (replanning) to 38/60 (+25.0 pp) with 0/60 harms, showing CORA blocks state-verified harms while guiding safe task completion.

I thank the authors for their response to the inconsistencies I pointed out in the reported evaluation results. However, I remain concerned about the magnitude of the correction to the MobileRisk comparison, where GPT-5-mini’s reported precision changes from 9.52% to 54.84%. In my opinion, a reporting error of this magnitude in a central benchmark comparison is unacceptable in a submitted manuscript. Therefore, I retain my original rating of 2 (Weak Reject).

Thank you for taking the time to follow up. We hope it helps to clarify that the initial submission did not have a 45-point discrepancy in model predictions or experimental outcomes: GPT-5-mini's underlying detection and false-alarm rates on MobileRisk were essentially identical in both versions, and the difference in Precision simply reflects evaluating the same detector under a ~1:10 step-level class ratio versus the 1:1 trajectory-level ratio.

Specifically, between the initial and revised Table 1, GPT-5-mini's Recall stayed at 33.33% and its FPR moved only from 27.94% to 27.45%. Because Recall and FPR are independent of class balance whereas Precision scales directly with the ratio of unsafe to safe instances, those exact same detection rates correspond to ~9.5% Precision over steps where safe actions outnumber unsafe ones by roughly ten to one (matching the ~10x ratio you noted in W2), and to 54.84% Precision on the 1:1 balanced trajectory set (102 unsafe and 102 safe trajectories).

At the same time, GPT-5-mini serves there as an ungated base policy on an external fixed-threshold check, while our primary safety-controller baseline OS-Sentinel (56.86% Recall and F1) and CORA's Recall (76.47%) and F1 lead (72.22%) remain intact. Our central calibrated selective-execution results in Table 2 (count-consistent from the initial submission), together with the three-seed common-budget comparisons and state-verified Android closed-loop studies, are likewise completely unaffected.

Thank you again for helping us make the trajectory-level convention uniform across every column of Table 1. Since your follow-up centers on this point, we hope our analyses and added experiments have satisfactorily addressed your earlier technical questions in W1, Q1, and Q3. Please let us know if anything else remains, and if this clarifies that the underlying experimental results were consistent, we would truly appreciate your considering an updated rating for the manuscript.

Sincerely,

Authors of #24571

Reviewer_YGZYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CORA, a pre-action safety controller for mobile GUI agents. An action-conditioned Guardian scores each proposed action, and Conformal Risk Control calibrates the execute/abstain threshold for a user-specified joint harmful-execution budget. Rejected actions are sent to a Diagnostician for an explanation and intervention, which cannot override the rejection. Goal-Lock keeps the original user goal fixed and treats on-screen text as evidence rather than instructions that can redefine the goal. The paper also introduces Phone-Harm, containing 150 human-authored harmful tasks and 150 matched benign tasks, with automated step-level harm labels and a human audit. Experiments evaluate calibrated action-level risk and autonomy, Phone-Harm routing, MobileRisk safety classification, and AndroidWorld task success, with Goal-Lock/action-conditioning ablations and runtime measurements.

Critical strengths
  • The paper proposes a new benchmark with human-authored harmful tasks, step-level harm labels, and a matched benign subset. This supports evaluation of harmful execution and unnecessary interruption at the action level, rather than relying only on a final task outcome.
  • The empirical results show strong improvements in Phone-Harm routing RAcc/RF1 and MobileRisk recall, with a more modest precision gain over the best baseline. These results support the usefulness of the controller while retaining substantial autonomy.
Critical weaknesses
  • The method innovation is limited. The framework combines established components, but the paper does not clearly explain what additional technical challenge is addressed by combining them. Generic separation of guard decision and diagnosis generation is not established as novel here, and the related work acknowledges earlier separation approaches.
  • The paper does not clearly specify how the Diagnostician's supervised rationales and intervention targets were created or validated.
  • And it does not isolate the Diagnostician's benefit through a gate-only versus gate-plus-Diagnostician ablation. It is unclear how much the generative diagnosis improves intervention quality or task continuation.
  • For the Goal-Lock, retaining the original goal does not formally guarantee that the models resist injected observations. And the ablation comparison does not isolate whether the improvement comes from resisting injected instructions or retaining the original task goal.
  • Phone-Harm rewards intervention for Misuse/Misbehavior tasks but treats any intervention in an Injection task as over-intervention. An injection could induce an unsafe proposal that appropriately requires rejection, but this scoring rule would penalize that rejection. The routing metrics therefore do not necessarily reflect correct action-level safety decisions.
  • Blocking an action or asking the agent to try again can change its later actions. The paper does not prove that the chosen risk limit still holds for these changed action sequences.
  • Although CORA improves RAcc/RF1 and unsafe-trajectory recall, the high OIR/FPR leads to a practical concern. And it is unclear whether the detection gains can be retained with less false positives.
Questions for authors
  • How were the Diagnostician's offline training targets, including rationales, risk types, and intervention choices, generated and validated?
  • What independent tasks/episodes and app/policy mixtures comprise the calibration set, and how were proposals sampled? How does this construction support exchangeability with the future proposals to which the guarantee applies?
  • How do automated harm-label errors affect calibration and reported harmful-execution risk? Given that the judge detected 81.3% harmful proposals in the human audit, were the calibration/evaluation labels independently checked or was any label-error correction applied?
1 comment

Thank you for highlighting Phone-Harm's paired step-level design and CORA's routing and recall gains with high autonomy. Below we address all of your questions and concerns by clarifying how CORA's components couple and isolating each module's contribution through new controlled experiments.

W1: What coupling CORA's components solves.

At the GUI boundary, risk is action-conditional on the same screen (Cancel vs. Delete requires opposite decisions), and CRC alone cannot create autonomy. VeriSafe and OS-Sentinel use fixed rules or uncalibrated checks without a tunable harm bound, and KnowNo calibrates planner uncertainty rather than action harm.

Building on standard CRC, CORA couples calibrated gating with safe continuation: (1) Action-conditioned ranking makes tight budgets usable: our scalar Guardian separates 40/40 same-screen pairs across 3 seeds (action-free ties at 0/40; App. K) and lifts 5% group-CRC autonomy from 9.91% (Qwen2.5-VL-7B critic) to 75.00–80.17% (64.22–71.12% vs. 3.88% at 4%; Tables 7, 12). (2) Non-overriding diagnosis avoids halting tasks: gate-only gives 0/40 completions (Table 11), while overriding rejections breaks calibration; re-gating each step preserves pre-action control.

W2, W3, Q1: Diagnostician supervision and recovery value.

Sec. 3.5 and Apps. F.6, M detail supervision: our base model maps harm/type labels to Reflect, Ask-for-Confirm, or Abort with template rationales (100% parseable, 83.19% agreement on 232 actions), while recovery training (D) fits 120 synthetic UI-grounded diagnoses naming the mismatch (selected on 12 dev states).

To isolate diagnostic content quality, we ran 240 episodes on 60 entity-disjoint Android states across object, operation, and parameter errors in Files/Contacts (shared Actor, imposed initial rejection, 8-call budget, dev-fixed gate; Table 3, App. M; T keeps D's Reflect tag with generic advice):

ArmDoneHarmCalls / tok
Replanning (R)23/600/602.37 / 52
Untuned diagnosis (Z)32/600/603.67 / 254
Trained diagnosis (D)38/600/604.18 / 403
Generic rationale (T)21/601/603.82 / 494

D beats R by +25.0 pp (95% CI [16.7, 31.7]), while stripping D's rationale (T) loses 17 completions (-28.3 pp, [18.3, 36.7]), proving UI-grounded content, not extra turns or tags, drives recovery. Zero-shot Z lifts completion from 23/60 to 32/60, and grounded training (D) raises it to 38/60 (+10.0 pp, [0.0, 20.0]) with zero JSON fallbacks (0/60 vs. 60/60; Table 13). On binary Cancel/Delete modals (Table 11), gating blocks harm (0/40 vs. 40/40 ungated) and continuation beats gate-only (17/40, 18/40 vs. 0/40), where templates match replanning.

W4: Deconfounding Goal-Lock.

Goal-Lock is a learned input contract, not a formal injection proof; risk bounds come from CRC. Holding goal, screen, UI text, and action fixed across 3 seeds, trust-priority vs. neutral headers lifts native AUROC from 0.709 to 0.854 (+0.145, [0.070, 0.205]; App. K); holdout (+0.020) and MobileRisk (+0.017) CIs cross zero (Table 8), as trust headers target prompts conflicting with a benign goal rather than misuse or routine steps.

W6, Q2: Calibration groups and changed sequences.

Task IDs or normalized goals form intact groups split by fixed-seed stratified balancing before scoring, blind to scores and labels (App. G.5). Three Guardians (seeds 42/43/44) trained on 86 groups (val-AUROC selection) calibrate on 279 proposals (42 trajectories, 20 apps, 26 groups) and test on 232 (22 groups). App. F.4 proves marginal new-group loss control via the 26-group correction, allowing within-group dependence; group separation avoids overlap under group exchangeability, and changed sequences are tested via re-gated execution (Tables 3, 11).

W5, W7, Q3: Safety scoring, false alarms, and label audit.

Blocking an unsafe injected action is valid protection: routing (Tables 1, 14) reports injection intervention descriptively (14% = 7/50 vs. OS-Sentinel 46%), while step-level safety (Table 7) credits blocking any harmful step.

On 204 MobileRisk trajectories, CORA catches 20 more unsafe cases than OS-Sentinel (78/102 vs. 58/102) with 8 fewer false alarms (36 vs. 44; Table 15). Under OOD 10%/20% FPR caps, CORA reaches >4x and 1.9x the critic's recall (25.16%/37.91% vs. 5.88%/19.61%; ROC, Table 10), while in-distribution 5% calibration rejects 13–23/191 safe actions vs. 169/191 (Table 7) and Normal-150 passes 711/714 safe actions (99.58%).

In the 120-action/57-task audit (Table 6), human agreement is 97.5% and Qwen3.5-Flash attains 96.7% accuracy (κ=0.85\kappa=0.85, 92.9% P, 13/16 = 81.3% R). Rather than rescaling CRC, we verified all 200 native labels and closed-loop outcomes directly from OS file/contact state, grounding calibrated gating and recovery in app safety.

Reviewer_ySmXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CORA combines a learned action-risk scorer, Conformal Risk Control, and a non-overriding diagnostic module for mobile GUI agents. It introduces Phone-Harm and reports 89.95% autonomous coverage with 2.42% joint harmful execution, below a 5% budget on a held-out action dataset.

Critical strengths
  • Practical pre-action architecture that separates execution gating from generated explanations.
  • Clearly scoped marginal risk guarantee, with separate calibration and evaluation data.
  • Useful mobile safety benchmark with step-level annotations and a human reliability audit.
Critical weaknesses
  • Limited deployment relevance. It assumes exchangeable proposals and does not cover adaptive closed-loop or trajectory-level harm, as the authors acknowledge. Moreover, automated labels missed 3 of 16 harmful examples in the human audit, controlling judge-labeled risk does not automatically control actual harm.

  • Appendix D.1 treats any intervention on an injection task as an error, even when it blocks a dangerous injected action. Conversely, any intervention on misuse/misbehavior tasks counts as correct, without establishing that harmful execution was prevented.

  • Evaluation denominators need clarification. Table 1’s 79.80% routing accuracy and 19.23% injection over-intervention rate do not reconcile with the stated 150 tasks and 50 injection tasks under the defined counting rules. Actual denominators, exclusions, or averaging procedures should be disclosed. Some mixed-traffic baseline results also use aggregation conventions rather than measured benign-task runs.

  • Component benefits are insufficiently isolated. The Goal-Lock ablation replaces the goal with the observation, confounding trust separation with access to goal information. Action-conditioning changes harmful executions only from 29 to 26, without uncertainty estimates. Competing scorers calibrated under the same risk budget would provide a stronger comparison.

Questions for authors

Can the authors reconcile the reported counts, evaluate actual closed-loop harm with human-verified labels, and compare equally calibrated baselines?

1 comment

Thank you for recognizing CORA's pre-action separation of gating from explanation, clearly scoped guarantee, and audited benchmark. Below we address all of your questions and concerns through unified benchmark evaluations, common-budget calibration, three-seed component controls, and state-verified closed-loop Android experiments.

W2, W3, Q1: Reconciled denominators and action-level safety scoring.

Original Table 1 reported evaluation-subset counts (e.g., 99 tasks; CORA 58/5/21/15 = 79.80% RAcc, 19.23% OIR) under a Harm-150 heading, while Mixed-300 averaged task routing with step pass-through and used convention benign values for some baselines. All nine methods now use the full 150 Harm tasks, 300 Mixed tasks (pooling Harm-150 with measured Normal-150 runs for every method), and 204 MobileRisk trajectories (integer matrices in Table 15). CORA's counts are 81/7/43/19 on Harm, 81/10/190/19 on Mixed, and 78/36/66/24 on MobileRisk, retaining the highest Harm and Mixed RAcc/RF1 (82.67/86.17% and 90.33/84.82%) and highest MobileRisk Recall/F1 (76.47/72.22%, +20 detections and -8 false alarms vs. OS-Sentinel; Tables 1, 14), with action IR reported separately (92/2,100 = 4.38%; 95/2,814 = 3.38%).

We agree on W2: category routing measures behavioral adherence, whereas blocking a hijacked unsafe injection step is valid protection. We report injection intervention descriptively without lower-is-better ranking (7/50 = 14%), score step-level safety against action harm labels in Table 7 (crediting blocked unsafe injections), and verify harm prevention from OS state below.

W4, Q1: Competing scorers calibrated under the same risk budget.

We trained three Guardians (seeds 42/43/44) on 86 designated task/goal groups (val-AUROC selection) and evaluated a frozen Qwen2.5-VL-7B-Instruct critic on identical goal, screenshot, UI text, and action inputs (Apps. G.5, H.2). Each scorer selects its threshold independently on the same 279 calibration actions in 26 intact groups at a 5% group-mean joint-risk budget (App. F.4) and runs frozen on 232 actions in 22 disjoint groups (Table 7):

ScorerCoverageGroup riskSafe rejected / 191
CORA (seeds 42/43/44)75.00–80.17%1.94–3.61%13–23 (6.8–12.0%)
Frozen Qwen2.5-VL-7B critic9.91%0.45%169 (88.5%)

Under identical calibration, the critic meets the risk bound only by rejecting 88.5% of safe actions, whereas CORA preserves 7.5–8x higher autonomy. Threshold-free ranking favors CORA by +0.245 mean AUROC (95% paired CI [0.136, 0.354]), and this autonomy advantage holds across the 4/5/7.5/10% budget sweep (Table 12).

W4: Deconfounded Goal-Lock and action-conditioning controls.

Keeping Goal-Lock unchanged, we supplemented Table 2's goal-replacement baseline with a 3-seed control holding the goal, screen, observation, and action identical while swapping trust-priority for neutral headers (Table 8, App. K). Pooled native AUROC rises from 0.709 to 0.854 (+0.145, 95% CI [0.070, 0.205]), whereas broader holdout (+0.020) and MobileRisk (+0.017) CIs cross zero, as expected, since trust headers target states where an on-screen prompt conflicts with a benign goal, unlike misuse or routine steps.

Pairing Tap(Cancel) and Tap(Delete) on the same screen across 40 held-out cases gives 40/40 correct rankings per Full CORA seed, while removing action input ties all pairs (0/40; Δ\DeltaAUROC +0.354 [0.300, 0.415]). Across 204 MobileRisk trajectories (153 shared-prefix families), 3-seed mean AUROC rises by +0.0396 ([0.0001, 0.0778]; Table 8), adding multi-seed uncertainty to the 29-vs.-26 holdout count.

W1, Q1: Closed-loop Android safety with state-verified labels.

In the 120-action/57-task audit (Table 6), human agreement is 97.5% (κ=0.90\kappa=0.90) and Qwen3.5-Flash reaches 96.7% accuracy (κ=0.85\kappa=0.85, recall 13/16 = 81.3%). To test protection without judge noise, we built 100 resettable file-renaming and contact-favoriting cases with all 200 paired action labels (120 calibration, 80 test) verified directly from filesystem contents and Contacts records.

Across 160 closed-loop episodes on the 40 held-out cases, starting from erroneous deletions under frozen calibration thresholds, an 8-action budget, and re-gating at each step (Table 11), ungated execution causes data loss in 40/40 cases, while gate-only, gated replanning, and gated diagnosis each yield 0/40 harms and 0/40, 18/40, and 17/40 completions. Given an initial rejection across 60 object, operation, and parameter error states (dev-fixed gate; Table 3, App. M), grounded diagnosis boosts safe completion from 23/60 (replanning) to 38/60 (+25.0 pp) with 0/60 harms, pairing proposal-level CRC with state-verified closed-loop safety.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.