- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on improving the safety of mobile GUI agents by controlling harmful action execution while preserving agent autonomy. Existing safety mechanisms often rely on heuristic thresholds and lack explicit guarantees on execution risks. To address this limitation, the authors propose CORA, a pre-action safety framework that formulates safeguarding as selective action execution. CORA employs an action-conditioned Guardian to estimate the risk of proposed actions and uses Conformal Risk Control (CRC) to calibrate execution thresholds under a user-specified risk budget. For rejected actions, a generative Diagnostician provides explanations and intervention suggestions without overriding the safety decision, while a Goal-Lock mechanism separates trusted user goals from untrusted interface content. The authors also introduce Phone-Harm, a benchmark containing 150 harmful tasks with step-level annotations and 150 matched benign tasks. Experiments show that CORA achieves 89.95% autonomous coverage with a 2.42% joint harmful-execution rate under a 5% risk budget on a disjoint holdout, while outperforming evaluated baselines in routing accuracy on Phone-Harm and F1 on MobileRisk.
Critical strengths
- The paper presents a well-defined risk-budgeted safety control framework.
The paper formulates GUI agent safety as selective action execution under an explicit risk budget. It separates the action-conditioned Guardian from the generative Diagnostician, ensuring that rejected actions cannot be autonomously re-enabled by generated explanations. The CRC calibration procedure is carefully specified, including the finite-sample correction, tied scores, and the hard-abstention case (Appendix F.3). The reported holdout results are also consistent with the underlying action counts (Table 2 and Appendix E). The design provides a clear execution criterion while separating risk estimation from intervention generation.
- The paper provides transparent experimental reporting and a clear discussion of its limitations.
The paper reports detailed experimental results on runtime overhead, annotation reliability, and the scope of its theoretical guarantees. It shows a relatively small average latency increase from 20.52 to 21.37 seconds per step (Table 5). The paper also validates the step-level harm judge against human annotations and reports task-clustered bootstrap confidence intervals (Appendix G.4). Furthermore, it explicitly acknowledges that the CRC guarantee is marginal, relies on exchangeability, and does not directly extend to distribution shifts or adaptive closed-loop execution.
Critical weaknesses
- The paper's risk-control guarantee does not fully capture the safety objectives of practical GUI agents.
The paper uses CRC to bound the joint harmful-execution probability (P(H \cap E)), rather than the probability of executing a harmful proposal (P(E \mid H)) or the risk of harm over a complete task. At the reported operating point, CORA still allows 22 of 76 harmful calibration proposals (28.9%) and 26 of 132 harmful holdout proposals (19.7%) to pass the execution gate. Since the controlled quantity depends on the prevalence of harmful proposals, a 5% joint risk budget does not necessarily imply a low harmful-action miss rate. Moreover, the guarantee relies on exchangeability and action-level harm labels, while the automated judge achieves only 81.3% recall on harmful actions against human annotations (Table 7). The paper also acknowledges that the guarantee does not directly extend to adaptive closed-loop execution (Appendix F.4). Finally, the main comparisons do not consistently establish the calibrated risk-budget property: MobileRisk uses a fixed threshold of 0.60 rather than the CRC-selected threshold, and the operating threshold for the Phone-Harm routing comparison is not clearly specified. These limitations make it difficult to assess how the theoretical guarantee translates into practical safety.
- The paper's main comparison results are difficult to reconcile with the stated evaluation protocol.
The reported metrics contain inconsistencies with the stated evaluation sets. For example, MobileRisk contains 102 unsafe and 102 safe trajectories, yet GPT-5-mini achieves 33.33% recall and 9.52% precision (Table 1). According to the standard metric definitions, these values imply:
TP = 34, FP ≈ 34 × (1/0.0952 − 1) = 323 > 102 (total safe trajectories).
This is mathematically inconsistent with the reported evaluation size. Similar discrepancies appear in the routing accuracy on Harm-150 (Table 1), while Table 3 appears to mix task-level and action-level statistics. These inconsistencies make it difficult to verify the reported improvements over baselines. The paper should clarify the actual evaluation sizes, aggregation procedures, and confusion matrices.
Questions for authors
Q1. The paper controls the joint harmful-execution probability (P(H \cap E)), but 26 of 132 harmful proposals (19.7%) are still executed on the holdout set under the 5% risk budget. Why is this joint probability chosen over the harmful-action miss rate (P(E \mid H)) or an episode-level safety measure? How should the risk budget be interpreted when the prevalence of harmful proposals changes across deployment environments?
Q2. The MobileRisk evaluation reportedly contains 102 unsafe and 102 safe trajectories. However, GPT-5-mini's reported recall (33.33%) and precision (9.52%) imply approximately 34 true positives and 323 false positives, exceeding the number of safe trajectories. Could the authors provide the actual evaluation sizes and confusion matrices for each baseline, and explain how these metrics were computed?
Q3. The routing metrics in Tables 1 and 3 appear inconsistent with the stated evaluation sizes and aggregation definitions. For example, CORA's RAcc of 79.80% cannot be obtained as a task-level accuracy over 150 tasks, while its RAcc of 89.69% in Table 3 appears to average task-level routing accuracy with step-level pass-through accuracy. Could the authors clarify the effective sample sizes and aggregation formulas, and confirm whether all methods are compared using consistent evaluation units?
3 comments
I thank the authors for their response to the inconsistencies I pointed out in the reported evaluation results. However, I remain concerned about the magnitude of the correction to the MobileRisk comparison, where GPT-5-mini’s reported precision changes from 9.52% to 54.84%. In my opinion, a reporting error of this magnitude in a central benchmark comparison is unacceptable in a submitted manuscript. Therefore, I retain my original rating of 2 (Weak Reject).
Thank you for taking the time to follow up. We hope it helps to clarify that the initial submission did not have a 45-point discrepancy in model predictions or experimental outcomes: GPT-5-mini's underlying detection and false-alarm rates on MobileRisk were essentially identical in both versions, and the difference in Precision simply reflects evaluating the same detector under a ~1:10 step-level class ratio versus the 1:1 trajectory-level ratio.
Specifically, between the initial and revised Table 1, GPT-5-mini's Recall stayed at 33.33% and its FPR moved only from 27.94% to 27.45%. Because Recall and FPR are independent of class balance whereas Precision scales directly with the ratio of unsafe to safe instances, those exact same detection rates correspond to ~9.5% Precision over steps where safe actions outnumber unsafe ones by roughly ten to one (matching the ~10x ratio you noted in W2), and to 54.84% Precision on the 1:1 balanced trajectory set (102 unsafe and 102 safe trajectories).
At the same time, GPT-5-mini serves there as an ungated base policy on an external fixed-threshold check, while our primary safety-controller baseline OS-Sentinel (56.86% Recall and F1) and CORA's Recall (76.47%) and F1 lead (72.22%) remain intact. Our central calibrated selective-execution results in Table 2 (count-consistent from the initial submission), together with the three-seed common-budget comparisons and state-verified Android closed-loop studies, are likewise completely unaffected.
Thank you again for helping us make the trajectory-level convention uniform across every column of Table 1. Since your follow-up centers on this point, we hope our analyses and added experiments have satisfactorily addressed your earlier technical questions in W1, Q1, and Q3. Please let us know if anything else remains, and if this clarifies that the underlying experimental results were consistent, we would truly appreciate your considering an updated rating for the manuscript.
Sincerely,
Authors of #24571
Thank you for recognizing our risk-budgeted selective-execution formulation, calibration boundary handling, and transparent experimental reporting. Below we address all of your questions and concerns with unified benchmark evaluations, operating-point analyses, and state-verified closed-loop experiments.
W2, Q2, Q3: Reconciled denominators, aggregation, and confusion matrices.
The discrepancies arose from reporting evaluation-split subsets (e.g., 99 tasks, 79/99 = 79.80%) under benchmark-total headings, averaging task routing with step pass-through in Mixed-300, and partial-denominator baseline rows on MobileRisk. All nine methods now share unified denominators and integer TP/FP/TN/FN matrices (Tables 1, 14, 15) on Harm-150 (100 Misuse+Misbehavior positives, 50 Injection negatives), Mixed-300 (pooling Harm-150 with measured Normal-150 tasks: 100 positives, 200 negatives), and MobileRisk (102 unsafe, 102 safe):
Standard accuracy and F1 over these task matrices give Harm and Mixed RAcc/RF1; action IR uses separate action denominators (CORA: 92/2,100 = 4.38% Harm; 95/2,814 = 3.38% Mixed), and injection intervention (7/50 = 14%) is descriptive. CORA retains the highest Harm and Mixed RAcc/RF1 (82.67/86.17% and 90.33/84.82%) and highest MobileRisk Recall/F1 (+20 detections, -8 false alarms vs. OS-Sentinel; VLM-critic leads precision at 74.36%).
W1, Q1: Why joint risk, conditional miss rates, and prevalence shifts.
In an online GUI stream (often mostly benign, as in Normal-150) bounding joint risk P(H∩E) controls autonomous harm per arriving proposal on the same denominator as Coverage P(E), staying well-defined even as pH=P(H)→0. Under consistent weighting (pH>0), P(H∩E)=pHP(E∣H): on our holdout (pH=132/1,075), a 5% joint budget (τ^=0.952) gives 26/1,075=2.42% joint risk and 26/132=19.70% misses at 89.95% coverage. Given a valid lower bound pmin≤pH, a miss target β maps to α=pminβ; under prevalence shift, τ^ is recalibrated on target traffic with the Guardian frozen (App. J).
Empirically, 19.70% reflects the permissive α=0.05 choice rather than scorer capacity: on the frozen holdout sweep (Table 9, App. L), tightening τ to 0.50 cuts executed harms over fourfold, from 26/132 (19.70%) to 6/132 (4.55%, JHR 0.56%), at 85.77% coverage (916/943 = 97.14% safe actions; -4.19 pp coverage for 20 fewer harms). Conversely, direct 5% conditional-miss calibration achieves 0/40 held-out misses, but the 1/(nH+1) finite-sample penalty on rare harmful events (nH=60) restricts acceptance to 3/80 globally and 14/80 by app (App. J), validating our joint-risk design, which avoids small-denominator over-conservatism while allowing tighter operating points.
W1: Calibrated common-budget comparison vs. fixed-threshold transfer.
Table 1 evaluates routing and zero-recalibration MobileRisk transfer (τ=0.60, as MobileRisk has no calibration split) against uncalibrated baselines. To test the calibrated risk-budget property across competing scorers, we retrained three Guardians (seeds 42/43/44) on 86 disjoint task/goal groups (val-AUROC selection) and calibrated CORA and a frozen
Qwen2.5-VL-7Bcritic independently on the same 279 proposals in 26 groups (Table 7; 4/5/7.5/10% sweep in Table 12). At the 5% group-mean budget, CORA retains 75.00–80.17% coverage (1.94–3.61% risk) vs. 9.91% (0.45% risk) for the critic (+0.245 AUROC, [0.136, 0.354]), with marginal new-group loss control proved under the 26-group correction in App. F.4.W1, Q1: Label reliability and state-verified closed-loop safety.
In the 120-action/57-task audit (Table 6), human agreement is 97.5% (κ=0.90) and
Qwen3.5-Flashreaches 96.7% accuracy (κ=0.85, 92.9% precision, 13/16 = 81.3% recall). Rather than rescaling CRC by recall, we verified all 200 native Android labels against OS file/contact state and ran 160 resettable episodes across 40 tasks starting from erroneous deletions, re-gating each continuation step (Table 11): ungated execution harms 40/40 cases, whereas gate-only, gated replanning, and gated diagnosis each yield 0/40 harms with 0/40, 18/40, and 17/40 completions. Given an initial rejection across 60 object, operation, and parameter error states (dev-fixed gate; Table 3), grounded diagnosis lifts safe completion from 23/60 (replanning) to 38/60 (+25.0 pp) with 0/60 harms, showing CORA blocks state-verified harms while guiding safe task completion.