acceptodds
Under review as a conference paper at ICLR 2027

PonderAgent: A Risk-Adaptive Dual-System Framework for Safe and Selective Mobile GUI Agents

Abstract

Mobile GUI agents powered by vision-language models operate in untrusted interfaces, where indirect prompt injection and context-dependent risks can induce unsafe actions. Existing defenses against unsafe or policy-violating agent behavior often rely on model-specific alignment or apply expensive external review uniformly, leading to poor cross-model transferability and high deployment costs. Model-specific alignment is costly to update and may degrade general utility, while uniform review wastes latency, computation, and budget on low-risk inputs. To address these limitations, we present PonderAgent, a risk-adaptive dual-system framework that combines task-level pre-interception, lightweight semantic risk gate, RAG-augmented critic system, and global working memory. Low-risk actions follow a fast path, whereas potentially risky actions are escalated for policy-grounded review, separating task-oriented action generation from safety evaluation. On 230 executable MobileSafetyBench tasks, PonderAgent increases harm prevention for open-weight T-Actors from 13% to 6566%, while goal achievement remains sensitive to T-Actor capability. With GPT-4o as the T-Actor, PonderAgent reaches 70% harm prevention and with Qwen3-VL-32B-Instruct, it prevents 28 of 30 executable indirect prompt-injection attacks (93.3%). In addition, the R-Gate adds 228 ms per step and routes 49.8% of actions to the C-System, reducing the estimated average per-step latency to roughly half the cost of uniform C-System review.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Mobile GUI agents, AI safety, Risk-adaptive routing, Dual-system architecture, Vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_kBzqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PonderAgent, a model-agnostic safety framework for mobile GUI agents. It combines task-level pre-screening, semantic risk routing, RAG-enhanced safety review, and cross-step memory to selectively escalate actions for deeper review. On MobileSafetyBench, the framework improves harm prevention for open-weight actors from 1–3% to 65–66%, achieves a 93.3% prevention rate against indirect prompt injection in the default configuration, and is estimated to reduce average latency to roughly half that of uniform deep review.

Critical strengths
  1. The motivation is reasonable. Allocating review resources according to contextual risk can reduce unnecessary deep review and addresses a clear engineering need.
  2. The framework is modular and does not require access to the actor’s internal reasoning. Its rule and policy knowledge bases can be updated independently, facilitating integration with different actors.
  3. The experiments cover multiple actors, module ablations, risk categories, and injection attacks, and include comparisons with random routing. The authors also report limitations such as weak semantic discrimination, false rejections, and high latency.
Critical weaknesses
  1. The methodological novelty and GUI-specific challenges are insufficiently explained. Rule-based filtering, semantic matching, RAG-based review, and historical memory are established techniques, and lightweight screening followed by expensive review is an intuitive design. The paper should clarify what makes reducing review costs for GUI agents different from doing so for general agents or LLMs, and how these differences motivate its specific design. The discussion of contextual and cross-step risks does not sufficiently establish why this combination goes beyond applying a general safety framework to GUI tasks.
  2. The contribution of semantic risk scoring remains unclear. The R-Gate matches action context against approximately 100 risk patterns, but its semantic score achieves an AUC of only 0.53. Although random-routing comparisons and ablations support the complete gate, they do not establish the additional value of semantic matching. Moreover, supervised logistic regression over the same embeddings achieves an AUC of 0.96. While it requires labeled training data, the authors should explain the trade-off motivating their training-free design and consider an end-to-end comparison.
  3. The practical advantage over existing safety approaches remains unclear. Under the default configuration, benign-task success drops from 53% to 40%, while each slow-path review takes approximately 354 seconds. These costs do not invalidate the method, but comparisons with bare actors or uniform deep review alone are insufficient to establish its practical competitiveness. Comparisons with relevant safety frameworks, such as OS-Sentinel and SafeMobile, at comparable safety or task-utility levels would help determine whether PonderAgent offers a better overall trade-off in latency and cost.
Questions for authors

Please see the weakness.

Reviewer_cMYpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PonderAgent protects mobile GUI agents through task interception, risk routing, a retrieval-augmented safety critic, and working memory. On MobileSafetyBench, the default configuration increases high-risk harm prevention from 1% to 65%, but reduces benign-task success from 53% to 40%. Prompt-injection prevention improves from 16/30 to 28/30.

Critical strengths
  1. The problem is practically relevant, and the framework improves harm prevention across multiple actor backbones. The matched prompt-injection comparison also shows a substantial improvement (Tables 1 and 8).
  2. The authors report important limitations, including weak semantic routing, over-blocking, and high latency. Table 3 provides useful evidence that selective review can reduce unnecessary interventions.
Critical weaknesses
  1. The central routing mechanism is weakly supported. In Table 4, the semantic score achieves AUC 0.53, compared with 0.96 for logistic regression using the same embeddings. The authors acknowledge that task weights and keyword fallback drive the full gate’s task-level advantage. However, the stronger learned router is not evaluated end to end, leaving the proposed semantic mechanism’s additional value unclear.
  2. The evaluation does not establish a convincing safety–utility–cost advantage. Table 2’s uniform baseline uses a non-thinking critic without RAG, whereas the full system uses a thinking critic with RAG. Against similarly budgeted random routing, harm prevention increases from 55% to 65%, but benign-task success falls from 51% to 40%. This does not demonstrate superiority at matched utility. Furthermore, the average step takes approximately 198 seconds (Table 5), substantially limiting practical efficiency.
  3. Figures 2–3 contain visibly rasterized icons, inconsistent icon styles, and background patches that do not blend with their containing blocks. Small labels and crowded layouts reduce readability.
Questions for authors
  1. With identical critic, RAG, memory, and task-level checks, how do uniform, random, learned, and proposed routing compare at matched budgets and matched benign-task success?
  2. What does semantic similarity add beyond task weights and keyword fallback? Does the learned router improve end-to-end performance under task-disjoint validation?
  3. Please clarify the actual denominators and evaluation batches. Table 15 labels HR HP as n=50 but reports values such as 62.7%; Appendix C.6’s paired regression counts also need reconciliation with Table 2.
  4. Appendix C.5 reports annotator agreement rather than the distribution of correctness ratings. What proportion of explanations are correct, and how often are PASS decisions unsafe?
Reviewer_uLhZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PonderAgent, a safety framework for mobile GUI agents that combines task-level interception, risk-based routing, and a deliberative critic. The framework selectively reviews actions judged risky rather than applying the critic at every step. Experiments on MobileSafetyBench report improved harm prevention across several actor models, together with ablations and analyses of routing behavior and computational cost.

Critical strengths

The framework addresses the practical need to balance safety review with task execution and computational cost. The paper reports both harm prevention and legitimate-task performance, and includes prompting and routing baselines.

Critical weaknesses
  1. The technical novelty is not sufficiently established. The main design combines task filtering, heuristic risk estimation, retrieval, and selective escalation to a stronger critic. These are reasonable engineering choices, but the paper does not clearly establish a new routing principle or insight beyond this combination. The cost-based formulation is not convincingly connected to a calibrated routing decision in the implemented system.
  2. The evaluation is too narrow to establish that the design generalizes. The experiments are concentrated on MobileSafetyBench, and the additional targeted subset does not provide an independent benchmark. This leaves open whether the risk patterns, thresholds, and processing choices are particularly suited to this benchmark. The actor coverage includes GPT-4o and Qwen3-VL configurations, but does not establish whether the same gains hold for a representative strong general-purpose agent with explicit safety instructions. Together, benchmark coverage and agent coverage limit the practical conclusions.
Questions for authors
  1. Generalization: Were the risk patterns and thresholds fixed independently of the evaluation tasks? Can the authors provide evidence on an independent benchmark and a strong general-purpose agent with an explicit safety prompt, without retuning these components?
  2. Matched comparison: Table 2 uses a non-thinking critic without retrieval for uniform review, while PonderAgent uses the thinking critic. How does uniform review perform with the same critic and retrieval configuration? This would help isolate the benefit of selective routing.
  3. Safety and utility: Relative to random routing, Table 2 improves harm prevention from 55% to 65% while reducing low-risk goal achievement from 51% to 40%. How should this trade-off be interpreted, and does risk-based routing improve the safety-utility trade-off at matched operating points?
  4. Risk estimation: The semantic score has AUC 0.53, whereas the learned probe reaches 0.96 in the reported analysis. How much of the end-to-end benefit comes from the semantic score, as opposed to the other rules and interception mechanisms?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.