acceptodds
Desk rejected at ICLR 2027

ContextGuard: Context-Conditioned Neuro-Symbolic Safety Guardrail for Fine-Grained Intervention in Mobile GUI Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Mobile GUI agents increasingly perform complex tasks autonomously, making runtime safety mechanisms that account for the evolving interaction context increasingly important. Existing guardrails typically either use context as auxiliary information for model-based risk prediction or enforce predefined safety rules over fixed representations. While model-based approaches can capture contextual risks, their safety judgments may rely on opaque external models and incur additional computational cost, whereas rule-based approaches provide explicit constraints but lack a unified mechanism for grounding evolving interaction context into constraint applicability. We present , a new context-conditioned neuro-symbolic runtime safety guardrail that makes interaction context part of the safety semantics itself. ContextGuard grounds heterogeneous mobile observations into soft semantic predicates capturing interface and action semantics as well as task-dependent and history-dependent context, and incorporates these predicates directly into explicit safety constraints. A neuro-symbolic reasoning module evaluates explicit safety constraints over these predicates, allowing contextual information to directly determine which constraints apply while retaining inspectable constraint-level reasoning. The resulting decisions are mapped to three runtime interventions: , , and . On a diverse evaluation set of 592 task trajectories spanning 10 real mobile applications and 10 safety risk categories under the AndroidLab evaluation setting, ContextGuard achieves 90.9% Binary Accuracy, 4.0% False Positive Rate, and 87.8% Intervention Accuracy, achieving the highest fine-grained Intervention Accuracy among the evaluated baselines. Ablation results show that incorporating task instructions and execution history improves Intervention Accuracy from 56.6% to 87.8%, while adding execution history introduces only 0.11 ms/task of additional guardrail overhead. These results demonstrate that ContextGuard integrates contextual semantics directly into safety constraints, combines flexible semantic grounding with interpretable neuro-symbolic reasoning, and maintains practical guardrail overhead for mobile-agent safety intervention. The source code is available at https://anonymous.4open.science/r/ContextGuard/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent Safety, Mobile Agents, Runtime Verification, Neuro-Symbolic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KERjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ContextGuard, a context-conditioned neuro-symbolic safety guardrail for mobile GUI agents. It grounds GUI observations, task instructions, proposed actions, and interaction history into soft semantic predicates, which are evaluated against predefined safety constraints to produce fine-grained runtime interventions (PERMIT, CONFIRM, or STOP). Experiments report improvements in safety detection and intervention accuracy over existing guardrail methods.

Critical strengths
  1. Meaningful problem setting. Context-dependent safety intervention is an important problem for mobile GUI agents, as the safety implications of an action can vary with user intent and interaction history.
  2. Interpretable framework. Combining neural semantic grounding with explicit safety constraints provides an interpretable mechanism for fine-grained intervention decisions.
Critical weaknesses
  1. Limited evidence of generalization and advantages over model-based approaches. The proposed neuro-symbolic framework relies on predefined safety predicates and constraints, raising concerns about its scalability and generalization to open-ended real-world agent scenarios. While such rules may work well within a predefined safety policy space, it remains unclear how effectively they can handle unseen applications, tasks, or safety risks. More importantly, the paper does not sufficiently demonstrate the advantages of this rule-dependent design over model-based approaches in more diverse and realistic settings.

  2. Insufficient experimental validation and methodological clarity. The main evaluation relies on a newly constructed benchmark with manually designed or audited safety labels, which may introduce subjectivity and potentially favor the proposed rule-based framework. Furthermore, the paper does not clearly specify the architecture, training procedure, or supervision of the Neural Grounding Model, which is a critical component of the proposed method and essential for assessing its effectiveness and reproducibility.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear Reviewer KERj,

Thank you again for your thoughtful feedback. We have carefully addressed the concerns raised in your review and provided additional clarifications and experimental results in our rebuttal. We have also uploaded the latest version of the revised manuscript PDF, which incorporates these changes.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our responses and the updated manuscript and share any remaining concerns or feedback before the discussion closes. We would be happy to provide further clarification if needed.

Thank you for your support and valuable time.

Thank you for the clarification. I understand that Neuro-Symbolic, or more specifically, rule-based approaches can help improve the interpretability and controllability of mobile agent safety. However, I still believe that, given the diversity and complexity of realistic mobile agent scenarios, model-based approaches should be prioritized over rule-based ones. The paper also does not provide sufficient evidence or experimental results to substantiate this choice. Therefore, I will maintain my current score.

Thank you for sharing your further thoughts. We agree that model-based approaches play an important role in addressing the diversity and complexity of realistic mobile agent scenarios. However, we believe that explicit rule-based reasoning offers complementary capabilities that are particularly valuable for runtime safety, where safety constraints and intervention decisions need to be interpretable, controllable, and verifiable.

Importantly, ContextGuard does not aim to establish the universal superiority of rule-based approaches over model-based ones. Instead, it demonstrates how context-conditioned symbolic reasoning can provide a principled mechanism for enforcing safety constraints during mobile agent execution. By grounding UI observations, task instructions, and interaction history into semantic predicates, ContextGuard explicitly conditions safety-rule applicability on task intent and workflow state, enabling fine-grained intervention decisions that are difficult to achieve through safety prediction alone.

Our experimental results provide empirical support for this design. On AndroidLab, we compare ContextGuard against four representative state-of-the-art model-based guardrails: Safiron, AgentAuditor, OS-Sentinel, and SeerGuard. ContextGuard achieves the best binary safety recognition and fine-grained intervention accuracy among all evaluated methods, outperforming these model-based baselines. We further evaluate ContextGuard on MobileSafetyBench, where it is compared with OS-Sentinel and SeerGuard. The results demonstrate its ability to transfer to a separate benchmark, including applications not encountered during AndroidLab evaluation, without benchmark-specific tuning. Together, these results provide strong empirical evidence for the effectiveness of context-conditioned neuro-symbolic reasoning in mobile agent runtime safety, including in comparison with state-of-the-art model-based approaches.

We view explicit symbolic safety reasoning not as a replacement for model-based approaches, but as an important complementary mechanism. Combining the flexible semantic understanding of neural models with the explicit constraints and controllable intervention provided by symbolic reasoning is a promising direction, and the current work establishes a concrete foundation for this direction. Thank you again for your thoughtful feedback.

Thank you for the thorough rebuttal, the error analysis, and the per-category breakdown. While these additions make the evaluation much clearer, your clarifications confirm my main soundness concerns. First, there is a big gap between how the paper is framed and what the system actually does: the draft presents ContextGuard as a neural grounding and differentiable reasoning approach, yet the rebuttal confirms it uses purely deterministic XML parsing, 26 hand-crafted rules with fixed weights, and no learned components. Second, the headline 87.8% intervention accuracy leans heavily on a separate lexical pattern matcher that handles the ~40% content-risk subset (where accuracy is 89–99%), whereas categories requiring actual GUI context reasoning perform much worse—such as Data Loss (64.7% recall), System Availability (62.5% recall), and Financial Loss (40.0% FPR). Combined with the alignment bias of testing hand-written rules and keyword patterns on an author-constructed benchmark whose labels match those exact outputs, the core methodological issues remain, so I will keep my current rating.

Reviewer_rpe5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ContextGuard, a runtime safety guardrail for mobile GUI agents. It maps interface observations, proposed actions, task instructions, and interaction history into soft semantic predicates, which are evaluated against explicit safety constraints to produce PERMIT, CONFIRM, or STOP decisions. The paper reports favorable safety detection and intervention performance on 592 AndroidLab trajectories, with additional binary safety evaluation on MobileSafetyBench. While the intervention setting is practically relevant and the reported results are promising, the methodological novelty remains insufficiently established.

Critical strengths
  1. Useful intervention choices. The method distinguishes between actions that need user confirmation and actions that should be stopped. This is useful in practice, since a valid action with important consequences should not always be treated in the same way as an inappropriate action.
  2. Good reported results and low runtime cost. ContextGuard achieves 87.8% intervention accuracy on the main dataset, outperforming the tested baselines, with a reported processing time of 240.85 ms per task. The additional results on MobileSafetyBench also support its ability to detect safety risks on another dataset.
Critical weaknesses
  1. The key function is not clearly explained, making the novelty difficult to assess. The grounding function ϕθ\phi_\theta determines properties such as whether an action matches the task, whether the task is complete, and whether an operation is repeated. However, the paper does not clearly explain how this function works, what model it uses, or how it produces confidence scores. These details are important because this function appears to handle much of the difficult contextual reasoning. The paper also needs to explain how this process differs from previous context-aware detection methods. Using contextual predicates as conditions in safety rules alone does not clearly show a new technical contribution.
  2. The ablation study does not sufficiently evaluate the method itself. The current study shows that adding task instructions and action history improves performance, but it does not show which parts of the proposed method lead to these improvements or why its way of using context is better than previous approaches. More ablation studies are needed to support the claimed contributions. However, the method currently lacks enough detail to identify suitable component-level comparisons. The authors should first explain the method more clearly and then evaluate the contributions of its main components.
  3. Actual examples would make the method easier to understand. The paper should include examples showing the GUI state, task, proposed action, relevant history, predicate scores, triggered rules, and final decision. It would be especially helpful to show how a similar action leads to different decisions when the context changes. These examples could be placed in the appendix.
Questions for authors

Please see the weakness.

3 comments
AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear Reviewer rpe5,

Thank you again for your thoughtful feedback. We have carefully addressed the concerns raised in your review and provided additional clarifications and experimental results in our rebuttal. We have also uploaded the latest version of the revised manuscript PDF, which incorporates these changes.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our responses and the updated manuscript and share any remaining concerns or feedback before the discussion closes. We would be happy to provide further clarification if needed.

Thank you for your support and valuable time.

Reviewer_rpe5OpenReview ↗

Official Review by Reviewer

Thank you for the rebutal. The ablation and worked example only show that interaction history helps within the proposed framework, but do not establish the necessity of the symbolic grounding design itself—the authors should compare against the simpler, functionally equivalent baseline of an LLM reasoning directly over interaction history, otherwise the unique contribution of the proposed mechanism remains unjustified. Therefore, I will maintain my current score.

Thank you for this important suggestion. We agree that a direct comparison with LLM reasoning over interaction history is needed to further examine the contribution of ContextGuard's symbolic grounding design.

To address this concern, we introduce a direct LLM baseline using gpt-oss-20b, which predicts intervention decisions from the task instruction, raw UI observation, proposed action, and preceding interaction history, without using ContextGuard's semantic predicates or safety rules. The baseline is evaluated on the same 592 fixed AndroidLab trajectories, using the same task-level intervention labels and prediction aggregation procedure as ContextGuard. Thus, both approaches have access to the same contextual information but differ in how it is represented and used for intervention decisions.

Comparison with Direct LLM Reasoning.

MethodBinary Acc.FPRRecallFNRF1Intervention Acc.
Direct LLM + History59.6%30.7%56.3%43.7%67.6%29.1%
ContextGuard90.9%4.0%89.1%10.9%93.6%87.8%

The results show that ContextGuard substantially outperforms direct LLM reasoning across all reported metrics. In particular, its Intervention Accuracy reaches 87.8%, compared with only 29.1% for the direct LLM baseline. This demonstrates that providing interaction history and other contextual information to an LLM alone is insufficient to achieve comparable fine-grained intervention performance. Explicit semantic grounding and symbolic rule composition provide a structured mechanism for interpreting GUI and workflow context and determining the applicability of safety constraints.

We have added the experimental setup and results to Appendix D. Thank you again for your thoughtful feedback.

Reviewer_cBfCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ContextGuard, a runtime guardrail for mobile GUI agents. At each step, the observation (UI state, task instruction, proposed action, history) is mapped by a neural grounding function ϕθ\phi_\theta to soft predicates. These cover interface/action semantics (e.g., submit, financial, external) and context (e.g., task_aligned, task_overreach, commitment_boundary, task_completed, repeated). A hand-specified knowledge base of conjunctive rules p1∧⋯∧pm⇒dkp_1 \wedge \dots \wedge p_m \Rightarrow d_k is evaluated with a t-norm. Rules whose activation exceeds a threshold δ\delta fire, and the output is chosen by the precedence STOP > CONFIRM > PERMIT. The method is evaluated on 592 author-constructed tasks derived from AndroidLab, covering 10 apps and 10 risk categories, using frozen agent trajectories and task-level labels (PERMIT / CONFIRM / STOP / CONFIRM+STOP). It reports 90.9% binary accuracy, 4.0% FPR, 87.8% intervention accuracy, and 240.85 ms/task overhead, outperforming five adapted baselines. An ablation shows that adding task and history context raises intervention accuracy from 56.6% to 87.8%. A binary-only external evaluation on MobileSafetyBench (200 tasks) is also included.

Critical strengths

The problem setup is reasonable. The paper builds its idea based on a fact that the same action can be fine in one context and risky in another. It also goes beyond a simple safe/unsafe label and decides between PERMIT, CONFIRM, and STOP. The ablation (Table 2) supports this: adding task and history context raises intervention accuracy from 56.6% to 87.8%. Binary accuracy barely changes. So context mainly helps the guardrail pick the right kind of intervention.

Critical weaknesses

The evaluation setup may favor the proposed method. The authors built the tasks, wrote the labels, and wrote the rules. The four label types match the rule outputs exactly. The baselines, on the other hand, need “output adapters” to fit these labels, and the adapters are not described. Even without any context, ContextGuard gets 56.6% intervention accuracy, which is close to the best baseline (64.9%). This suggests part of the gain comes from the rules matching the benchmark, not from context modeling. It is also unclear whether the rules were changed for MobileSafetyBench.

The metrics and analysis are not enough to support the claims. About 75% of tasks are unsafe, so binary accuracy is misleading. AgentSpec gets 75.5% accuracy but has a 95.3% FPR. It flags almost everything and is barely better than always answering “unsafe.” For ContextGuard, the numbers imply it misses about 48 unsafe tasks (recall ≈ 89%). For a safety tool, missed risks matter more than false alarms, but the paper focuses on FPR. The paper should provide: per-category results; an error analysis; example audit traces, even though interpretability is a main claim; hardware details for the timing results.

Questions for authors

The key neural part is not described. The paper does not explain what ϕθ\phi_\theta actually is. We don’t know: Other details are also missing: which t-norm is used, the value of δ\delta, how many rules are in KK, and who wrote them. The paper calls the reasoner “differentiable,” but nothing seems to be trained through it. Without these details, readers cannot judge or reproduce the results. its architecture or size; whether it is trained, and on what data; how it produces confidence scores; how it decides predicates like task_aligned from free-text instructions without an LLM.

It is unclear how content-level risks are detected. About 40% of the benchmark (218 tasks) is about harmful content: illegal activities, ethical risks, misinformation, and harmful generation. Most of these are Email tasks labeled STOP. Detecting them requires judging what a message says. None of the listed predicates (submit, delete, financial, credential, etc.) covers message content. The paper does not explain how ContextGuard handles this large part of the benchmark.

2 comments
AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear Reviewer cBfC,

Thank you again for your thoughtful feedback. We have carefully addressed the concerns raised in your review and provided additional clarifications and experimental results in our rebuttal. We have also uploaded the latest version of the revised manuscript PDF, which incorporates these changes.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our responses and the updated manuscript and share any remaining concerns or feedback before the discussion closes. We would be happy to provide further clarification if needed.

Thank you for your support and valuable time.

AuthorsOpenReview ↗

Response to Reviewer cBfC (Follow-up Comment Posted in Reviewer KERj's Thread)

Note: The follow-up comment was posted in Reviewer KERj's thread. This response addresses the concerns raised in Reviewer cBfC's review.

Thank you for your further assessment and for acknowledging the additional evaluation and error analysis. We would like to clarify several aspects of ContextGuard's methodology and evaluation.

1. Grounding and reasoning framework.

We believe the distinction between the framework design and its current experimental instantiation is important here. The characterization of ContextGuard as merely deterministic XML parsing overlooks the semantic grounding process and its role in the overall reasoning framework. ContextGuard does not simply extract UI elements or match keywords in XML. Instead, it integrates information from the UI hierarchy, task instructions, proposed actions, and interaction history to establish semantic predicates that represent the current interaction context. These predicates explicitly condition the applicability of symbolic safety rules and determine fine-grained intervention decisions. This design connects semantic grounding with symbolic safety reasoning, allowing grounded predicates to condition the applicability of safety rules.

In the current experimental implementation, this grounding process is realized deterministically using Android UI XML and parsed interaction information, rather than a separately trained neural grounding model. This is an implementation choice that allows us to focus the evaluation on context-conditioned symbolic safety reasoning, rather than jointly evaluating semantic perception and safety reasoning. Similarly, the safety rules are manually specified with fixed weights. The differentiable formulation characterizes the mathematical properties of the reasoning operators, rather than implying that rule weights are learned or optimized in the reported experiments. Therefore, the absence of learned components in the current implementation does not make the framework equivalent to XML parsing or invalidate its neuro-symbolic formulation. The contribution lies in explicitly connecting grounded interaction semantics with context-conditioned symbolic safety constraints to support fine-grained runtime intervention.

2. Content-level risk recognition and context-conditioned reasoning.

We would like to clarify that the lexical pattern matcher is only one component of content-level risk recognition, not an independent intervention mechanism. Content-related risks are also assessed within ContextGuard's overall context-conditioned reasoning framework. The presence of risk-related content alone does not determine the final intervention; the decision considers the relevant task, UI, and interaction context together with the applicable safety rules. Therefore, the performance on content-risk categories should not be attributed solely to lexical pattern matching, nor should content-level risk recognition be treated as separate from contextual reasoning.

We acknowledge the performance variations across risk categories, particularly the lower Recall on Data Loss (64.7%) and System Availability (62.5%), and the higher FPR on Financial Loss (40.0%). These results highlight specific scenarios where the current grounding and rule configuration could be further improved. Nevertheless, ContextGuard achieves 90.9% Binary Accuracy, 93.6% F1, 4.0% FPR, and 87.8% Intervention Accuracy across the full evaluation set, demonstrating strong overall performance across the evaluated risk categories.

3. Benchmark construction and evaluation.

The ground-truth intervention requirements are manually designed or reviewed based on task semantics and the expected consequences of agent actions, rather than generated from ContextGuard's predictions. Moreover, all evaluated methods are tested on the same frozen trajectories and assessed against the same annotations, ensuring a consistent basis for comparison.

Beyond AndroidLab, we also evaluate ContextGuard on MobileSafetyBench, a separate benchmark containing 200 tasks across 20 risk categories, including applications not encountered in AndroidLab. The ContextGuard method, safety-rule configuration, and other experimental settings remain unchanged, without benchmark-specific tuning. This additional evaluation provides evidence of the framework's ability to transfer to a different benchmark and application settings.

Overall, ContextGuard provides an explicit mechanism for grounding mobile interaction context and applying symbolic safety constraints to produce fine-grained runtime interventions. Thank you again for your thoughtful feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time, thoughtful feedback, and constructive suggestions. We appreciate the opportunity to clarify our work and address the concerns raised during the discussion. Below, we summarize the paper and the main points addressed in our rebuttal, along with the corresponding manuscript revisions.

Paper Summary

ContextGuard is a context-conditioned neuro-symbolic runtime safety guardrail for mobile GUI agents. It integrates UI observations, task instructions, proposed actions, and interaction history to ground the current interaction context into semantic predicates. These predicates condition symbolic safety rules, enabling the guardrail to reason about task intent, workflow progression, and potential action consequences before producing fine-grained intervention decisions. Unlike approaches that directly predict safety outcomes from model inputs, ContextGuard explicitly represents contextual information and applies structured symbolic reasoning to determine the appropriate intervention.

Rebuttal Summary and Revisions

During the discussion, we addressed the reviewers' main concerns through methodological clarifications and additional experiments:

  • Reviewer cBfC: Clarified the evaluation protocol, benchmark construction, grounding implementation, and content-level risk detection mechanism. Added per-category results, error analysis, worked examples, and runtime details.
  • Reviewer rpe5: Further clarified the contributions of semantic grounding and context-conditioned symbolic reasoning through ablation analysis and worked examples. Added a direct LLM baseline to examine the effectiveness of explicit grounding and symbolic reasoning.
  • Reviewer KERj: Clarified the framework's generalization scope, grounding implementation, and complementary advantages over model-based approaches. Further explained the existing cross-benchmark evaluation and comparisons with representative model-based guardrails.

Manuscript Revisions: Added Recall and False Negative Rate to Section 4 (Experiments); added runtime measurement configurations to Appendix B.5 (Evaluation Metrics); added Appendix B.6 (Further Analysis), Appendix D (Direct LLM Reasoning with Interaction History), and Appendix E (Case Study).

We sincerely appreciate the reviewers' constructive comments, which have helped us improve the clarity and presentation of the paper. We hope our clarifications and revisions adequately address the concerns raised.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.