acceptodds
Under review as a conference paper at ICLR 2027

Praxis: Policy-State Control for Lifecycle Guardrails of Web Agents

16 views

Abstract

LLM-based agents increasingly act in consequential environments—invoking tools, moving money, and modifying external state on a user's behalf—and a growing body of guardrails monitors them for safety. Most existing guardrails share a common abstraction: the guard acts as a judge that maps agent behavior to a safety verdict. We argue that judgment is necessary but insufficient for effective guarding. First, trajectory-conditioned judgment alone does not explicitly maintain policy obligations across decision points, making long-horizon enforcement difficult. Second, a verdict offers limited support for task continuation: it may identify unsafe behavior without providing the actionable guidance needed for the target agent to recover and proceed safely. Consequently, correct safety judgments do not necessarily translate into successful task completion. In this work, we recast agent guarding from safety judgment to compliance control and instantiate this formulation as Praxis. Praxis compiles long-form natural-language policies into executable rule automata, maintains their policy state throughout the trajectory, and selects the least disruptive admissible intervention to preserve task progress while enforcing compliance. The LLM is confined to semantic interpretation and evidence grounding, while lifecycle transitions and intervention selection are performed deterministically. This separation turns policy semantics into persistent control state rather than information that must be reconstructed at every decision point, and naturally supports prevention, cross-step obligations, and post-action remediation within a unified framework. We further introduce PraxisBench, comprising 4,200 decision points across six domains derived from benchmark trajectories and long-form policies, to support training and evaluation of guardrails beyond binary safety judgment. Experiments show that Praxis achieves the highest completion-under-policy and lowest over-blocking on PraxisBench while maintaining strong violation-detection performance on external guardrail benchmarks. These results suggest that agent guardrails can move beyond passive safety judges toward policy-grounded controllers that preserve both compliance and task progress. Code and data will be released upon publication

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent Safety, Policy Enforcement, Long-Horizon Agent Control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QAgCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper Praxis compiles natural-language policies into stateful automata and uses them to guide interventions during agent execution. The paper also introduces PraxisBench and reports improvements in policy-compliant task completion across several benchmarks. They also introduce PraxisBench, containing 4,200 decision points across six domains

Critical strengths
  1. The motivation is convincing: detecting violations alone is not enough to help agents continue safely.
  2. Persistent obligation tracking and support for both prevention and recovery form a useful framework.
  3. The reported results are promising, and the ablations help explain the contributions of individual components.
Critical weaknesses
  1. Some accuracy, recall, and FPR values in Table 1 are difficult to reconcile with the definitions in Appendix B. Please clarify the aggregation procedure and whether methods use the same evaluation samples.
  2. A relevant missing work is Wen et al., Learning Efficient Guardrails for Compliance ICML 2026, which introduces PolicyGuardBench and PolicyGuard-4B. Its focus differs from lifecycle control, but the overlap in policy-compliance detection warrants discussion. A comparison on compatible detection settings would help clarify Praxis’s added value.
  3. It remains unclear how concrete interventions are generated and checked when multiple rules apply. Combining intervention categories does not necessarily resolve conflicting requirements.
Questions for authors
  1. How are the ST-WebAgentBench metrics aggregated, and do all methods use the same evaluation samples?
  2. How are interventions checked for compliance when multiple rules apply?
  3. Could you compare Praxis with PolicyGuard-4B in detection accuracy and runtime cost?

I would consider raising my score if the authors clarify the evaluation issues and strengthen the experimental evidence.

Reviewer_mJF7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Praxis, a runtime guardrail that treats policy compliance as stateful execution control. Natural-language rules are compiled into prohibition/obligation automata; an LLM grounds predicates in trajectory evidence using three-valued logic, while deterministic transitions track outstanding obligations and select admissible interventions. The controller supports both pre-action prevention and post-action remediation. The paper also introduces PraxisBench, containing 4,200 annotated decision points across six domains and 2,000 atomic rules. Evaluation reports improved completion under policy and reduced over-blocking on PraxisBench, alongside results on ST-WebAgentBench and AgentDojo. Ablations investigate compilation, persistent state, retrieval, intervention structure, and lifecycle coverage.

Critical strengths
  1. Persistent obligations provide a useful control abstraction. Separating activation, commitment, satisfaction, and recovery addresses a real limitation of repeatedly reconstructing obligations from trajectory history. Preserving unresolved rules when retrieval relevance changes is particularly sensible. The state-update ablation and activation-to-commit analysis in Table 4 and Appendix B.4 directly investigate this motivation and report larger benefits for longer gaps.

  2. The benchmark evaluates more than binary safety judgments. PraxisBench includes evidence, admissible interventions, cross-step obligations, and pre-/post-action decisions, with human verification of every retained decision point and double annotation of 20% of samples. Evaluating compliant completion alongside violations and over-blocking is consequential because it distinguishes useful guarding from indiscriminate blocking.

Critical weaknesses
  1. The reported detection metrics require reconciliation before the external-benchmark comparison can be assessed. Appendix B.1 defines accuracy, recall, and FPR from confusion-matrix counts. For a common labeled evaluation set with unsafe prevalence (p), these satisfy [ \mathrm{Accuracy}=p,\mathrm{Recall}+(1-p)(1-\mathrm{FPR}). ] However, the ST-WebAgentBench values in Table 1 imply (p\approx13.5%) for Praxis, (25.2%) for StepGuard, and (71.2%) for Direct LLM Guard. These differences cannot be explained by rounding. Macro-averaging or method-dependent evaluation trajectories could explain different implied prevalences, but the aggregation and evaluation populations are not specified. This matters because Section 5.2 interprets the comparison as improved unsafe-action discrimination. For PraxisBench, the number of evaluated trajectories and uncertainty estimates are also missing, making the 69.2% versus 67.8% CuP comparison with StepGuard difficult to assess.

  2. A central part of the deterministic intervention mechanism is underspecified. Equation (7) assumes a rule-specific admissible set (V_i[s]), but Appendix A.3 mainly provides broad candidate intervention types, and Appendix A.2 does not explicitly describe how intervention eligibility is represented in the extracted specification. For example, Table 8 lists Feedback, Transform, Require, and Deny for a pre-action violation, whereas the approval example in Figure 4 admits only Require and Deny. This distinction determines whether the controller requests genuine external approval or selects a cheaper but inadequate response. Similarly, selecting an intervention label does not specify how a compliant transformed action or recovery payload is generated and checked against all active rules. Appendix C.3 describes annotation-time admissibility through factual/counterfactual questions, but I could not identify an equally explicit runtime algorithm. This limits reproducibility and obscures the contribution of the proposed controller relative to unspecified intervention machinery.

Questions for authors
  1. Evaluation clarification: Are all ST-WebAgentBench methods evaluated on identical labeled decision points? Please provide the confusion matrices and aggregation procedure explaining the prevalences implied by Table 1. If evaluation uses method-dependent trajectories, how is action-discrimination performance compared on matched examples?

  2. Method clarification: Please provide an executable specification or pseudocode for constructing (V_i[s]), including the approval example’s exclusion of Feedback and Transform. How are intervention payloads generated and checked against every active rule? Do these steps require additional LLM calls or manually specified templates?

  3. Evidence clarification: How many distinct tasks and trajectories underlie the PraxisBench test results, and can you provide a paired uncertainty estimate for CuP against StepGuard? Since multiple trajectories are collected per task in Appendix C.2, does the trajectory-level split also separate task instances and policy documents, or are these shared across partitions?

Reviewer_M4f4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies stateful guardrails for web agents, aiming to improve task completion under policy constraints by selecting the cheapest admissible intervention. An LLM interprets policy rules offline and grounds factual predicates online, while deterministic state transitions and policy admissibility determine the available interventions. The paper also proposes PraxisBench, with 4,200 decision points and 2,000 atomic rules across six domains, for evaluating policy-compliant execution and intervention decisions. Experiments are conducted on PraxisBench, ST-WebAgentBench, and AgentDojo. The reported results show higher policy-compliant completion on PraxisBench and higher accuracy and recall on ST-WebAgentBench than the compared methods, with competitive results on AgentDojo.

Critical strengths
  • This paper studies stateful guardrails while trying to improve successful task completion under policy constraints, with deterministic transitions after policy interpretation and factual grounding. This matters when an earlier policy requirement must remain active until a later action.
  • It defines a pool of intervention types and selects the least disruptive admissible intervention under its cost definition. This supports task continuation through requesting missing evidence, correcting an action, or recovering from an executed action. The accompanying benchmark evaluates intervention decisions and completion under policy, making both compliance and task progress part of the evaluation.
Critical weaknesses
  • The intervention types are predefined, and the cheapest is selected from the admissible set, but the concrete corrective actions are unclear. For examples, for Transform, which replaces the proposed action with a compliant alternative, what are the alternatives? How are they found and validated? I did not find an explicit algorithm for generating their concrete tool operations and arguments, or a stated requirement to recheck every replacement before execution. The composition rule resolves whether execution is held, undone, or escalated, but does not fully explain how incompatible concrete corrections are reconciled.
  • For stateful guarding, context length may be a limitation. The paper limits the current observation and retrieves relevant rules, but does not explain how the full trajectory history or accumulated unresolved obligations are managed at context limits. It also does not evaluate token use or latency as they grow.
  • The intervention cost is a predefined disruption score, but its calibration against actual delay, user effort, or lost task progress is unclear. Although different cost preferences are evaluated, the results do not establish the practical cost benefit of selecting the “cheapest” intervention.
  • The empirical gains over the strongest existing guarded baselines are marginal.
Questions for authors
  • Which specific obligation-tracking and intervention-control capabilities distinguish Praxis from the closest existing runtime or stateful guards?
  • Would DTap [1] be a suitable benchmark for evaluating Praxis?
  • During the end2end evaluation, how are policy violations and over-blocking scored on the resulting trajectories? Are the evaluation labels determined independently of Praxis’s predicted rules and states?

[1] Chen, Zhaorun, et al. "DecodingTrust-Agent platform (DTap): A controllable and interactive red-teaming platform for AI agents." arXiv preprint arXiv:2605.04808 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.