- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper Praxis compiles natural-language policies into stateful automata and uses them to guide interventions during agent execution. The paper also introduces PraxisBench and reports improvements in policy-compliant task completion across several benchmarks. They also introduce PraxisBench, containing 4,200 decision points across six domains
Critical strengths
- The motivation is convincing: detecting violations alone is not enough to help agents continue safely.
- Persistent obligation tracking and support for both prevention and recovery form a useful framework.
- The reported results are promising, and the ablations help explain the contributions of individual components.
Critical weaknesses
- Some accuracy, recall, and FPR values in Table 1 are difficult to reconcile with the definitions in Appendix B. Please clarify the aggregation procedure and whether methods use the same evaluation samples.
- A relevant missing work is Wen et al., Learning Efficient Guardrails for Compliance ICML 2026, which introduces PolicyGuardBench and PolicyGuard-4B. Its focus differs from lifecycle control, but the overlap in policy-compliance detection warrants discussion. A comparison on compatible detection settings would help clarify Praxis’s added value.
- It remains unclear how concrete interventions are generated and checked when multiple rules apply. Combining intervention categories does not necessarily resolve conflicting requirements.
Questions for authors
- How are the ST-WebAgentBench metrics aggregated, and do all methods use the same evaluation samples?
- How are interventions checked for compliance when multiple rules apply?
- Could you compare Praxis with PolicyGuard-4B in detection accuracy and runtime cost?
I would consider raising my score if the authors clarify the evaluation issues and strengthen the experimental evidence.