- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
SafeSym builds a PDDL world model from a GUI finite-state machine, checks syntax and goal reachability, matches actions against a fixed safety-rule library, and injects the resulting obligations as preconditions or auxiliary check actions. A planner then returns a plan that meets the goal only on paths satisfying the injected constraints. The plan is given to a GUI agent as decision-time guidance. On web tasks drawn from AutoWebWorld, across several backbone models, SafeSym reports higher obligation compliance than two external guard methods while keeping task success comparable and making no extra runtime guard calls. A progressive ablation attributes most of the gain to static constraint injection, with a further gain from dynamic matching.
Critical strengths
1.Safety is placed in the model, and its cost is measured. Injecting obligations as preconditions or check actions before planning is a concrete alternative to a second model vetting each chosen action. Table 2 makes the benefit operational: zero external guard calls and roughly half the online model calls per task of the two guard baselines. This matters because runtime guard cost is a real barrier to deploying agent safety mechanisms, and the paper reports it directly.
- The ablation and the uncertainty reporting are informative and honest. Unconstrained planning barely changes obligation metrics, static constraint injection produces the large jump, and dynamic instantiation adds a further gain on the obligation types that need parameter binding. The appendix reports paired bootstrap intervals, including one that spans zero for obligation coverage against SafePred. Appendix F states that the guarantee is relative to the supplied symbolic model and rule library, and that it does not certify external sanitizers or approval workflows. This lets a reader see which parts of the gain are solid.
Critical weaknesses
The setting is narrow, and the safety notion is procedural. The method assumes an accurate FSM of the target site, a fixed library covering five obligation types, and tasks where the obligation check can be expressed as a typed support action. Robustness to incomplete or noisy transition models is not tested, and the paper lists this itself. The metric rewards emitting the right check in the right order and does not verify that the check had its intended effect. Together these limit how far the results transfer to open-web GUI agents.
Questions for authors
On episodes where the planner returns a constraint-satisfying plan, how often does the low-level agent omit or reorder a required check, and is that the main source of the 57.9% OSR?
2 comments
Follow-up on Rebuttal Discussion
Dear Reviewer YLr6,
Thank you again for your thoughtful feedback and positive assessment. We have carefully addressed the concerns raised in your review and provided additional analysis of execution trajectories to clarify the omission and ordering of safety checks in our rebuttal. We have also uploaded an updated manuscript PDF incorporating these revisions.
As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our responses and share any remaining concerns or feedback. We would be happy to provide further clarification if needed. If our responses have satisfactorily addressed your concerns, we would also appreciate your consideration of an updated assessment.
Thank you for your valuable time and consideration.
Response to Reviewer YLr6
Thank you for recognizing SafeSym's runtime efficiency, informative ablations, and transparent uncertainty and scope reporting. We address the two aspects of the stated weakness and then answer the execution-gap question with an additional log analysis.
W1: Symbolic-model and rule scope
SafeSym targets known or model-accessible GUIs, with guarantees defined relative to the supplied symbolic model and encoded policy. PDDL represents the combinatorial state space implicitly through predicates and action schemas rather than enumerating every GUI state. Workflow specifications, application instrumentation, developer interfaces, or upstream exploration can provide the model; when transitions are updated, SafeSym reruns obligation grounding, compilation, validation, and planning.
AutoWebWorld's executable sites and reference FSMs let us isolate this policy-to-model transformation across 24 websites, 345 multi-step tasks, five application domains, and four actor backbones. Within the supplied model, every constrained plan must establish the required obligation predicates before governed actions. The five obligation types are reusable safety-procedure interfaces rather than a closed list of risks: new policies can add triggers, state predicates, templates, or typed procedures while retaining the same planner. We will make this applicability and update path more explicit in the revision.
W2: Procedural compliance and external effects
The current evaluation measures typed-obligation orchestration during GUI execution; deployment-specific handler correctness is enforced at a separate interface. In the symbolic model, a
checkis a stateful support action: its predicate is initially false, the governed action requires it, the support action establishes it, and the governed action consumes or resets it. OSR (Succ.) and OCov (Succ.) are computed from recorded actor trajectories, measuring required-type coverage, prescribed ordering, and invalid types rather than copying compliance from the plan.A deployment can bind the same typed interface to an authorization service, validator, sanitizer, content moderator, or human-approval workflow, and establish the predicate only after that procedure reports success. SafeSym therefore determines which procedure is required, binds it to the governed action, and enforces its planning-time lifecycle; the deployment handler verifies its external effect. We will state this division directly when defining the guarantee and metrics.
Q1: What explains the 57.9% OSR (Succ.)?
The 57.9% OSR (Succ.) is explained by actors omitting required check types, not by additional order-only failures. Across 1,370 completed SafeSym episodes with nonempty reference safe plans, 762 are successful. Every OSR (Succ.) failure among these successful episodes coincides with a missing required obligation type, with no additional order-only failures when all required types are present:
The 42.1% macro-average missing-type rate is the complement of the reported 57.9% OSR (Succ.). Across all 1,370 completed episodes, including unsuccessful executions, 845 (61.7% pooled) miss at least one required type. Only one episode (0.073%) exhibits a first-occurrence ordering inversion among observed required types, and it also misses a required type. No invalid check types occur in the recorded and successfully parsed execution trajectories. Thus, ordering contributes no additional OSR (Succ.) failures beyond those already failing coverage.
This localizes the measured gap to actor-level realization: constrained plans satisfy the compiled obligations, while GUI actors sometimes omit required check types during execution. The variation across backbones further shows that execution-level realization depends on actor capability. We will add this breakdown to the revision and explicitly distinguish planning-level constraint satisfaction from actor-level compliance.