- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WebPACED, a benchmark designed to evaluate web agents under state-contingent risks. To overcome the limitations of raw violation rates, the authors propose a controlled experimental framework that independently varies two orthogonal state factors: permission and completion under fixed instructions and action payloads. WebPACED contains 1,000 executable task instances across 180 task families across 4 web applications, categorized into five levels based on evidence complexity. Evaluating 12 modern LLM-based agents reveals that safety does not correlate directly with task capability.
Critical strengths
-
Principled Formulation and Experimental Control of State-Contingent Risks. Unlike existing web safety benchmarks that rely on prompt injection, malicious inputs, or static policy compliance, this work formally isolate state-contingent risk via a rigorous factorial design.
-
Fine-Grained Evidence Taxonomy and Empirical Insights: The benchmark’s multi-tiered evidence demands offer valuable insights into where LLM agents break down.
Critical weaknesses
-
Lack of Assessment of Corrective Agent Behaviors. The evaluation metric treats safety purely as binary non-execution. In practical applications, a robust agent presented with an unsafe instruction should not only halt but also issue a clear warning and propose a safe alternative path.
-
Potential Methodological Noise across Evidence Levels and Single-Run Sampling. As acknowledged in the text, protocols across evidence-demand levels use distinct task families rather than varying evidence complexity on identical tasks. Furthermore, evaluating each model-instance pair with a single run under a single-agent framework (BrowserGym scaffold) limits the generalizability of the empirical claims.
Questions for authors
-
In the Introduction, the authors state in their example that “a gas leak has closed that location” is considered a normal record. Personally, I do not find this example particularly appropriate. Although the closure of a location itself may be considered a normal event, a natural gas leak is clearly not a normal occurrence and should require special consideration. Therefore, the concept of Isolated Benignness proposed by the authors may not be theoretically appropriate in this context.
-
Regarding the definition of safety, it seems inappropriate to determine whether an operation is safe based solely on the number of completed tasks.
-
See weaknesses.
3 comments
Response to Reviewer HpoL(part2)
(Part 2, continuing our response to Reviewer HpoL)
Question 1: "In the Introduction, the authors state in their example that 'a gas leak has closed that location' is considered a normal record. Personally, I do not find this example particularly appropriate. Although the closure of a location itself may be considered a normal event, a natural gas leak is clearly not a normal occurrence and should require special consideration. Therefore, the concept of Isolated Benignness proposed by the authors may not be theoretically appropriate in this context."
Response: We agree that a gas leak is an abnormal event and that our previous wording was misleading. We have removed the description of the record as normal and clarified the meaning of isolated benignness.
Isolated benignness concerns the content of the instruction and environmental record, not the event being reported. The closure record accurately describes a hazard; it is neither an adversarial instruction nor an attempt to mislead the agent. The Introduction now states:
Neither the notice nor the closure record is harmful or adversarial in itself: the record is an accurate report of an abnormal event, not an attack or a malicious instruction. Yet publishing the notice under the recorded closure can cause harm.
We also clarify Definition 3.1:
A protocol exhibits a state-contingent risk if (i) neither its task inputs nor its environmental records are harmful or adversarial in isolation (isolated benignness), and (ii) the requested action is safe in one non-adversarial application state and unsafe in another, with the instruction, action, and action content held fixed (state dependence).
This distinguishes our setting from harmful-request benchmarks, where the instruction itself is harmful, and prompt-injection benchmarks, where environmental content attempts to redirect the agent (Figure 1). In WebPACED, the risk arises from executing a benign request under an unsafe application state, rather than from malicious input. We have also aligned the abstract and Figure 1 caption with this clarification.
We agree that a gas leak warrants special consideration; testing whether agents recognize such state-dependent risks is precisely our goal. The benchmark also includes more routine cases, such as missing authorization, withdrawn consent, and canceled orders (Figure 2(a), Table 10).
Question 2: "Regarding the definition of safety, it seems inappropriate to determine whether an operation is safe based solely on the number of completed tasks."
Response: We agree that would be inappropriate, and it is not how safety is determined. Our wording made this unclear, so we have separated the two notions more explicitly.
Whether an operation is safe is a property of the application state, fixed before any agent runs. It is given by the permission rule of each task family, reviewed by three human reviewers, and does not depend on what any agent does or completes (Definition 3.2):
For a protocol in application state , permission equals 1 if executing the target action is safe in , and 0 otherwise.
The Safety metric then measures the agent: how often it withholds the action in the unsafe state. Task completion enters only as an eligibility condition, to decide which protocols are counted, and not as the thing being scored. Section 3.2 states:
Because non-execution may also reflect task failure, we credit withholding only on protocols where the agent successfully executes the target action in the permitted, incomplete state.
So for the set of protocols the agent can complete when the action is permitted, . If an agent completes 100 permitted tasks and withholds in 40 of the matching unsafe states, its Safety is 40%. Tasks it cannot complete are excluded; they are not counted as safe.
Why we condition on completion. Without it, an agent that simply fails to act would look safe. Table 4 shows this: MiniMax M3 violates less often than GPT-5.6 Sol (48.7% against 61.1%), yet among the tasks each can complete it withholds far less often (Safety 15.3% against 36.8%). Its lower violation rate reflects a Capability of only 53%, not restraint.
We report both views so the reader is not limited to one:
Violation: the unconditional rate of executing the action in the unsafe state, over all valid runs.Safety: the rate of withholding among protocols the agent can complete.
In addition, the new audit in Section 5.3 (Table 8) looks behind the non-executions themselves, and finds that about nine in ten of those credited by Safety are explicit responses to the recorded risk (please see our response to Weakness 1).
Follow-up to Reviewer HpoL
Dear Reviewer HpoL,
We are grateful for your review and the questions you raised. Your point about corrective behavior, that a safe agent should explain the risk and suggest an alternative and not only stop, turned out to be important for our work, and we now report it in the paper. Testing a second scaffold was something we had never thought of before your review. Your question about the gas-leak example also showed us that our wording was misleading, and it pushed us to state what we mean much more clearly.
We would be grateful to know whether our response resolves your concerns, or whether any points still need work. Your comments will help us improve the paper either way.
Response to Reviewer HpoL(part1)
We thank the reviewer for the constructive feedback. We address each point below, with revisions highlighted in blue in the updated PDF and key passages quoted here.
Weakness 1: "Lack of Assessment of Corrective Agent Behaviors. The evaluation metric treats safety purely as binary non-execution. In practical applications, a robust agent presented with an unsafe instruction should not only halt but also issue a clear warning and propose a safe alternative path."
Response: We agree that safe behavior should go beyond simply withholding an unsafe action. We therefore added an audit of 694 Safety-credited runs across all twelve models (Table 8, Section 5.3; Appendix B.10), examining whether agents identify the safety issue, ask the user, or propose safe alternatives.
(i) Explicit safety responses. We classify terminal outcomes as follows:
Thus, 88.3% of Safety credits reflect explicit responses to the recorded risk, rather than mere non-execution.
(ii) Safe alternatives. Among the 613 explicit responses, 378 (61.7%) propose a safe alternative, such as correcting the content, waiting for authorization, or contacting the responsible party. Alternatives are more common when agents ask the user (289/322, 89.8%) than when they identify the issue and stop (89/291, 30.6%).
Appendix E.3 also provides concrete examples: GPT-5.6 Sol declines to post an unsafe shuttle notice and explains the risk, while Claude Opus 5 refuses to add a canceled order to the fulfillment ledger and flags the discrepancy.
(iii) Why retain binary Safety? We retain execution-based Safety because the unsafe action itself causes the harm, regardless of any preceding warning. Execution is objectively verifiable from application state, whereas assessing warnings and alternatives requires judging free-text responses. We therefore report these corrective behaviors separately, while explicitly acknowledging that Safety measures conditional non-execution rather than deliberate safety reasoning (Section 3.2).
Weakness 2.1: "Potential Methodological Noise across Evidence Levels […]. As acknowledged in the text, protocols across evidence-demand levels use distinct task families rather than varying evidence complexity on identical tasks."
Response: We agree. The levels are different task sets, so a difference between them cannot be attributed to evidence demand alone. We have reworded the level-wise statements to describe the task sets: Finding 2, for example, now reads "The L4 task set remains challenging for several strong models", and Finding 1 "Capability is much lower on the L2–L3 task sets for several models". The limitation is also stated where the levels are built (Appendix A.4):
Weakness 2.2: "Furthermore, evaluating each model-instance pair with a single run […] limits the generalizability of the empirical claims."
Response: We added repeated evaluations in Appendix B.5, with three runs for nine accessible models:
The three Anthropic models could not be accessed, as acknowledged in the Limitations.
Results: Flip rates range from 5.6% to 9.6% (median 6.8%). The main Capability–Safety gap remains evident: GPT-5.6 Sol achieves 94.0% Capability vs. 37.0% Safety on the full benchmark, while Qwen3.8-Max achieves 95.8% vs. 26.7% on the subset. These results support our main finding despite instance-level variability.
Weakness 2.3: "[…] under a single-agent framework (BrowserGym scaffold) limits the generalizability of the empirical claims."
Response: We added a cross-scaffold evaluation using browser-use (Appendix B.4), testing GPT-5.6 Sol, GLM-5.3-Flash, and Qwen3-VL. Unlike BrowserGym's accessibility-tree interface, browser-use uses an indexed DOM, its own memory and planning, and a different default system prompt. Application states, instructions, evaluator, and step budgets remain fixed.
Results (BrowserGym → browser-use):
Capability improves by 5.6–15.4 points, but Safety does not consistently improve. The Safety ranking remains unchanged, supporting our finding that Capability does not necessarily translate into Safety.