- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies agent failures arising under benign user intent without a third-party attacker. It proposes a risk-model representation connecting triggers, inherent failures, and unsafe actions, and organizes risks using the relationship between reachable goal states and safe states. A three-stage construction pipeline turns these models into executable tasks in an extended AgentCanary environment. The resulting NARA-Bench contains 1,800 tasks spanning six risk models, 12 scenario scopes, 45 tools, and seven outcome categories. Eleven agent configurations are evaluated using task-specific rubrics for safety outcome, risk awareness, and utility. The results document substantial failures on the selected tasks, including cases in which agents recognize a risk but still act unsafely.
Critical strengths
-
A substantial executable resource for a consequential failure class. The benchmark moves beyond a collection of textual risk scenarios by providing tasks, environment assets, tools, and task-specific grading rubrics. Its balanced coverage of 72 risk-model–scenario combinations and evaluation of 11 agent configurations make it potentially useful for systematically testing failures under benign requests. The construction funnel and six detailed case studies help make the resource inspectable (Sections 4–6; Appendices C, H, and N).
-
Separating outcome, awareness, and utility yields useful diagnostics. The evaluation distinguishes recognizing a hazard from preventing it and from completing the legitimate task. For the highest-scoring configuration, Figure 4 reports 84.3 for awareness versus 68.2 for outcome, with 26% of executions in the recognized-failure category. This provides a concrete reason to evaluate actual tool-mediated behavior rather than rely on verbal risk recognition alone. Appendix G's human comparison is also a useful initial check on trajectory grading.
Critical weaknesses
-
The benchmark's selection procedure limits the interpretation of failure rates and rankings. Appendix H selects the 25 lowest-safety-scoring eligible tasks in each combination after execution with GLM-4.7/OpenClaw; this is a deliberately failure-enriched stress test. The construction is reasonable for discovering failure modes, but the resulting 49% Level 1/2 rate for the strongest evaluated agent is not an estimate of failure frequency under ordinary benign use. Selection sensitivity is not quantified, so the stability of rankings under a different selector or an unfiltered audited pool is unclear. This matters because the paper uses the results to support broad claims about safety in non-adversarial workflows. Appendix G's 100-trajectory validation helps, but 97% agreement concerns the combined outcome/awareness label used for selection, whereas the headline Level 1/2 rate is defined by outcome shortfalls; the latter needs its own validation.
-
The proposed unified framework conflates terminal-state safety with safe execution. Section 3.3 defines a task-completion path as safe solely when its final state belongs to the safe-state set, and Case (1) consequently rules out risk when all goal states are safe. An execution can violate a constraint before returning to a safe final state, or cause harm without achieving the goal at all. Unless states explicitly retain a persistent record of every relevant violation, the formulation misses these behaviors. This is especially relevant for an agent benchmark whose rubrics inspect full trajectories and irreversible disclosures or actions. The framework should use trajectory-level safety, or state and justify the necessary history-augmentation assumption, and explain how incomplete executions are classified.
Questions for authors
- Evaluation clarification: How do outcome-failure rates and agent rankings change on an unfiltered sample of the 2,410 audit-valid tasks, or under a different task selector? Please distinguish conclusions about stress-test vulnerability from claims about deployment prevalence.
- Grading clarification: How were the 100 human-scored trajectories sampled across agents and risk models, how many annotators were involved, and were they independent of rubric tuning? Can you report outcome-only failure-label agreement and a confusion matrix, separately from the combined selection label?
- Formal clarification: Does the state in Section 3.3 include the full relevant execution history and persistent violation flags? How does the framework classify a transient violation followed by recovery, or an unsafe execution that never reaches the goal?
5 comments
Author Response (Part 2)
W1-2&Q2: Appendix G's 100-trajectory validation helps, but 97% agreement concerns the combined outcome/awareness label used for selection, whereas the headline Level 1/2 rate is defined by outcome shortfalls; the latter needs its own validation. Grading clarification: How were the 100 human-scored trajectories sampled across agents and risk models, how many annotators were involved, and were they independent of rubric tuning? Can you report outcome-only failure-label agreement and a confusion matrix, separately from the combined selection label?
R2: Thank you for these valuable comment and question! We have validated both the safety outcome and safety awareness dimensions against human scoring. We provide the following details.
- Regarding human validation details
- Sampling: We randomly sample 25 trajectories from each of four agent configurations (Fable-5.1 + Claude Code, GPT-5.6-Sol + Codex, Kimi-K3 + OpenClaw, and DeepSeek-V4-Pro + OpenClaw), totaling 100 trajectories covering all six risk models.
- Annotation: Three annotators independently label the trajectories and then reach consensus through discussion. None participated in rubric design or tuning, and both agent identities and LLM judge outputs are hidden during annotation.
- Regarding safety outcome and awareness agreement
- Evaluation criterion: Scores below 1.0 are classified as failures, while scores equal to 1.0 are classified as safe.
- Results: As shown in Tables 2 and 3, GPT-5.6-Sol achieves 94% agreement on safety outcome and 92% on safety awareness with human consensus, respectively.
- Level 1/2 reliability: The outcome-only agreement directly supports the reliability of the combined Level 1/2 failure rate, while the awareness-only agreement provides further evidence for distinguishing between Levels 1 and 2.
- Revision: We add the human validation protocol and separate agreement results, including both confusion matrices, in Appendix G.
Table 2. Confusion matrix for safety outcome classification
| Human\LLM | LLM: outcome failure | LLM: outcome safe |
|---|---|---|
| Human: outcome failure (68) | 65 | 3 |
| Human: outcome safe (32) | 3 | 29 |
Table 3. Confusion matrix for safety awareness classification
| Human\LLM | LLM: awareness failure | LLM: awareness safe |
|---|---|---|
| Human: awareness failure (46) | 43 | 3 |
| Human: awareness safe (54) | 5 | 49 |
Author Response (Part 3)
W2&Q3: The proposed unified framework conflates terminal-state safety with safe execution. Section 3.3 defines a task-completion path as safe solely when its final state belongs to the safe-state set, and Case (1) consequently rules out risk when all goal states are safe. An execution can violate a constraint before returning to a safe final state, or cause harm without achieving the goal at all. Unless states explicitly retain a persistent record of every relevant violation, the formulation misses these behaviors. This is especially relevant for an agent benchmark whose rubrics inspect full trajectories and irreversible disclosures or actions. The framework should use trajectory-level safety, or state and justify the necessary history-augmentation assumption, and explain how incomplete executions are classified. Formal clarification: Does the state in Section 3.3 include the full relevant execution history and persistent violation flags? How does the framework classify a transient violation followed by recovery, or an unsafe execution that never reaches the goal?
R3: Thank you for these insightful comment and question! We apologize for the lack of clarity in our original presentation of the unified risk model framework. We provide the following clarifications.
- Regarding intermediate safety violations: Our framework adopts history-aware environment states to capture safety violations throughout execution.
- State definition: In our intended framework, each environment state captures both the current environment configuration and the relevant execution history, including persistent records of prior safety violations. Once a safety violation occurs, its record is retained in all subsequent states, which therefore remain outside the safe-state set. Thus, final-state safety implies safety throughout the execution.
- Revision: We clarify this definition in Section 3.3.
- Regarding harmful unsuccessful executions: Our framework classifies abstract Risk Models rather than individual execution outcomes.
- Classification criterion: Risk Models are classified based on the inherent relationship between task goals and safety constraints, specifically whether safe or unsafe task-completion paths exist, rather than whether a particular execution succeeds.
- Scope: The same Risk Model may manifest as either a successful execution with safety violations or an unsuccessful execution that causes harm. Such executions remain subject to trajectory-level safety evaluation, while their associated Risk Models are classified based on the safe reachability of the task goal.
- Revision: We clarify this distinction in Section 3.3.
- Regarding Case (1): This case represents an idealized setting which requires a highly robust environment with comprehensive safety constraints.
- Clarification: Under this idealized assumption, safety is preserved regardless of how the agent executes, including incomplete executions. Thus, no safety risk arises, and no Risk Model falls into this case.
- Revision: We clarify this assumption in Section 3.3.
Author Response (Part 1)
Thank you for your careful review and thoughtful comments! We are encouraged by your positive comments on our clear presentation, substantial executable benchmark, comprehensive evaluation, and insightful diagnostics. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.
W1-1&Q1: The benchmark's selection procedure limits the interpretation of failure rates and rankings. Appendix H selects the 25 lowest-safety-scoring eligible tasks in each combination after execution with GLM-4.7/OpenClaw; this is a deliberately failure-enriched stress test. The construction is reasonable for discovering failure modes, but the resulting 49% Level 1/2 rate for the strongest evaluated agent is not an estimate of failure frequency under ordinary benign use. Selection sensitivity is not quantified, so the stability of rankings under a different selector or an unfiltered audited pool is unclear. This matters because the paper uses the results to support broad claims about safety in non-adversarial workflows. Evaluation clarification: How do outcome-failure rates and agent rankings change on an unfiltered sample of the 2,410 audit-valid tasks, or under a different task selector? Please distinguish conclusions about stress-test vulnerability from claims about deployment prevalence.
R1: Thank you for these insightful comment and question! We would like to clarify the motivation behind our task selection, the scope of our findings, and the reliability of our results.
Table 1. Safety Performance on the 1,800 tasks and the full 2,410 audited tasks.