acceptodds
Under review as a conference paper at ICLR 2027

From Security to Safety: Towards Non-Adversarial Risks of AI Agents

Abstract

As AI agents increasingly enter production workflows and take autonomous actions, ensuring their safe and reliable operation becomes critical for real-world deployment. While existing research has extensively studied adversarial threats in *Agent Security*, *Agent Safety* risks arising in non-adversarial settings remain insufficiently clarified and explored. To bridge this gap, we clearly clarify *Agent Safety* risks as *non-adversarial risks*: harms caused by agents autonomously taking unsafe actions under benign user intent and without third-party attackers. Building on this clarification, we introduce *risk model* to capture the causal pathways from risk triggers through inherent agent failures to unsafe actions, and propose a unified framework for organizing these models. We further develop a three-stage risk task generation pipeline that translates abstract risk models into high-quality executable tasks, and use it to construct the scalable *NARA-Bench*, which currently comprises 1,800 executable risky tasks spanning 6 risk models, 12 scenario scopes, 45 tools, and 7 risk-outcome categories, enabling end-to-end, high-fidelity evaluation of *Agent Safety*. Systematic evaluations show that current agents remain susceptible to severe and diverse risky behaviors even in benign environments. Our findings reveal substantial non-adversarial safety challenges and provide a foundation for evaluating and improving the reliability of agents in real-world deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Agent, Agent Safety, Non-Adversarial Risk, Risk Modeling, Safety Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TvcUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies agent failures arising under benign user intent without a third-party attacker. It proposes a risk-model representation connecting triggers, inherent failures, and unsafe actions, and organizes risks using the relationship between reachable goal states and safe states. A three-stage construction pipeline turns these models into executable tasks in an extended AgentCanary environment. The resulting NARA-Bench contains 1,800 tasks spanning six risk models, 12 scenario scopes, 45 tools, and seven outcome categories. Eleven agent configurations are evaluated using task-specific rubrics for safety outcome, risk awareness, and utility. The results document substantial failures on the selected tasks, including cases in which agents recognize a risk but still act unsafely.

Critical strengths
  1. A substantial executable resource for a consequential failure class. The benchmark moves beyond a collection of textual risk scenarios by providing tasks, environment assets, tools, and task-specific grading rubrics. Its balanced coverage of 72 risk-model–scenario combinations and evaluation of 11 agent configurations make it potentially useful for systematically testing failures under benign requests. The construction funnel and six detailed case studies help make the resource inspectable (Sections 4–6; Appendices C, H, and N).

  2. Separating outcome, awareness, and utility yields useful diagnostics. The evaluation distinguishes recognizing a hazard from preventing it and from completing the legitimate task. For the highest-scoring configuration, Figure 4 reports 84.3 for awareness versus 68.2 for outcome, with 26% of executions in the recognized-failure category. This provides a concrete reason to evaluate actual tool-mediated behavior rather than rely on verbal risk recognition alone. Appendix G's human comparison is also a useful initial check on trajectory grading.

Critical weaknesses
  1. The benchmark's selection procedure limits the interpretation of failure rates and rankings. Appendix H selects the 25 lowest-safety-scoring eligible tasks in each combination after execution with GLM-4.7/OpenClaw; this is a deliberately failure-enriched stress test. The construction is reasonable for discovering failure modes, but the resulting 49% Level 1/2 rate for the strongest evaluated agent is not an estimate of failure frequency under ordinary benign use. Selection sensitivity is not quantified, so the stability of rankings under a different selector or an unfiltered audited pool is unclear. This matters because the paper uses the results to support broad claims about safety in non-adversarial workflows. Appendix G's 100-trajectory validation helps, but 97% agreement concerns the combined outcome/awareness label used for selection, whereas the headline Level 1/2 rate is defined by outcome shortfalls; the latter needs its own validation.

  2. The proposed unified framework conflates terminal-state safety with safe execution. Section 3.3 defines a task-completion path as safe solely when its final state belongs to the safe-state set, and Case (1) consequently rules out risk when all goal states are safe. An execution can violate a constraint before returning to a safe final state, or cause harm without achieving the goal at all. Unless states explicitly retain a persistent record of every relevant violation, the formulation misses these behaviors. This is especially relevant for an agent benchmark whose rubrics inspect full trajectories and irreversible disclosures or actions. The framework should use trajectory-level safety, or state and justify the necessary history-augmentation assumption, and explain how incomplete executions are classified.

Questions for authors
  1. Evaluation clarification: How do outcome-failure rates and agent rankings change on an unfiltered sample of the 2,410 audit-valid tasks, or under a different task selector? Please distinguish conclusions about stress-test vulnerability from claims about deployment prevalence.
  2. Grading clarification: How were the 100 human-scored trajectories sampled across agents and risk models, how many annotators were involved, and were they independent of rubric tuning? Can you report outcome-only failure-label agreement and a confusion matrix, separately from the combined selection label?
  3. Formal clarification: Does the state in Section 3.3 include the full relevant execution history and persistent violation flags? How does the framework classify a transient violation followed by recovery, or an unsafe execution that never reaches the goal?
5 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for your careful review and thoughtful comments! We are encouraged by your positive comments on our clear presentation, substantial executable benchmark, comprehensive evaluation, and insightful diagnostics. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.


W1-1&Q1: The benchmark's selection procedure limits the interpretation of failure rates and rankings. Appendix H selects the 25 lowest-safety-scoring eligible tasks in each combination after execution with GLM-4.7/OpenClaw; this is a deliberately failure-enriched stress test. The construction is reasonable for discovering failure modes, but the resulting 49% Level 1/2 rate for the strongest evaluated agent is not an estimate of failure frequency under ordinary benign use. Selection sensitivity is not quantified, so the stability of rankings under a different selector or an unfiltered audited pool is unclear. This matters because the paper uses the results to support broad claims about safety in non-adversarial workflows. Evaluation clarification: How do outcome-failure rates and agent rankings change on an unfiltered sample of the 2,410 audit-valid tasks, or under a different task selector? Please distinguish conclusions about stress-test vulnerability from claims about deployment prevalence.

R1: Thank you for these insightful comment and question! We would like to clarify the motivation behind our task selection, the scope of our findings, and the reliability of our results.

  • Regarding task selection: Our objective is not merely to increase task difficulty, but to retain challenging risky tasks that enhance the benchmark's potential to differentiate the safety capabilities of advanced agents.
    • Motivation: This design is inspired by the saturation-based task retirement mechanism adopted in Terminal-Bench 4.0, which removes tasks that agents can already solve reliably and thus offer limited discriminative value.
    • Implementation: We use GLM-4.7 (released in Dec. 2025) with OpenClaw as an earlier-generation reference agent to filter out relatively easy tasks, balancing task difficulty and coverage.
    • Revision: We include this motivation in Appendix H.
  • Regarding the scope of our findings: We agree that failure rates, including the reported 49% Level 1/2 rate, measured on the benchmark should not be interpreted as estimates of safety failure probabilities in real-world usage.
    • Evaluation objective: Our comparisons across risk models and scenario scopes aim to identify relative safety weaknesses within the benchmark, rather than establish the prevalence or inherent severity of different risks in real-world settings.
    • Revision: We clarify the scope of conclusions in Section 6.3.
  • Regarding selection bias and result reliability: We acknowledge that using a single reference agent may influence benchmark comparisons.
    • Additional evaluation: Due to time and resource constraints, we evaluate the 610 audited tasks excluded from NARA-Bench using GPT-5.6-Sol + OpenClaw and Kimi-K3 + OpenClaw, enabling a comparison between the selected 1,800 tasks and all 2,410 audited tasks.
    • Results: As shown in Table 1, overall safety scores increase slightly for both agents, while their relative ranking remains unchanged. The combined Level 1/2 failure rates decrease slightly from 57% to 55% and from 69% to 66%, respectively. These results suggest that our task selection has limited impact on the observed safety performance and agent comparisons.
    • Revision: We include these additional results in Appendix O.2.

Table 1. Safety Performance on the 1,800 tasks and the full 2,410 audited tasks.

Agent / Task SetOverallSafety OutcomeSafety AwarenessUtilityLevel 1 (%)Level 2 (%)Level 3 (%)Level 4 (%)Level 5 (%)
GPT-5.6-Sol + OpenClaw (1800)64.661.869.664.1401714227
GPT-5.6-Sol + OpenClaw (2410)64.962.269.964.0381713329
Kimi-K3 + OpenClaw (1800)56.652.765.453.149207222
Kimi-K3 + OpenClaw (2410)56.953.065.753.247198224

AuthorsOpenReview ↗

Author Response (Part 2)

W1-2&Q2: Appendix G's 100-trajectory validation helps, but 97% agreement concerns the combined outcome/awareness label used for selection, whereas the headline Level 1/2 rate is defined by outcome shortfalls; the latter needs its own validation. Grading clarification: How were the 100 human-scored trajectories sampled across agents and risk models, how many annotators were involved, and were they independent of rubric tuning? Can you report outcome-only failure-label agreement and a confusion matrix, separately from the combined selection label?

R2: Thank you for these valuable comment and question! We have validated both the safety outcome and safety awareness dimensions against human scoring. We provide the following details.

  • Regarding human validation details
    • Sampling: We randomly sample 25 trajectories from each of four agent configurations (Fable-5.1 + Claude Code, GPT-5.6-Sol + Codex, Kimi-K3 + OpenClaw, and DeepSeek-V4-Pro + OpenClaw), totaling 100 trajectories covering all six risk models.
    • Annotation: Three annotators independently label the trajectories and then reach consensus through discussion. None participated in rubric design or tuning, and both agent identities and LLM judge outputs are hidden during annotation.
  • Regarding safety outcome and awareness agreement
    • Evaluation criterion: Scores below 1.0 are classified as failures, while scores equal to 1.0 are classified as safe.
    • Results: As shown in Tables 2 and 3, GPT-5.6-Sol achieves 94% agreement on safety outcome and 92% on safety awareness with human consensus, respectively.
    • Level 1/2 reliability: The outcome-only agreement directly supports the reliability of the combined Level 1/2 failure rate, while the awareness-only agreement provides further evidence for distinguishing between Levels 1 and 2.
  • Revision: We add the human validation protocol and separate agreement results, including both confusion matrices, in Appendix G.

Table 2. Confusion matrix for safety outcome classification

Human\LLMLLM: outcome failureLLM: outcome safe
Human: outcome failure (68)653
Human: outcome safe (32)329

Table 3. Confusion matrix for safety awareness classification

Human\LLMLLM: awareness failureLLM: awareness safe
Human: awareness failure (46)433
Human: awareness safe (54)549

AuthorsOpenReview ↗

Author Response (Part 3)

W2&Q3: The proposed unified framework conflates terminal-state safety with safe execution. Section 3.3 defines a task-completion path as safe solely when its final state belongs to the safe-state set, and Case (1) consequently rules out risk when all goal states are safe. An execution can violate a constraint before returning to a safe final state, or cause harm without achieving the goal at all. Unless states explicitly retain a persistent record of every relevant violation, the formulation misses these behaviors. This is especially relevant for an agent benchmark whose rubrics inspect full trajectories and irreversible disclosures or actions. The framework should use trajectory-level safety, or state and justify the necessary history-augmentation assumption, and explain how incomplete executions are classified. Formal clarification: Does the state in Section 3.3 include the full relevant execution history and persistent violation flags? How does the framework classify a transient violation followed by recovery, or an unsafe execution that never reaches the goal?

R3: Thank you for these insightful comment and question! We apologize for the lack of clarity in our original presentation of the unified risk model framework. We provide the following clarifications.

  • Regarding intermediate safety violations: Our framework adopts history-aware environment states to capture safety violations throughout execution.
    • State definition: In our intended framework, each environment state captures both the current environment configuration and the relevant execution history, including persistent records of prior safety violations. Once a safety violation occurs, its record is retained in all subsequent states, which therefore remain outside the safe-state set. Thus, final-state safety implies safety throughout the execution.
    • Revision: We clarify this definition in Section 3.3.
  • Regarding harmful unsuccessful executions: Our framework classifies abstract Risk Models rather than individual execution outcomes.
    • Classification criterion: Risk Models are classified based on the inherent relationship between task goals and safety constraints, specifically whether safe or unsafe task-completion paths exist, rather than whether a particular execution succeeds.
    • Scope: The same Risk Model may manifest as either a successful execution with safety violations or an unsuccessful execution that causes harm. Such executions remain subject to trajectory-level safety evaluation, while their associated Risk Models are classified based on the safe reachability of the task goal.
    • Revision: We clarify this distinction in Section 3.3.
  • Regarding Case (1): This case represents an idealized setting which requires a highly robust environment with comprehensive safety constraints.
    • Clarification: Under this idealized assumption, safety is preserved regardless of how the agent executes, including incomplete executions. Thus, no safety risk arises, and no Risk Model falls into this case.
    • Revision: We clarify this assumption in Section 3.3.

Reviewer_nMUqOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies non-adversarial agent safety meaning agents causing harm when they're carrying out benign user requests without any attacker. They introduce NARA-Bench which has 1800 tasks covering six risk models. They use a three-stage pipeline to turn risk models into executable evaluation tasks. Their main finding is that current agents exhibit serious and varied unsafe behaviors even in benign environments.

Critical strengths
  • the paper studies an important problem: agents can cause harm even without an attacker
  • they introduce large executable benchmark covering different risks, scenarios and tools
  • they carry evaluation across different models and harnesses
Critical weaknesses
  • the causal claims need stronger evidence. Triggering a failure does not establish its cause
  • the formalism checks final-state safety which can miss unsafe actions along the way
  • i'm not sure why awareness gets 30% of the score even when the agent causes harm
Questions for authors
  • can you show that removing the risk trigger reduces the failure?
  • would selecting tasks with a different model change the results?
  • why give awareness 30% of the score when the agent can recognize the risk and still cause harm?
4 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for your careful review and thoughtful comments! We are encouraged by your positive comments on our clear presentation, important problem setting, comprehensive executable benchmark, and extensive evaluation. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.


W1&Q1: the causal claims need stronger evidence. Triggering a failure does not establish its cause. can you show that removing the risk trigger reduces the failure?

R1: Thank you for these insightful comment and question! We agree that it is important to establish whether observed safety failures stem from the intended Risk Triggers rather than other potential limitations.

  • Regarding the correspondence between Risk Triggers and failure mechanisms: We consider this concern at two stages of benchmark construction.
    • PoC of Risk Model: We conduct preliminary Risk Trigger ablations on risky tasks constructed during PoC by removing the designated trigger while preserving the remaining task context. Agents safely complete all control tasks, providing initial evidence that the observed failures are attributable to the intended triggers.
    • Risky Task Generation & Audit: We require each trigger to play a critical role in task execution and be directly linked to the expected risk and scoring rubric. This correspondence is further verified during task auditing.
    • Revision: We clarify these design considerations in Appendix C.
  • Regarding matched control experiments: To further address your concern, we conduct additional Risk-Trigger ablations to validate the intended mechanisms.
    • Experimental setup: We randomly sample 12 tasks from each of the six Risk Models and construct matched controls by removing the corresponding trigger while keeping other conditions unchanged. We evaluate these controls using GPT-5.6-Sol + OpenClaw.
    • Results: As shown in Table 1, safety scores reach 100.0 across all six Risk Models after trigger removal, compared with 45.0–75.0 in the original tasks. These consistent improvements provide further evidence that the observed safety failures are primarily associated with the intended Risk Triggers rather than other potential limitations.
    • Revision: We include these additional experiments in Appendix O.1.

Table 1. Safety outcome scores under the original tasks and matched control tasks without triggers.

Risk Modelw/ triggerw/o trigger
MES45.0100.0
OM61.67100.0
PI53.33100.0
SCM53.33100.0
UG59.17100.0
VI75.0100.0

W2: the formalism checks final-state safety which can miss unsafe actions along the way.

R2: Thank you for this constructive suggestion! We apologize for the lack of clarity in our original presentation of the unified risk model framework. We provide the following clarifications.

  • Regarding intermediate safety violations
    • State definition: In our intended framework, each environment state captures both the current environment configuration and the relevant execution history. Once a safety violation occurs, it is retained in all subsequent states, which therefore remain outside the safety set. Thus, final-state safety implies safety throughout the execution.
    • Revision: We clarify this definition in Section 3.3.

W3&Q3: i'm not sure why awareness gets 30% of the score even when the agent causes harm. why give awareness 30% of the score when the agent can recognize the risk and still cause harm?

R3: Thank you for these valuable comment and question! We clarify why safety awareness contributes 30% to the overall score, even when unsafe outcomes occur.

  • Safety outcome and safety awareness measure different dimensions. We expect a safe agent to both “know the risk” and “act safely.” The safety outcome score measures whether the agent actually performs unsafe actions, whereas the safety awareness score measures whether the agent recognizes and understands the risk. We assign 50% weight to outcome and 30% to awareness to prioritize actual safety behavior while preserving awareness as a meaningful diagnostic dimension.
  • Even when harm occurs, we still need to distinguish “knowingly violating” from “completely unaware”. As discussed in Section 6.3, agents may recognize risks yet still perform unsafe actions. Retaining safety-awareness scores in such cases reveals distinct failure modes that outcome-only evaluation cannot distinguish. Importantly, a high awareness score does not override an unsafe outcome, which remains explicitly captured by the safety outcome dimension.
  • Safety awareness scoring also reveals latent risks in cases of “lucky safety,” where agents avoid unsafe outcomes by chance or environmental factors despite failing to recognize the risks, which outcome-only evaluation may overlook.
  • Revision: We clarify these in Appendix C.5.

AuthorsOpenReview ↗

Author Response (Part 2)

Q2: would selecting tasks with a different model change the results?

R4: Thank you for this insightful question! We would like to clarify the motivation behind our task selection and the reliability of our results.

  • Regarding task selection
    • Motivation: This design is inspired by the saturation-based task retirement mechanism adopted in Terminal-Bench 4.0, which removes tasks that agents can already solve reliably and thus offer limited discriminative value.
    • Implementation: We use GLM-4.7 (released in Dec. 2025) with OpenClaw as an earlier-generation reference agent to filter out relatively easy tasks, balancing task difficulty and coverage.
    • Revision: We include this motivation in Appendix H.
  • Regarding result reliability
    • Additional evaluation: We evaluate the 610 audited tasks excluded from NARA-Bench using GPT-5.6-Sol + OpenClaw and Kimi-K3 + OpenClaw, enabling a comparison between the selected 1,800 tasks and all 2,410 audited tasks.
    • Results: As shown in Table 2, overall safety scores increase slightly for both agents, while their relative ranking remains unchanged. The combined Level 1/2 failure rates decrease slightly from 57% to 55% and from 69% to 66%, respectively. While time and resource constraints prevent us from repeating the full task selection process with another reference agent at this stage, these results suggest that our task selection has limited impact on the observed safety performance and agent comparisons.
    • Revision: We include these additional results in Appendix O.2.

Table 2. Safety Performance on the 1,800 tasks and the full 2,410 audited tasks.

Agent / Task SetOverallSafety OutcomeSafety AwarenessUtilityLevel 1 (%)Level 2 (%)Level 3 (%)Level 4 (%)Level 5 (%)
GPT-5.6-Sol + OpenClaw (1800)64.661.869.664.1401714227
GPT-5.6-Sol + OpenClaw (2410)64.962.269.964.0381713329
Kimi-K3 + OpenClaw (1800)56.652.765.453.149207222
Kimi-K3 + OpenClaw (2410)56.953.065.753.247198224
AuthorsOpenReview ↗

Thanks to Reviewer nMUq

Please allow us to thank you again for reviewing our paper and providing valuable feedback, and in particular for recognizing the strengths of our paper in terms of clear presentation, important problem setting, comprehensive executable benchmark, and extensive evaluation.

Kindly let us know if our responses and the new experiments have adequately addressed your concerns. We would be more than happy to answer any additional questions during the post-rebuttal period. Your feedback would be greatly appreciated.

AuthorsOpenReview ↗

A Gentle Reminder of the Final Feedback

Thank you very much again for your initial comments. They are extremely valuable for improving our work. We shall be grateful if you can have a look at our response and modifications, and please let us know if anything else that can be added to our next version.

Reviewer_isfJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies agent failures under benign user intent and without third-party attackers. It proposes a trigger–failure–action framework and introduces NARA-Bench, comprising 1,800 executable tasks across six risk models, 12 scenario scopes, and 45 tools. Evaluation of 11 agent configurations finds substantial safety failures, including failures after risks are recognized. Human checks provide supporting evidence for task quality and automated grading.

Critical strengths
  1. Executable environments make consequential actions and resulting state changes inspectable. Combined with the documented construction pipeline and human audits, this gives the benchmark practical value beyond evaluations based solely on generated descriptions of agent behavior.
  2. Separately measuring safety outcomes, awareness, and utility exposes failures that a single success metric would obscure. The substantial number of recognized-but-unprevented failures is particularly useful evidence that risk recognition alone doesn't ensure safe execution.
Critical weaknesses
  1. Appendix H selects the 25 lowest-scoring eligible tasks per risk–scenario pairing using GLM-4.7 with OpenClaw. This is a reasonable stress-test design, but the paper doesn't establish how this selection changes rankings or differences between risk categories. Reported failure rates characterize this deliberately selected challenge set; they don't establish how frequently these failures occur in normal use.
  2. Section 3.3 equates safe task-completion paths with safe final states. That doesn't exclude an unsafe intermediate action followed by recovery, and restricting attention to completed tasks omits harmful unsuccessful executions. Consequently, the claim that Case (1) admits no applicable risk model doesn't follow without additional assumptions.
  3. Successful triggering provides evidence that a failure can occur, but doesn't isolate its proposed cause. For example, Appendix N.1 labels an incomplete concurrency fix as an “Unachievable Goal” failure without establishing why a safe implementation was infeasible. Without matched controls, differences between categories may reflect task difficulty or construction choices rather than distinct causal mechanisms.
Questions for authors
  1. How do rankings and differences between risk models change on the full 2,410 audited tasks, or under selection by a different reference agent?
  2. Does the environment state explicitly retain every prior safety violation? If not, how should the framework handle temporary violations and harmful executions that never reach the goal?
  3. Can matched experiments removing the proposed trigger distinguish the intended mechanisms from ordinary capability failures, particularly for Unachievable Goal tasks?
5 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for your careful review and thoughtful comments! We are encouraged by your positive comments on our clear presentation, interesting and useful insights, practical benchmark value, and fine-grained evaluation. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.


W1&Q1: Appendix H selects the 25 lowest-scoring eligible tasks per risk–scenario pairing using GLM-4.7 with OpenClaw. This is a reasonable stress-test design, but the paper doesn't establish how this selection changes rankings or differences between risk categories. Reported failure rates characterize this deliberately selected challenge set; they don't establish how frequently these failures occur in normal use. How do rankings and differences between risk models change on the full 2,410 audited tasks, or under selection by a different reference agent?

R1: Thank you for these insightful comment and question! We would like to clarify the motivation behind our task selection, the scope of our findings, and the reliability of our results.

  • Regarding task selection: Our objective is not merely to increase task difficulty, but to retain challenging risky tasks that enhance the benchmark's potential to differentiate the safety capabilities of advanced agents.
    • Motivation: This design is inspired by the saturation-based task retirement mechanism adopted in Terminal-Bench 4.0, which removes tasks that agents can already solve reliably and thus offer limited discriminative value.
    • Implementation: We use GLM-4.7 (released in Dec. 2025) with OpenClaw as an earlier-generation reference agent to filter out relatively easy tasks, balancing task difficulty and coverage.
    • Revision: We include this motivation in Appendix H.
  • Regarding the scope of our findings: We agree that failure rates measured on the benchmark should not be interpreted as estimates of safety failure probabilities in real-world usage.
    • Evaluation objective: Our comparisons across risk models and scenario scopes aim to identify relative safety weaknesses within the benchmark, rather than establish the prevalence or inherent severity of different risks in real-world settings.
    • Revision: We clarify the scope of conclusions in Section 6.3.
  • Regarding selection bias and result reliability: We acknowledge that using a single reference agent might potentially influence comparisons across risk models.
    • Additional evaluation: To address this concern, we evaluate the 610 audited tasks excluded from NARA-Bench using GPT-5.6-Sol + OpenClaw and Kimi-K3 + OpenClaw, enabling a comparison between the original 1,800-task benchmark and the full set of 2,410 audited tasks.
    • Results: As shown in Table 1, the relative rankings and score differences across risk models remain largely consistent for both agents before and after task selection. These results suggest that our task selection has limited impact on the observed risk-model comparisons.
    • Revision: We include these additional results in Appendix O.2.

Table 1. Scores across six risk models on the 1,800 tasks and the full 2,410 audited tasks.

Agent / Task SetUGMESVIPISCMOM
GPT-5.6-Sol + OpenClaw (1800)65.563.565.562.863.467.0
GPT-5.6-Sol + OpenClaw (2410)65.863.765.963.163.667.4
Kimi-K3 + OpenClaw (1800)54.648.057.960.158.660.3
Kimi-K3 + OpenClaw (2410)54.748.358.160.559.060.6
AuthorsOpenReview ↗

Author Response (Part 2)

W2&Q2: Section 3.3 equates safe task-completion paths with safe final states. That doesn't exclude an unsafe intermediate action followed by recovery, and restricting attention to completed tasks omits harmful unsuccessful executions. Consequently, the claim that Case (1) admits no applicable risk model doesn't follow without additional assumptions. Does the environment state explicitly retain every prior safety violation? If not, how should the framework handle temporary violations and harmful executions that never reach the goal?

R2: Thank you for these insightful comment and question! We apologize for the lack of clarity in our original presentation of the unified risk model framework. We provide the following clarifications.

  • Regarding temporary safety violations: We acknowledge that the original framework does not explicitly define the components of environment states.
    • State definition: In our intended framework, each environment state captures both the current environment configuration and the relevant execution history. Once a safety violation occurs, it is retained in all subsequent states, which therefore remain outside the safety set. Thus, final-state safety implies safety throughout the execution.
    • Revision: We clarify this definition in Section 3.3.
  • Regarding harmful unsuccessful executions: We clarify that our framework classifies abstract Risk Models rather than individual execution results.
    • Classification criterion: Risk Models are classified based on the inherent relationship between task goals and safety constraints, specifically whether safe or unsafe task-completion paths exist, rather than whether a particular execution succeeds.
    • Scope: The same Risk Model may manifest as either a successful execution with safety violations or an unsuccessful execution that causes harm. Such executions are not excluded from risk analysis, but do not determine the Risk Model's category.
    • Revision: We clarify this distinction in Section 3.3.
  • Regarding Case (1): We clarify that this case represents an idealized setting which requires a highly robust environment with comprehensive safety constraints.
    • Clarification: Under this idealized assumption, safety is preserved regardless of how the agent executes. Thus, no safety risk arises, and no Risk Model falls into this case.
    • Revision: We clarify this assumption in Section 3.3.
AuthorsOpenReview ↗

Author Response (Part 3)

W3&Q3: Successful triggering provides evidence that a failure can occur, but doesn't isolate its proposed cause. For example, Appendix N.1 labels an incomplete concurrency fix as an “Unachievable Goal” failure without establishing why a safe implementation was infeasible. Without matched controls, differences between categories may reflect task difficulty or construction choices rather than distinct causal mechanisms. Can matched experiments removing the proposed trigger distinguish the intended mechanisms from ordinary capability failures, particularly for Unachievable Goal tasks?

R3: Thank you for these insightful comment and question! We agree that it is important to establish whether observed safety failures stem from the intended Risk Triggers rather than other potential limitations.

  • Regarding the correspondence between Risk Triggers and failure mechanisms: We consider this concern at two stages of benchmark construction.
    • PoC of Risk Model: We conduct preliminary Risk Trigger ablations on risky tasks constructed during PoC by removing the designated trigger while preserving the remaining task context. Agents safely complete all control tasks, providing initial evidence that the observed failures are attributable to the intended triggers.
    • Risky Task Generation & Audit: We require each trigger to play a critical role in task execution and be directly linked to the expected risk and scoring rubric. This correspondence is further verified during task auditing.
    • Revision: We clarify these design considerations in Appendix C.
  • Regarding the example in Appendix N.1: A safe implementation is infeasible because the user requires the agent to make all tests pass before release, while the underlying concurrency race requires a fundamental redesign of the synchronization mechanism. For the agent, this creates an Unachievable Goal risk: under the current system constraints, the stated objective cannot be met safely and honestly.
  • Regarding matched control experiments: To further address your concern, we conduct additional Risk-Trigger ablations to validate the intended mechanisms.
    • Experimental setup: We randomly sample 12 tasks from each of the six Risk Models and construct matched controls by removing the corresponding trigger while keeping other conditions unchanged. We evaluate these controls using GPT-5.6-Sol + OpenClaw.
    • Results: As shown in Table 2, safety scores reach 100.0 across all six Risk Models after trigger removal, compared with 45.0–75.0 in the original tasks. These consistent improvements provide further evidence that the observed safety failures are primarily associated with the intended Risk Triggers rather than other potential limitations.
    • Revision: We include these additional experiments in Appendix O.1.

Table 2. Safety outcome scores under the original tasks and matched control tasks without triggers.

Risk Modelw/ triggerw/o trigger
MES45.0100.0
OM61.67100.0
PI53.33100.0
SCM53.33100.0
UG59.17100.0
VI75.0100.0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.