acceptodds
Under review as a conference paper at ICLR 2027

Tracing Paths to Risk: Automated Safety Evaluation for Tool-used Agents via Risk-Path Exploration

Abstract

LLM agents increasingly act through external tools, extending safety risks beyond generated content to actions in the environment. Existing agent safety evaluations often rely on manually constructed or fixed test cases, which sparsely sample the execution space of each risk. However, the same risk may arise from different tool-use trajectories, environment states, and trigger conditions, leaving many feasible failure modes unexplored. To address this limitation, we propose a risk-path-driven framework for automated agent safety evaluation that treats each risk category as a family of reachable execution paths. Given an agent and a target risk, the framework constructs a compact risk-relevant graph by pruning irrelevant tools, enabling efficient search for feasible and diverse risk paths that are instantiated into executable test cases. It further refines failed cases using execution feedback, allowing test cases to adapt to the target agent’s behavior. During evaluation, a verifier determines whether the target risk has been triggered using tool-call traces and environment-state evidence, rather than relying solely on the agent's self-reports or final responses. Experiments across 349 task scenarios, 7 agent configurations, and 8 risk categories show that our framework achieves a 91.25% average risk discovery rate and uncovers diverse path-dependent safety failures that fixed test cases may miss.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Security, Agent Security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hMBXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RiskMaze, an automated safety evaluation framework for tool-based LLM agents. It constructs risk-relevant tool graphs to explore diverse risk paths, generates executable test cases, and iteratively refines them using execution feedback. The framework evaluates safety violations based on observed tool-use trajectories and environment states.

Critical strengths
  1. The paper addresses an important problem in agent safety evaluation, particularly the need to assess risks arising from tool interactions and environment state changes rather than solely from final responses.

  2. The proposed framework provides a relatively complete automated testing pipeline, covering test generation, execution-guided refinement, and risk verification.

Critical weaknesses
  1. Unclear research problem and limited methodological novelty. The paper's central motivation is to improve risk-path coverage, but it is unclear what fundamental research challenge the proposed framework addresses beyond generating diverse safety test cases. The overall method appears to be a combination of relatively standard techniques, including LLM-based tool analysis, graph-based path enumeration, test generation, and execution-guided refinement. Moreover, the risk-path search is limited to a maximum of three tool calls, raising questions about its applicability to complex, long-horizon agent workflows.

  2. Relatively outdated experimental setting and limited validation on modern agents. The experiments primarily rely on GPT-4o and GLM-9B as testing backbones, with additional analyses involving even older models. Although newer models are included as evaluation targets, the paper does not sufficiently investigate whether the proposed framework remains beneficial with stronger, more recent testing backbones. More importantly, the evaluation is based on 349 predefined task scenarios and simulated tool environments, leaving its effectiveness in realistic, complex agent settings unclear.

Questions for authors

N/A

Reviewer_Twt7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RiskMaze, an automated safety evaluation framework for tool-using agents. The authors argue that the same risk can be triggered through many different tool-use paths, while existing benchmarks typically cover only a few fixed instances. RiskMaze first prunes the tool set to those relevant to the target risk and builds a tool graph. It then searches the graph for action sequences that match risk templates and turns each path into an executable task. Each task is then iteratively refined using execution feedback from the target agent. Finally, the refined task is re-executed in a fresh simulated sandbox to judge the final result. On 349 Agent-SafetyBench scenarios, RiskMaze reports average risk discovery rates of 92.75% and 89.75% with GPT-4o and GLM-9B as the testing backbones. The paper also reports path structural diversity, pipeline reliability, a prompt-injection extension, and a human audit of the automatic verifier.

Critical strengths

Strengths

  1. Path-level formulation of risk testing. The paper treats each risk as a family of tool-use paths and makes path coverage an explicit evaluation objective. This addresses a real gap in existing agent safety benchmarks, which rely on sparse, fixed test cases. I think this is the paper’s most reusable idea.

  2. Complete and scalable pipeline. RiskMaze runs end to end, from path search and test generation to feedback-guided refinement and execution-grounded judgment. It covers 349 scenarios and 7 target models with stage acceptance rates above 98%, so it is practically usable as testing infrastructure.

Critical weaknesses

Weaknesses

W1. Missing baselines and ablations, and a limited evaluation environment.

The paper does not compare RiskMaze against the original Agent-SafetyBench test cases, other agent safety benchmarks, or automated red-teaming methods. It also does not include ablations of its core components, such as path search, tool pruning, and refinement. Since each path is refined against the target agent for up to 10 rounds, the reported RDR is closer to an attack success rate after repeated attempts. It therefore does not show that path-driven search discovers more risks than existing approaches. In addition, all experiments run in simulated tool sandboxes. Using real MCP servers (e.g., file systems, databases, email) would make the evaluation more realistic.

W2. Tool-use errors are conflated with safety failures, and stronger models are not evaluated.

Several risk categories, such as Function Misuse, Hallucination, and Ambiguous Instruction, may reflect limitations in tool use or instruction following. I believe not all such cases can be attributed to an agent willingly performing a risky action. The paper also does not break down outcomes into refusals and execution failures. Moreover, the testing backbones (GPT-4o, GLM-9B) and most target models (GPT-4o, Llama3.1-8B, Qwen2.5-7B, GLM-9B) are 2024-era models with relatively limited capability. Therefore, many cases that require multiple refinement rounds may indicate that the refiner is finding a path the agent is capable of following, rather than discovering a failure of safety alignment. The paper does not test this with newer, stronger models. This matters because without separating “cannot do” from “should not do,” RDR does not reliably reflect agent safety, and the conclusions offer limited guidance for improving models.

Questions for authors

Q1. What is the RDR at round 0 (without refinement), and how many refinement rounds are needed on average? Reporting cumulative RDR per round would clarify how much comes from path search versus repeated refinement.

Q2. Among cases where the risk is not triggered, what proportion are refusals, confirmation requests, and execution failures?

Q3. Among executions that actually trigger a risk, how many distinct tool sequences are observed, and how often do they match the intended path?

Q4. Could the authors evaluate stronger, more recent models (e.g., Claude Opus/Sonnet 5, GPT-5.6 Sol/Terra) as targets and testing backbones, to exclude capability-driven failures and further demonstrate the framework's generalizability?

Reviewer_dLCHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RiskMaze, an automated safety evaluation framework for tool-based LLM agents. It explores risk-related tool paths, generates and iteratively refines executable tests, and identifies safety violations from execution evidence. Experiments across multiple scenarios, risk categories, and target agents demonstrate high risk discovery rates and structural diversity in generated tool sequences.

Critical strengths
  1. The paper addresses the limited coverage of different execution paths leading to the same risk in fixed test suites. The motivation is clear, and path-oriented safety evaluation has practical value.
  2. The framework integrates tool filtering, path search, test generation, and feedback-guided refinement, with risk assessment grounded in execution evidence. The overall pipeline is relatively complete.
  3. Experiments cover 349 scenarios, 8 risk categories, and 7 target agents. The evaluation also examines path diversity, execution validity, and human verification, providing multiple perspectives on the framework's performance.
Critical weaknesses
  1. The diversity metrics in Figure 2 are based on intended tool sequences, while RDR counts risk occurrences even when agents follow different paths. Thus, high path diversity and RDR may still correspond to repeated actual risk-triggering trajectories, leaving the discovery of distinct effective risk paths unverified.
  2. The framework combines path search, test generation, and feedback-guided refinement, but lacks a comparison with direct test generation under the same refinement budget. Since Table 3 already demonstrates the effectiveness of iterative refinement, the independent contribution of path search remains unclear.
  3. The tool graph is primarily constructed using semantic compatibility, workflow rules, and risk templates, while subsequent validation focuses on schema compliance and path consistency. Plausible tool sequences do not necessarily imply risk reachability under actual environment states, permissions, and data dependencies. This gap has not been directly evaluated.
Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.