acceptodds
Under review as a conference paper at ICLR 2027

X-AgentRed: Red-Teaming Personal Agents in Evolving Multi-Service Ecosystems

Abstract

Personal agents increasingly connect to services across the web and applications by using tools. As the effects of their actions accumulate and interact across services, unintended or harmful outcomes can arise when agents fulfill benign user requests. Proactively identifying these compositional failures requires exhaustively exploring diverse cross-service paths. Existing red-teaming methods often simply prompt LLMs to identify potentially harmful paths from task and environment descriptions or decompose harmful objectives specified by humans or LLMs into seemingly benign action combinations. However, generation is biased toward patterns learned during training, potentially favoring familiar tool combinations and leaving novel cross-service interactions underexplored. To address this, we propose X-AgentRed, a self-evolving red-teaming framework that discovers harmful tool call paths through a graph of tool dependencies and expands and refines them using execution feedback. By analyzing each tool's effects, potential harms, and preconditions for execution, the red-teaming agent constructs a directed graph that captures how one tool's effects can satisfy another's execution preconditions. Recipes sampled from this graph are instantiated as realistic scenarios in which agents can produce harmful outcomes. Execution feedback enables self-improvement by diversifying successful paths by substituting benign-services and refining preconditions at blocked steps in unsuccessful paths to enable further execution. Across four frontier agentic models, X-AgentRed achieves 1.77× the harmful path rate of the baseline and explores a more diverse set of tools, yielding 2.99× the tool coverage on average.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent Red-Teaming, Tool-Use Agents, Agent Safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cYamOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes X-AgentRed, a red-teaming framework for discovering harmful behavior in personal agents operating across multiple services. It constructs a HarmFlow graph based on tool preconditions, effects, and potential harms, then samples paths to generate scenarios with benign user requests. Execution feedback is used to diversify successful paths and refine unsuccessful ones. The authors evaluate the framework on four target models in a simulated environment containing 30 services and 914 tools, reporting higher harmful path rates and broader tool coverage than STAC and TRACE.

Critical strengths

The paper studies an important problem. As agents gain access to more tools and services, harmful outcomes may arise from their interactions even when the user's request is benign. Studying these risks is relevant to the deployment of personal agents.

The HarmFlow graph provides an explicit representation of dependencies between tools. This makes the proposed tool combinations easier to inspect and provides a reasonable way to organize the search for potentially harmful workflows.

The evaluation covers four target models and a broad range of tools. The reported improvements in both harmful path rate and tool coverage are encouraging. The additional analyses of workflow diversity and judge sensitivity also help readers understand the results and their limitations.

Critical weaknesses

In lines 87–90, the authors present their motivation: “This reliance on LLMs’ prior knowledge introduces a potential source of bias: LLMs are prone to favoring familiar patterns learned during training.” However, this paper does not conduct experiments to demonstrate that this bias exists. The authors further state that this bias “can concentrate exploration on common tool combinations while leaving novel cross-service interactions under-explored.” However, some existing works already investigate cross-service interactions, e.g., ChainFuzzer [1] and SCR-Bench [2].

The traces generated by X-AgentRed mainly rely on LLM simulation. Feedback from the real world is much more complex than feedback from an LLM simulation, so there is a gap between the synthetic traces and real-world scenarios. We do not know whether the phenomena observed in these synthetic scenarios also occur in the real world.

There are no ablation studies. The current experiments cannot demonstrate the effectiveness of the HarmFlow graph. The authors should isolate the contributions of the prompts, the HarmFlow graph, and other components, and show how each component affects the results.

[1] ChainFuzzer: Greybox Fuzzing for Workflow-Level Multi-Tool Vulnerabilities in LLM Agents

[2] Benign in Isolation, Harmful in Composition: Security Risks in Agent Skill Ecosystems

Questions for authors
  1. The example in Figure 1(a) is a bit confusing. Whether the action is harmful depends on certain conditions: the agent must clearly know that the file is internal and cannot be shared externally. Sharing it externally under these conditions would be harmful. If the agent cannot determine whether the file can be shared externally, or if only the user knows that it cannot, then the agent is simply following the user's instructions, and there is no problem with the agent itself. The authors also state that “Long-horizon tool use is an under-explored failure mode.” However, this example involves only a few steps, so I am not fully convinced that it illustrates a long-horizon task, particularly compared with tasks in benchmarks such as Terminal-Bench.

  2. To synthesize such traces, how do you get the models, especially Claude Opus, to follow your instructions without refusing the requests?

7 comments
AuthorsOpenReview ↗

Response to Reviewer cYam (1/2)

We appreciate the reviewer for taking the time to review our paper.


Q1 · Clarifying the example in Figure 1(a)

Sorry for the confusion. Figure 1(a) is a simplified conceptual example to help readers understand how a multi-service path can bypass a known sharing restriction. It assumes that the agent knows the report is internal and would refuse a direct request to share it externally. However, asking the agent to email the weekly report to the client contact in a shared document leads it through document lookup, contact retrieval, and email delivery. Along this longer path, the agent may fail to apply the restriction and disclose the internal report to an external recipient. We have added the sharing restriction directly to Figure 1(a) to clarify the example. Actual cases from our experiments were presented in Appendices C.1 and C.4.


Q2 · Why refusals are uncommon and how they are counted

X-AgentRed is designed to uncover failures that emerge as agents carry out benign requests over multiple turns. It generates these requests from sampled recipes, rather than constructing explicitly harmful prompts (lines 298–300). With this design, we have observed only one refusal while generating 250 paths. If a refusal occurs later during target execution or judging, we count the path as a failure in HPR (Appendix A.5, lines 801–803).


W1 · Evidence for LLMs’ tendency to favor familiar tool combinations

Baselines that rely on LLMs to generate tool paths concentrate their use on a limited set of tools and services, as shown in Figure 4(a–c). Tool coverage is 41.5% for X-AgentRed, compared with 13.9% for STAC and 12.7% for TRACE. The ten most-used tools account for 24.3% of its tool use, versus 43.2% and 63.3% for the baselines, while its paths span 5.0 services on average, versus 2.7 and 1.5 (Figure 6b). Figure 4(d) shows lower upper-tail path overlap than STAC (95th-percentile Jaccard similarity: 0.224 versus 0.525), although TRACE also has low overlap (0.170). These comparisons establish differences in exploration behavior but do not isolate the contribution of training priors.

We thank the reviewer for pointing out these studies [1, 2] and have added both to Related Work (Section 2). However, ChainFuzzer identifies software vulnerabilities within a given agent application through tool-code analysis and payload fuzzing, while SCR-Bench evaluates three predefined forms of risk from skill composition: capability flow, trust transfer, and authorization confusion. Both study compositional risks, but our scope is the discovery of harmful paths across an evolving ecosystem of personal-agent services under benign user requests.

[1] Wu et al., “ChainFuzzer: Greybox Fuzzing for Workflow-Level Multi-Tool Vulnerabilities in LLM Agents,” arXiv, ’26.

[2] Xie et al., “Benign in Isolation, Harmful in Composition: Security Risks in Agent Skill Ecosystems,” arXiv, ’26.


W2 · Applicability of HarmFlow beyond the simulated environment

We agree that our environment, built with Amazon’s ToolSimulator and Claude Opus 4.8 (Section 4.1), cannot fully reproduce production services. However, large-scale evaluation across live services is difficult because it requires managing accounts and permissions, restoring initial states, and containing harmful side effects. Simulated or controlled environments offer a practical way to address these constraints and are widely used in recent agent research.

However, the core principle of HarmFlow construction applies equally to real services. It links tool effects to other tools’ execution preconditions using prior knowledge of their high-level functions. In our simulated environment, we constructed HarmFlow from this knowledge, without providing individual tool descriptions or source code. For example, a customer lookup tool can provide the recipient address required by an email-sending tool. This dependency can be identified without reproducing either service’s internal implementation. Service-specific permissions and execution constraints would then be validated in the actual environment. Our experiments assessed discovery in the simulator, so live-service success rates remain to be evaluated.


AuthorsOpenReview ↗

Response to Reviewer cYam (2/2)

W3 · Ablation study of HarmFlow and self-improvement

We have conducted component ablations by removing one component at a time. We compare the full framework, which combines HarmFlow with self-improvement, with two ablated variants. In w/o self-improvement, we retain HarmFlow for recipe sampling but disable feedback-based augmentation and refinement. In w/o HarmFlow, self-improvement remains enabled, but path generation, augmentation, and refinement are performed directly by an LLM without a graph, following the baselines’ LLM-based approach. Removing HarmFlow lowers HPR on all three targets and roughly halves tool coverage. Removing self-improvement also lowers HPR, despite slightly higher coverage, showing that feedback improves harmful-path discovery beyond expanding tool coverage. The full framework’s slightly lower tool coverage is expected, since augmentation and refinement modify existing paths locally, whereas w/o self-improvement relies entirely on sampling new paths from the graph.

MethodHPR (%) ↑Tool coverage (%) ↑
Claude Opus 4.8Claude Opus 5GPT-5.6 SolClaude Opus 4.8Claude Opus 5GPT-5.6 Sol
X-AgentRed9.611.214.829.636.233.4
  w/o self-improvement9.28.412.133.338.436.2
  w/o HarmFlow5.23.611.213.817.216.7

Without HarmFlow, tool coverage is close to STAC on both targets and to TRACE on GPT-5.6 Sol. The full framework covers about twice as many tools as the variant without HarmFlow, supporting the graph’s contribution to broader exploration. We have added these experiments and their analysis in Section 4.7 (Tables 5 and 6).

MethodTool coverage (%) ↑
Claude Opus 4.8GPT-5.6 Sol
X-AgentRed29.633.4
  w/o HarmFlow13.816.7
STAC15.114.0
TRACE8.015.3
Area_Chair_PxTLOpenReview ↗

Authors' Rebuttal Available - Please Review and Comment

Dear reviewer: The authors have submitted their rebuttal. Please review their responses to see if your concerns have been addressed, and provide your comments or join the discussion at your earliest convenience.

Reviewer_JFzTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose X-AgentRed, a framework that uses a static prerequisite–effect–harm dependency graph (HarmFlow) built from 914 APIs to generate cross-service attack recipes. Through a dual-loop evolutionary mechanism featuring "successful path replacement + blocked path reconstruction," X-AgentRed automatically discovers security vulnerabilities in personal AI agents during multi-tool execution.

Critical strengths
  • The paper targets security risks in long-horizon tool usage within multi-service ecosystems—a critical yet relatively understudied safety boundary.
  • Decoupling multi-step path search into local dependency relations effectively mitigates search limitations caused by relying solely on the prior biases of LLMs.
  • The dual-feedback mechanism, combining successful path replacement and blocked-step repair, is logically sound and structurally clear.
Critical weaknesses

​ My primary concern lies in the execution budget disparity across experiments. The authors evaluated 250 scenarios per model, but X-AgentRed averaged 23.2 tool calls per episode, whereas STAC and TRACE averaged 7.8 and 3.3, respectively. Across 250 scenarios, X-AgentRed accumulated roughly 5,800 target interactions—significantly higher than STAC (~1,950) and TRACE (~825). Consequently, it is difficult to determine whether the improvements in HPR and coverage stem from HarmFlow's search efficiency or simply from a higher interaction budget. Could the authors provide comparative dynamic results under a strictly matched total interaction budget (e.g., fixing the total number of tool calls)?

​ Comparing against only STAC and TRACE may not fully demonstrate the proposed method's advantages over recent work. Relevant recent methods are omitted, such as T-MAP [1], which also employs trajectory-aware evolutionary search for multi-step tool execution, and recent red-teaming work exploring tool-chain abuse within the Model Context Protocol (MCP) ecosystem [2]. Why were these attack paradigms based on trajectory evolution or real-protocol ecosystems omitted from the comparison and discussion? Could the authors clarify the specific advantages of an explicit dependency graph over trajectory-aware evolutionary search under an identical budget?

​ HarmFlow constructs edges purely via pairwise "effect–prerequisite" matching, which fails to enforce global entity binding (e.g., user/account IDs) or context permissions across long chains. What is the initial global executability rate of raw abstract recipes sampled directly from HarmFlow prior to LLM-driven contextual adaptation and path repair? What fraction of generated paths turn into dead ends due to global context conflicts? Scenarios are sampled with a specific target harm hh, yet the success metric H(τ)H(\tau) only checks whether "any harm" was triggered. If a scenario designed for Harm A ultimately triggers an unrelated Harm B, is it counted as a success? Does the current HPR measure "any-harm discovery rate" or "target-harm discovery rate"? This needs explicit clarification. Using Claude Opus 4.8 for both environment simulation and target evaluation alongside Claude Opus 5 as a judge creates overlapping model families. Has the potential intra-family bias in linguistic and logical priors been evaluated, and to what extent does it impact evaluation consistency?

​ HarmFlow relies entirely on a static topology. In practice, real-world APIs frequently undergo schema updates, parameter shifts, or temporary service outages. How robust is the framework when migrating from closed, simulated environments to open, dynamic API ecosystems? How does it handle the reconstruction overhead when underlying APIs change dynamically? Furthermore, while the authors frame "Long-horizon tool use is an under-explored failure mode" in Figure 1 as primary motivation, they do not sufficiently discuss existing literature on long-chain planning failures, multi-turn agent security, and complex workflow vulnerabilities (e.g., Agent-SafetyBench [3] and Crescendo [4]).

[1] Lee, et al. (2026). T-MAP: Red-Teaming LLM Agents with Trajectory-aware Evolutionary Search.

[2] He, et al. (2026). Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools.

[3] Zhang, Z., et al. (2025). Agent-SafetyBench: Evaluating the Safety of LLM Agents.

[4] Russinovich, M., et al. (2025). Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack.

Questions for authors

please refer to Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer JFzT (1/2)

We appreciate the reviewer for the thoughtful review and constructive feedback.


W1 · Generation and execution costs of longer harmful paths

X-AgentRed was specifically designed to discover longer, more complex harmful paths across services, extending exploration beyond the shorter paths typically found by existing red-teaming methods or human red-teamers (lines 94–102). These longer paths naturally require more tool calls to execute than the baseline paths. As these methods generate candidate paths for red-teaming, generation cost per candidate path provides a more direct comparison of their computational efficiency than the target agent’s execution cost. As reported in Section 4.5 (Table 4), X-AgentRed was substantially more efficient at generating candidate paths, requiring 22.10 API calls per path on average, compared with 55.00 for STAC and 103.24 for TRACE.


W2 · Comparison with T-MAP and differences from AutoMalTool

T-MAP evolves adversarial prompts toward harmful objectives using execution feedback and a tool-call graph built from observed trajectories. We have run T-MAP against Claude Opus 5 and obtained an HPR of 0.8%, compared with 11.2% for X-AgentRed. We attribute this gap to T-MAP’s generated prompts, which more explicitly request harmful actions and are therefore more readily detected and refused by recent frontier models’ safety guardrails. X-AgentRed instead generates benign user requests that can lead to harmful outcomes through interactions across multiple services, without explicitly requesting harmful behavior.

AutoMalTool generates malicious MCP tools by injecting instructions into their descriptions and iteratively refining them to manipulate tool arguments or the interpretation of tool outputs. This requires control over tool descriptions and therefore does not match our setting, where tool definitions remain fixed and we study harmful outcomes that arise from cross-service interactions under benign user requests. We have added both T-MAP and AutoMalTool to Section 2.


W3-1 · Recipe executability and dead ends caused by global context conflicts

As described in Section 3.2 (lines 290–300), recipes are ordered tool sequences without concrete arguments or a user prompt and are therefore not directly executable. Scenario instantiation turns the abstract recipe into a benign user prompt that the target agent can execute.

We have also measured how often global context conflicts cause dead ends after scenario instantiation. For X-AgentRed, conflicts arose in approximately 3–6% of paths. While executing the generated prompts, target agents recovered in roughly half of these cases by selecting another tool within the same service or adjusting the arguments. Only 2.0% of paths on each target reached a dead end, compared with 47.2% for STAC on Claude Opus 4.8 and 58.2% on GPT-5.6 Sol. Naively relying on LLMs to generate tool chains can introduce invalid arguments, including values that do not exist in the environment. X-AgentRed’s lower dead-end rate supports the effectiveness of its scenario instantiation step in producing executable paths.

MethodClaude Opus 4.8GPT-5.6 Sol
No conflict (%)Recovered (%)Dead end (%)No conflict (%)Recovered (%)Dead end (%)
X-AgentRed96.81.22.094.43.62.0
STAC50.82.047.238.23.658.2

W3-2 · HPR under Any Harm and Target Harm criteria

Sorry for the confusion. In previous Table 2, we reported Any Harm HPR, which counts paths that produce a confirmed harmful outcome. To address this question, we have also evaluated Target Harm HPR, which requires the observed harm to match the harm specified in the recipe. X-AgentRed achieves the highest Target Harm HPR on all four targets, averaging 12.1%, compared with 6.9% for STAC and 1.7% for TRACE. Its advantage therefore extends to realizing the harm specified in the recipe. We have added these results and their analysis to the main experiments in Section 4.2 (Table 3).

TargetAny Harm (%)Target Harm (%)
TRACESTACOursTRACESTACOurs
Claude Opus 4.80.46.07.60.03.03.2
GPT-5.6 Sol2.414.019.61.29.614.8
DeepSeek-V4-Pro2.09.618.01.66.810.4
GLM-5.26.012.431.64.08.020.0
Average2.710.519.21.76.912.1

AuthorsOpenReview ↗

Response to Reviewer JFzT (2/2)

W3-3 · Assessing potential bias when the judge and target share a model family

We have compared the two judges on the same X-AgentRed paths for each target. On DeepSeek and GLM, which belong to neither judge’s model family, the Claude : GPT ratios of harmful labels are similar at 1.29:1 and 1.33:1. For Claude and GPT paths, the ratios are 1:1.20 and 1.95:1, respectively. Relative to the pattern on DeepSeek and GLM, these results suggest that each judge is less likely to label paths from its own model family as harmful.

To avoid relying on a single model family, our main evaluation used both a Claude judge and a GPT judge and counted a path as harmful only when both agreed (Section 4.6). We have also conducted a human evaluation with 30 expert red-teamers and found 82.8% agreement with the combined judge labels, providing independent support for these evaluations. We have added the judge comparison to Appendix B.7 (Table 13) and the human evaluation to Section 4.2 and Appendix B.6.

Target agentClaude judge (%)GPT judge (%)Ratio (Claude : GPT)
Claude Opus 4.816.419.61 : 1.20
GPT-5.6 Sol46.023.61.95 : 1
DeepSeek-V4-Pro28.822.41.29 : 1
GLM-5.252.839.61.33 : 1

W4-1 · Cost of updating HarmFlow when tools are added, modified, or removed

Removing a tool requires deleting its node and incident edges, without rechecking unrelated tool pairs. With n tools after an update and k new or modified tools, only dependencies involving changed tools need to be checked, assuming the other tool annotations remain valid: Cfull=n2,Cupdate=n2−(n−k)2=2nk−k2.C_{\mathrm{full}}=n^2,\qquad C_{\mathrm{update}}=n^2-(n-k)^2=2nk-k^2. Updating a small subset of tools reuses most existing dependencies. For 10 changed tools among 914, this requires about 46× fewer pairwise checks than a full rebuild. Periodically batching changes can avoid rechecking the same dependencies after successive updates. Efficient graph maintenance techniques, such as incremental coarsening [1] and local repair of graph indexes [2], could further reduce this computational overhead. We have added an analysis of HarmFlow’s maintenance costs to Appendix A.2.

[1] Gupta et al., “Adapting to Evolving Graphs: A Scalable Framework for Dynamic Coarsening,” ICML ’26.

[2] Mishra et al., “Graph-based Nearest Neighbors with Dynamic Updates via Random Walks,” ICLR ’26.


W4-2 · Relation to Agent-SafetyBench and Crescendo

Thank you for these references. Agent-SafetyBench [1] benchmarks tool-use failures such as acting without required information, ignoring constraints, or trusting unsafe tool outputs. Crescendo [2] uses a model’s earlier replies to justify increasingly harmful follow-up requests. Both address failures across interactions. In contrast, X-AgentRed links tool effects to execution preconditions across services, using the resulting dependency graph and execution feedback to discover new harmful paths under benign user requests. We have clarified these similarities and differences in Section 2.

[1] Zhang et al., “Agent-SafetyBench: Evaluating the Safety of LLM Agents,” arXiv, 2024.

[2] Russinovich et al., “Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack,” USENIX Security ’25.

Thank you for the responses. I will keep my score.

We appreciate the reviewer’s follow-up. Could the reviewer clarify whether any of the original concerns remain unresolved or whether there are any additional concerns? We would be happy to address them.

Reviewer_zQTAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

X-AgentRed uses a tool-dependency graph and execution feedback to discover harmful cross-service behavior arising from benign requests. In a simulated ecosystem of 914 tools across 30 services, it improves harmful-scenario discovery and tool coverage over two adapted baselines across four target agents.

Critical strengths

The graph provides a concrete search structure for a consequential problem: harmful interactions across services during otherwise benign tasks. Higher discovery rates and broader tool coverage across all four targets support the usefulness of the complete framework.

Critical weaknesses
  1. HarmFlow's specific contribution remains uncertain. Figure 5 compares feedback-conditioned scenario populations, while Figure 7 measures tool overlap without checking ordered dependencies. These analyses support discovery by the complete system but do not isolate the benefit of graph guidance. A graph ablation would strengthen that explanation; the current claims should distinguish the observed system gains from the proposed mechanism.
  2. Small gains need caution: the single-run Claude result is 19 versus 15 successes (Table 2; Appendix A.4). Adaptive batches are not independent replications.
  3. The principal validity limitation is calibration of simulated harms. The judges assess simulator-produced states, and GPT-target HPR changes from 19.6% under AND to 50.0% under OR (Table 9). This establishes sensitivity, not incorrect labels. The existing safeguards support a simulation study, but an independent audit of recorded cases would strengthen claims about executable, unauthorized harm.
  4. TRACE's declared source–sink paths are compared with executed paths in Figure 4(b–d) (Appendix B.3). This disclosed mismatch limits those diversity comparisons; it does not invalidate the X-AgentRed–STAC execution comparison.
Questions for authors

Which existing evidence supports a distinct benefit from graph guidance beyond the complete system's gains?

3 comments
AuthorsOpenReview ↗

Response to Reviewer zQTA

We thank the reviewer for the careful review and constructive suggestions.


Q1 / W1 · Ablation study of HarmFlow’s contribution

To directly assess HarmFlow’s contribution, we have conducted the ablation study suggested by the reviewer. We compare the full framework, which combines HarmFlow with self-improvement, with two ablated variants. In w/o self-improvement, we retain HarmFlow for recipe sampling but disable feedback-based augmentation and refinement. In w/o HarmFlow, self-improvement remains enabled, but path generation, augmentation, and refinement are performed directly by an LLM without a graph, following the baselines’ LLM-based approach. Comparing the full framework with w/o HarmFlow therefore tests the contribution of graph guidance across path generation and self-improvement.

Removing HarmFlow reduces HPR by 3.6–7.6 percentage points across all three targets and roughly halves tool coverage. The full framework therefore finds more harmful paths while exploring a wider set of tools than the variant using direct LLM-based generation and updates. These results provide evidence for the benefit of graph guidance beyond the overall system gains reported in the original figures. The full framework’s slightly lower tool coverage is expected, since augmentation and refinement modify existing paths locally, whereas w/o self-improvement relies entirely on sampling new paths from the graph.

MethodHPR (%) ↑Tool coverage (%) ↑
Claude Opus 4.8Claude Opus 5GPT-5.6 SolClaude Opus 4.8Claude Opus 5GPT-5.6 Sol
X-AgentRed9.611.214.829.636.233.4
  w/o self-improvement9.28.412.133.338.436.2
  w/o HarmFlow5.23.611.213.817.216.7

The second table shows that, without HarmFlow, tool coverage falls close to STAC on both targets and to TRACE on GPT-5.6 Sol. The full framework covers about twice as many tools as the variant without HarmFlow, linking the coverage advantage to graph guidance. We have added these experiments and their analysis in Section 4.7 (Tables 5 and 6).

MethodTool coverage (%) ↑
Claude Opus 4.8GPT-5.6 Sol
X-AgentRed29.633.4
  w/o HarmFlow13.816.7
STAC15.114.0
TRACE8.015.3

W2 · Human evaluation

We have conducted a human evaluation with 30 expert red-teamers. The red-teamers evaluated whether each path produced a harmful outcome, without seeing the method or judge labels. On GPT-5.6 Sol paths across all three methods, human labels agreed with the combined judge labels in 82.8% of cases. The LLM-judges more often labeled ambiguous situations as harmful, reflecting a more conservative assessment of risk than the red-teamers. We have summarized these findings in Section 4.2 and provided detailed analysis in Appendix B.6.


W3 · Updating TRACE’s evaluation from source–sink paths to executed paths

Sorry for the confusion. We have changed TRACE’s evaluation from source–sink paths to executed paths, matching our evaluation of X-AgentRed and STAC. Using executed paths yields worse results for TRACE on all three reported metrics, with lower tool coverage, fewer services per path, and a higher top-10 tool usage share. The table summarizes these changes, averaged across four targets. X-AgentRed retains higher tool coverage and more services per path, while HPR is unchanged. We have updated the corresponding results and analyses in Table 2 and Figures 4 and 6.

MetricSource–sink pathsExecuted paths (Updated)
Tool coverage (%)23.712.7
Mean services per path1.91.5
Top-10 tool usage share (%)48.963.3

Thanks for the detailed response. I think these comments clarify my concerns and will keep my ratings.

We appreciate the reviewer’s time and follow-up. We are glad that our response addressed the reviewer's concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.