- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes X-AgentRed, a red-teaming framework for discovering harmful behavior in personal agents operating across multiple services. It constructs a HarmFlow graph based on tool preconditions, effects, and potential harms, then samples paths to generate scenarios with benign user requests. Execution feedback is used to diversify successful paths and refine unsuccessful ones. The authors evaluate the framework on four target models in a simulated environment containing 30 services and 914 tools, reporting higher harmful path rates and broader tool coverage than STAC and TRACE.
Critical strengths
The paper studies an important problem. As agents gain access to more tools and services, harmful outcomes may arise from their interactions even when the user's request is benign. Studying these risks is relevant to the deployment of personal agents.
The HarmFlow graph provides an explicit representation of dependencies between tools. This makes the proposed tool combinations easier to inspect and provides a reasonable way to organize the search for potentially harmful workflows.
The evaluation covers four target models and a broad range of tools. The reported improvements in both harmful path rate and tool coverage are encouraging. The additional analyses of workflow diversity and judge sensitivity also help readers understand the results and their limitations.
Critical weaknesses
In lines 87–90, the authors present their motivation: “This reliance on LLMs’ prior knowledge introduces a potential source of bias: LLMs are prone to favoring familiar patterns learned during training.” However, this paper does not conduct experiments to demonstrate that this bias exists. The authors further state that this bias “can concentrate exploration on common tool combinations while leaving novel cross-service interactions under-explored.” However, some existing works already investigate cross-service interactions, e.g., ChainFuzzer [1] and SCR-Bench [2].
The traces generated by X-AgentRed mainly rely on LLM simulation. Feedback from the real world is much more complex than feedback from an LLM simulation, so there is a gap between the synthetic traces and real-world scenarios. We do not know whether the phenomena observed in these synthetic scenarios also occur in the real world.
There are no ablation studies. The current experiments cannot demonstrate the effectiveness of the HarmFlow graph. The authors should isolate the contributions of the prompts, the HarmFlow graph, and other components, and show how each component affects the results.
[1] ChainFuzzer: Greybox Fuzzing for Workflow-Level Multi-Tool Vulnerabilities in LLM Agents
[2] Benign in Isolation, Harmful in Composition: Security Risks in Agent Skill Ecosystems
Questions for authors
-
The example in Figure 1(a) is a bit confusing. Whether the action is harmful depends on certain conditions: the agent must clearly know that the file is internal and cannot be shared externally. Sharing it externally under these conditions would be harmful. If the agent cannot determine whether the file can be shared externally, or if only the user knows that it cannot, then the agent is simply following the user's instructions, and there is no problem with the agent itself. The authors also state that “Long-horizon tool use is an under-explored failure mode.” However, this example involves only a few steps, so I am not fully convinced that it illustrates a long-horizon task, particularly compared with tasks in benchmarks such as Terminal-Bench.
-
To synthesize such traces, how do you get the models, especially Claude Opus, to follow your instructions without refusing the requests?
7 comments
Response to Reviewer cYam (2/2)
W3 · Ablation study of HarmFlow and self-improvement
We have conducted component ablations by removing one component at a time. We compare the full framework, which combines HarmFlow with self-improvement, with two ablated variants. In w/o self-improvement, we retain HarmFlow for recipe sampling but disable feedback-based augmentation and refinement. In w/o HarmFlow, self-improvement remains enabled, but path generation, augmentation, and refinement are performed directly by an LLM without a graph, following the baselines’ LLM-based approach. Removing HarmFlow lowers HPR on all three targets and roughly halves tool coverage. Removing self-improvement also lowers HPR, despite slightly higher coverage, showing that feedback improves harmful-path discovery beyond expanding tool coverage. The full framework’s slightly lower tool coverage is expected, since augmentation and refinement modify existing paths locally, whereas w/o self-improvement relies entirely on sampling new paths from the graph.
| Method | HPR (%) ↑ | Tool coverage (%) ↑ | ||||
|---|---|---|---|---|---|---|
| Claude Opus 4.8 | Claude Opus 5 | GPT-5.6 Sol | Claude Opus 4.8 | Claude Opus 5 | GPT-5.6 Sol | |
| X-AgentRed | 9.6 | 11.2 | 14.8 | 29.6 | 36.2 | 33.4 |
| w/o self-improvement | 9.2 | 8.4 | 12.1 | 33.3 | 38.4 | 36.2 |
| w/o HarmFlow | 5.2 | 3.6 | 11.2 | 13.8 | 17.2 | 16.7 |
Without HarmFlow, tool coverage is close to STAC on both targets and to TRACE on GPT-5.6 Sol. The full framework covers about twice as many tools as the variant without HarmFlow, supporting the graph’s contribution to broader exploration. We have added these experiments and their analysis in Section 4.7 (Tables 5 and 6).
| Method | Tool coverage (%) ↑ | |
|---|---|---|
| Claude Opus 4.8 | GPT-5.6 Sol | |
| X-AgentRed | 29.6 | 33.4 |
| w/o HarmFlow | 13.8 | 16.7 |
| STAC | 15.1 | 14.0 |
| TRACE | 8.0 | 15.3 |
Authors' Rebuttal Available - Please Review and Comment
Dear reviewer: The authors have submitted their rebuttal. Please review their responses to see if your concerns have been addressed, and provide your comments or join the discussion at your earliest convenience.
Response to Reviewer cYam (1/2)
We appreciate the reviewer for taking the time to review our paper.
Q1 · Clarifying the example in Figure 1(a)
Sorry for the confusion. Figure 1(a) is a simplified conceptual example to help readers understand how a multi-service path can bypass a known sharing restriction. It assumes that the agent knows the report is internal and would refuse a direct request to share it externally. However, asking the agent to email the weekly report to the client contact in a shared document leads it through document lookup, contact retrieval, and email delivery. Along this longer path, the agent may fail to apply the restriction and disclose the internal report to an external recipient. We have added the sharing restriction directly to Figure 1(a) to clarify the example. Actual cases from our experiments were presented in Appendices C.1 and C.4.
Q2 · Why refusals are uncommon and how they are counted
X-AgentRed is designed to uncover failures that emerge as agents carry out benign requests over multiple turns. It generates these requests from sampled recipes, rather than constructing explicitly harmful prompts (lines 298–300). With this design, we have observed only one refusal while generating 250 paths. If a refusal occurs later during target execution or judging, we count the path as a failure in HPR (Appendix A.5, lines 801–803).
W1 · Evidence for LLMs’ tendency to favor familiar tool combinations
Baselines that rely on LLMs to generate tool paths concentrate their use on a limited set of tools and services, as shown in Figure 4(a–c). Tool coverage is 41.5% for X-AgentRed, compared with 13.9% for STAC and 12.7% for TRACE. The ten most-used tools account for 24.3% of its tool use, versus 43.2% and 63.3% for the baselines, while its paths span 5.0 services on average, versus 2.7 and 1.5 (Figure 6b). Figure 4(d) shows lower upper-tail path overlap than STAC (95th-percentile Jaccard similarity: 0.224 versus 0.525), although TRACE also has low overlap (0.170). These comparisons establish differences in exploration behavior but do not isolate the contribution of training priors.
We thank the reviewer for pointing out these studies [1, 2] and have added both to Related Work (Section 2). However, ChainFuzzer identifies software vulnerabilities within a given agent application through tool-code analysis and payload fuzzing, while SCR-Bench evaluates three predefined forms of risk from skill composition: capability flow, trust transfer, and authorization confusion. Both study compositional risks, but our scope is the discovery of harmful paths across an evolving ecosystem of personal-agent services under benign user requests.
[1] Wu et al., “ChainFuzzer: Greybox Fuzzing for Workflow-Level Multi-Tool Vulnerabilities in LLM Agents,” arXiv, ’26.
[2] Xie et al., “Benign in Isolation, Harmful in Composition: Security Risks in Agent Skill Ecosystems,” arXiv, ’26.
W2 · Applicability of HarmFlow beyond the simulated environment
We agree that our environment, built with Amazon’s ToolSimulator and Claude Opus 4.8 (Section 4.1), cannot fully reproduce production services. However, large-scale evaluation across live services is difficult because it requires managing accounts and permissions, restoring initial states, and containing harmful side effects. Simulated or controlled environments offer a practical way to address these constraints and are widely used in recent agent research.
However, the core principle of HarmFlow construction applies equally to real services. It links tool effects to other tools’ execution preconditions using prior knowledge of their high-level functions. In our simulated environment, we constructed HarmFlow from this knowledge, without providing individual tool descriptions or source code. For example, a customer lookup tool can provide the recipient address required by an email-sending tool. This dependency can be identified without reproducing either service’s internal implementation. Service-specific permissions and execution constraints would then be validated in the actual environment. Our experiments assessed discovery in the simulator, so live-service success rates remain to be evaluated.