acceptodds
Under review as a conference paper at ICLR 2027

Survival of the Flawed: How Agent Harness Evolution Inherit, Amplify, and Generalize Adversarial Trajectories

Abstract

Self-evolving LLM agents improve autonomously by mining their own execution trajectories, diagnosing failures, and revising their components. This paradigm has recently been extended to agent safety: safety-oriented harness evolution frameworks learn safety boundaries directly from rollouts and rewrite the agent's safety harness without human oversight. However, these frameworks implicitly trust the trajectories they learn from. We show that this trust turns the evolution loop itself into a new attack surface: by injecting a small number of crafted trajectories into the evolution pool, through malicious user inputs, compromised tool backends, or tampered logs, an adversary with no access to model weights or memory can make the evolver write attacker-chosen rules into the harness. We present TrajPoison, a diagnosis-aware poisoning framework that synthesizes a target edit disguised as a legitimate safety repair, iteratively refines poisoning trajectories against a surrogate evolver until they induce that edit, and adds support trajectories so the edit passes validation. To counter this threat, we propose EvoGuard, a runtime evidence-grounded defense that jointly examines the historical trajectories behind each self-update, the knowledge-state transition it induces, and cross-evidence corroboration, suppressing anomalous updates before they are consolidated. Experiments on Agent-SafetyBench and AgentHarm show that, with only 5% poisoned trajectories, TrajPoison increases the unsafe rate by up to 1.97 in the permissive mode and reduces utility to as low as 0.60 in the restrictive mode, persisting for 20 rounds after the poison is purged. EvoGuard reduces the attack success rate by 96.8% while preserving the benefits of benign evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Self-evolving, Agent Security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TmytOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how trajectory poisoning can subvert the safety harness of self-evolving LLM agents. It proposes TRAJPOISON, an attack framework that presents attacker-chosen harness edits as legitimate safety repairs, refines poisoned trajectories using diagnostic feedback from a surrogate evolution engine, and adds supporting trajectories to help the edits pass validation. The attack can weaken safety constraints or induce excessive restrictions that reduce utility. The paper also introduces EvoGuard, a defense that examines the evidence supporting candidate updates, replays tasks to check whether safety-relevant observations reproduce, and rejects updates that expand permitted actions on harmful probes in a private invariant suite. Experiments show that TRAJPOISON degrades safety or utility, with these effects transferring to a held-out benchmark and other agent backbones. In the primary evaluation setting, EvoGuard substantially mitigates the attacks while largely preserving the benefits of benign evolution.

Critical strengths
  1. The threat model is plausible, and the problem addresses a real-world problem.

  2. The evaluation covers both poisoning modes and checks transfer across benchmarks and agent backbones. These settings fit the claims about attack effectiveness.

Critical weaknesses
  1. The proposed methods are insufficiently validated. This work promised to compare TRAJPOISON with existing poisoning methods, but it doesn't actually test against poisoning baselines. The promised ablation in the Reproducibility Statement is also not provided. Moreover, it does not directly evaluate the evidential asymmetry underlying EVOGUARD.

  2. The claim about persistence lacks experimental support. The abstract and introduction claim the attack effects last for 20 rounds after removal, but the experiment applies poisoning in every round.

Questions for authors
  1. How is C(∆) calculated, and where does the threshold come from?

  2. Besides safety-invariant verification, what results show that evidence dispersion and reproducibility score can distinguish benign patches from poisoned patches and improve EVOGUARD's effectiveness?

  3. Which experimental results support the 96.8% reduction in attack success rate claimed in the abstract and the 96.8% reduction in harmful rate claimed in the introduction?

  4. What experimental results support the claimed persistence for 20 rounds after the poison is purged?

  5. How does TRAJPOISON perform compared with existing poisoning attacks, and what experimental results show the contribution of each stage?

Reviewer_EW1wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies trajectory poisoning in self-evolving agent safety harnesses. It proposes TrajPoison, which defines target edits disguised as safety repairs, refines poisoned trajectories using diagnostic feedback, and adds supporting examples to help harmful updates pass validation. The paper also proposes EvoGuard, which verifies updates through evidence tracing, trajectory replay, and independent safety checks. Experiments on Agent-SafetyBench and AgentHarm show that poisoning 5% of trajectories per round can substantially reduce safety or utility. The effects also transfer across tasks and model backbones, while EvoGuard mitigates the attacks.

Critical strengths
  1. The problem has practical importance. Safety mechanisms can themselves change as agents learn from experience. Protecting the update process is therefore important alongside protecting individual tasks. The paper clearly identifies this risk and offers useful guidance for designing safer self-evolving agents.

  2. The evaluation considers both safety and utility. The paper studies attacks that weaken safety rules and attacks that cause excessive refusal. For example, restrictive poisoning reduces SHE’s safe task completion rate from 59.4% to 35.6%. This shows that a lower unsafe behavior rate does not necessarily imply better overall performance.

Critical weaknesses
  1. The attacker’s access in the experiments does not clearly match the practical scenarios discussed. The main experiments directly replace complete trajectories in the experience pool. The synthetic trajectories in the appendix also include model reasoning, tool calls, and safety judgments. An attacker who can only submit user requests or control tool responses may not be able to modify these records. For example, the paper does not show a complete case where a misleading tool response produces the required trajectory and ultimately leads to an accepted harmful rule. The results mainly support a setting where experience records can be tampered with; the evidence for attacks with more limited access is insufficient.
  2. The experiments show that poisoning causes harm, but do not establish the proposed methods’ advantages over simpler alternatives. For example, Table 1 shows an increase in unsafe behavior from 18.5% to 36.4%, but provides no direct comparison with simple trajectory poisoning or existing attacks under the same poisoning rate and query budget. The benefit of diagnostic feedback is therefore unclear. Similarly, the defense evaluation does not compare EvoGuard with independent safety testing alone, leaving the added value of evidence tracing and replay uncertain. These omissions weaken the evidence for the methodological contributions.
Questions for authors
  1. What exactly must the attacker control? Can you provide a complete example using only user input or a tool response, showing how the malicious content enters a trajectory, induces a rule change, passes the original validation, and affects later tasks? If the experiments require direct changes to reasoning records or evaluation labels, please state this requirement clearly.

  2. How much does the full method improve over simpler alternatives? Under the same poisoning rate and query budget, how does TrajPoison compare with simple trajectory poisoning or existing attacks? How much additional protection does the full EvoGuard provide over independent safety testing alone?

Reviewer_vky5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a new agent safety issue: when an agent automatically modifies the safety rules based on the historical execution records, an attacker can mislead the agent to write down the wrong safety judgment into its own harness through editing a piece of historical records. The authors proposed the TrajPosion attacker framework to mislead the agent system into making a wrong judgment by faking somehow reasonable execution traces. They also propose EvoGuard to trace the evidence, reexecute historical tasks, and check whether the safety rules have been abnormally changed. The experiments also demonstrate their effectiveness.

Critical strengths
  1. The research problem is critical for a self-evolving agent. Whether the attack can evolve from one pollution to a long-term safety rule. They point out permissive poisoning and restrictive poisoning and extend the common issue into safety harness evolution.

  2. This submission not only finds the safety issue, but also proposes a defensive strategy. TrajPoison aims to design new attacks, and EvoGuard tries to check whether every rule update is supported.

  3. The experiments show that the risk directly exists in the harness layer, not just in the single foundation model layer.

Critical weaknesses
  1. The submission only did experiments on one major self-evolution framework. It will be better to test it on more different frameworks with different diagnosis, update, and validation mechanisms.

  2. It will be useful to add more ablation studies of EvoGuard on robustness.

  3. The submission claims persistent corruption for 20 rounds after poison removal and mitigation by EvoGuard, but the experiments do not seem to isolate or quantify these effects.

Questions for authors

Please refer to the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.