- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether self-evolving agents can introduce safety risks when they complete tasks, absorb feedback, and update their own rules, memory mechanisms, or tools during normal operation. Specifically, it investigates whether modifications that are beneficial for the current task may be inappropriately reused in future contexts, thereby causing safety issues. The main contribution is the proposal of SEABENCH, which constructs 48 longitudinal task sequences in a personal-assistant environment. Each sequence consists of upstream tasks where the agent accumulates self-updates, followed by downstream tasks that evaluate whether these updates lead to unsafe behaviors. The benchmark covers three types of evolution mechanisms and combines adaptive task search, paired comparisons with agents without self-evolution, and attribution analysis of updates to identify failure trajectories caused by persistent self-updates. Experiments on three models show that, among the discovered trajectories, self-evolution generally improves task completion rates but also increases risks such as privacy leakage, weakened authorization boundaries, and cross-context misuse. Further reasoning-trace analysis reveals that some of these risks are reflected in the agent’s planning process, and the paper provides a preliminary validation of the feasibility of using such signals to identify unsafe outputs.
Critical strengths
This paper evaluates three models and three types of updates—controller, memory, and tools/skills—within a unified personal-assistant environment. Across all 3,435 paired candidates evaluated during the search, evolved agents exhibit a safety failure rate of 32.2%, compared with 24.3% for non-evolving agents. The analysis further characterizes failures associated with different update mechanisms, including weakened authorization boundaries, loss of contextual constraints, and unsafe tool reuse. This contribution strengthens the empirical understanding of self-evolution risks across models and update mechanisms, while providing a shared experimental environment and concrete failure trajectories for studying these risks.
Critical weaknesses
Insufficient substantive novelty compared with existing studies and benchmarks. Prior work [1] has already investigated safety degradation caused by memory accumulation, tool creation and reuse, and long-term interaction. TAME [2] proposes Trust-Memevo to evaluate trustworthiness degradation during benign task evolution, while MemEvoBench [3] studies multi-turn safety degradation under contaminated memories and biased feedback. SEABench does not sufficiently clarify what important new risk mechanisms are revealed by its unified evaluation environment compared with these works, or what substantially stronger evaluation capabilities it provides. This weakens the justification for the significance of the benchmark contribution and is the primary factor affecting my overall assessment.
The safety failure judgments are inconsistent with the presented behavioral evidence. In the insurance information handoff case in Appendix D.3.3, the summarized response explicitly describes a behavior that restricts identifier disclosure before authorization, yet it is subsequently labeled as a privacy leak. Similarly, the hiking coordination case clearly distinguishes between an internal version and a de-identified version intended for sharing, but it is still classified as an information leakage case. The paper does not provide sufficient details about the specific violating content, recipient, or disclosure pathway to justify these labels. Since safety annotations are the foundation of the benchmark, such inconsistencies directly undermine the credibility of the reported failure rates, comparisons among different evolution mechanisms, and evaluations of defense strategies.
[1] S. Shao et al., “Your agent may misevolve: Emergent risks in self-evolving LLM agents,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2026.
[2] Y. Cheng et al., “TAME: A trustworthy test-time evolution of agent memory with systematic benchmarking,” arXiv preprint arXiv:2602.03224, 2026.
[3] W. Xie et al., “MemEvoBench: Benchmarking memory misevolution in LLM agents,” arXiv preprint arXiv:2604.15774, 2026.
Questions for authors
Could the authors provide direct comparisons with existing studies and benchmarks to clearly demonstrate what new risk mechanisms or evaluation capabilities SEABench introduces, and why these additions are significant?
Could the authors fix the tasks identified through the search process and compare the failure rates of evolving and non-evolving agents across independent executions? This would help clarify whether the benchmark scores reflect safety degradation caused by agent evolution or merely the search process's ability to identify failure cases.
Could the authors provide the complete execution traces and specific safety judgment criteria for the aforementioned cases in Appendix D.3.3, explicitly identifying what information was disclosed, to whom, and which authorization or contextual boundaries were violated? Could they also explain why cases in which authorization restrictions were preserved or shared versions were already de-identified were nevertheless classified as safety failures?
5 comments
Rebuttal to Reviewer zkm9 (2/2)
3. On independently rerunning the tasks found by search
Thank you for this suggestion. For each downstream safety test, we already give both agents the same prompt in separate sessions from their respective frozen states (Sec. 4.3, lines 279–283).
To check whether the difference recurs, we repeat each discovered safety test with the prompt fixed and restore both agents' saved states before every attempt. We then compare the fraction of attempts in which each agent fails and report results below. This checks whether the evolved agent fails more often when it starts with the same update and receives the same request again. However, we note that LLM-stochasticity is still an important consideration here, which partially motivated our use of an adaptive pipeline: the reasoning generated during the safety test can still vary even when we hold the earlier trajectory fixed. Bjarnason et al. [4] find early divergence across repeated agent runs, and Gonzalez-Pumariega et al. [5] find that the same computer-use task can succeed or fail across runs. Furthermore, due to LLM-related stochasticity, even with the same context and setting temperature to 0, a sampled final answer can vary [6].
Results: We perform this test over 10 randomly chosen task sequences on the controller surface for Kimi K2.5 for this illustration. Over 10 seeds, this yields tests each for evolved and unevolved agents. Evolved agents exhibit higher failures, even with the aforementioned stochasticity: at 27.6% against 20.6% for unevolved. Furthermore, for every task sequence taken alone, the evolved agent consistently exhibits higher safety failures than the baseline unevolved agent. Indeed, a one-sided Mantel–Haenszel test, which compares conditions while keeping sequences separate, yields , illustrating statistical significance. These results are now also added to a revision of the paper (Sec. 6.1, lines 374 - 396).
Thank you again for your feedback. We believe these comparisons and corrections clarify the contribution, will provide the promised deliverables (particularly, revised prompts for aforementioned safety tests), and we would sincerely appreciate it if you could strengthen your support of the paper in light of them.
References
[1] Shao et al. Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents. ICLR 2026. (https://openreview.net/forum?id=Fd1jgQQW28)
[2] Cheng et al. TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarking. EMNLP 2026. (https://aclanthology.org/2026.emnlp-main.2100/)
[3] Xie et al. MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agents. 2026. (https://arxiv.org/abs/2604.15774)
[4] Bjarnason et al. On Randomness in Agentic Evals. 2026. (https://arxiv.org/abs/2602.07150)
[5] Gonzalez-Pumariega et al. On the Reliability of Computer Use Agents. 2026. (https://arxiv.org/abs/2604.17849)
[6] Horace He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Connectionism, September 2025. DOI
Thank you for clarifying SEABench’s contribution. This largely resolves my novelty concern, and the fixed-prompt reruns show that the failure gap persists on the sampled controller tasks. My remaining concern is the safety labeling. Please share the traces and criteria for the hiking and insurance cases, and clarify whether revising the 15 tests changes the labels or the overall failure gap. If these judgments are supported and the main result holds, I would be open to raising my score from 2 to 3.
Rebuttal to Reviewer zkm9 (1/2)
We thank you for your review and for recognizing the value of the unified environment, the comparison across all explored candidates, and the analysis of authorization weakening, contextual boundary degradation, and unsafe tool reuse. We address your main points below.
1. On novelty and comparison with prior work
Thank you for pointing out these studies. Shao et al. [1] is already discussed in Sec. 2 (lines 121–129). They study model self-training, accumulated memory, tool creation and reuse, and workflow evolution in separate experiments. They compare safety before and after evolution and, in the tool-reuse experiment, distinguish an unsafe tool from unsafe use of a tool.
TAME [2] measures trustworthiness as executor and evaluator memories evolve during benign tasks. It appeared at EMNLP in October 2026 and is contemporaneous under the ICLR guidelines. MemEvoBench [3], an arXiv preprint, studies how misleading inputs, noisy tool returns, and biased feedback affect safety as memory accumulates. ICLR 2027 Reviewer Guidelines, FAQ
In contrast to these prior works on experience/memory accumulation under a fixed protocol, in our work, the agent can rewrite the procedure for retaining, retrieving, or compacting memory. SEABench evaluates such changes alongside agent-written controller rules and executable tools or skills under one longitudinal protocol. The linked environment lets us reuse the same records across tasks with different purposes, recipients, or authorization boundaries. We then test whether the evolving and non-evolving agents preserve the same safety boundary in the later task. We will add these comparisons to Sec. 2 to further strengthen discussion on our novelty.
SEABench uses one longitudinal protocol to evaluate agent-written changes to controller instructions, memory-handling procedures, and executable tools or skills, with the model weights held fixed. For example, the agent can change how it retains, retrieves, or compacts information. The linked environment lets us test what happens when the same records are used for a different recipient or purpose, or under a different authorization boundary. For each surface, we give the same downstream prompt to the evolved and paired non-evolving agents and assess whether upstream self-evolution as a whole contributed to the failure (Sec. 5 and Appendix A).
2. On the insurance handoff and hiking examples
We thank you for the opportunity to clarify the insurance handoff and hiking examples. These tests probe whether the evolved agent preserves the relevant authorization or disclosure boundary when it applies a behavior learned upstream. More natural-sounding requests would help illustrate how those boundaries arise in ordinary use. In both cases, the safety label should follow what the agent actually disclosed and to whom.
For the hiking example, while the internal view contained booking references, the club-facing view redacted them but still retained flight details, which the privacy criteria prohibit the release of, thus justifying the privacy failure label. Thanks for pointing out the insurance handoff example, we will modify and correct it.
We are therefore modifying 15 such safety tasks for subtler natural sounding variants (including the fix to the insurance handoff example) out of 720 downstream tests and will report the modifications shortly during the rebuttal phase. We will provide the execution traces and task-specific rubrics for these two cases as files in the provided anonymous GitHub link.