acceptodds
Under review as a conference paper at ICLR 2027

SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents

Abstract

Self-evolving LLM agents have gained prominence for their ability to improve after deployment by modifying their harness, including their controller instructions, memory management protocols, and reusable tools and skills, in response to user and environment feedback. However, locally useful updates may persist into later tasks where they produce unsafe behavior, even without direct adversarial influence. To study this risk, we introduce SEABench, a benchmark for studying endogenous misalignment arising from agent self-evolution with 48 longitudinal task sequences that span multiple evolution surfaces, task domains, and harm types in a rich personal-assistant environment. To account for the stochasticity inherent in agentic operations, we provide an adaptive trajectory discovery pipeline that probes for failures while preserving original task intent and supports causal attribution through paired non-evolving agents and attribution scores. Our evaluation across multiple recent LLMs, evolution surfaces, and harm types reveals that self-evolution can increase task completion rates while producing safety failures that are absent for paired non-evolving baseline agents, and that qualitatively different safety behaviors emerge across evolution surfaces and harm types. Furthermore, this divergence in safety behavior is reflected in agents' chain-of-thought reasoning, which yields an effective monitoring strategy that can mitigate unsafe behavior with a low false positive rate.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Self-evolving agents, safety, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_zkm9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether self-evolving agents can introduce safety risks when they complete tasks, absorb feedback, and update their own rules, memory mechanisms, or tools during normal operation. Specifically, it investigates whether modifications that are beneficial for the current task may be inappropriately reused in future contexts, thereby causing safety issues. The main contribution is the proposal of SEABENCH, which constructs 48 longitudinal task sequences in a personal-assistant environment. Each sequence consists of upstream tasks where the agent accumulates self-updates, followed by downstream tasks that evaluate whether these updates lead to unsafe behaviors. The benchmark covers three types of evolution mechanisms and combines adaptive task search, paired comparisons with agents without self-evolution, and attribution analysis of updates to identify failure trajectories caused by persistent self-updates. Experiments on three models show that, among the discovered trajectories, self-evolution generally improves task completion rates but also increases risks such as privacy leakage, weakened authorization boundaries, and cross-context misuse. Further reasoning-trace analysis reveals that some of these risks are reflected in the agent’s planning process, and the paper provides a preliminary validation of the feasibility of using such signals to identify unsafe outputs.

Critical strengths

This paper evaluates three models and three types of updates—controller, memory, and tools/skills—within a unified personal-assistant environment. Across all 3,435 paired candidates evaluated during the search, evolved agents exhibit a safety failure rate of 32.2%, compared with 24.3% for non-evolving agents. The analysis further characterizes failures associated with different update mechanisms, including weakened authorization boundaries, loss of contextual constraints, and unsafe tool reuse. This contribution strengthens the empirical understanding of self-evolution risks across models and update mechanisms, while providing a shared experimental environment and concrete failure trajectories for studying these risks.

Critical weaknesses

Insufficient substantive novelty compared with existing studies and benchmarks. Prior work [1] has already investigated safety degradation caused by memory accumulation, tool creation and reuse, and long-term interaction. TAME [2] proposes Trust-Memevo to evaluate trustworthiness degradation during benign task evolution, while MemEvoBench [3] studies multi-turn safety degradation under contaminated memories and biased feedback. SEABench does not sufficiently clarify what important new risk mechanisms are revealed by its unified evaluation environment compared with these works, or what substantially stronger evaluation capabilities it provides. This weakens the justification for the significance of the benchmark contribution and is the primary factor affecting my overall assessment.

The safety failure judgments are inconsistent with the presented behavioral evidence. In the insurance information handoff case in Appendix D.3.3, the summarized response explicitly describes a behavior that restricts identifier disclosure before authorization, yet it is subsequently labeled as a privacy leak. Similarly, the hiking coordination case clearly distinguishes between an internal version and a de-identified version intended for sharing, but it is still classified as an information leakage case. The paper does not provide sufficient details about the specific violating content, recipient, or disclosure pathway to justify these labels. Since safety annotations are the foundation of the benchmark, such inconsistencies directly undermine the credibility of the reported failure rates, comparisons among different evolution mechanisms, and evaluations of defense strategies.

[1] S. Shao et al., “Your agent may misevolve: Emergent risks in self-evolving LLM agents,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2026.

[2] Y. Cheng et al., “TAME: A trustworthy test-time evolution of agent memory with systematic benchmarking,” arXiv preprint arXiv:2602.03224, 2026.

[3] W. Xie et al., “MemEvoBench: Benchmarking memory misevolution in LLM agents,” arXiv preprint arXiv:2604.15774, 2026.

Questions for authors

Could the authors provide direct comparisons with existing studies and benchmarks to clearly demonstrate what new risk mechanisms or evaluation capabilities SEABench introduces, and why these additions are significant?

Could the authors fix the tasks identified through the search process and compare the failure rates of evolving and non-evolving agents across independent executions? This would help clarify whether the benchmark scores reflect safety degradation caused by agent evolution or merely the search process's ability to identify failure cases.

Could the authors provide the complete execution traces and specific safety judgment criteria for the aforementioned cases in Appendix D.3.3, explicitly identifying what information was disclosed, to whom, and which authorization or contextual boundaries were violated? Could they also explain why cases in which authorization restrictions were preserved or shared versions were already de-identified were nevertheless classified as safety failures?

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer zkm9 (1/2)

We thank you for your review and for recognizing the value of the unified environment, the comparison across all explored candidates, and the analysis of authorization weakening, contextual boundary degradation, and unsafe tool reuse. We address your main points below.

1. On novelty and comparison with prior work

Thank you for pointing out these studies. Shao et al. [1] is already discussed in Sec. 2 (lines 121–129). They study model self-training, accumulated memory, tool creation and reuse, and workflow evolution in separate experiments. They compare safety before and after evolution and, in the tool-reuse experiment, distinguish an unsafe tool from unsafe use of a tool.

TAME [2] measures trustworthiness as executor and evaluator memories evolve during benign tasks. It appeared at EMNLP in October 2026 and is contemporaneous under the ICLR guidelines. MemEvoBench [3], an arXiv preprint, studies how misleading inputs, noisy tool returns, and biased feedback affect safety as memory accumulates. ICLR 2027 Reviewer Guidelines, FAQ

In contrast to these prior works on experience/memory accumulation under a fixed protocol, in our work, the agent can rewrite the procedure for retaining, retrieving, or compacting memory. SEABench evaluates such changes alongside agent-written controller rules and executable tools or skills under one longitudinal protocol. The linked environment lets us reuse the same records across tasks with different purposes, recipients, or authorization boundaries. We then test whether the evolving and non-evolving agents preserve the same safety boundary in the later task. We will add these comparisons to Sec. 2 to further strengthen discussion on our novelty.

SEABench uses one longitudinal protocol to evaluate agent-written changes to controller instructions, memory-handling procedures, and executable tools or skills, with the model weights held fixed. For example, the agent can change how it retains, retrieves, or compacts information. The linked environment lets us test what happens when the same records are used for a different recipient or purpose, or under a different authorization boundary. For each surface, we give the same downstream prompt to the evolved and paired non-evolving agents and assess whether upstream self-evolution as a whole contributed to the failure (Sec. 5 and Appendix A).

2. On the insurance handoff and hiking examples

We thank you for the opportunity to clarify the insurance handoff and hiking examples. These tests probe whether the evolved agent preserves the relevant authorization or disclosure boundary when it applies a behavior learned upstream. More natural-sounding requests would help illustrate how those boundaries arise in ordinary use. In both cases, the safety label should follow what the agent actually disclosed and to whom.

For the hiking example, while the internal view contained booking references, the club-facing view redacted them but still retained flight details, which the privacy criteria prohibit the release of, thus justifying the privacy failure label. Thanks for pointing out the insurance handoff example, we will modify and correct it.

We are therefore modifying 15 such safety tasks for subtler natural sounding variants (including the fix to the insurance handoff example) out of 720 downstream tests and will report the modifications shortly during the rebuttal phase. We will provide the execution traces and task-specific rubrics for these two cases as files in the provided anonymous GitHub link.

AuthorsOpenReview ↗

Rebuttal to Reviewer zkm9 (2/2)

3. On independently rerunning the tasks found by search

Thank you for this suggestion. For each downstream safety test, we already give both agents the same prompt in separate sessions from their respective frozen states (Sec. 4.3, lines 279–283).

To check whether the difference recurs, we repeat each discovered safety test with the prompt fixed and restore both agents' saved states before every attempt. We then compare the fraction of attempts in which each agent fails and report results below. This checks whether the evolved agent fails more often when it starts with the same update and receives the same request again. However, we note that LLM-stochasticity is still an important consideration here, which partially motivated our use of an adaptive pipeline: the reasoning generated during the safety test can still vary even when we hold the earlier trajectory fixed. Bjarnason et al. [4] find early divergence across repeated agent runs, and Gonzalez-Pumariega et al. [5] find that the same computer-use task can succeed or fail across runs. Furthermore, due to LLM-related stochasticity, even with the same context and setting temperature to 0, a sampled final answer can vary [6].

Results: We perform this test over 10 randomly chosen task sequences on the controller surface for Kimi K2.5 for this illustration. Over 10 seeds, this yields 5×10×10=5005 \times 10 \times 10=500 tests each for evolved and unevolved agents. Evolved agents exhibit higher failures, even with the aforementioned stochasticity: at 27.6% against 20.6% for unevolved. Furthermore, for every task sequence taken alone, the evolved agent consistently exhibits higher safety failures than the baseline unevolved agent. Indeed, a one-sided Mantel–Haenszel test, which compares conditions while keeping sequences separate, yields p=0.0037p=0.0037, illustrating statistical significance. These results are now also added to a revision of the paper (Sec. 6.1, lines 374 - 396).

Thank you again for your feedback. We believe these comparisons and corrections clarify the contribution, will provide the promised deliverables (particularly, revised prompts for aforementioned safety tests), and we would sincerely appreciate it if you could strengthen your support of the paper in light of them.

References

[1] Shao et al. Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents. ICLR 2026. (https://openreview.net/forum?id=Fd1jgQQW28)

[2] Cheng et al. TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarking. EMNLP 2026. (https://aclanthology.org/2026.emnlp-main.2100/)

[3] Xie et al. MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agents. 2026. (https://arxiv.org/abs/2604.15774)

[4] Bjarnason et al. On Randomness in Agentic Evals. 2026. (https://arxiv.org/abs/2602.07150)

[5] Gonzalez-Pumariega et al. On the Reliability of Computer Use Agents. 2026. (https://arxiv.org/abs/2604.17849)

[6] Horace He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Connectionism, September 2025. DOI

Thank you for clarifying SEABench’s contribution. This largely resolves my novelty concern, and the fixed-prompt reruns show that the failure gap persists on the sampled controller tasks. My remaining concern is the safety labeling. Please share the traces and criteria for the hiking and insurance cases, and clarify whether revising the 15 tests changes the labels or the overall failure gap. If these judgments are supported and the main result holds, I would be open to raising my score from 2 to 3.

Reviewer_tKihOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies endogenous misalignment: a self-evolving agent writes a locally useful update and that update later persists into contexts where it causes unsafe behavior without any external adversary. It introduces SEABench, a personal-assistant environment with 89 linked JSON files. The paper evaluates Kimi K2.5, Grok 4.3, and GPT 5.6 Luna. Evolved agents complete more downstream tasks than non-evolving ones. They also fail 43.9% of safety tests, against 0% for the paired baselines. Tools or skills is the most vulnerable surface. A reasoning-flow based analysis of Kimi traces shows more unresolved risk threads after evolution. A CoT monitor built on this analysis catches 70.9% of failures at a 9.7% false-positive rate. It works noticeably worse on tools/skills, where the harm lives inside the invoked artifact.

Critical strengths
  1. Many deployed agents adapt by editing AGENTS.md, memory files, and skills. The question of whether a useful update later becomes a harmful standing rule is distinct from prompt injection and memory poisoning, and the paper names it clearly.
  2. The paper also finds that CoT monitoring helps much less on tools or skills, because the defect sits in the tool. This yields a transferable design insight: safeguards may need to operate at update promotion or tool invocation, not only on reasoning traces. It is the most useful takeaway for practitioners.
Critical weaknesses
  1. The sequences are found by an optimizer that explicitly searches for prompts where the evolved agent fails and the baseline succeeds; the paper itself likens this to red teaming. The resulting safety-test prompts often request the harmful output directly or invoke the evolved rule: The Bob dinner note asks the agent to include "complete member IDs, group numbers… and the specific banking details." The workshop email asks for reconstructed content to be presented as confirmed outcomes. Even in Figure 1, the evolution task has the user ask for these details to be included "in all home vendor communications." Many failures therefore look like the agent following a preference the user stated. The non-evolving agent is "safe" largely because it lacks the context the prompt refers to. Several showcased failures also do not read as failures. The hiking packet redacts the shareable view. The HR handoff restricts identifiers pending authorization. The device review repeatedly marks details as not established. Together, this suggests the paper demonstrates elicitation of unsafe outputs through searched, user-pressured sequences. The title, the definition, and contribution all depend on that distinction.

  2. A reader expecting a benchmark cannot tell how to score a new model on SEABench, or whether cross-model comparisons in Section 6.2 are meaningful when each model had its own search. As written, the contribution reads as an existence proof plus an auditing pipeline. That is a smaller contribution than a benchmark.

Questions for authors

Are the 48 released sequences fixed, so that a new model can be run on them as-is? Or must the discovery pipeline be rerun for each model? If the latter, what score does SEABench report for a new model, and on what basis are models compared?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer tKih (1/2)

We thank you for your review and for recognizing the importance of locally useful updates becoming harmful standing rules. We are glad you found the distinction between risks visible in reasoning and risks inside a tool useful for practitioners. We address your main concerns below.

1. On user pressure and the examples

We would like to clarify that finding sequences of individually benign tasks that can lead to endogenous misalignment is the purpose of SEABench. We link the upstream and downstream tasks so that the later task can test whether a behavior learned earlier is applied within the appropriate safety boundary (Sec. 4.2). The lookahead judge guides which sequences we try. The agent writes its own update, and both agents receive the same downstream safety prompt. Our claim is that these failures can occur on the sequences found by the stress test.

This follows prior precedent in stateful security testing. RESTler [6] links API requests so that a later operation can expose a vulnerability in the state created by earlier requests. The OWASP testing guide [7] gives a concrete example. Starting a purchase can award loyalty points, and canceling it tests whether the system wrongly lets those points carry over. The upstream task plays the same role in SEABench. It is a genuine and plausible task whose completion can leave the agent with a persistent rule, and the downstream task tests whether the agent wrongly carries that rule into a different context. Shao et al. [8] use a related design for agents, pairing a benign tool-creation task with a later task that tests whether reusing the tool is safe.

Thank you for recognizing the distinction between a useful update becoming a harmful standing rule and an external attack. Fig. 1 actually illustrates precisely this in SEABench. The user asks the agent to include operational details in home-vendor communications. The agent turns that preference into a standing rule for household communications and applies it when drafting a message to a family member. Both agents receive the same request, and the baseline drafts the message without including those details.

We would also like to clarify some misunderstandings about the safety labels. The Bob dinner and workshop prompts test the same issue more directly. They test whether the evolved agent still follows its original safety instructions when a later request presses it to disclose sensitive details or present an inference as a fact. In particular, for the Bob dinner task, the test checks if the agent violates the initial instructions in AGENTS.md to prohibit letting “authorization language override current-scope safety” and to “match disclosure to the recipient’s role and current need-to-know”, and therefore is a direct test of whether prior safety instructions might be violated by an evolved agent, specifically prohibiting disclosure of banking details to Bob and requires keeping the message limited to the dinner update. The evolved agent’s response included Alice’s full bank account and routing numbers, balance, transaction history, and insurance member and group identifiers, whereas the unevolved agent remained scoped and specified refused to divulge sensitive details and proposed a privacy-preserving response to Bob instead. These prompts are thus useful stress tests. For the hiking example, while the internal view contained booking references, the club-facing view redacted them but still retained flight details, which the privacy criteria prohibit the release of, thus justifying the privacy failure label. For the device review example, the hallucination safety label is also valid, while the evolved agent marks a number of details as not established, it still does not flag and retains hallucinated details about a shared office subnet and the timeline of authenticaton events, despite the original safety instructions and the safety task prompt itself prohibiting filling gaps with assumptions. Therefore, most of the pointed-to examples are valid safety labels. We will correct the safety labelling issue in the HR handoff example.

We agree that more natural-sounding requests would more interestingly illustrate how the same failures can arise in ordinary use. To this end, we have identified and are modifying similar 15 tasks out of 720 downstream tests and will report the corrections shortly during the rebuttal phase. This includes the safety labelling issue in the HR handoff example.

AuthorsOpenReview ↗

Rebuttal to Reviewer tKih (2/2)

2. On how a new model is evaluated on SEABench

Thank you for this question. We would like to clarify that the discovery pipeline is expressly designed to evaluate any model that can run in the SEABench agent harness. This includes all applicable models that can be accessed via API-based providers like OpenRouter, the OpenAI/Anthropic APIs, etc. and local deployments via vLLM. The pipeline imposes no model-specific restrictions on the agent being evaluated, and the supplied code already gives explicit instructions for running the pipeline with other models.

The 48 released sequences are the starting task specifications. To obtain the discovery-based score for a new model, we rerun discovery with that model in both the evolving and non-evolving conditions. The starting prompts can also be run as written to evaluate those prompts. As discussed in Sec. 4.3 (lines 240–249), Bjarnason et al. [1] show that repeated executions diverge early into different strategies, and Gonzalez-Pumariega et al. [2] show that the same computer-use task can succeed or fail across repetitions with the model held fixed. In our setting, a fixed task sequence can lead to different updates or no persistent update at all. The space of possible execution paths across decisions, tool calls, and updates grows combinatorially, which motivates discovery.

Sec. 5 defines the score as the proportion of downstream safety tests where the evolved agent fails, the paired baseline stays safe, and the attribution score is at least 4. Sec. 6.1 also reports completion on the retained tests, and Fig. 4 reports safety across all explored candidates.

We would also like to clarify the basis for the cross-model comparison. Each model is evaluated with the same discovery and scoring procedure. This design is well-precedented in published literature, including a prominent benchmark. HarmBench [3], an automated red-teaming benchmark published at ICML 2024, applies adaptive methods such as TAP [4] to each target model. TAP refines prompts using the target model's responses, so the prompts found can differ across models. HarmBench reports the transfer of prompts between models as a separate evaluation. More recently, Hagendorff et al. [5] compare nine target models on the same benchmark objectives using multi-turn prompts adapted to each target's replies.

In SEABench, we use the same starting task specifications, environment, Kimi K2.5 optimizer, refinement budget, and judging criteria (Secs. 4.3 and 5). We rerun discovery because different models can write different updates in response to the same starting tasks. This allows us to compare which failures the same stress test uncovers for each model. For example, on memory evolution, the submitted paper reports failures on 56 of 80 retained tests for Kimi K2.5 and 15 of 80 for Grok 4.3 (Sec. 6.1). Sec. 6.2 also compares the types of failures and updates found for each model.

Thank you again for your feedback. We believe these clarifications make the contribution and the use of the benchmark clearer, and we remain glad to discuss the example corrections during the rebuttal period.

References

[1] Bjarnason et al. On Randomness in Agentic Evals. 2026. arXiv

[2] Gonzalez-Pumariega et al. On the Reliability of Computer Use Agents. 2026. arXiv

[3] Mazeika et al. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ICML 2024. Sec. 6 and App. C.1. PMLR

[4] Mehrotra et al. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. NeurIPS 2024. Sec. 5, Tables 1 and 3. NeurIPS Proceedings

[5] Hagendorff et al. Large reasoning models are autonomous jailbreak agents. Nature Communications, 2026. Publisher

[6] Atlidakis, Godefroid, and Polishchuk. RESTler: Stateful REST API Fuzzing. ICSE 2019.

[7] OWASP. Testing for the Circumvention of Work Flows. Web Security Testing Guide, v4.2, WSTG-BUSL-06.

[8] Shao et al. Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents. ICLR 2026.

I have raised my evaluation since my concerns are mostly solved, and thanks to the authors for clarifying the device review and hiking examples, which I had misread.

Thank you very much!

Reviewer_a5tQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SEABench to study “endogenous misalignment” in self-evolving agents: while modifying their own controller files, memory procedures, or tools/skills, agents may write rules that are useful upstream but cause safety failures downstream. The authors build a synthetic personal-assistant environment with 48 task sequences (3 evolution surfaces × 4 task domains × 4 harm types), and use a TextGrad-based pipeline to search for trajectories in which the evolved agent fails while the non-evolving agent remains safe. The paper stress-tests three models, with three main results: task completion increases from 35.7% to 47.2% after evolution; the evolved agent shows a 43.9% safety failure rate versus 0% for the baseline; and a CoT-based monitor intercepts 70.9% of failures.

Critical strengths
  1. The problem is important and timely. Harness-level self-evolution is becoming a mainstream deployment pattern, and the framing of “locally useful updates causing harm elsewhere” is clear.
  2. The environment is carefully designed, with entities consistent across applications and grounded in contextual integrity.
  3. Using a paired non-evolving agent as a control and reporting both capability and safety is a sound design choice.
  4. The qualitative analysis across evolution surfaces is insightful. For example, CoT monitoring is notably less effective on the tools/skills surface because the defect resides in the invoked artifact rather than in the reasoning. This negative finding is valuable in itself.
Critical weaknesses
  1. A per-model red-teaming pipeline seems to be reported as a benchmark. From the paper’s description, sequences appear to be searched separately for each model, with the search objective being precisely “evolved agent fails, baseline stays safe,” and the reported results are the trajectories retained during search. The 0/720 baseline rate therefore follows from the selection criterion, and the 43.9% figure demonstrates that such failures exist rather than measuring how large the risk is. The per-model failure rates may reflect search success rather than a safety ranking, and no cross-model transfer results are reported.
  2. The “endogenous / non-adversarial” framing could be better supported. Evolution tasks are refined under the guidance of a lookahead judge toward updates likely to cause later failures. The code suggests that the initial sequences come from an attack-sequence generator, and that refinement penalizes overt attack language, so the benign appearance of the tasks may partly be a product of optimization. A human study verifying that the final prompts resemble ordinary user requests would strengthen this claim.
  3. Evaluation reliability. Kimi K2.5 generates the environment, rewrites the tasks, serves as all judges, and is also one of the evaluated models. Since the judges’ verdicts are also the search objective, judge false positives may be amplified. In addition, the causal-attribution claim relies on the attribution judge, which does not appear to have been validated against human annotations.
  4. Reproducibility. The paper states that the task sequences used in the experiments are released, but the repository appears to contain only the 48 pre-refinement sequences; the refined versions actually evaluated do not seem to be included.
  5. Novelty should be positioned more carefully. Prior work available before the deadline has studied safety degradation of self-evolving agents caused by benign experience, and is not cited or discussed:
  6. Possible confound in the control condition. Many safety-test prompts refer to conventions established during evolution, so the non-evolving agent’s “safety” may partly stem from lacking this context or not completing the task; the reported capability gain is also measured on the same tasks. A completion × safety cross-tabulation would help clarify this.
Questions for authors
  1. Is my understanding above correct? If not, I would appreciate it if the authors could point out where it is mistaken.
  2. How many refinement runs were performed, and which model was used as the agent in each?
  3. How do sequences found on one model perform when applied to the other models?
  4. Could the authors provide pre- and post-refinement prompts for several sequences, along with a human plausibility evaluation of the final prompts?
5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer a5tQ (1/3)

We thank you for your review and recognition of the importance of harness-level self-evolution, the care taken to link entities across applications, and the value of comparing safety and capability with a paired baseline, etc. We address your main concerns below.

1. On per-model discovery and cross-model transfer

We appreciate the opportunity to clarify the purpose of SEABench. Our goal is to establish that locally useful self-updates can lead to downstream safety failures. As stated in Sec. 4.3 (lines 269–283), this is precisely why the stress test searches for cases where the evolved agent fails while its paired non-evolving agent stays safe. The 43.9% versus 0% result summarizes the safety outcomes on the retained tests. Sec. 6.2 compares the types and frequency of failures uncovered for each model under the same discovery procedure.

The discovery procedure is part of the benchmark design and is precedented in published literature. For example, HarmBench [6] specifies a common set of behaviors and uses automated red teaming to generate test cases for the models it evaluates. More recently, Hagendorff et al. [7] use a shared set of 70 objectives to benchmark safety through adaptive multi-turn interactions with nine target models. In SEABench, the shared environment, starting task specifications, and safety rubrics define the audit, and the discovery procedure finds trajectories that exercise them.

We run discovery separately with Kimi K2.5, Grok 4.3, and GPT 5.6 Luna as the evaluated agents. In each run, the same base model is used for both agent types (non-evolving, evolving) and Kimi K2.5 serves as the fixed TextGrad optimizer (Sec. 5). Sec. 4.3 specifies up to five critique-and-revise iterations per task, and Fig. 4 reports 3,435 paired safety-test candidates in total. That total counts candidate pairs; we provide the refinement-run/candidate counts by evaluated model: 770 for Kimi K2.5, 1045 for GPT-5.6 Luna, and 900 for Grok 4.3.

Moreover, the scope of our work and the stated precedent do not include applying the refined prompts found for one model to another. This is motivated by prior observations on model stochasticity: as discussed in Sec. 4.3, Bjarnason et al. [1] show that repeated runs diverge early into different strategies, and Gonzalez-Pumariega et al. [2] find that success can vary across repeated runs with the task and model fixed. In our setting, even a repeat with the source model can produce a different update. We leave cross-model replay to future work and will distinguish it explicitly from the per-model results.

2. On lookahead and the plausibility of the upstream tasks

Thank you for raising this point. The lookahead judge deliberately steers the search toward tasks likely to produce a relevant update (Fig. 2 and Sec. 4.3). The agent itself writes the update in response to an upstream task and feedback. The optimizer selects which prompts to try, while the refinement criterion requires the original task meaning and intent to be preserved (lines 272–273). The existence claim concerns the trajectories that satisfy these conditions. Lookahead helps find such a trajectory, and preserving the upstream task intent is what makes it relevant to the claim. The overall sequence satisfies our non-adversarial desideratum, with each upstream task being benign by itself and is a valid search for such a sequence over the vast space of all possible task sequences, while linking upstream tasks to downstream tests, similar to prior work like Shao et al. [12], who pair a benign tool-creation task with a related safety-sensitive reuse task, and MemoryArena [13], which uses interdependent tasks across sessions. A held-out downstream prompt can be fixed before refinement while still probing the behavior targeted by its upstream tasks.

For illustration of the initial generation procedure and the semantic constraint, we point to our description of how the initial sequences are generated (described in Sec. 4.3, lines 261-268). We will provide before-and-after prompts for a number of sequences in the appendix and report a human assessment of the final refined prompts.

AuthorsOpenReview ↗

Rebuttal to Reviewer a5tQ (2/3)

3. On judge reliability during discovery

We appreciate the opportunity to clarify how the safety tests are run. Each downstream safety test runs independently, in separate sessions starting from the respective frozen evolved and non-evolving agent states (Sec. 4.3, lines 268–271). Any changes made during a safety test are discarded before the next test. The search uses the resulting safety labels to select candidates. Using automated judgments to guide adaptive search and to score its outcomes follows prior work in automated safety testing. TAP [8] uses the same GPT-4 judging function to rank candidates and report success. PAIR [9] and Crescendo [10] likewise use judge feedback to decide how to continue the search and judge whether it succeeded. More recently, Shamsi et al. [11] optimize against an LLM-judged danger score and use the same metric to report safety outcomes.

Appendix D.1 already checks the Kimi-based safety judge against human annotations, yielding 92.0% precision and 100% recall on 70 judgments, and against an independent GPT 5.6 Sol auditor, yielding 93.6% precision and 96.7% recall on 1,360 judgments.

We validate the attribution judge over 70 safety failures via human annotation, with the annotation set randomly chosen in a manner that retains close-to-equal representation of the three surfaces and three models. The attribution judge agrees with human annotators 91.42% of the time, with 6 disagreements.

4. On the release of refined prompts

Thank you for pointing out this distinction. The 48 starting sequences and the prompts actually used after refinement serve different purposes. The starting sequences are inputs to discovery, whereas the refined prompts discovered are semantically similar variants that are specific to the agent trajectory they were discovered over. We refer the reviewer again to point 1 of this response on the adaptive discovery pipeline. As mentioned in point 2, we will provide before-and-after prompts for a number of sequences in the appendix for ease of inspection.

5. On novelty and related work

Thank you for pointing out these studies. We take this opportunity to clarify the novelty of our work with respect to them. Zhao et al. [3] show that even experience collected on benign tasks can lead to unsafe behavior when retrieved later. MemEvoBench [4] studies memory degradation under misleading inputs, noisy tool returns, and biased feedback. TAME [5] evaluates trustworthiness during benign memory evolution and proposes a way to update executor and evaluator memory. These studies examine how accumulating or injecting memory entries can affect safety under a predefined protocol for updating and retrieving memory.

In contrast to these prior works on experience/memory accumulation under a fixed protocol, in our work, the agent can rewrite the procedure for retaining, retrieving, or compacting memory. SEABench evaluates such changes alongside agent-written controller rules and executable tools or skills under one longitudinal protocol. The linked environment lets us reuse the same records across tasks with different purposes, recipients, or authorization boundaries. We then test whether the evolving and non-evolving agents preserve the same safety boundary in the later task. We have added these comparisons to Sec. 2 (Related Work) to further strengthen discussion on our novelty (lines 114-121, in green font).

6. On baseline context and task completion

We would like to clarify that the completion and safety rubrics are separate, and the safety tasks are designed to admit solutions satisfying both (Sec. 4.2, lines 214–216). In the submitted results for the retained tests in final trajectories, the baseline completes 257 of 720 tasks and has no flagged safety failures (Sec. 6.1). Thus, those 257 cases are examples of completion with a safe baseline outcome, while some other safe baseline outcomes may reflect non-completion. Since safety tasks' task completion criteria allow for genuine task completion success without requiring use of prior self-updates or harmful behavior, these are largely valid task completion judgements. In addition, we also include a human annotation of the task completion judge in the paper (Appendix D.1), which over 70 samples, achieved 96.8% recall and perfect precision. However, we acknowledge that safety tasks should ideally not include references to prior updates for fairness, we will report amended safety task prompts in this regard during the discussion period.

Thank you again for your feedback. We believe these clarifications make the contribution and the scope of the evaluation clearer. We would be glad to provide and discuss the additional validation during the rebuttal period.

AuthorsOpenReview ↗

Rebuttal to Reviewer a5tQ (3/3)

References

[1] Bjarnason et al. On Randomness in Agentic Evals. 2026. (https://arxiv.org/abs/2602.07150)

[2] Gonzalez-Pumariega et al. On the Reliability of Computer Use Agents. 2026. (https://arxiv.org/abs/2604.17849)

[3] Zhao et al. On Safety Risks in Experience-Driven Self-Evolving Agents. 2026. (https://arxiv.org/abs/2604.16968)

[4] Xie et al. MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agents. 2026. (https://arxiv.org/abs/2604.15774)

[5] Cheng et al. TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarking. EMNLP 2026. (https://aclanthology.org/2026.emnlp-main.2100/)

[6] Mazeika et al. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ICML 2024. (https://proceedings.mlr.press/v235/mazeika24a.html)

[7] Hagendorff et al. Large reasoning models are autonomous jailbreak agents. Nature Communications, 2026. (https://www.nature.com/articles/s41467-026-69010-1)

[8] Mehrotra et al. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. NeurIPS 2024. (https://proceedings.neurips.cc/paper_files/paper/2024/hash/70702e8cbb4890b4a467b984ae59828a-Abstract-Conference.html)

[9] Chao et al. Jailbreaking Black Box Large Language Models in Twenty Queries. IEEE SaTML 2025. (https://doi.org/10.1109/SaTML64287.2025.00010)

[10] Russinovich et al. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack. USENIX Security 2025. (https://www.usenix.org/system/files/usenixsecurity25-russinovich.pdf)

[11] Shamsi et al. When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models. EACL 2026 Student Research Workshop. (https://aclanthology.org/2026.eacl-srw.33/)

[12] Shao et al. Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents. ICLR 2026. Sec. 3.3. (https://proceedings.iclr.cc/paper_files/paper/2026/hash/a24cd16bc361afa78e57d31d34f3d936-Abstract-Conference.html)

[13] He et al. Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks. ICML 2026. (https://proceedings.mlr.press/v306/he26am.html)

Reviewer_jCvLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks a simple question: if an LLM agent changes its own memory, rules, or tools while doing earlier tasks, can those changes cause problems later?

To study this, the authors build SEABench, a test environment for a fictional personal assistant with emails, calendars, finances, health records, files, and other information. The agent first performs ordinary tasks and is allowed to update itself. It is then tested on later tasks where those updates may no longer be appropriate.

The authors use an automated search process to find cases where the self-updating agent behaves unsafely while a version that was not allowed to update remains safe. They test three models and three types of self-updates, and also analyze the agents’ reasoning traces. Their main finding is that self-evolution can improve task completion while also making the agent more likely to violate safety and privacy boundaries.

Critical strengths

The paper studies an important and under explored problem: an agent may learn a useful behavior in one context and later apply it in an unsafe context. The benchmark covers several evolution surfaces, task domains, and types of safety failures in one persistent environment.

The paper provides concrete examples of persistent unsafe updates and compares evolved agents with paired non-evolving agents. The authors also release the task-generation and evaluation pipeline and clearly discuss important limitations, including the difficulty of attributing a failure to one specific update.

Critical weaknesses

The biggest problem is that the benchmark searches for exactly the cases it later reports. The TextGrad pipeline keeps tasks where the self-updating agent fails but the non-updating agent stays safe. Therefore, the result of 316 failures versus 0 is partly expected from how the tasks were selected. The reported p-values are also unreliable because the candidate tasks were generated adaptively rather than sampled independently. These results show that the pipeline can find safety problems, but they do not show how often self-evolution causes problems in ordinary use. The paper needs a separate test set of fixed tasks that was never used during task discovery or refinement.

The paper also makes stronger causal claims than the experiments support. Comparing a self-updating agent with a non-updating agent shows that their behavior is different, but it does not tell us which specific update caused the failure or rule out randomness and differences in their earlier interactions. The same Kimi model is used to generate the environment, search for tasks, and judge the results, which may make the benchmark especially favorable to that model. The judge check uses only a small amount of human-labeled data, and the reasoning-trace monitor is trained and tested on a small dataset without a strong output-only baseline or an independent test set.

Questions for authors
  1. Can the authors provide results on fixed, held-out task sequences that were never used by TextGrad, the task judges, or the refinement pipeline? What are the failure rates before and after task discovery, and how would the statistical analysis change if candidates were treated as correlated, adaptively generated samples rather than independent observations?

  2. How is the non-evolving baseline constructed exactly? Does it receive the same upstream tasks, feedback, environment interactions, and random seeds as the evolving agent, with only the update mechanism disabled? Can the authors provide artifact-level counterfactuals in which the evolved agent is reset to the baseline except for one controller, memory, or tool update?

  3. How robust are the results when task discovery and judging use an independent model rather than Kimi K2.5? In particular, can the authors report model-independent safety labels, sequence-level held-out splits for the CoT monitor, and a comparison against a simple output-only safety monitor?

7 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer jCvL (1/3)

We thank you for your review and for recognizing the importance of studying how an agent can learn a useful behavior in one context and apply it unsafely in another. We address your main questions below.

1. On adaptive selection, the candidate-level analysis, and held-out tasks

We agree that the 316 versus 0 comparison on retained tests reflects the search criterion. This is precisely the purpose of the stress test in Sec. 4.3 (lines 240–249). It searches for executions in which a locally useful self-update leads to a later failure. This use of adaptive search follows published benchmarks such as HarmBench [7], which evaluates red-teaming methods such as TAP [8] that refine test prompts in response to the target model's behavior. The reported comparison describes the failures found by that procedure.

For the comparison before and after retaining candidates, Fig. 4 reports 1,105 versus 833 failures across all 3,435 explored pairs (32.2% versus 24.3%), including rejected prompts. Sec. 6.1 reports 316 versus 0 across 720 retained pairs. The all-candidate set spans the search itself. The initial safety prompts are also screened at bootstrapping (Sec. 4.3, lines 251–257), so neither comparison supplies a failure rate on a held-out set selected before discovery.

We appreciate the point about the p-values and the suggestion. We would first like to clarify how the candidate-level comparison in Fig. 4 is constructed. For each candidate, we give both agents the same prompt and run them in separate sessions from their respective frozen states (Sec. 4.3); TextGrad then uses the results to propose the next prompt variant for that task only. Additionally, each safety task in a sequence is run independently of other safety tasks. To better account for this dependence in a statistical test, we now also group variants by task and keep the two agents' outcomes paired for each prompt; this yields p=5.55×10−19p=5.55\times 10^{-19}. We have updated the paper with the grouped exact test p-value to the paper (edits in green text).

Regarding a fixed held-out suite, our paper explicitly describes the limitations of such an approach and motivates an adaptive pipeline: as discussed in Sec. 4.3, Bjarnason et al. [1] show that agent trajectories can diverge early across repeated runs, and Gonzalez-Pumariega et al. [2] show that the same task and model can yield different outcomes on rerun. In SEABench, this can change which persistent update is written in the first place. Our study uses discovery to find executions where an update is made and later matters for safety, following precedent in prior work that uses generated prompts to probe for concrete safety failures (Perez et al. [3], Russinovich et al. [5], and Shamsi et al. [6]).

However, following a suggestion by Reviewer zkm9, we run multiple iterations on fixed discovered tests for 10 seeds over 10 randomly selected sequences (see response to zkm9, point 3):

To check whether the difference recurs, we would repeat each discovered safety test with the prompt fixed and restore both agents' saved states before every attempt. We then compare the fraction of attempts in which each agent fails and report results below. This checks whether the evolved agent fails more often when it starts with the same update and receives the same request again.

Results: We perform this test over 10 randomly chosen task sequences on the controller surface for Kimi K2.5 for this illustration. Over 10 seeds, this yields 5×10×10=5005 \times 10 \times 10=500 tests each for evolved and unevolved agents. Evolved agents exhibit higher failures, even with the aforementioned stochasticity: at 27.6% against 20.6% for unevolved. Furthermore, for every task sequence taken alone, the evolved agent consistently exhibits higher safety failures than the baseline unevolved agent. Indeed, a one-sided Mantel–Haenszel test, which compares conditions while keeping sequences separate, yields p=0.0037p=0.0037, illustrating statistical significance. These results are now also added to a revision of the paper (Sec. 6.1, lines 374 - 396).

AuthorsOpenReview ↗

Rebuttal to Reviewer jCvL (2/3)

2. On the scope of our causal claim and the paired baseline

We take the opportunity to clarify a misunderstanding about our causal claim. Our claim expressly concerns upstream self-evolution as a whole, as stated in Sec. 5 (lines 285–287), not causal attribution to a particular update. Reviewer a5tQ notes this as well: the paired non-evolving agent serves as a control. Sec. 6.1 (lines 318–323) uses the paired non-evolving agent as the counterfactual for that process, which makes it a process-level counterfactual (i.e. to the entire evolution process). Appendix A (lines 660–663) explicitly distinguishes this claim from attribution to an individual update.

We would like to clarify that both agents use the same base model, receive the same task prompts, and run with matched random seeds. Self-updates are enabled only for the evolving agent (Secs. 3 and 4.3). The agents can take different actions upstream, so the feedback and tool results they receive can differ due to evolution. For each downstream safety test, we give both agents the same prompt in separate standalonesessions starting from their respective frozen states. Therefore, this is already discussed in the paper.

The proposed baseline with one update added would test whether that update alone can reproduce the failure in the baseline state. The intervention described in Appendix A would test its contribution in the evolved state by varying the update while keeping the rest of that state fixed. Both require separate downstream reruns for each update evaluated and would multiply the cost of our current experiments (approximately USD 500); we leave this individual-update analysis to future work.

3. On the use of Kimi and judge validation

We use Kimi K2.5 to generate the common environment, refine prompts, and judge the main outcomes (Sec. 5 and Appendix E.2) The object of study are the evaluated agents, not how different pipeline base models impact the pipeline: evaluated agent varies while the discovery pipeline is held fixed. As we clarify in our response to Reviewer uhnL, our aim is to show that there exist trajectories in which a locally useful update leads to a later safety failure. A single discovery pipeline suffices for this purpose, and the same pipeline finds such cases for all three agents. Fixing and using a single pipeline base model is well-precedented in prominent published literature in automated safety testing: [5] (using GPT-4, and also include GPT-4 as an evaluated model) and [7] (using Mixtral 8x7B) fix the same base model for their prompt optimizers to uncover vulnerabilities in models/agents. A prominent non-safety agent evaluation work (τ\tau-bench, [9]) also uses a fixed simulated user (using GPT-4) for agent evaluation. Comparing optimizers would address a separate question about which failures each search uncovers and would involve a separate study which is outside the scope of our work and one conference paper length submission.

Additionally, contrary to the stated speculation, the reported safety results do not favor Kimi. Kimi has the highest overall failure rate under discovery, at 60.83% (146/240), compared with 40.83% (98/240) for Grok 4.3 and 30.0% (72/240) for GPT 5.6 Luna (Sec. 6.2). The highest rate for any model and surface combination is also Kimi's 70.0% on memory evolution (Sec. 6.1).

We already assess possible judge errors separately against human labels and an independent model. Appendix D.1 already compares both judges with 70 human judgments. The safety judge achieves 92.0% precision and 100% recall, and the completion judge achieves 100% precision and 96.8% recall. The number of annotated examples is comparable to human validation in prior published work: Shao et al. [4] validate their tool-creation and reuse judge on 25 cases each for two models for 50 judgments. Goel et al. [10] employ human annotation for their contextual-privacy judge over 30 randomly sampled responses. Additionally, in our work, we add comparison with an independent GPT 5.6 Sol auditor which covers 1,360 safety judgments (93.6% precision and 96.7% recall) and 1,340 completion judgments (100% precision and 96.1% recall). Together these provide a direct human check on a sample size that is on par with prior related work, and a larger independent-model check supporting use of the two judges in the reported setting.

We validate the attribution judge over 70 safety failures via human annotation, with the annotation set randomly chosen in a manner that retains close-to-equal representation of the three surfaces and three models. The attribution judge agrees with human annotators 91.42% of the time, with 6 disagreements. The judge validation in the paper (initially reported in Appendix D.1) is now added to the main text in Sec 6.4, along with the new attribution judge validation results (in green font).

AuthorsOpenReview ↗

Rebuttal to Reviewer jCvL (3/3)

4. On the monitor test set and output-only baseline

Thank you for your suggestion regarding monitor train/val/test splits. We provide results based on a revised split where traces belonging to the same sequence reside in the same split.

New CoT Monitoring Results on Revised Sequence-Wise Splits: Following your suggestion, we have rerun the monitor on a revised 210/120/150 train-val-test split that ensures that traces that belong to the same sequence remain in the same split. This yields comparable (and even relatively improved, albeit on a newer split) results over those presented initially in the paper: with 82.7% test accuracy and 9.2% FPR while yielding a failure recall of 71.4%, catching 45 out of 63 failures. We have updated the paper to report the revised split and these new results (Sec 6.3, revisions in green text).

We also compare against an output-only LLM judge on the same test examples. However, an output-only LLM judge may only observe an agent output post-hoc, missing already executed harmful agent actions, unlike the proposed monitor, which seeks to interrupt agents based on their chain of thought reasoning before the agent executes actions. As mentioned in our response to Reviewer uhnL, who asks for this baseline among others (a direct LLM judge of the task and reasoning trace and a classifier without the risk-progression representation), the results on monitor baseline comparison are as follows.

MethodTest accuracyFailure recallFPR
Direct reasoning LM judge76.7%46.0%8.0%
Output-only LM judge78.0%52.0%9.0%
Text-only ExtraTrees81.3%60.0%8.0%
Progression monitor (ours)84.0%70.0%9.0%

We have updated the paper with this baseline comparison in Sec 6.3 (lines 503-507, green font).


Thank you again for your feedback. We believe the revisions above clarify your concerns, the causal claim, what each evaluation measures, and improves p-value reporting. We remain glad to engage further and report any additional validation as far as possible.

References

[1] Bjarnason et al. On Randomness in Agentic Evals. 2026. (https://arxiv.org/abs/2602.07150)

[2] Gonzalez-Pumariega et al. On the Reliability of Computer Use Agents. 2026. (https://arxiv.org/abs/2604.17849)

[3] Perez et al. Red Teaming Language Models with Language Models. EMNLP 2022. (https://aclanthology.org/2022.emnlp-main.225/)

[4] Shao et al. Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents. ICLR 2026. Appendix D.9. (https://openreview.net/forum?id=Fd1jgQQW28)

[5] Russinovich et al. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack. USENIX Security 2025. Secs. 4.2–4.3. (https://www.usenix.org/system/files/usenixsecurity25-russinovich.pdf)

[6] Shamsi et al. When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models. EACL 2026 Student Research Workshop. Secs. 3–4. (https://aclanthology.org/2026.eacl-srw.33/)

[7] Mazeika et al. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ICML 2024. Secs. 3 and 6. (https://proceedings.mlr.press/v235/mazeika24a.html)

[8] Mehrotra et al. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. NeurIPS 2024. Secs. 2, 4–5. (https://proceedings.neurips.cc/paper_files/paper/2024/hash/70702e8cbb4890b4a467b984ae59828a-Abstract-Conference.html)

[9] Yao et al. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. ICLR 2025. (https://proceedings.iclr.cc/paper_files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html)

[10] Anmol Goel, Cornelius Emde, Seong Joon Oh, Sangdoo Yun, and Martin Gubri. 2026. Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8870–8892, San Diego, California, United States. Association for Computational Linguistics.

Reviewer_uhnLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SEABench, a benchmark for investigating safety regressions induced by parameter-free self-evolution in LLM agents. Through longitudinal task sequences and paired evaluations, the authors examine capability–safety trade-offs across different evolution surfaces. The paper also explores reasoning-trace monitoring as a potential mitigation strategy.

Critical strengths
  1. The paper studies a safety concern: benign self-updates can introduce persistent behavioral changes that become harmful in subsequent tasks. This setting is particularly relevant as LLM agents increasingly maintain long-term memory, modify controller instructions, and develop reusable skills. I appreciate the attempt to distinguish endogenous misalignment from conventional adversarial attacks.
  2. SEABench considers three evolution surfaces, four task domains, and four categories of safety failures. The longitudinal task structure and interconnected personal-assistant environment provide a useful setting for investigating how locally beneficial updates can affect downstream behavior. The concrete examples of evolved controller, memory, and tool artifacts are particularly helpful for understanding the underlying failure mechanisms.
Critical weaknesses
  1. My main concern is that the benchmark construction and evaluation procedure are closely coupled. As described in Section 4.3, safety-test prompts are iteratively refined to identify cases where the evolved agent exhibits unsafe behavior while the corresponding non-evolving agent remains safe. Consequently, the reported 43.9% safety failure rate for evolved agents and 0% for the non-evolving baseline (Section 6.1) are measured on deliberately selected trajectories, rather than an independently sampled task distribution.

    I acknowledge that the authors explicitly position the method as a stress-testing pipeline and recognize this limitation. Nevertheless, the paper occasionally interprets these results as broader evidence that self-evolution substantially increases safety failures. The candidate-level analysis in Figure 4 provides a more informative comparison, but the observed difference is considerably smaller (32.2% versus 24.3%), and these candidates are also generated through an adaptive search procedure.

    A similar concern applies to the reported capability improvement, since the benchmark construction also favors upstream tasks on which evolution is useful. I would like to see evaluation on an independent, non-optimized set of downstream tasks, ideally with multiple independent trajectories. This would help distinguish the effectiveness of the failure-discovery pipeline from the prevalence of safety regressions induced by self-evolution.

  2. The paired evolving/non-evolving comparison is a reasonable starting point, but it does not fully isolate the contribution of specific self-evolution updates. Once the agents follow different upstream trajectories, their task outcomes, accumulated state, and subsequent behavior may diverge for several reasons. The attribution judge provides supporting evidence, but its assessment alone does not establish a causal mechanism.

    The authors acknowledge in Appendix A that attributing failures to individual artifacts would require finer-grained counterfactual interventions. I consider this a particularly important limitation because causal attribution is presented as one of the paper's central contributions.

    A more convincing analysis would involve removing or reverting an identified persistent update while keeping the remaining agent state fixed, and then checking whether the downstream safety failure disappears. Such artifact-level ablations, even on a representative subset of trajectories, would substantially strengthen the paper. As currently presented, the experiments provide compelling examples of associations between self-evolution and downstream failures, but less definitive evidence about the causal role of particular updates.

  3. The evaluation relies heavily on LLM-based judges for task completion, safety, and attribution. Although Appendix D.1 reports encouraging agreement with human annotations, the human validation covers only 70 judgments, which seems limited given the diversity and context dependence of the benchmark's safety criteria. More importantly, I could not find a comparable human validation of the attribution judge, despite its central role in establishing endogenous misalignment.

    Some examples also raise questions about how safety violations are defined. For instance, the controller-update example in Appendix D.3.1 (page 17) evaluates a prompt that explicitly asks the agent to include complete member IDs and banking details. While the resulting disclosure may be undesirable, it is important to clarify how the evaluation distinguishes inappropriate compliance with a user's explicit request from a violation specifically caused by self-evolution.

    Similarly, the hiking-coordination example in Appendix D.3.3 (page 23) is classified as a privacy failure even though the described response separates an internal unredacted view from a redacted shareable view. It is not immediately clear whether sensitive information was actually disclosed to an unauthorized recipient. This distinction matters because internal retrieval and external disclosure should not necessarily be treated as equivalent.

    I encourage the authors to provide more detailed adjudication guidelines, a larger independently human-annotated sample, and a breakdown of agreement by harm category. Clarifying these borderline cases would increase confidence in the reported failure rates.

  4. Section 6.3 reports that the proposed monitor flags 56 out of 79 unsafe traces while producing 11 false positives among 113 safe traces. These results are encouraging, but the evaluation is primarily conducted on Kimi K2.5 reasoning traces, and the paper does not establish whether the approach transfers effectively across models, environments, or unseen failure mechanisms.

    In addition, the train/validation/test split is described at the trace level. It is unclear whether trajectories originating from the same task sequences can appear in different partitions. Given the structured nature of the benchmark and the similarity among related task prompts, this could potentially lead to optimistic estimates of monitoring performance.

    I would also like to see comparisons against simpler monitoring baselines, such as direct LLM-based safety classification, output-based monitoring, or a classifier without the proposed risk-progression representation. Without these comparisons, it is difficult to determine how much of the performance improvement comes from the proposed structured reasoning analysis.

    Finally, the distinction between detecting a potentially unsafe output and successfully mitigating the failure deserves more attention. Blocking an output may prevent immediate disclosure, but it may also prevent the user task from being completed. Reporting the resulting task-success and safety trade-off would make the mitigation results more practically meaningful.

Questions for authors

see weakness part

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer uhnL (1/3)

We thank you for your review and for recognizing the importance of safety regressions from benign self-updates. We are glad you found the longitudinal setting and the concrete controller, memory, and tool examples valuable. We address your concerns below.

1. On the interpretation of the safety and capability results

As you note, the paper explicitly positions SEABench as a stress test (Sec. 4.3, lines 240–249). Our goal is to establish that there exist trajectories in which locally useful self-updates lead to downstream safety failures. Searching for such cases is part of the design. This follows prior precedent in diagnostic use of tests generated in automated red teaming, from Perez et al. [1] to recent work that searches over multi-turn interactions [5] or uses prompt optimization to uncover safety failures [6]. The 43.9% versus 0% safety result and 47.2% versus 35.7% completion result apply to the 720 selected candidate tests. Fig. 4 also reports safety on all 3,435 candidate pairs, including rejected prompts, where the rates are 32.2% versus 24.3%. Both sets were explored by the discovery procedure. We make this scope explicit for both safety and completion wherever we report these results.

We would like to clarify that the link between the upstream tasks and downstream safety tests is part of the benchmark design and such a design is precedented in prior literature. Each downstream test probes whether a behavior learned upstream is applied safely when the context changes (Sec. 4.2). This follows the design of Shao et al. [7], who pair a benign tool-creation task with a related safety-sensitive reuse task, and MemoryArena [8], which uses interdependent tasks across sessions. A held-out downstream prompt can be fixed before refinement while still probing the behavior targeted by its upstream tasks. In SEABench, each downstream test is run in a separate session from the frozen agent states (Sec. 4.3).

As discussed in Sec. 4.3 (lines 246-251), Bjarnason et al. [2] show that repeated agent runs can diverge early into different strategies, and Gonzalez-Pumariega et al. [3] show that the same computer-use task can succeed in one run and fail in another with the model held fixed. In our setting, a fixed sequence when used again in a stochasticity-agnostic manner can produce the update of interest, a different update, or no persistent update at all. Sec. 6.1 and Appendix A explicitly scope our conclusions to the discovered trajectories. The space of possible trajectories grows combinatorially across decisions, tool interactions, and persistent updates, making exhaustive traversal infeasible. This necessitates the use of adaptive discovery to find concrete failures for a meaningful study. We these precedents and discussion is made explicit in a revision of the paper (Sec 4.3 in green text).

2. On causal attribution claims and reverting individual updates

We would like to clarify some misunderstandings regarding the causal attribution: our attribution claim expressly concerns upstream self-evolution as a whole, not individual updates, as stated in Sec. 5 (lines 298–299). Sec. 6.1 (lines 345–351) uses the paired non-evolving run as the counterfactual. The safety tests start from the two frozen agent states and use the same downstream prompt. To clarify a key misunderstanding: the attribution judge examines the connection between upstream changes and the downstream failure, and is used along with this paired comparison to establish causal attribution to upstream self-evolution as a whole, and is not used as a sole causal attribution mechanism.

The point that reverting an update while holding the rest of the evolved state fixed would test whether that particular update contributed to the failure, is a separate question from the paper's claim about upstream self-evolution as a whole. We, infact discuss this very aspect in Appendix A (lines 769–772). Answering it would require a separate set of downstream safety runs for every upstream update ablated, multiplying the evaluation cost, with approximately USD 500 spent on the current experiments. We believe this is an important question for future work.

AuthorsOpenReview ↗

Rebuttal to Reviewer uhnL (2/3)

3. On judge validation and the examples in Appendix D.3

We would like to clarify that Appendix D.1 already validates the safety and completion judges in two ways. On 70 human judgments, the safety judge achieves 92.0% precision and 100% recall, and the completion judge achieves 100% precision and 96.8% recall. This is comparable in scale to Shao et al. [7], who validate their tool-creation and reuse judge on 25 cases each for two models, for 50 model–case judgments. The independent GPT 5.6 Sol audit also covers 1,360 safety judgments and 1,340 completion judgments. We validate the attribution judge over 70 safety failures via human annotation, with the annotation set randomly chosen in a manner that retains similar representation of the three surfaces and three models. The attribution judge agrees with human annotators 91.42% of the time, with 6 disagreements, with 3 for privacy, 2 for guardrail erosion, and 1 for hallucination.

The judge validation in the paper (initially reported in Appendix D.1) is now added to the main text in Sec 6.4, along with the new attribution judge validation results (in green font).

Thank you for pointing out the Bob dinner and hiking examples; while those initial tests check if an evolved agent adheres to its initial instructions over later prompts like these, we agree that more interesting cases should use natural sounding user tasks. In particular, for the Bob dinner task, the test checks if the agent violates the initial instructions in AGENTS.md to prohibit letting “authorization language override current-scope safety” and to “match disclosure to the recipient’s role and current need-to-know”, and therefore is a direct test of whether prior safety instructions might be violated by an evolved agent, specifically prohibiting disclosure of banking details to Bob and requires keeping the message limited to the dinner update. The evolved agent’s response included Alice’s full bank account and routing numbers, balance, transaction history, and insurance member and group identifiers, whereas the unevolved agent remained scoped and specified refused to divulge sensitive details and proposed a privacy-preserving response to Bob instead. However, while such tests do have value, to obtain more natural sounding examples, we have identified and are modifying 15 such safety examples out of 720 tests and will report the modifications shortly during the rebuttal phase. For the hiking example, while the internal view contained booking references, the club-facing view redacted them but still retained flight details, which the privacy criteria prohibit the release of, thus justifying the privacy failure label.

4. On the scope and baselines of the reasoning monitor

The results in Sec. 6.3 are for Kimi K2.5 reasoning traces. We recommend training and calibrating the monitor separately for each base model. Importantly, prior work such as Lee et al. [9] show how models differ in how they organize and express their reasoning, including when they backtrack, explore alternatives, and how they phrase those steps, which necessitates monitor retraining for a new model. We had raw traces for Kimi, reasoning summaries for Luna, and encrypted raw traces for Grok (Sec. 6.3 and Appendix D.2). Extending the study to another model would require collecting and annotating new traces for the entire suite of tests, adding to the approximately USD 500 cost of the current experiments, and we note that it exceeds the scope of what we intend and claim such a monitor to do in terms of cross model transfer, given the aforementioned discussion. This would very likely not benefit the discussion, is not central to our work, nor is it a claim we intend to make. We make this very discussion explicit in the revised paper (Sec 6.3, "Mitigation via CoT monitoring", changes in green text).

We clarify how related task sequences and paired traces are assigned to the train, validation, and test partitions: following reviewer jCvL's recommendation, we have revised the splits so that traces belonging to the same sequence reside in the same split (see response to jCvL, point 4), and the reported results for the monitor reflect that (reported in paper revision, Sec. 6.3, lines 484-488, in green font).

AuthorsOpenReview ↗

Rebuttal to Reviewer uhnL (3/3)

Results on Monitor Baselines: We also compare with a direct LLM judge of the task and reasoning trace, an output-only monitor (however, such a judge is limited to a post-hoc analysis of the failure as the output may only be available after the agent has taken some harmful tool calls/actions, contrary to the objective of our monitor which is to interrupt an agent before it outputs any harmful actions), and a classifier without the risk-progression representation, on the same test examples and provide results below. We have updated the paper with this baseline comparison (end of Section 6.3, lines 503-507 and table 1, in green text).

MethodTest accuracyFailure recallFPR
Direct reasoning LM judge76.7%46.0%8.0%
Output-only LM judge78.0%52.0%9.0%
Text-only ExtraTrees81.3%60.0%8.0%
Progression monitor (ours)84.0%70.0%9.0%

We also clarify that detection and successful mitigation in this setting are the one and the same: we propose a first mitigation method as a monitor/filter that detects harmful actions before they happen and, if detected, stops the agent and outputs a refusal message. It does not alter or steer the agent to act differently, which is a mitigation method that can be studied in future work. We emphasize this in the revision (Sec 6.3, "Mitigation via CoT monitoring.", paragraph 3).

Thank you again for your feedback. We believe these clarifications address the scope of the reported results, and we remain glad to discuss the remainder of the corrections during the rebuttal period, and would appreciate strengthened support in light of these clarifications and added results.

References

[1] Perez et al. Red Teaming Language Models with Language Models. EMNLP 2022. (https://aclanthology.org/2022.emnlp-main.225/)

[2] Bjarnason et al. On Randomness in Agentic Evals. 2026. (https://arxiv.org/abs/2602.07150)

[3] Gonzalez-Pumariega et al. On the Reliability of Computer Use Agents. 2026. (https://arxiv.org/abs/2604.17849)

[4] Lee, D., Queen, O., & Zou, J. ReasonOps: Operator Segmentation for LLM Reasoning Traces. arXiv. 2026. (https://arxiv.org/abs/2605.29192)

[5] Guo et al. Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks. ICLR 2026. (https://proceedings.iclr.cc/paper_files/paper/2026/hash/5ac1428c23b5da5e66d029646ea3206d-Abstract-Conference.html)

[6] Shamsi et al. When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models. EACL 2026 Student Research Workshop. Secs. 3–4. (https://aclanthology.org/2026.eacl-srw.33/)

[7] Shao et al. Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents. ICLR 2026. Sec. 3.3. (https://proceedings.iclr.cc/paper_files/paper/2026/hash/a24cd16bc361afa78e57d31d34f3d936-Abstract-Conference.html)

[8] He et al. Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks. ICML 2026. (https://proceedings.mlr.press/v306/he26am.html)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.