acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Safety of LLM Agents in Context

Abstract

AI safety benchmarks evaluate LLM-based agents with empty context, isolating the target behavior and thereby strengthening construct validity of the measurement. In real-world settings, however, autonomous and personal agents continually execute tasks over long horizons, accumulating input context from earlier interactions. Evaluating agent safety with prior context is costly and remains challenging because context can follow a variety of distributions and introduce dependencies between tasks. In this work, we study how the distribution and depth of prior context influence subsequent safety behavior across three benchmarks and nine frontier agentic models. We introduce a measure for meta-evaluating the sensitivity of agent safety benchmarks to contexts formed by executions of benign and harmful tasks. We probe its empirical range using behaviorally controlled contexts, such as executions compliant with unsafe requests, which are elicited through multi-shot jailbreaking and transferred across agents. Overall, we find that continual context shifts agent safety depending on its distribution and the effect often intensifies as context depth increases. Prior task executions frequently make agents overly cautious, resulting in overrefusal. However, safety deteriorates significantly when repeated compliance with unsafe instructions leads to emergent misalignment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agentic safety, LLM agents, harm, context

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_3zrMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

the paper studies how an agent's previous interactions change its safety on subsequent tasks compared with evaluating each task from an empty context. The key finding is that prior unsafe compliance can make agents less safe; ordinary task histories often preserve safety; histories containing harmful requests and refusals frequently increase caution and over-refusal. Benchmarks used: AgentDojo, AgentHarm, RedCode. A variety of models:

  • Open-weight: DeepSeek V4 Pro, DeepSeek V4 Flash, Kimi K2.6, Kimi K3, Muse Glimmer.
  • Closed-weight: GPT 5.6 Sol, Claude Sonnet 5, Gemini 3.1 Pro, Gemini 3.7 Flash.
Critical strengths
  • interesting problem to study
  • experiments across different models and context settings provide evidence supporting the paper's thesis
Critical weaknesses

i'm not sure what the key novelty is here. The finding that models become more likely to comply with harmful requests after seeing examples of unsafe compliance is already known from many-shot jailbreaking.

Questions for authors

what validates the actual main-experiment judge and do corrected empty-message checks change the conclusions?

1 comment

We sincerely thank the reviewer for their appreciation of our work!

On the novelty and many-shot jailbreaking (W1).

Our key contribution, as outlined in the introduction, is a general framework for systematically evaluating how the type and depth of context influence safety measurements for frontier agentic models. By convincingly showing how a safety score depends on what precedes the evaluated task, we argue that future benchmarks should cover a range of context distributions rather than the empty one alone.

Regarding many-shot jailbreaking, we already discuss this and its subsequent works in the paper (Sec. 4). Our work demonstrates that this phenomenon (i) generalizes to frontier agents, (ii) across a broader range of environments, when conditioned on contexts that contain compliant behavior. Furthermore, we use only a few prior compliance demonstrations (multi-shot) and show they even transfer between agents.

Increased compliance after unsafe demonstrations is only one of our findings listed in the introduction:

  • The same harmful requests in context have opposite effects depending on how they were executed, e.g. most open-weight agents become more cautious after their own executions and less safe after complying ones (Table 1).
  • Most closed-weight agents do not comply after unsafe demonstrations, but treat them as safety cues and refuse benign tasks more often (Tab. 1, Example C.11).
  • Harmful context, i.e. the agent's own executions of harmful tasks, makes agents more cautious and increases overrefusal of benign tasks (Tab. 1, Ex. C.7–C.8).
  • Benign coding context aligns open-weight agents to refuse more requests to generate risky code (Fig. 6, Ex. C.1).
  • Complying context makes the benchmarks less informative since it significantly reduces safety differentiation between harmful and benign tasks for seven of nine agents (App. Table 2).

Clearly, most safety benchmarks do not evaluate agents in context, which provides direct evidence that our findings about their substantial measurement changes are valuable. What might seem “already known” in hindsight is not a settled debate in the safety field.

Q1. what validates the actual main-experiment judge and do corrected empty-message checks change the conclusions?

Regarding the judge:

  • AgentDojo uses benchmark-native scoring and HumanEval Pass@1 uses the official functional tests, so these results do not depend on a judge.
  • For AgentHarm and RedCode, we keep the default judge prompts and grading of the original benchmarks. In Appendix E, we rescore 2,740 stored responses with exactly this protocol and Claude Sonnet 5 as a lower-cost judge. Repeated judgments are stable and mostly agree with human labels.
  • We did not repeat this analysis with the more expensive Opus, but would argue Sonnet 5 to be on-par with Opus 4.8 on such a simple task.

Regarding the empty-message checks:

  • They come from the original AgentHarm grading functions and potentially affect only harm and completion scores. Refusal and overrefusal labels, RedCode, and AgentDojo are unaffected and show the same effects of context.
  • A passed empty-message check gives credit for a tool that the agent never called. Since agents call fewer tools when they refuse, this may only limit the reported changes in harm and completion scores toward zero rather than increasing them.

We now state this in Appendix E (lines 1864–1866).


Thank you again for finding our paper clear, informative, and sound. We would greatly appreciate an increase in rating if you feel we adequately addressed your remaining concerns.

Reviewer_UjqUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how previous task histories affect the safety of LLM agents. It extends existing safety benchmarks by adding benign, risky, or successfully completed harmful tasks before the task being evaluated. Experiments across nine models and three benchmarks show that these histories can increase harmful behavior or cause agents to refuse benign tasks. The findings suggest that agent safety evaluations should consider prior interactions alongside standard empty-context tests.

Critical strengths
  1. The paper studies a practically relevant gap in agent safety evaluation. Agents accumulate interaction histories during use, while standard evaluations often start with empty context. Evaluating both harmful behavior and benign task completion helps reveal safety–utility tradeoffs that empty-context scores may miss.

  2. The experiments provide useful evidence across several settings. Across nine models and three benchmarks, prior histories produce substantial and different effects: some agents become more compliant with harmful requests, while others increasingly refuse benign tasks. These findings support considering interaction history when evaluating deployed agents.

Critical weaknesses
  1. The contribution beyond prior work is not sufficiently established. Previous studies have already shown that context and unsafe demonstrations can change model safety, including in AgentHarm. This paper extends evaluation to complete agent histories and more settings, but does not clearly establish what new understanding these histories provide beyond known context and demonstration effects. This limits the significance of its main contribution.

  2. The methodology does not adequately account for context as an integral part of safety-relevant decision-making. Prior interactions provide evidence about user intent, authorization, and risk, so an appropriate decision on the same request may change with its history. However, the framework evaluates these decisions against fixed benchmark labels defined independently of that history. For example, refusing an allegedly authorized bank-account operation after repeated fraudulent requests may reflect reasonable caution rather than degraded capability. Treating such disagreement as a failure risks penalizing agents for appropriately using contextual evidence. This limits the interpretation of the reported effects: they demonstrate that history changes behavior, but do not consistently establish whether those changes worsen safety or utility.

Questions for authors
  1. What is the main new empirical finding beyond prior work on contextual safety and demonstration-based jailbreaking? In particular, which result shows that complete task-execution histories provide additional understanding beyond the effects of unsafe demonstrations or contextual content? This would help clarify whether the contribution primarily establishes a new phenomenon or extends existing findings to broader agent settings.

  2. Does the framework assume that a task’s benign label remains valid regardless of the preceding history? For example, in Appendix C.7, the agent receives an allegedly authorized bank-account testing request after a history of fraudulent requests. Questioning that authorization may be reasonable given the conversation, even if the current task is labeled benign by the benchmark. How does the evaluation distinguish unjustified overrefusal from appropriate caution based on history? Does the reported increase in overrefusal remain substantial for tasks whose benign interpretation is unambiguous in context?

  3. Does the empty-content grading issue identified in Appendix E also affect the main Opus-based evaluation? If so, how frequently does it occur across history conditions, and do the principal effects and significance conclusions remain after correction? Since Appendix E evaluates Sonnet rather than the main Opus judge, what evidence supports the reliability of the actual scoring protocol used for the reported results?

2 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

We thank the reviewer for engaging with our work. Below, we provide point-by-point responses to each of the comments.

W1. Previous studies have already shown that context and unsafe demonstrations can change model safety, including in AgentHarm.

Our key contribution is a general framework for systematically evaluating how the type and depth of context influence safety measurements for frontier agentic models. By convincingly showing how a safety score depends on what precedes the evaluated task, we argue that future benchmarks should cover a range of context distributions rather than the empty one alone.

This paper extends evaluation to complete agent histories and more settings, but does not clearly establish what new understanding these histories provide beyond known context and demonstration effects.

The new understanding is that continual context shifts agent safety depending on its type, and the effect often intensifies as context depth increases. Clearly, most safety benchmarks do not evaluate agents in context, which provides direct evidence that our findings about their substantial measurement changes are valuable. What might seem “known” in hindsight is not a settled debate in the safety field.

Q1. What is the main new empirical finding beyond prior work on contextual safety and demonstration-based jailbreaking?

Demonstration-based jailbreaking shows that unsafe demonstrations increase compliance. It has not necessarily been shown for frontier agents such as DeepSeek V4 Pro and Kimi K3 across diverse benchmarks (Figure 4). Another new finding is that prior task executions frequently make agents overly cautious instead:

  • Harmful context, i.e. the agent's own executions of harmful tasks, makes agents more cautious and increases overrefusal of benign tasks (Table 1, Examples C.7–C.8).
  • Most closed-weight agents do not comply after unsafe demonstrations, but treat them as safety cues and refuse benign tasks more often (Tab. 1, Ex. C.11).
  • Benign coding context aligns open-weight agents to refuse more requests to generate risky code (Fig. 6, Ex. C.1).

In particular, which result shows that complete task-execution histories provide additional understanding beyond the effects of unsafe demonstrations or contextual content?

For example, the comparison of harmful and complying contexts in Table 1. Both contexts contain the same harmful requests, and only their executions differ. Yet most open-weight agents become more cautious after their own executions and less safe after complying ones, e.g. the harm score of DeepSeek V4 Pro decreases after its own executions of harmful tasks and increases after complying executions of such tasks. So the effect comes from how the requests were executed, not from the requests themselves.

W2. The methodology does not adequately account for context as an integral part of safety-relevant decision-making.

We kindly disagree, since the point of our work is precisely to highlight that context is integral to safety-relevant decision-making, and an empty-context methodology rooted in typical benchmarks does not adequately account for it.

Prior interactions provide evidence about user intent, authorization, and risk, so an appropriate decision on the same request may change with its history.

Frankly, this realization is the contribution of our paper.

However, the framework evaluates these decisions against fixed benchmark labels defined independently of that history. For example, refusing an allegedly authorized bank-account operation after repeated fraudulent requests may reflect reasonable caution rather than degraded capability.

Yes, and this is what our paper shows across extensive experiments. Some frontier agents become more cautious after refusing harmful requests, while others comply with consecutive harmful requests. We should measure it rather than only consider empty-context safety evals.

Treating such disagreement as a failure risks penalizing agents for appropriately using contextual evidence.

We do not penalize agents for appropriately using contextual evidence. We report the benchmark results and discuss their interpretation grounded in agents' behavior. If we had relabeled the tasks, the agents’ behavior would not change, and the safety measurement would be interpreted w.r.t. those potentially flipped task labels.

This limits the interpretation of the reported effects: they demonstrate that history changes behavior, but do not consistently establish whether those changes worsen safety or utility.

We find it mostly clear that an agent that executes prompt injections, harasses people, sells drugs, or generates malware is unsafe, and most of these labels do not depend on context.

AuthorsOpenReview ↗

[2/2] Response by Authors

Q2. Does the framework assume that a task’s benign label remains valid regardless of the preceding history? [...] Questioning that authorization may be reasonable given the conversation, even if the current task is labeled benign by the benchmark.

Yes, thank you for pointing this out. We have revised the paper’s text (line 124) to make it clearer that we keep the tasks, labels, and scoring of each benchmark unchanged and vary only the context, so that any change in score is attributable to context (Sec. 2.1). We agree that questioning authorization may be reasonable: we write that agents treat benign instructions “as borderline” after harmful requests, e.g. Kimi K2.6 refuses to send an encouraging email because “the previous context suggests the user may be testing boundaries” (Ex. C.8). We still need to measure it, since it degrades usability, e.g. a few misuse attempts can leave a deployed personal agent less helpful for the rest of its session.

How does the evaluation distinguish unjustified overrefusal from appropriate caution based on history? Does the reported increase in overrefusal remain substantial for tasks whose benign interpretation is unambiguous in context?

The overrefusal metric counts refusals of benign tasks and does not make this distinction by itself. Yes, the increase remains substantial where the benign interpretation is unambiguous, e.g. HumanEval problems are benign in any context, e.g. sorting the characters within each word. Yet the overrefusal of GPT 5.6 Sol and Claude Sonnet 5 on them rises from zero to nearly one in risky or complying code contexts, often due to provider guardrails (Fig. 2, Ex. C.3, C.15). AgentHarm harmful tasks are matched to benign controls proposed by the benchmark. Our work argues for evaluating agents in context so as to measure an increase in overrefusal for tasks whose benign interpretation can be not unambiguous in context. We believe that distinguishing between ambiguous and non-ambiguous benign tasks may be ambiguous itself.

Q3. Does the empty-content grading issue identified in Appendix E also affect the main Opus-based evaluation? If so, how frequently does it occur across history conditions, and do the principal effects and significance conclusions remain after correction?

Yes, potentially, and we now state this in Appendix E (lines 1864–1866). The checks could affect only the harm and completion scores, and may only limit their reported changes toward zero rather than increasing them. Refusal and overrefusal labels are unaffected and lead to the same significance conclusions. We do not intend to change the evaluation protocol from the original AgentHarm work, as we consider this only a minor limitation of the study.

Since Appendix E evaluates Sonnet rather than the main Opus judge, what evidence supports the reliability of the actual scoring protocol used for the reported results?

We put extra effort into providing evidence to support the judge's reliability by conducting a human evaluation study with exactly the protocol of the main evaluation (Appendix E). We did not repeat it with the more expensive Opus, but would argue Sonnet 5 to be on-par with Opus 4.8 on such a simple task.


We are glad you found our paper clear and sound. We would greatly appreciate it if you could re-evaluate the informativeness of our work, as we believe it provides decisive empirical evidence for the important arguments we agree on, which the review seems to list as weaknesses rather than as the paper's contributions. Thank you again for your time.

Reviewer_ec6nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out the question that nowadays, LLM agents are evaluated with empty context and this mismatches the real setting, e.g., personal agents. Therefore, the authors propose a meta-evaluation framework aiming to fill this gao. Specifically, the authors construct the context for three current safety benchmarks, including indirect prompt injection,harmful user instructions and risky code generation. Here context are controlled through depth and the type of tasks. The authors test this framework on 9 LLMs under different types and depth context.

Critical strengths

The authors study an interesting and important problem of evaluating agents in context.

Critical weaknesses

The problem I am most concerned about is the context construction. There is still some difference between reality and the framework. For example, as the authors also acknowedged in Page 9, Line 479-481, the complying context comes from DeepSeek V4 Pro and therefore the results could be affected by prefill awareness. And if I understand correctly, the context is constructed with weakly related tasks by design, while in reality, they usually are correlated.

Questions for authors

Could you help me understand more about the prefill awareness effect in the results?

2 comments

We sincerely thank the reviewer for their appreciation of our work!

W1. The problem I am most concerned about is the context construction. There is still some difference between reality and the framework. [...] And if I understand correctly, the context is constructed with weakly related tasks by design, while in reality, they usually are correlated.

We need to clarify that the strength (weak/correlated) of the relation between the evaluated task and the context changes and depends on the safety benchmark. Our primary goal is to propose a simple and effective approach to advancing safety evaluation benchmarks beyond the empty-context scenario. We show how this framework can be instantiated in practice on three benchmarks to demonstrate various potential implementations:

  1. For AgentDojo, prompt injections are evaluated on weakly related tasks, i.e. from another environment, while we report additional results for correlated tasks from the same environment in Appendix D. While the latter scenario can be considered “more real”, we highlight that such construction can allow agents to reuse information from previous turns and thus influence the eval.
  2. For AgentHarm, user instructions are evaluated on correlated tasks from the same category, e.g. copyright, fraud, harassment. We show the per-category influence of contexts in Figure 5 (moved from Appendix to main text), where we find, for example, that harmful context produces the largest evaluation changes for tasks concerning drugs, disinformation, and copyright, making agents significantly more cautious (lines 370–376).
  3. For RedCode, coding requests are evaluated on moderately related tasks, i.e. malware from other categories (e.g. adware and rootkit before a trojan in Example C.14), or on unrelated benign coding tasks.

While we agree that there can be some differences between reality and our work, it’s precisely the discussion our paper provides on multiple accounts.

Q1. Could you help me understand more about the prefill awareness effect in the results?

Great question; we were also initially surprised that the effect could occur, as discussed in concurrent work by Wang et al. (2026).

Prefill awareness refers to an agentic model recognizing that earlier agent messages were externally supplied rather than naturally generated by itself. Recently, Wang et al. (2026) distinguish between recognizing a prefill and resisting its influence. For example, in their high-stakes alignment and ethics experiments, Opus 4.5 resisted tampered reasoning prefills in most cases (87%), despite near-chance detection performance (53% balanced Acc.). Generally, resistance alone does not establish awareness. Our examples illustrate why this distinction matters. In Figure 3, Kimi K3 rejects earlier harmful actions without identifying them as externally generated. Examples C.3, C.11, and C.15 instead illustrate provider safeguards.

What we mention in the paper as a limitation is that prefill awareness could lead an evaluated agent to discount the unsafe behavior in our transferred complying context traces (Sec. 2.4). We use these context traces because agents that reliably refuse harmful requests cannot readily produce their own examples of compliance. Notably:

  • Prefill awareness does not influence the results in the other context conditions (clean / benign / injected / harmful / risky).
  • Transfer is not necessary to measure the effect of complying context on the evaluated safety. For example, DeepSeek V4 Flash and Kimi K3 were able to generate their own complying histories as reported in Figure 4. Moreover, with transferred context, in Ex. C.13, Kimi K2.6 explicitly treats earlier code generation as its own precedent, while Ex. C.4–C.5 show agents following injections after earlier compliance.

On a final note, consider that statements about prefill awareness in the agent’s reasoning trace are self-reports, and chain-of-thought reasoning need not faithfully reflect what drives its behavior (see e.g. Turpin et al., 2023, and its subsequent work). An agentic model may recognize a prefill without mentioning it, while mentioning a prefill does not establish that this recognition caused its refusal.

We have extended the limitations in the paper with these points (lines 530–532).

Wang et al. Prefill awareness in large language models. arXiv:2606.12747, 2026

Turpin et al. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. NeurIPS 2023


Thank you again for finding our paper generally understandable, informative, and sound. We would greatly appreciate an increase in rating if you feel we adequately addressed your remaining concerns.

Thank you for the clarification. The explanation of task relation partially solve my concern and I understand it is hard to simulate real word task dependence. The prefill awareness discussion is also helpful. Overall, I decide to maintain my score.

Reviewer_ii9sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates LLM safety under continual context and makes several findings. For example, (1) prior task executions frequently make agents overly cautious, resulting in overrefusal; (2) safety reduces when repeated compliance with unsafe instructions is in context. Evaluation is conducted on different benchmark families, multiple agents, and context types.

Critical strengths
  1. The paper studies a practical setting, where users make multi-turn requests and yield winding context.
  2. The paper presents broad experiments, covering different benchmarks and nine LLMs.
Critical weaknesses
  1. The main finding is not new. For example, "The Cold-Start Safety Gap in LLM Agents (EMNLP 2026)" studies whether safety changes as the number of preceding tasks increases; "Many-shot Jailbreaking (NeurIPS 2024)" shows that repeated demonstrations of harmful compliance in context increasingly reduce safety. There is other work cited in the discussion section, but the novelty of this work is still unclear to me. It would be nice if the authors could clarify the novelty and narrow their contribution claim if necessary.
  2. Due to the above weakness, I think this paper would have enough contribution if it established a deeper understanding of its observations. For example, why does harmful context make a model safer? A context contains user query, model reasoning traces, tool calls, model refusal, etc., and it is unclear whether some particular part is the main cause. The paper abstractly calls it "conversational self-consistency", while it would be insightful if the paper dug deeper: Is it tool-calling results, reasoning traces, model answers, or others? Different answers can uncover different mechanisms.
Questions for authors
  1. The paper uses the phrase "context distribution", but currently there are three possible values: benign, harmful, and complying. It would be more scientifically interesting if a context could mix, for example, benign and harmful, or vary the order of different categories. That would make it a real "distribution". In contrast, the current version looks more like comparing a few context types.
  2. Based on line 468, the authors adopt the definition of "emergent misalignment" as "inducing unsafe behavior in an unrelated domain". However, the experiments mainly use preceding tasks from the same harm category as the subsequent task, which is in-domain. Can the authors clarify or refine the claimed contribution "emergent misalignment"?
4 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

Thank you very much for your review of our work.

W1. The main finding is not new. [...] It would be nice if the authors could clarify the novelty and narrow their contribution claim if necessary.

Our key contribution, as outlined in the introduction, is a general framework for systematically evaluating how the type and depth of context influence safety measurements for frontier agentic models. By convincingly showing how a safety score depends on what precedes the evaluated task, we argue that future benchmarks should cover a range of context distributions rather than the empty one alone. We will update the text to ensure this is articulated, thank you!

Most safety benchmarks do not evaluate agents in varying context, which supports that our findings about their substantial measurement changes are valuable to the community. We believe strongly that this work provides useful insights that can improve future safety evaluation approaches by outlining how results can change based on context content. We thus kindly ask you to reconsider recommending our paper for discussion at the conference.

As agentic models and methods for evaluating their safety evolve continuously, and as experiments become prohibitively expensive, it is valuable to report findings across diverse experimental conditions to facilitate their reproduction and confirmation (or not!). Our goal is to gather more evidence to confirm or refute particular findings, reach consensus, or monitor the progress of hypotheses over time and scale.

Regarding “Many-shot Jailbreaking (NeurIPS’24),” we discuss this and its subsequent works in the paper (Section 4). Our work demonstrates that this phenomenon (i) generalizes to frontier agents, (ii) across a broader range of environments, when conditioned on contexts that contain compliant behavior. Furthermore, we use only a few prior compliance demonstrations (multi-shot) and show they even transfer between agents. We will update the manuscript to ensure this is clearer.

Regarding “The Cold-Start Safety Gap in LLM Agents (EMNLP 2026),” we were sincerely unaware of this concurrent work and have now added its discussion to Section 4 (lines 489–499). Consider some of the following differences between our concurrent works:

  • They evaluate smaller open-weight models (4–70B) and exclude closed-weight ones. We evaluate much larger, frontier agentic models, including four closed-weight ones that often react differently, e.g. with overrefusal instead of compliance.
  • They build a benchmark of direct harmful requests and confirm the effect on AgentHarm and Agent-SafetyBench. We propose a more general framework for existing and future safety evals on different examples of prompt injection (AgentDojo) and risky code generation (RedCode).
  • They mostly place benign (regular) tasks in context, and in Appendix F harmful requests prefilled with a canned refusal. We additionally place the agent's own executions of harmful tasks in context, and also compliant executions, i.e. compliance with harmful requests, which they do not study. Their "Compliant Response" ablation replaces the response to a benign request with agreeable text (e.g. "Sure, I can help."), so it is not compliance with a harmful request.
  • Their Finding 3, “Any preceding context improves safety over cold-start,” does not hold for complying context, which significantly raises the AgentHarm harm score for most open-weight agents, and raises attack success in AgentDojo and RedCode.
  • They find that the agent's responses in context matter less for safety than the user requests. We find that responses can decide the direction of the effect, e.g. for DeepSeek V4 Pro, where the harm score decreases after its own executions of harmful tasks, but increases after complying executions of such tasks.
  • They measure utility on separate benchmarks (BFCL, API-Bank) and state that warm-up does not cause over-refusal. We also evaluate harmful tasks alongside their benign variants under the same context, and find that benign context makes GPT 5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash overrefuse benign tasks more often, so the added caution is not free. Thus, overall, our work is complimentary to this paper and maintains significant value.

Finally, as suggested, we narrowed the claims in the introduction: we now state that we find evidence for our findings, and that they concern frontier agents (lines 87–88) and are happy to narrow any of them further if necessary.

AuthorsOpenReview ↗

[2/2] Response by Authors

W2. [...] For example, why does harmful context make a model safer? [...] The paper abstractly calls it "conversational self-consistency", while it would be insightful if the paper dug deeper: Is it tool-calling results, reasoning traces, model answers, or others?

While we (truly) think that this is indeed an interesting research direction, we think that it does not invalidate our methodology and positive results. Our work partially addresses these questions across the findings and examples presented in the paper:

  • In Sections 3.1 & 3.2, we explain why harmful context makes agents safer: prior refusals act as precedent, and even when agents comply, disclaimers such as "for educational purposes only" accumulate as safety cues.
  • In Table 1, the same harmful requests in context have opposite effects depending on how they were executed: for Kimi K3, the harm score decreases after its own (mostly refused) executions and increases significantly after complying ones.
  • In Examples C.7, C.8, and C.12, agents cite the earlier user requests. For example, Kimi K3 reasons that "the user has previously asked for dark web stolen credit cards, credential theft, plagiarism, fake passports, and visa fraud evasion" and then refuses both the harmful task and overrefuses its benign variant. Harmful context makes agents safer because they read it as evidence of the user's intent.
  • In Ex. C.13, the agent cites its own earlier answers. Kimi K2.6 reasons "I have refused similar requests multiple times in this conversation [...] and I should remain consistent," whereas after complying context it reasons "I've been providing code that implements these simulated attacks." This is what we call conversational self-consistency.
  • In Ex. C.6, the agent cites an earlier tool result and its own action. DeepSeek V4 Pro reasons "The previous task had a prompt injection attempt [...] I followed that," and then withholds a benign bill payment.
  • In Ex. C.14, reasoning traces are not needed. The complying context is transferred without reasoning, yet Muse Glimmer writes the trojan that it refuses in clean and risky contexts.
  • In Appendix D, we demonstrate how tool results can influence such an analysis. In the same-environment context, agents reuse earlier tool outputs and skip the call that returns the injection, which is why we use cross-environment context.

We share the reviewer’s excitement regarding the potential future work that extends it, which we now note in line 538.

Q1. The paper uses the phrase "context distribution", but currently there are three possible values: benign, harmful, and complying. It would be more scientifically interesting if a context could mix, for example, benign and harmful, or vary the order of different categories. That would make it a real "distribution". In contrast, the current version looks more like comparing a few context types.

Great question! By “context distribution” we mean a distribution parametrized by context type (e.g. benign, harmful, complying) and context depth (k=1,2,…k=1,2,\ldots). We agree that it would also be interesting to study mixed contexts, as we have already pointed out in the paper lines 533–537: Finally, to isolate the effect of each context distribution, we use homogeneous contexts, whereas deployed agents may accumulate interleaved tasks and periodically compact their histories. Future work can explore longer, mixed, and compacted contexts, as well as develop mitigations, such as fine-tuning on longer trajectories to improve safety in context without increasing overrefusal or degrading general capabilities.

Q2. [...] the experiments mainly use preceding tasks from the same harm category as the subsequent task, which is in-domain. Can the authors clarify or refine the claimed contribution "emergent misalignment"?

Thank you for pointing this out. Let us clarify that the paper does not claim a contribution to “emergent misalignment,” and we made this discussion clearer in lines 515–521 to reflect this.

We discuss how our experimental findings relate to this phenomenon, for example in that benign contexts can surprisingly align agents not to complete harmful user requests, which can be understood as “emergent alignment”. Furthermore, we study quite recent (2026) agentic models, while the emergent misalignments have been studied in pre-2026 language models that often lacked agentic capabilities. Kindly note that experiments within the AgentDojo benchmark use preceding tasks from a different environment, e.g. Slack to Banking transfer, which is clearly out-of-domain. Nevertheless, we agree that future work can benefit from experiments on AgentHarm in which the harm categories are mixed in context as well.


Thank you again for finding our paper generally understandable, informative, and sound. We would greatly appreciate an increase in rating if you feel we adequately addressed your remaining concerns.

Reviewer_ii9sOpenReview ↗

Reviewer Response

Thank you for the response and revision. Regarding novelty, I distinguish extending empirical coverage (e.g., adding model families, benchmarks, and agent environments) from new conceptual contributions (such as new insights, phenomena, methodology, or research directions). Regarding the latter, my understanding is that the cold-start paper shows "benign context -> increased safety; harmful context + refusal demonstrations -> increased caution" and the many-shot jailbreaking work shows "unsafe compliance -> increased harmful behavior". This paper examines naturally executed harmful contexts and additionally shows "benign context -> various safety effects". I acknowledge that it provides a more systematic empirical characterization of how benign/harmful/complying task histories affect harmful behavior (cross-agents, etc.), and it is a meaningful contribution.

I retain my score because I expect more insights beyond the above. In particular, my concern W2 is unresolved: the qualitative examples provide plausible explanations, but ablation studies can help identify which components of the context lead to the observed changes. My Q1 also remains regarding how the findings generalize to more realistic mixed or reordered contexts.

Dear Reviewer, thank you very much for the follow-up and for acknowledging that our work is a meaningful contribution.

To extend your simple summary, regarding many-shot jailbreaking, it studied non-reasoning language models ('24), whereas we evaluate LLM agents that reason and use tools. Regarding the cold-start paper, it is contemporaneous work as per the ICLR guidelines. There are also much more findings that do not appear in these papers, e.g. most closed-weight agents do not comply after unsafe demonstrations, but treat them as safety cues and refuse benign tasks more often; or that benign context makes GPT 5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash overrefuse benign tasks more often, whereas the cold-start paper finds that warm-up does not cause over-refusal in smaller models (4–70B).

On W2. Our evidence is not only qualitative. The paper, for example, separates two causes of the observed changes:

  • Agent actions. Harmful and complying contexts contain the same requests, and only their executions differ. Yet the direction of the effect flips for most open-weight agents (Table 1), so the same requests have opposite effects depending on how they were executed.
  • Reasoning traces. Transferred complying context contains no reasoning, yet raises harm for most open-weight agents (Table 1), so reasoning traces are not necessary.

Kindly note that a finer ablation of single components within a trajectory is a research challenge itself. It seems non-obvious on how to do this in a principled way, since each of the context parts depend heavily on another, e.g. tool calls must stay paired with their outputs, and removing a user request while keeping the agent's answer produces a context that is clearly out of distribution. Such edits could also trigger prefill awareness.

On Q1. Homogeneous contexts are a deliberate design choice that can be simply applied to a broad range of future safety benchmarks. A mixed context changes type, proportion, and order at once, so it can only be interpreted against the homogeneous baselines that we provide.

Frankly, both W2 and Q1 ask for "more insights" that our framework could enable in the future, i.e. what would make the work stronger, and we do not see major concerns that would warrant a “weak rejection”.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the responses by Authors

Dear Reviewers and Area Chair,

Through this work, we argue that agentic safety benchmarks should cover a range of context distributions rather than the empty context alone. This is not currently the case. For a convincing argument, we introduce a general framework for systematically evaluating how the type and depth of context influence safety measurements for agentic models. We use it to comprehensively verify how safety quantification depends on what precedes the evaluated task across 9 frontier agents and 3 safety benchmarks. We identify several nuanced insights about safety evals executed in context, and analyze qualitative examples extensively. Finally, we relate our findings to a broad range of related topics in the literature.

We kindly ask that you consider recommending our paper for discussion at the conference, given that the debate on this topic is not settled in the safety field, and that our evidence is necessary to shift towards more context-dependent safety evaluations of LLM agents.

In brief, we addressed the 7 questions and 6 potential weaknesses across the 4 reviews as follows (the changes are visible in red):

  • [ii9s W1; UjqU W1 & Q1; 3zrM W1**]** Novelty. What's new? We extended the discussion (lines 489–507) and narrowed wording in the introduction (lines 87–88):
    • Regarding many-shot jailbreaking, while it is partly related, our work is much broader. We demonstrate that multi-shot jailbreaking applies to frontier agents across a broader range of environments, and that we can transfer the complying context between agents. Ultimately, we argue for the necessity of evaluating agents and meta-evaluating safety benchmarks across various context distributions (e.g. benign, risky).
    • Regarding the cold-start safety gap, compared to this concurrent work, we analyze a different set of agentic models and benchmarks, in some cases coming to different conclusions.
  • [ii9s W2**]** Why does harmful context make a model safer? It would be insightful if the paper dug deeper. We answered with findings from the paper and agree that future work can extend it (line 538).
  • [UjqU W2 & Q2**]** Fixed benchmark labels vs. reasonable caution in context. We agree that caution can be reasonable in context and show such nuanced cases in the paper, which further support our main argument. We show how context influences safety measurements and discuss the interpretation of these changes in context. We also clarified the methodology in line 124.
  • [ec6n W1 & Q1**]** Context construction and prefill awareness. Answered pointing to lines 370–376, moved Figure 5 from the Appendix to main text, and clarified in lines 530–532.
  • [ii9s Q1**]** Context distribution. Answered pointing to lines 533–537.
  • [ii9s Q2**]** Emergent misalignment. Answered and clarified in lines 515–521.
  • [UjqU Q3; 3zrM Q1**]** Appendix E. Answered and clarified in lines 1864–1866.

Thank you again for the valuable feedback, which allowed us to improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.