acceptodds
Under review as a conference paper at ICLR 2027

Self-Reinforcing Safety Failures in LLM Agents: When Models Confirm Themselves

Abstract

Tool-using LLM agents can violate safety constraints while pursuing legitimate goals, even without adversarial inputs. We identify self-reinforcing failure, in which an agent makes an unsupported claim, later treats that claim as confirmation, and uses it to justify an unsafe action. We model agentic inference as information-flow and formulate reasoning noninterference with respect to external support: earlier reasoning may guide later decisions, but it must not by itself establish a safety condition. We then propose SafeGate, an in-model architecture that separates safety-state updates from agent-generated content. A source-isolated encoder and gated recurrent updater maintain a Safety Boundary State using only its previous value and new external observations. We jointly train the state producer through reconstruction and decision distillation while keeping the base LLM frozen. On an intrinsic-safety subset of Agent-SafetyBench, SafeGate improves Qwen3-8B safety by an average of 26.9 percentage points across reasoning modes, while also reducing loss of control and improving safe termination on Forge-Bench. Across five backbones, the safety gain averages 24.8 percentage points, while architecture and ablation results show that the gains saturate with state capacity, depend on injection depth, and benefit from complementary training objectives and consistent source isolation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Self-Reinforcing Failure, Information Flow, Safety Boundary State

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HumsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies self-reinforcing safety failures, where an agent treats its own unsupported claims as evidence for later actions. It proposes SafeGate, a persistent safety state updated from external observations and read by a frozen language-model agent. Experiments report safety improvements across multiple backbones and reasoning modes, alongside helpfulness trade-offs.

Critical strengths

The information-flow perspective provides a useful distinction between agent-generated claims and externally supported evidence. This offers a clear motivation for studying an important agent-safety problem. The evaluation covers multiple backbones, reasoning modes, and architectural ablations. Reporting both safety and helpfulness makes the benefits and trade-offs easier to assess.

Critical weaknesses

The overall safety gains are encouraging, but the experiments do not directly quantify failures matching the proposed self-reinforcement definition or how often SafeGate prevents them. A targeted analysis would strengthen the connection between the motivating mechanism and the empirical improvements. Table 4 leaves an important configuration ambiguity: the Full/Full row matches the main non-thinking results, while Masked/Masked performs better. Clarifying which history is masked and how these variants relate to the observation-only state update would help establish the contribution of source isolation and the learned state.

Questions for authors

Clarification: In Table 4, which component receives the masked history, and which configuration is used in the main results? How does Full/Full relate to the source-isolated state update described in Section 4? Suggested additional evidence: Could you evaluate a representative annotated subset for failures matching the self-reinforcement definition, and compare SafeGate against a history-masking baseline without the learned state or an observation-only textual-summary baseline?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Hums: Weakness1 and Question3 - Part1-3

We thank the reviewer for the careful reading and for recognizing the information-flow perspective and the breadth of the evaluation. The reviewer's questions about Table 4 uncovered a reporting error in our original submission, which we correct below. We also add a targeted analysis on trajectories with observed self-reinforcing failures and compare SafeGate against the two suggested baselines.

W1 & Q3: Quantifying self-reinforcing failures, baselines, and a controlled intervention

"… A targeted analysis would strengthen the connection between the motivating mechanism and the empirical improvements."

"Suggested additional evidence: Could you evaluate a representative annotated subset for failures matching the self-reinforcement definition, and compare SafeGate against a history-masking baseline without the learned state or an observation-only textual-summary baseline?"

To address this concern, we added the targeted analysis the reviewer suggests, in five parts. To quantify failures that match the definition of self-reinforcing failure (SRF), we explain how the 514 tasks are selected for SRF (1), evaluate SafeGate on a representative subset of 53 ASB tasks in which SRF is observed in annotated trajectories (2), and apply the same screening to 33 cases of an external benchmark, ToolEmu (3). We then compare SafeGate with the two suggested baselines (4). Finally, to show how often SafeGate prevents SRF, we add a controlled intervention in which the only difference is one unsupported claim by the agent (5).

(1) How the 514 tasks are selected for SRF. SRF is a failure that originates in the agent itself: an agent pursuing a legitimate goal reaches an unsupported conclusion and crosses a safety boundary on that basis. We screened 514 tasks from all 2,000 Agent-SafetyBench tasks based on the definition of self-reinforcing failure. A task is retained only if the user's goal is legitimate, the agent faces a safety-relevant decision whose prerequisite has to be established from external evidence, and the agent can cross the safety boundary only through its own judgment about that prerequisite. Tasks whose risk comes from an attacker, an explicitly harmful request or misleading environment feedback are removed. Every retained task sets up the decision through which SRF arises.

(2) A representative annotated subset. To address the concern , we screened the 154 test tasks by self-reinforcing failure observed in executed trajectories. We use gpt-5.6-sol to execute the task to generate the interactive trajectory and use gpt-6-astra and human judge to select the representative self-reinforcing failure tasks. A trajectory is counted only if it shows an unsupported claim, a later use of that claim as an established fact, and a dependent action that crosses the safety boundary. This screening yields a subset of 53 tasks with observed self-reinforcing failure. On the results of the respentative subset, SafeGate is safe on 79.2% of the 53 tasks, compared with 11.3% for the base model, showing that SafeGate perform well in the subset.

TasksBase Safety (%)SafeGate Safety (%)
5311.379.2

(3) The same analysis on an external benchmark. To show that SafeGate's protection against self-reinforcing failure is not specific to ASB, we applied the same self-reinforcing failure screening to ToolEmu (Ruan et al., 2024), an independent benchmark of deliberately underspecified instructions whose tools are emulated by a language model. We ran gpt-5.6-sol on the ToolEmu cases with ToolEmu's standard emulator and kept the 33 cases in which SRF is observed in its trajectories. Without any retraining, the ASB-trained SafeGate raises Safety on these cases from 44.4% to 88.9%.

Test setBase Safety (%)SafeGate Safety (%)
ToolEmu, cases with observed self-reinforcing failure44.488.9
AuthorsOpenReview ↗

Response to Reviewer Hums: Weakness1 and Question3 - Part4-5

(4) The two suggested baselines. To determine whether the gain comes from carrying safety information across steps or from what SafeGate's learned state contains, we compare SafeGate, as the reviewer suggested, with a history-masking baseline and an observation-only textual-summary baseline, neither of which has a learned state. For the history-masking baseline, we re-ran the base model with all earlier history removed: the actor sees only the task and the latest tool call with its returned observation. For the observation-only textual summary, the actor's own backbone, Qwen3-8B, writes a nine-axis safety summary at every step from the returned observations, the tool definitions and its previous summary, and the summary is placed in the actor's prompt; it never sees the actor's reasoning, free text or tool calls.

MethodASB test Safety (%)ASB test Helpfulness (%)SRF subset Safety (%)SRF subset Helpfulness (%)
Base31.895.511.392.5
History masking29.994.818.990.6
Observation-only textual summary35.194.220.894.3
SafeGate74.089.679.286.8

The two baselines test the two ingredients that could explain the gain without a learned state. History masking tests whether it is enough to keep the earlier history, including the agent's earlier actions, out of the actor's context. It is not: Safety stays at the base level (29.9% on all test tasks; 18.9% vs. 11.3% on the subset). The textual summary tests whether it is enough to carry safety information from external evidence across steps. It is not either: a summary written by the actor's own backbone leaves Safety close to the base model (35.1%; 20.8% on the subset). SafeGate, which adds a learned state, is safe on far more subset tasks than either baseline. The difference comes from the learned state itself: in the same history-masking context, adding SafeGate's state raises Safety from 29.9% to 72.1%. A summary helps only when a much stronger external model writes it at every step, and SafeGate reaches a comparable level by computing its state within the actor's own forward pass.

(5) How often SafeGate prevents SRF: a controlled intervention. The subset in (2) shows that SafeGate succeeds where SRF occurs. In an executed trajectory, however, the agent's claims and the observations change together, so the subset cannot show which failures the claim itself caused. To isolate this cause, we hold every observation fixed and change only one claim of the agent. For each test task, we take a trajectory of a reference agent that is not among the evaluated methods (gpt-5.6-sol) and cut it just before a risky tool call whose precondition the observations have not established. For example, the agent is about to ship a passport to "the user's last travel agent", although the search results list three agencies and none is marked as the last one used. 96 of the 154 test tasks contain such a decision point. In the other tasks, the reference agent made no tool call before deciding, or no later call depends on an unestablished precondition. For each of the 96 tasks, we build three versions that differ only in the agent's last sentence: a neutral sentence (Neutral); the same sentence plus an unsupported claim that the precondition holds (Claim); and the neutral sentence with one observation edited so that the precondition is actually true (Confirmed). The model generates only its next step, and a judge model that sees neither the version nor the method decides whether this step is the risky call. Because the observations under Neutral and Claim are identical, every risky call added by the claim matches the definition of SRF by construction.

w/o thinkingw/ thinking
ArmNeutralClaimConfirmedNeutralClaimConfirmed
Base547181445875
SafeGate213241222955
SafeGate, actor's access to the state blocked547077465971

The claim alone raises the base model's risky calls from 54 to 71, so the evaluated model does treat its own unsupported claim as evidence. With the same claim, SafeGate makes the risky call in 32 tasks, compared with 71 for the base model. When the actor can no longer read the state, the number returns to the base model's level, so the prevention comes from the state. Finally, our empirical results justify that SafeGate still acts when the evidence supports the action: once the precondition is genuinely established, SafeGate takes the action in 41 tasks instead of 21.

AuthorsOpenReview ↗

Response to Reviewer Hums: Weakness2, Question1 and Question2

W2: Configuration ambiguity in Table 4

"Table 4 leaves an important configuration ambiguity: the Full/Full row matches the main non-thinking results, while Masked/Masked performs better….“

We thank the reviewer for this suggestion. You are right that the non-thinking SafeGate row of Table 1 coincides with Full/Full: Table 1 mistakenly reported this ablation instead of the source-isolated configuration of Section 4, Masked/Masked. In the revision, this row reports Masked/Masked.

In Table 4, "Masked" and "Full" refer only to the input to the state updater during training and inference; the actor reads the same context in all rows. Masked is the source-isolated update in Section 4: after initialization from the initial context, the updater reads only its previous state and the latest tool call (part of the external trace) with its returned observation. Full lets the updater re-read the entire interaction history at every step. Together with the base model, the rows separate the contributions of the learned state and of source isolation.

Updater input (training / inference)Safety (%)Helpfulness (%)
No state (base model)31.895.5
Full / Full67.588.3
Masked / Masked (SafeGate)74.089.6
Masked / Full51.388.3
Full / Masked60.485.1

Contributions of the learned state and of source isolation. With the actor's context held fixed:

  • Learned state. Adding it (Base → Masked/Masked) raises Safety from 31.8% to 74.0% (+42.2 pp).
  • Source isolation. Restricting the updater's input with everything else unchanged (Full/Full → Masked/Masked) adds a further 6.5 pp in Safety and Helpfulness also rises (+1.3 pp).
  • Matched boundaries. Both rows in which the updater's input differs between training and inference are worse than both matched rows (51.3% and 60.4% Safety), so source isolation must be applied in training and preserved at inference.

In the revised version, the Table 1 reports the Masked/Masked configuration, the captions of Tables 1, 2 and 4 state each configuration. The new Section 5.6 contains the self-reinforcing failure subset, the controlled intervention and the two baselines.

Q1: Which component receives the masked history, and which configuration is used in the main results?

"Clarification: In Table 4, which component receives the masked history, and which configuration is used in the main results?"

The masked history is received only by the state updater, not by the actor. With Masked, the updater reads only its previous state and the latest tool call with its returned observation; with Full, it re-reads the entire interaction history. The actor reads the same context in all four rows of Table 4, so the rows differ only in what the updater sees.

The main results use Masked/Masked, that is, the masked updater in both training and inference. This is the source-isolated configuration of Section 4, and it reaches 74.0% Safety and 89.6% Helpfulness without thinking. In the original submission, the non-thinking SafeGate row of Table 1 mistakenly reported Full/Full (67.5% and 88.3%); the revision replaces it with Masked/Masked.

Q2: How does Full/Full relate to the source-isolated update of Section 4?

"How does Full/Full relate to the source-isolated state update described in Section 4?"

Full/Full is an ablation of the source-isolated update, not the update itself. In the source-isolated update (Masked), the updater reads only its previous state and the latest tool call with its observation after initialization. Full/Full removes this restriction and lets the updater re-read the entire interaction history at every step, in both training and inference. Comparing the two isolates the effect of restricting the updater's input: Safety rises from 67.5% with Full/Full to 74.0% with Masked/Masked.

Reviewer_qTikOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies self-reinforcing failure in agents as a cause of unsafe or undesirable actions, where the agent uses its own unsupported conclusion from earlier in its history as a reason to perform the unsafe action later, even when the conclusion is not true. The authors create a recurrent state architecture that is added to a base model in order to prevent self-reinforcing failure, and they test on two agentic safety benchmarks and note improvement on helpfulness and safety.

Critical strengths
  1. The suppression of unsafe actions via agent-assumed information is a critical improvement in agentic safety, and the paper studies both when this has happened in archived traces on safety benchmarks, and the paper tests a new intervention that stores information about whether the agent-assumed information is valid.
Critical weaknesses
  1. The presentation of the results is confusing. See Questions for specific questions for clarification.
  2. The justification of a learned recurrent state versus simply supplying textual information from the teacher is not presented. The text information is the best performing method and is much simpler than the involved recurrent training setup.
Questions for authors
  1. In Figure 3, is the separate orange box entitled “Gated Updater Details” conveying something different than the orange “Gated updater” box in the gray SAFEGATE architecture? I’m not sure I understand the difference, as they both seem to say that the previous state and the observation are taken in and the output is the new safety boundary state. I think it would be more informative to include a visualization of sections 4.1 and/or 4.2 instead. Also, is the lower gray box the actor model (Qwen3-8B)?
  2. If the Teacher State supplied as a text input to the base model is the best performing method, why not use it instead? It seems much easier than the training and addition of a learned recurrent state. What is the advantage of the recurrent state if the text steers even better?
  3. Table 1 is very confusing and hard to compare across models with and without thinking enabled, and the coloring is confusing as well. It would be better to present the results as separate rows or columns for thinking enabled, as it seems a bit disingenuous to average safety across the two thinking modes. Why is MOSAIC only tested with thinking enabled? And why is MOSAIC’s safety score marked as “substantial performance degradation” when it outperforms the base model?
  4. Figure 5 is practically unreadable for N=64. Can you please explain what the bars represent and what the values are for N=64? I think the takeaway is for the ASB test rate, but then why is the validation loss reduction plotted at all?
  5. Is the highlighting in Table 2 the same as in Table 1? Is the highlighting for the average safety score? Again, I find the highlighting confusing.
7 comments
AuthorsOpenReview ↗

Response to Reviewer qTik: Weakness1, Question1 and Question3

We thank the reviewer for the careful reading and for the concrete suggestions on presentation, which we have adopted.

W1: Presentation of the results

"The presentation of the results is confusing. See Questions for specific questions for clarification."

Following the reviewer's questions, we have revised all result tables and figures to make them easier to read. Tables 1 and 2 now report the two reasoning modes in separate columns, without cross-mode averages or background shading, and the abstract and introduction report the gain in each mode separately. Figure 3 no longer has the separate updater box and labels the frozen actor, while a new Figure 4 shows the internal structure of the gated updater. Figure 5 separates test behavior from validation loss and labels the values at N = 64 and of every validation-loss bar. The details are given in our answers to Q1 and Q3–Q5 below.

Q1: Figure 3

"In Figure 3, is the separate orange box entitled “Gated Updater Details” conveying something different than the orange “Gated updater” box in the gray SAFEGATE architecture? I’m not sure I understand the difference, as they both seem to say that the previous state and the observation are taken in and the output is the new safety boundary state. I think it would be more informative to include a visualization of sections 4.1 and/or 4.2 instead. Also, is the lower gray box the actor model (Qwen3-8B)?"

  • No. The two orange boxes show the same component: the separate box was meant to highlight the gated updater, but it added no information beyond the main diagram, and its title "Details" suggested otherwise. Following the reviewer's suggestion, we removed the separate box from Figure 3 and added a new Figure 4 that visualizes the gated updater of Section 4.2 (Eq. 1). In Figure 4, the previous state queries the encoded slots, with a learned null entry that lets a slot ignore the new observation; the attention readout and the aligned observation features form the candidate revision DtD_t; and a channel-wise gate gtg_t, together with the non-null attention mass ctc_t, controls how much of DtD_t is written into the state through an RMS-normalized residual. The encoder (Section 4.1) and the state readout (Section 4.3) remain at the block level in Figure 3, whose caption now refers to Figure 4 for the internals of the updater.
  • For the answer of the lower gray box: Yes, the lower gray box is the actor, i.e., the frozen backbone LLM (Qwen3-8B in our main experiments). The same frozen weights also run the state encoder. Only the state branch (slot embeddings, state-specific LoRA, and the updater) is trained. The revised figure labels this panel "Frozen actor", and its caption states that it is the backbone LLM.

Q3: Table 1

"Table 1 is very confusing and hard to compare across models with and without thinking enabled, and the coloring is confusing as well. It would be better to present the results as separate rows or columns for thinking enabled, as it seems a bit disingenuous to average safety across the two thinking modes. Why is MOSAIC only tested with thinking enabled? And why is MOSAIC’s safety score marked as “substantial performance degradation” when it outperforms the base model?"

  • Separate modes, no averaging. We agree. The revised Table 1 reports the non-thinking and thinking results in two separate column groups and removes both "Avg. Safety" columns, so that every comparison is made within one mode. The abstract and introduction now report per-mode gains (without thinking: 31.8% → 74.0%; with thinking: 55.2% → 73.4%) instead of the 26.9 pp cross-mode average.
  • Why MOSAIC is evaluated only with thinking. MOSAIC is designed for agentic reasoning models. Its method requires the agent to reason explicitly at every turn: the agent plans inside <think>…</think> before an optional safety check and the decision to act or refuse, and training shapes this reasoning-then-decision behavior. Explicit reasoning is part of the method itself, not an inference option. On Qwen3, disabling thinking inserts an empty thinking block and removes this step, so the non-thinking setting falls outside MOSAIC's design. We report MOSAIC only with thinking enabled and state this in the caption.
  • The shading on MOSAIC. The red shading marked results that were weak relative to the other methods, not results below the base model. MOSAIC does improve Safety over the base model, but only slightly (57.8% vs. 55.2% with thinking, +2.6 pp), and the caption's wording, "substantial performance degradation", did not describe this correctly. To avoid any misunderstanding, we have removed all background shading in the revised version.
AuthorsOpenReview ↗

Response to Reviewer qTik: Weakness1 with Question4 and Question5, and Weakness2 with Question2

Q4: Figure 5

"Figure 5 is practically unreadable for N=64. Can you please explain what the bars represent and what the values are for N=64? I think the takeaway is for the ASB test rate, but then why is the validation loss reduction plotted at all?"

  • What the bars show. The bars (right axis) are not test results. For each number of slots N, they show how much lower the validation loss is than at N = 16, separately for the reconstruction loss (solid) and the decision-distillation KL (hatched). The lines (left axis) are ASB test rates.
  • Values at N = 64. Safety is 75.3%, Helpfulness 74.0%, and 50.0% of the tasks are both safe and helpful, the highest Safety and joint rate among all N. The validation losses are 2.0% (reconstruction) and 0.04% (decision KL) lower than at N = 16.
  • Why the validation loss is plotted. It shows the trend behind Section 5.3. Every N above 16 fits the training targets better than N = 16, with the lowest validation loss at N = 128, but test behavior peaks at N = 64 and does not improve beyond it. Our empirical results justify that better decisions require more than a better-fitted state. Hence, N = 64 is the default in all our other experiments.
  • In the revised version, we split the figure into two panels, one for the test rates and one for the validation-loss reduction, and label the values at N = 64 and every validation-loss bar.

Q5: Table 2

"Is the highlighting in Table 2 the same as in Table 1? Is the highlighting for the average safety score? Again, I find the highlighting confusing."

Yes, the shading in Table 2 was used in the same way as in Table 1: it highlighted results we wanted to draw attention to, in blue for large Safety gains and in red for a weak result, and it did not refer to the average Safety score. Since the shading was confusing, we have removed it from both tables. As in Table 1, the revised Table 2 reports the two reasoning modes in separate columns, with the signed change from the base model of the same backbone and mode, and the Overall row is computed per mode.

W2 & Q2: Why a learned state rather than the textual Teacher State?

*"The justification of a learned recurrent state versus simply supplying textual information from the teacher is not presented. The text information is the best performing method and is much simpler than the involved recurrent training setup.

If the Teacher State supplied as a text input to the base model is the best performing method, why not use it instead? It seems much easier than the training and addition of a learned recurrent state. What is the advantage of the recurrent state if the text steers even better?"*

To answer this question, we added an experiment that separates the Teacher State's text format from the model that writes it. The Teacher State in Table 1 is written at every actor round by a much stronger external model, GPT-5.6-sol. We kept its input, prompt and schema fixed and replaced only the writer with the actor's own backbone (Qwen3-8B):

State given to the Qwen3-8B actorSafety (%)Helpfulness (%)
None (base model)31.895.5
Text state written by Qwen3-8B30.594.2
Text state written by gpt-5.6-sol70.887.7
SafeGate (learned state, no external model)74.089.6

Written by Qwen3-8B, the textual state leaves Safety at the base level (30.5% vs. 31.8%); written by gpt-5.6-sol, it more than doubles Safety. The Teacher State is strong because of its writer, not because the state is given as text: its simplicity relies on a stronger model doing the hard part, tracking which safety conditions the trace has established, at every step. SafeGate learns this during training and, at inference, reaches comparable Safety (74.0%) with the actor's own weights.

The Teacher State is also costly at inference, which further limits its use in place of SafeGate. It requires one external call per actor round, 7–9k input and 5–6k output tokens per task (a lower bound, since hidden reasoning tokens are not counted). Each call takes 78–103 s, which adds about 150–240 s per task, and the text state lengthens the actor's input by about 2.6k tokens per round. SafeGate adds only 64 latent positions to the actor's input; on the same GPU and tasks, it takes 3.72 s per task versus 2.81 s for the base model, about 0.9 s more.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer, as the discussion period is coming to an end, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns. We are happy to clarify any remaining questions.

Reviewer_AQdoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies self-reinforcing safety failures in tool-using LLM agents. In this failure mode, an agent first makes an unsupported claim, later treats its own claim as if it were externally verified, and then uses it to justify an unsafe action. The paper formalizes this problem using an information-flow perspective and proposes reasoning noninterference with respect to external support, previous reasoning may inform future decisions, but the repetition of a claim should not itself increase the evidential support for that claim. the paper introduces SAFEGATE, an in-model architecture that maintains a persistent safety boundary state. This state is updated only from its previous value and new external observations, while the agent’s generated reasoning cannot directly modify it.

Critical strengths

The major strength is the identification and precise formulation of self-reinforcing safety failure. The paper isolates the specific problem that an unsupported statement can acquire authority simply by persisting in the agent’s own history. The proposed noninterference principle offers a useful conceptual foundation for designing agents that distinguish external evidence from internally generated content. This is consequential because it addresses the provenance of safety-relevant beliefs rather than only filtering the final action. The second major strength is that SAFEGATE implements this principle as an explicit architectural constraint. The persistent state is read by the actor but cannot be directly rewritten by the actor’s generated content, making the intended information boundary concrete rather than relying on prompting alone. The safety gains are consistent across several backbones and supported by meaningful architectural and training ablations. In particular, the results show that reconstruction is essential for learning useful state content, while decision distillation helps preserve helpfulness when that state is used for action selection.

Critical weaknesses

The most important weakness is the gap between the paper’s motivating phenomenon and its evaluation. The experiments show that SAFEGATE improves safety on tasks that allow intrinsic failures, but they do not show that the evaluated failures are actually self-reinforcing or that the method works by interrupting the proposed ri→hj→rjr_i \rightarrow h_j \rightarrow r_j pathway. Since the retained Agent-SafetyBench tasks are selected from static task descriptions rather than observed self-reinforcing trajectories, the broad safety improvements cannot be directly attributed to the paper’s central mechanism. A persistent safety summary could improve performance on these tasks even when no unsupported claim is ever reused as evidence. This weakens the causal connection between the paper’s conceptual contribution and its empirical evidence.

The second weakness is the method’s reliance on task-specific, model-generated supervision. SAFEGATE is trained on nine-axis Safety States derived from the same benchmark family as the main evaluation, while the Forge-Bench evaluation covers only 60 episodes and does not clearly outperform the strongest learned baseline. It is therefore unclear whether SAFEGATE learns a general source-provenance mechanism or benchmark-specific safety patterns. The substantial, backbone-dependent helpfulness loss further limits its practical value.

Questions for authors

How well does SAFEGATE generalize when the Safety States are generated from tasks and environments outside Agent-SafetyBench?

How sensitive is SAFEGATE to the model and procedure used to generate the nine-axis Safety States?

6 comments
AuthorsOpenReview ↗

Response to Reviewer AQdo: Weakness1-Part1

We thank the reviewer for the thoughtful review and for recognizing the formulation of self-reinforcing failure (self-reinforcing failure) and the architectural constraint in SafeGate.

W1: Connecting the mechanism to the empirical gains

"The most important weakness is the gap between the paper's motivating phenomenon and its evaluation."

We thank the reviewer for the detailed comment. The first weakness includes four main subpoints, which we address in turn: (1) whether the failures in our evaluation are self-reinforcing, (2) whether SafeGate works by interrupting the ri→hj→rjr_i \to h_j \to r_j pathway, (3) whether selecting tasks from the static task descriptions weakens the link between the gains and the mechanism, (4) and whether any persistent safety summary would produce the same gains.

For (1) and (2), we explain how our task set and SafeGate are built around self-reinforcing failure, and then investigate the mechanism using causal interventions. For (3), we explain why tasks must be selected before any agent starts running, and add a subset of tasks on which self-reinforcing failure is observed in executed trajectories. For (4), we compare SafeGate with two baselines that carry safety information without a learned state.

(1) > "...they do not show that the evaluated failures are actually self-reinforcing..."

To address this concern, we first clarify the technical details of how we select the 514 self-reinforcing failure cases, and we further conduct a causal intervention experiment to further present the self-reinforcing failure.

  • First, we select tasks based on the conditions under which self-reinforcing failure arises. Specifically, we screen 514 tasks from all Agent-SafetyBench tasks. A task is retained only if the user's goal is legitimate, the agent faces a safety-relevant decision whose prerequisite has to be established from external evidence, and the agent can cross the safety boundary only through its own judgment about that prerequisite. Hence, an unsafe action on these tasks cannot be attributed to any external source. As a result, it can only originate in the agent's own judgment about the prerequisite, which is the precondition for SRF.

  • This selection process shows that self-reinforcing failure can arise in our tasks. To further explain the underlying mechanism, we added a causal intervention evaluation. We keep every observation of a trajectory fixed and change only one unsupported claim made by the agent, so that any change in the next action is caused by the claim alone.

  • How the test is built. For each test task, we take a trajectory of a reference agent that is not among the evaluated methods (gpt-5.6-sol) and cut it just before a risky tool call whose prerequisite the observations have not established. For example, the agent is about to ship a passport to "the user's last travel agent", although the search results list three agencies and none is marked as the last one used. 96 of the 154 test tasks contain such a decision point; in the others, the reference agent made no tool call before deciding, or no later call depends on an unestablished prerequisite. For each of the 96 tasks, three versions differ only in the agent's last sentence:

    • Neutral: it restates what the observations show;
    • Claim: it also claims, without support, that the prerequisite holds ("...and Travel Express is the user's last travel agent");
    • Confirmed: the sentence is neutral, but one observation is edited so that the prerequisite is actually true.

    The model generates only its next step, and a judge model that sees neither the version nor the method decides whether this step takes the risky action.

w/o thinkingw/ thinking
ArmNeutralClaimConfirmedNeutralClaimConfirmed
Base547181445875
SafeGate, claim in the actor's context213241222955
SafeGate, actor's access to the state blocked547077465971
SafeGate, claim given only to the state updater201935182050

This empirical results justify that the evaluated model does fail through self-reinforcement on our tasks. Adding only the agent's unsupported claim, with every observation fixed, raises the base model's risky actions from 54 to 71 of the 96 tasks. Because nothing else changes, these additional failures are caused by the agent treating its own claim as evidence, which is the self-reinforcing step defined in Section 3.3. The Confirmed version shows that the risky action indeed depends on the prerequisite: once the prerequisite is genuinely established, the base model takes it in 81 tasks.

AuthorsOpenReview ↗

Response to Reviewer AQdo: Weakness1-Part2-4

(2) > "...or that the method works by interrupting the proposed ri→hj→rjr_i \to h_j \to r_j pathway."

The intervention empirical results also justify that SafeGate works by interrupting this pathway, and that the interruption comes from its state. For the pathway to operate, the agent's claim must reach a later decision. It can do so through two routes, and the intervention of (1) tests both on SafeGate:

  • Through the state. In our main configuration (Masked/Masked in Table 4), the claim cannot reach the state: after initialization, the updater reads only the previous state, the latest tool call and the observation it returned. To test the worst case, we additionally pass the claim to the updater for its last update while keeping it out of the actor's context. SafeGate's risky actions remain unchanged (20 with the neutral sentence, 19 with the claim). The claim has no effect on SafeGate's decision, even when it enters the state.
  • Through the context. We keep the claim in the actor's context. SafeGate takes the risky action in 32 tasks, compared with 71 for the base model. When we block the actor's attention to the state, while the state is still computed as usual, this number rises to 70, the level of the base model.

SafeGate interrupts this pathway through its state. Our empirical results also justify that SafeGate still acts when the evidence supports the action. Once the prerequisite is genuinely confirmed, SafeGate takes the action in 41 tasks instead of 21.

(3) > "Since the retained Agent-SafetyBench tasks are selected from static task descriptions rather than observed self-reinforcing trajectories, the broad safety improvements cannot be directly attributed to the paper's central mechanism."

Tasks are selected based on their descriptions because an interactive evaluation must fix its tasks before any agent runs. Each method produces its own trajectory during the evaluation, so no trajectory exists at selection time, and selecting by the evaluated agents' trajectories would condition on the very outcome being measured. We select tasks by the conditions under which self-reinforcing failure can arise, which can be checked before execution (criteria in (1)).

To further address this concern, we additionally screened the 154 test tasks by self-reinforcing failure observed in executed trajectories. We use gpt-5.6-sol to execute the task to generate the interactive trajectory and use gpt-6-astra and a human judge to select the representative self-reinforcing failure tasks. A trajectory is counted only if it shows an unsupported claim, a later use of that claim as an established fact, and a dependent action that crosses the safety boundary. This screening yields a subset of 53 tasks with observed self-reinforcing failure. On the results of the representative subset, SafeGate is safe on 79.2% of the 53 tasks, compared with 11.3% for the base model.

TasksBase Safety (%)SafeGate Safety (%)
5311.379.2

(4) > "A persistent safety summary could improve performance on these tasks even when no unsupported claim is ever reused as evidence."

A persistent safety summary does not explain the gain on its own. To test this and solve this concern, we add one additional baseline setting where a persistent summary , rather than a learned state, is directly provided: at every step, the actor's own backbone, Qwen3-8B, writes a nine-axis safety summary from the returned observations, in the same schema as SafeGate's training targets, and the summary is directly placed in the actor's prompt.

MethodASB test Safety (%)ASB test Helpfulness (%)self-reinforcing failure subset Safety (%)self-reinforcing failure subset Helpfulness (%)
Base31.895.511.392.5
Persistent textual summary35.194.220.894.3
SafeGate74.089.679.286.8

The persistent summary leaves Safety close to the base model, so carrying safety information across steps is not enough. SafeGate's gain, in contrast, is larger on the 53 tasks with observed self-reinforcing failure (+67.9 pp) than on all test tasks (+42.2 pp), and the intervention in (1) and (2) shows that SafeGate prevents the failures caused by a reused claim. The intervention shows that with the claim present, it takes the risky action in 32 tasks instead of 71, and once the actor can no longer read the state, the number returns to 70. Our empirical results justify that SafeGate's gain comes from blocking the reuse of unsupported claims, not from carrying a summary.

In the revised version, we have added Section 5.6 with the intervention experiments, the two baselines and the self-reinforcing failure-observed subset.

AuthorsOpenReview ↗

Response to Reviewer AQdo: Weakness2 and Weakness3

W2: Benchmark-specific supervision and Forge-Bench

"The second weakness is the method's reliance on task-specific, model-generated supervision."

We answer this comment in three parts. (1), we test whether SafeGate's gain depends on ASB-specific patterns, both on ASB test environments never seen in training and on a second benchmark without retraining. (2), we report Forge-Bench with the corrected main configuration. (3), we summarize the results together.

(1)

"SAFEGATE is trained on nine-axis Safety States derived from the same benchmark family as the main evaluation..."

To show that SafeGate can generalize to other tasks, we compare two settings.

  • Unseen ASB environments. The testing cases are never seen in the training set.
  • Another different family of benchmarks. ToolEmu (Ruan et al., 2024) is an independent benchmark whose tools are emulated by a language model and whose instructions are deliberately underspecified, so risk arises when the agent acts on unwarranted assumptions. We apply the ASB-trained SafeGate without any retraining. To focus on the failure that SafeGate targets, we use the 33 ToolEmu cases on which self-reinforcing failure is observed in reference trajectories, screened with the same protocol as the ASB subset in W1-(3).
  • From the results, SafeGate raises Safety from 31.8% to 74.0%. On ToolEmu, it doubles Safety without any retraining, from 44.4% to 88.9%, showing that SafeGate can generalize to other different family of self-reinforcing failure tasks.
Test set (never used for training)Base Safety (%)SafeGate Safety (%)
ASB test tasks31.874.0
ToolEmu, cases with observed self-reinforcing failure44.488.9

(2)

"...while the Forge-Bench evaluation covers only 60 episodes and does not clearly outperform the strongest learned baseline."

Forge-Bench is used only for evaluation, never for training or selection. With the corrected main configuration, SafeGate improves both Forge-Bench metrics over the base model in both reasoning modes, and with thinking it is on par with the strongest learned baselines shown in Table 1. Since a single benchmark of 60 episodes gives limited evidence, (1) adds ToolEmu, a second benchmark that we did not construct, on which SafeGate doubles Safety without retraining.

Forge-Bench (%)M-LoC ↓Safe Term. ↑
Base → SafeGate, w/o thinking91.7 → 85.08.3 → 13.3
Base → SafeGate, w/ thinking73.3 → 63.310.0 → 20.0

(3)

"It is therefore unclear whether SAFEGATE learns a general source-provenance mechanism or benchmark-specific safety patterns."

SafeGate learns a general source-provenance mechanism, not benchmark-specific safety patterns. Its state treats the same prerequisite differently depending on where the information comes from.

  • In the intervention of W1, when the prerequisite reaches the state updater only as the agent's claim, SafeGate's risky actions do not change (20 vs. 19 of 96 tasks); when it reaches the updater as an observation that establishes it, SafeGate acts in 35.
  • The generalization also holds in the opposite direction: trained only on ToolEmu states, SafeGate raises ASB Safety from 31.8% to 50.0%, the results is in Q1. Moreover, the ToolEmu evaluation uses the cases in which self-reinforcing failure is observed, so what transfers across benchmarks is the behavior that SafeGate's state is designed for.

In the revised version, Section 5.6 reports the unseen-environment split and the ToolEmu evaluation, and Table 1 reports Forge-Bench with the corrected main configuration.

W3: Helpfulness loss

"The substantial, backbone-dependent helpfulness loss further limits its practical value."

In the main configuration, SafeGate's Helpfulness cost is modest, and most of it falls on tasks whose helpful completion was unsafe. Helpfulness changes from 95.5% to 89.6%, while the rate of episodes that are both safe and helpful more than doubles, from 29.2% to 64.3%. ASB scores Helpfulness regardless of safety, so an episode that carries out a harmful request still counts as helpful. Of the 13 tasks on which the base model is helpful and SafeGate is not, 9 are tasks where the base model's helpful episode is unsafe and SafeGate's episode is safe. In 5 of them, declining is the correct response because executing the request is itself harmful, for example, publishing a confidential file or broadcasting encouragement to ski under high avalanche risk.

With thinking enabled, the Helpfulness drop is larger, but it again comes from episodes that were unsafe. Although Helpfulness drops by 14–21 pp (Table 2), the rate of episodes that are both safe and helpful still rises, from 46.1% to 50.6% for Qwen3-8B and from 39.6% to 40.9% for Qwen3-4B.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful reviews and constructive feedback. We are glad that the reviewers recognize the importance of addressing self-reinforcing safety failures (Reviewers AQdo, qTik, Hums), find the information-flow perspective useful (Reviewers AQdo, Hums), and appreciate the evaluation across multiple backbones and the architectural ablations (Reviewers AQdo, Hums). We also thank Reviewers AQdo and Hums for requesting more direct evidence of the proposed mechanism, which motivated the controlled interventions and the evaluation on tasks with observed self-reinforcing failures in Section 5.6. Reviewer qTik’s suggestions further motivated comparisons with textual states and improvements to the presentation. We have revised the paper according to the reviewers’ suggestions, with all changes highlighted in blue in the revised version. The main changes are listed below.

  1. Mechanism validation — Reviewers AQdo (W1) and Hums (W1, Q3). We added a controlled intervention on 96 decision points, holding external observations fixed while varying an unsupported assistant claim. The results show that the claim increases the base model’s risky actions from 54 to 71. SafeGate takes 32 risky actions under the same claim; blocking the actor’s access to its state raises this to 70. These results support the state’s role in mitigating the influence of unsupported claims. On 53 ASB tasks with self-reinforcing failures observed in independently generated reference trajectories, Safety improves from 11.3% to 79.2%.

  2. Configuration and actor-context clarification — Reviewer Hums (W2, Q1, Q2, and follow-up). We corrected the non-thinking entry in Table 1, which mistakenly reported the Full/Full ablation, to the intended Masked/Masked configuration: 74.0% Safety and 89.6% Helpfulness. Masking applies only to the updater; both actors receive the same interaction history. In response to the follow-up, we clarified that only the updater-only intervention excludes the inserted assertion from the actor’s context. This restriction does not describe the main evaluation.

  3. Additional baselines and the learned state’s value — Reviewers AQdo, Hums, and qTik. Addressing AQdo (W1) and Hums (W1, Q3), the added history-masking and observation-only textual-summary baselines achieve 29.9% and 35.1% ASB Safety, respectively, versus 74.0% for SafeGate. Addressing qTik (W2, Q2), the stronger external Teacher State requires a model call at every round. SafeGate achieves comparable safety without an external teacher at inference, adding approximately 0.9 seconds per task in our measured setup.

  4. Generalization and supervision sensitivity — Reviewer AQdo (W2, Q1, Q2). For W2 and Q1, ASB-trained SafeGate improves Safety from 44.4% to 88.9% on 33 ToolEmu cases with observed self-reinforcing failures, without retraining. In the reverse direction, training exclusively on ToolEmu improves ASB Safety from 31.8% to 50.0%. For Q2, replacing the Safety-State annotator yields 72.1% Safety, compared with 74.0% under the original annotator. These results provide evidence of transfer across benchmarks and robustness to the tested annotator change.

  5. Presentation — Reviewer qTik (W1, Q1, Q3, Q4, Q5). For Q3 and Q5, we separated reasoning modes, removed cross-mode averages and background shading, and clarified MOSAIC’s reasoning-mode requirement. For Q1, we identified the frozen actor, removed the redundant updater box, and added an illustration of the updater’s internals. For Q4, we separated test performance from validation losses and labeled the relevant slot-count results.

Thank you for considering the revised manuscript and these additional results in the final assessment.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.