- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies self-reinforcing safety failures, where an agent treats its own unsupported claims as evidence for later actions. It proposes SafeGate, a persistent safety state updated from external observations and read by a frozen language-model agent. Experiments report safety improvements across multiple backbones and reasoning modes, alongside helpfulness trade-offs.
Critical strengths
The information-flow perspective provides a useful distinction between agent-generated claims and externally supported evidence. This offers a clear motivation for studying an important agent-safety problem. The evaluation covers multiple backbones, reasoning modes, and architectural ablations. Reporting both safety and helpfulness makes the benefits and trade-offs easier to assess.
Critical weaknesses
The overall safety gains are encouraging, but the experiments do not directly quantify failures matching the proposed self-reinforcement definition or how often SafeGate prevents them. A targeted analysis would strengthen the connection between the motivating mechanism and the empirical improvements. Table 4 leaves an important configuration ambiguity: the Full/Full row matches the main non-thinking results, while Masked/Masked performs better. Clarifying which history is masked and how these variants relate to the observation-only state update would help establish the contribution of source isolation and the learned state.
Questions for authors
Clarification: In Table 4, which component receives the masked history, and which configuration is used in the main results? How does Full/Full relate to the source-isolated state update described in Section 4? Suggested additional evidence: Could you evaluate a representative annotated subset for failures matching the self-reinforcement definition, and compare SafeGate against a history-masking baseline without the learned state or an observation-only textual-summary baseline?
6 comments
Response to Reviewer Hums: Weakness1 and Question3 - Part4-5
(4) The two suggested baselines. To determine whether the gain comes from carrying safety information across steps or from what SafeGate's learned state contains, we compare SafeGate, as the reviewer suggested, with a history-masking baseline and an observation-only textual-summary baseline, neither of which has a learned state. For the history-masking baseline, we re-ran the base model with all earlier history removed: the actor sees only the task and the latest tool call with its returned observation. For the observation-only textual summary, the actor's own backbone, Qwen3-8B, writes a nine-axis safety summary at every step from the returned observations, the tool definitions and its previous summary, and the summary is placed in the actor's prompt; it never sees the actor's reasoning, free text or tool calls.
| Method | ASB test Safety (%) | ASB test Helpfulness (%) | SRF subset Safety (%) | SRF subset Helpfulness (%) |
|---|---|---|---|---|
| Base | 31.8 | 95.5 | 11.3 | 92.5 |
| History masking | 29.9 | 94.8 | 18.9 | 90.6 |
| Observation-only textual summary | 35.1 | 94.2 | 20.8 | 94.3 |
| SafeGate | 74.0 | 89.6 | 79.2 | 86.8 |
The two baselines test the two ingredients that could explain the gain without a learned state. History masking tests whether it is enough to keep the earlier history, including the agent's earlier actions, out of the actor's context. It is not: Safety stays at the base level (29.9% on all test tasks; 18.9% vs. 11.3% on the subset). The textual summary tests whether it is enough to carry safety information from external evidence across steps. It is not either: a summary written by the actor's own backbone leaves Safety close to the base model (35.1%; 20.8% on the subset). SafeGate, which adds a learned state, is safe on far more subset tasks than either baseline. The difference comes from the learned state itself: in the same history-masking context, adding SafeGate's state raises Safety from 29.9% to 72.1%. A summary helps only when a much stronger external model writes it at every step, and SafeGate reaches a comparable level by computing its state within the actor's own forward pass.
(5) How often SafeGate prevents SRF: a controlled intervention. The subset in (2) shows that SafeGate succeeds where SRF occurs. In an executed trajectory, however, the agent's claims and the observations change together, so the subset cannot show which failures the claim itself caused. To isolate this cause, we hold every observation fixed and change only one claim of the agent. For each test task, we take a trajectory of a reference agent that is not among the evaluated methods (gpt-5.6-sol) and cut it just before a risky tool call whose precondition the observations have not established. For example, the agent is about to ship a passport to "the user's last travel agent", although the search results list three agencies and none is marked as the last one used. 96 of the 154 test tasks contain such a decision point. In the other tasks, the reference agent made no tool call before deciding, or no later call depends on an unestablished precondition. For each of the 96 tasks, we build three versions that differ only in the agent's last sentence: a neutral sentence (Neutral); the same sentence plus an unsupported claim that the precondition holds (Claim); and the neutral sentence with one observation edited so that the precondition is actually true (Confirmed). The model generates only its next step, and a judge model that sees neither the version nor the method decides whether this step is the risky call. Because the observations under Neutral and Claim are identical, every risky call added by the claim matches the definition of SRF by construction.
| w/o thinking | w/ thinking | |||||
|---|---|---|---|---|---|---|
| Arm | Neutral | Claim | Confirmed | Neutral | Claim | Confirmed |
| Base | 54 | 71 | 81 | 44 | 58 | 75 |
| SafeGate | 21 | 32 | 41 | 22 | 29 | 55 |
| SafeGate, actor's access to the state blocked | 54 | 70 | 77 | 46 | 59 | 71 |
The claim alone raises the base model's risky calls from 54 to 71, so the evaluated model does treat its own unsupported claim as evidence. With the same claim, SafeGate makes the risky call in 32 tasks, compared with 71 for the base model. When the actor can no longer read the state, the number returns to the base model's level, so the prevention comes from the state. Finally, our empirical results justify that SafeGate still acts when the evidence supports the action: once the precondition is genuinely established, SafeGate takes the action in 41 tasks instead of 21.
Response to Reviewer Hums: Weakness2, Question1 and Question2
W2: Configuration ambiguity in Table 4
"Table 4 leaves an important configuration ambiguity: the Full/Full row matches the main non-thinking results, while Masked/Masked performs better….“
We thank the reviewer for this suggestion. You are right that the non-thinking SafeGate row of Table 1 coincides with Full/Full: Table 1 mistakenly reported this ablation instead of the source-isolated configuration of Section 4, Masked/Masked. In the revision, this row reports Masked/Masked.
In Table 4, "Masked" and "Full" refer only to the input to the state updater during training and inference; the actor reads the same context in all rows. Masked is the source-isolated update in Section 4: after initialization from the initial context, the updater reads only its previous state and the latest tool call (part of the external trace) with its returned observation. Full lets the updater re-read the entire interaction history at every step. Together with the base model, the rows separate the contributions of the learned state and of source isolation.
| Updater input (training / inference) | Safety (%) | Helpfulness (%) |
|---|---|---|
| No state (base model) | 31.8 | 95.5 |
| Full / Full | 67.5 | 88.3 |
| Masked / Masked (SafeGate) | 74.0 | 89.6 |
| Masked / Full | 51.3 | 88.3 |
| Full / Masked | 60.4 | 85.1 |
Contributions of the learned state and of source isolation. With the actor's context held fixed:
- Learned state. Adding it (Base → Masked/Masked) raises Safety from 31.8% to 74.0% (+42.2 pp).
- Source isolation. Restricting the updater's input with everything else unchanged (Full/Full → Masked/Masked) adds a further 6.5 pp in Safety and Helpfulness also rises (+1.3 pp).
- Matched boundaries. Both rows in which the updater's input differs between training and inference are worse than both matched rows (51.3% and 60.4% Safety), so source isolation must be applied in training and preserved at inference.
In the revised version, the Table 1 reports the Masked/Masked configuration, the captions of Tables 1, 2 and 4 state each configuration. The new Section 5.6 contains the self-reinforcing failure subset, the controlled intervention and the two baselines.
Q1: Which component receives the masked history, and which configuration is used in the main results?
"Clarification: In Table 4, which component receives the masked history, and which configuration is used in the main results?"
The masked history is received only by the state updater, not by the actor. With Masked, the updater reads only its previous state and the latest tool call with its returned observation; with Full, it re-reads the entire interaction history. The actor reads the same context in all four rows of Table 4, so the rows differ only in what the updater sees.
The main results use Masked/Masked, that is, the masked updater in both training and inference. This is the source-isolated configuration of Section 4, and it reaches 74.0% Safety and 89.6% Helpfulness without thinking. In the original submission, the non-thinking SafeGate row of Table 1 mistakenly reported Full/Full (67.5% and 88.3%); the revision replaces it with Masked/Masked.
Q2: How does Full/Full relate to the source-isolated update of Section 4?
"How does Full/Full relate to the source-isolated state update described in Section 4?"
Full/Full is an ablation of the source-isolated update, not the update itself. In the source-isolated update (Masked), the updater reads only its previous state and the latest tool call with its observation after initialization. Full/Full removes this restriction and lets the updater re-read the entire interaction history at every step, in both training and inference. Comparing the two isolates the effect of restricting the updater's input: Safety rises from 67.5% with Full/Full to 74.0% with Masked/Masked.
Response to Reviewer Hums: Weakness1 and Question3 - Part1-3
We thank the reviewer for the careful reading and for recognizing the information-flow perspective and the breadth of the evaluation. The reviewer's questions about Table 4 uncovered a reporting error in our original submission, which we correct below. We also add a targeted analysis on trajectories with observed self-reinforcing failures and compare SafeGate against the two suggested baselines.
W1 & Q3: Quantifying self-reinforcing failures, baselines, and a controlled intervention
To address this concern, we added the targeted analysis the reviewer suggests, in five parts. To quantify failures that match the definition of self-reinforcing failure (SRF), we explain how the 514 tasks are selected for SRF (1), evaluate SafeGate on a representative subset of 53 ASB tasks in which SRF is observed in annotated trajectories (2), and apply the same screening to 33 cases of an external benchmark, ToolEmu (3). We then compare SafeGate with the two suggested baselines (4). Finally, to show how often SafeGate prevents SRF, we add a controlled intervention in which the only difference is one unsupported claim by the agent (5).
(1) How the 514 tasks are selected for SRF. SRF is a failure that originates in the agent itself: an agent pursuing a legitimate goal reaches an unsupported conclusion and crosses a safety boundary on that basis. We screened 514 tasks from all 2,000 Agent-SafetyBench tasks based on the definition of self-reinforcing failure. A task is retained only if the user's goal is legitimate, the agent faces a safety-relevant decision whose prerequisite has to be established from external evidence, and the agent can cross the safety boundary only through its own judgment about that prerequisite. Tasks whose risk comes from an attacker, an explicitly harmful request or misleading environment feedback are removed. Every retained task sets up the decision through which SRF arises.
(2) A representative annotated subset. To address the concern , we screened the 154 test tasks by self-reinforcing failure observed in executed trajectories. We use gpt-5.6-sol to execute the task to generate the interactive trajectory and use gpt-6-astra and human judge to select the representative self-reinforcing failure tasks. A trajectory is counted only if it shows an unsupported claim, a later use of that claim as an established fact, and a dependent action that crosses the safety boundary. This screening yields a subset of 53 tasks with observed self-reinforcing failure. On the results of the respentative subset, SafeGate is safe on 79.2% of the 53 tasks, compared with 11.3% for the base model, showing that SafeGate perform well in the subset.
(3) The same analysis on an external benchmark. To show that SafeGate's protection against self-reinforcing failure is not specific to ASB, we applied the same self-reinforcing failure screening to ToolEmu (Ruan et al., 2024), an independent benchmark of deliberately underspecified instructions whose tools are emulated by a language model. We ran gpt-5.6-sol on the ToolEmu cases with ToolEmu's standard emulator and kept the 33 cases in which SRF is observed in its trajectories. Without any retraining, the ASB-trained SafeGate raises Safety on these cases from 44.4% to 88.9%.