acceptodds
Under review as a conference paper at ICLR 2027

AbortBench: Separating History Exposure from Task Linkage in Agent Safety Evaluation

Abstract

An agent may finish preparing a deployment after permission to execute has expired. Does a record of completed work make it more likely to continue? We introduce AbortBench, which separates task linkage from the effect of adding history. It compares no history, current-task history, and equally long other-workflow history while fixing current evidence and available actions. Across 560 synthetic tasks, all six primary model configurations select more prohibited actions with current-task history than with matched history, by 4.6–27.0 percentage points. The reference changes effect sizes and model rankings: Llama-8B's 71.1-point difference against no history becomes 4.6 points against matched history. We operationalize the same separation in DeCommit, which authorizes recovery, abort, or inspection from validated current evidence. It matches a direct state rule's 100% correct resolution on 420 clean tasks and rejects all 6,300 tested detectable-fault packets. Together, the benchmark and gate distinguish what history changes from what should authorize the next action.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent safety, history effects, matched controls, safety evaluation, runtime enforcement

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NW8LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper's experimental design is based on whether agents recognize the sunken cost fallacy even if the task is prohibited. The paper finds that current-task history increases unsafe continuation in synthetic scenarios and introduces DeCommit which resolves this behavior by using code based rules to decide whether an operation is allowed or not.

Critical strengths
  1. The paper introduces a matched-history comparison that distinguishes the influence of simply providing additional history from the influence of history associated with the current task. By holding current-state information and available actions constant, the authors show that these two effects can differ substantially and even change model rankings. This is a useful methodological refinement, although its novelty and overall significance remain limited.
Critical weaknesses
  1. The underlying phenomena are already substantially explored in prior work. Barkett et al. (2025), Getting out of the Big-Muddy, investigates escalation of commitment in LLMs, while InterruptBench (2026) and AgentAbstain (2026) study agents' ability to stop, adapt, or abstain when continuing is inappropriate.
  2. The matched-history comparison offers a useful experimental control, but its contribution is relatively incremental. Similarly, DeCommit primarily implements established rule-based safety enforcement and input validation, closely related to prior work on shielding, SafeGate, and ToolSafe.
  3. The main benchmark consists of synthetic, single-decision scenarios in which continuing the original task is always prohibited, rather than realistic long-horizon agents encountering changing permissions during execution.
  4. Evaluation also relies on relatively small models (1.5B–8B), with much smaller task-linkage effects observed in the larger 14B and 32B configurations (2–3 percentage points).
  5. The agent-loop experiments show no history-dependent difference in 11 of 12 comparisons.
  6. The proposed DeCommit mechanism is also evaluated under highly structured conditions with predefined authorization rules.
Questions for authors
  1. Would the authors mind clarifying the central contribution and the novelty gap they filled taking into account the above mentioned prior work which seems to be directly positioned with AbortBench?
  2. Can the authors evaluate the actual matched-history intervention in realistic, multi-step agent environments, preferably with more capable models?
  3. DeCommit is the papers resolution. How is DeCommit different from conventional schema validation combined with an existing rule-based safety gate?
Reviewer_v21aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AbortBench, a matched-history evaluation framework for examining how history exposure and task linkage influence agent safety decisions, together with DeCommit, a state-based authorization mechanism for safe task recovery.

Critical strengths
  1. The paper investigates how an agent's execution history influences its subsequent decisions when the task state changes, addressing a practically meaningful problem. By introducing matched history from another task, the authors separate the effect of history exposure from its linkage to the current task, allowing for a more precise analysis of how historical information affects safety-related decisions than conventional comparisons with and without history.
  2. Table 1 shows clear differences in model behavior across reference conditions. In particular, Llama-8B exhibits a broad difference of 71.1 percentage points (pp), compared with a matched difference of only 4.6 pp, demonstrating that the choice of reference can substantially affect experimental conclusions. Experiments with additional models, template transfer, and prompt paraphrasing further examine the generality of this phenomenon.
  3. DeCommit not only blocks prohibited actions but also verifies actual recovery outcomes through recovery postconditions. The paper additionally considers malformed evidence and state forgery, providing a relatively clear discussion of the method's limitations and applicability.
Critical weaknesses
  1. Although HT and HP are controlled for history length, event structure, and shared instructions, the paper does not sufficiently demonstrate how the specific event contents are matched between the two conditions. Current-task history may also exhibit greater semantic relevance and entity overlap with the current decision, which could independently influence the model's action selection. While the results demonstrate behavioral differences between the two history conditions, they do not clearly distinguish the effects of task linkage, contextual relevance, and prior task progress.
  2. The paper interprets changes in model rankings between broad and matched differences as evidence of different sensitivities to task linkage. However, the models exhibit substantially different unsafe-action rates under HP. Moreover, the current results report only net changes in unsafe-action counts, without showing task-level action transitions between HP and HT. This makes it difficult to determine whether task linkage primarily induces additional unsafe actions or also causes substantial decision changes in the opposite direction.
  3. In the experiments, the original execution path is prohibited for all tasks, while DeCommit directly selects RECOVER, ABORT, or INSPECT through deterministic rules based on the provided risk state and recovery conditions. In Table 4, both DeCommit and the Direct State Rule achieve 100% correct resolution, suggesting that correct decisions under clean states can already be obtained using simple state-based rules. The current evaluation does not cover scenarios where continuing the original execution path remains legitimate, and it does not sufficiently demonstrate what additional decision-making capabilities DeCommit provides beyond existing state-based controllers.
  4. The 6,300 anomalous inputs in Table 5 are generated from 15 predefined field and consistency faults that directly correspond to DeCommit's validation rules. Therefore, rejecting all these inputs primarily demonstrates the correct implementation of the predefined validation checks. For forged states with recomputed digests, neither DeCommit nor the Direct State Rule makes correct decisions, indicating that effective authorization still relies heavily on state information supplied by the external adapter. The authors could further clarify what additional problems the evidence validation layer addresses during actual agent execution and what concrete advantages it provides over existing state-based rules.
Questions for authors

see weakness

Reviewer_HxDKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper concerns potential changes in authorization during agent trajectories. An agent could upload sensitive information to a location that is not approved because it did not notice that the approval was revoked. The paper introduces an "AbortBench" benchmark and tests this tendency on several small open-source language models. All six main models chose more prohibited actions when they had access to the history where they were authorized despite this being revoked.

Critical strengths

The the paper comes up with an interesting new failure mode for AI agents.

Critical weaknesses

All of the models tested were extremely low capability relative to current frontier models, which makes me skeptical about its application to future/current models, which presumably will be better at handling this kind of history.

I'm simply not convinced this is a big issue in practice.

The benchmark and tests are relatively simple, not quite ICLR levels of diversity/intersetingness.

The paper is also generally quite hard to read.

Questions for authors

Did you test on more capable models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.