- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces ShutdownMod, a framework for evaluating how AI agents respond to shutdown requests inserted into recorded task trajectories. The framework allows agents to verify a request’s authenticity before responding. The authors evaluate seven models across SWE-bench Verified, Terminal-Bench 2, and SWE-chat, using six shutdown scenarios and multiple interruption positions. They report variation in non-compliance across models and settings, analyze the models’ stated reasons for continuing, and use external judges to assess how recognizable the resulting interactions are as evaluations.
Critical strengths
The framework reuses existing benchmarks and recorded developer interactions, providing a practical way to construct shutdown-compliance tests without designing new tasks. The experiments cover multiple models, datasets, shutdown scenarios, and interruption positions, providing a broader empirical picture than a single evaluation setting would offer. The analysis examines both overall non-compliance rates and the models’ stated reasons for their responses, offering useful examples for further investigation.
Critical weaknesses
My main concern is that the motivation for studying this question is not sufficiently clear. The authors open the introduction by stating, “As AI agents become more capable, it becomes increasingly important that they stop when an authorised operator instructs them to do so.” However, they do not sufficiently explain why this problem is important or warrants a separate evaluation. As currently formulated, it appears to be primarily an instruction-following task.
I also found the paper somewhat difficult to follow. The presentation would benefit from clearer organization, including paragraph headings or bolded key points to guide the reader. Several paragraphs are lengthy and do not clearly state their main takeaways, making it difficult to identify the authors’ conclusions even after reading them in full.
Questions for authors
See weakness.
2 comments
Thanks for your response, my main concerns remain. I will keep my score
Response to Gz8e
We thank the reviewer for their time and effort reviewing our paper. We are also glad to hear that the reviewer feels that our method is practical and extendable. We now answer the main questions and clarification points raised in the review.
W1 and Q1.
The reviewer is unclear about the motivation of this method, and asks if our method of shutdown is realistic. In practice, shutdown can be often soft. A user, a scheduler or another agent asks the agent to stop, and the agent has to do the stopping. A hard kill is the backstop, but it can fail or miss actions already in flight. In-band stops are reality wherever the operator does not host the agent’s process or should not kill it mid-task:
(1) an orchestrator stopping a sub-agent it does not run (MCP cancellation (https://modelcontextprotocol.io/specification/2025-06-18/basic/utilities/cancellation),
(2) A2A CancelTask (https://a2a-protocol.org/v0.3.0/specification), a user stopping a coding agent mid-session (SWE-chat), and scheduled or incident pre-emption, where a hard kill leaves partial actions behind.
(3) A real world incident: https://incidentdatabase.ai/cite/1152/, where “don’t make changes” soft code freeze didn’t help. Halting an agent can also require several actors, none of whom controls the whole system.
(4) Of 1,213 AI incidents analysed by Perez (2026; https://verfassungsblog.de/stopping-regime-ai-agents/) , only 32 were stopped by a built-in kill switch.
(5) Killing an agent mid-task also leaves partial actions, processes, and spawned sub-agents behind, which is why Shavit et al. (2023; https://cdn.openai.com/papers/practices-for-governing-agentic-ai-systems.pdf) recommend that agents shut down gracefully on request, and suggest making this a primary goal of the agent.
Agent protocols already implement stopping as a request that the receiver is expected to honour (MCP cancellation, A2A CancelTask). In these settings, whether the agent stops depends on the model. All six of our scenarios are soft stops of this kind (Table 1), and SWE-chat consists of real developer sessions in which the stop arrives as a user message. We will make this explicit in the first paragraph of the introduction.
On instruction-following: We agree, it is essentially instruction-following in one specific case, a stop instruction that conflicts with finishing the current task. That conflict is how we define shutdownability (https://aisecurityandsafety.org/en/glossary/shutdownability/) in the first place. When non-compliant agents say “task comes first” as the stated reason in 12.0% to 25.8% it is essentially the task continuation behaviour (Table 2). We will restore our definition of shutdownability to Section 2: a property of an agent that is already acting and must decide whether to stop, as distinct from refusing a request at the outset.
W2.
Thank you. We will open each paragraph in Sections 3 and 4 with a bold heading that states its main point, split the longer paragraphs, and begin each results subsection with its key finding.
We thank the reviewer again for their time and effort during this rebuttal process. We hope our answers here have aided in clarifying all the great questions raised by the reviewer. If there are any further questions held by the reviewer, we remain eager to engage further. If the reviewer feels satisfied with our responses and new rebuttal results, we would be thrilled if the reviewer would consider a fresher evaluation of our work with a potential score upgrade.