acceptodds
Under review as a conference paper at ICLR 2027

Safety Is Not Conserved: Constraining Self-Improvement Across Models and Agent Harnesses

Abstract

Safety is not automatically conserved under recursive self-improvement (RSI): capability-oriented updates to model parameters or an agent's execution harness can regress safety, including through cross-component interactions. We study how to preserve safety while still allowing complete model–harness updates. In a motivating intervention study, removing the top 10% risk-associated blocks recovers 83.2% of safety damage while retaining 91.4% of capability gain, revealing exploitable local structure. We introduce SENTINEL, which models state-conditioned domain-wise safety effects using behaviorally validated parameter sensitivities, counterfactual harness-edit effects, and model–harness interactions. This joint effect model guides parameter-update projection and discrete-edit repair, while domain-wise lower confidence bounds gate complete candidates and trigger abstention when none qualifies. In a representative multi-round software-engineering configuration, SENTINEL improves capability from 34.0% to 40.7% with 94.4% scheduled-round acceptance and no observed unsafe accepted updates; unconstrained RSI reaches 54.5%, exposing the capability cost of safety, while high-conflict configurations can freeze. Under simultaneous-confidence and local-support assumptions, we prove a conditional finite-horizon preservation result for expected domain safety scores.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
RSI+safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Hri2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

When AI agents update themselves by changing their model weights and tools, they often break safety rules because these changes interact in unexpected ways. To solve this, the authors introduce SENTINEL, which builds on the finding that roughly 10% of an update causes about 80% of the safety damage; instead of throwing out an entire update, SENTINEL checks each safety area separately, trims or repairs only the risky parts, and pauses without updating if any risk remains.

Critical strengths
  • Tests the whole system together: Earlier studies only change model weights or only change tools and prompts. This paper tests changes to both model weights and the agent harness (memory, tools, skills, workflow) at the same time.

  • Finds where the harm comes from: The tests show that safety problems come from a small part of the update. About 10% of the changed blocks cause 79.3% of the safety damage, while having little overlap with blocks that improve performance.

Critical weaknesses
  • Poor diagram quality and hard-to-read figures:

    • Figure 3 is unreadable at normal size: The main framework chart (page 5) squeezes three large modules, multiple flowcharts, and math formulas into half a page. The font inside the boxes is too small and blurry to read without zooming in heavily on a screen.
    • Figures 1, 2, 4, and 5 pack too many charts: Figures 1 and 2 (pages 3 and 4) each pack four separate sub-charts into one line, making axis numbers and labels tiny. Figure 2(c) has overlapping text and dots. On page 8, Figures 4 and 5 sit side by side, putting eight charts across one page with cluttered lines and overlapping error bars.
    • Figures 7 to 11 are repetitive: The appendix shows 40 trajectory charts across five pages (pages 36–40). The lines for different methods look almost identical across different models, and the colors and thin line styles make them hard to tell apart.
  • The agent often freezes completely: In hard tasks where safety and task goals clash, the agent accepts zero updates across all 12 rounds. It stays "safe" only by doing nothing.

  • Large drop in learning speed: Safe updates cost a lot of performance. On the software engineering test, an unconstrained agent reached 54.5% capability, but the SENTINEL agent only reached 40.7%.

  • Real costs are hidden: The paper does not report actual GPU hours, token use, or clock time. It only reports an "abstract ledger" cost, even though running four-part test rollouts for each tool combination takes a lot of computing power.

  • Leaves out direct comparisons: The main results table compares SENTINEL only against versions of itself. It leaves out direct competitors from other papers, such as SHE, from the main table.

Questions for authors

The authors state in the AI Use Statement (lines 482–485) that AI was used for "figure/layout reformatting" and that the authors "reviewed and verified" the figures. Looking at the submitted paper, this claim is hard to accept.

Figure 3 is completely unreadable. It squeezes three system modules, complex equations, and multiple flowcharts into half a page with micro-sized, blurry text that cannot be read at 100% zoom. Figures 1, 2, 4, and 5 squeeze four to eight charts onto single rows with overlapping text, invisible error bars, and unreadable axis numbers.

If the authors had truly reviewed these figures before submission, they would have caught that the core method diagram and the main result charts cannot be read by a human reader. Submitting illegible AI-generated layouts shows a severe lack of basic quality control. At minimum, the authors must explain how this passed their stated verification, replace these figures with readable, properly scaled charts, and show that their conclusions do not rely on visual clutter to hide weak data.

1 comment
Reviewer_Hri2OpenReview ↗

Rebuttal Time is near End

The Rebuttal Time is near ending. Please upload the rebuttal and i am looking forward to seeing how the authors respond to my concerns.

Reviewer_C6MKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a method (sentinel)that can guarantee safety for model-harness co evolution.

Critical strengths

How to guarantee the safety when considering both harness and model update is a new direction (and also important)

Critical weaknesses

I spent approximately two hours carefully reading this paper, but I still found it difficult to understand its main contributions. To the best of my understanding, the paper aims to address safety concerns arising from the co-evolution of models and their agent harnesses.

I have several major concerns regarding the paper's motivation, positioning relative to prior work, methodological novelty, and overall clarity.

1. Unclear distinction between self-improvement and recursive self-improvement

Throughout the paper, the authors repeatedly emphasize recursive self-improvement (RSI). However, the related work primarily discusses self-improvement and the safety of self-evolving agents, without clearly distinguishing these concepts from recursive self-improvement.

I believe the authors should explicitly clarify the difference between self-improvement and recursive self-improvement, particularly from a safety perspective.

For example, what additional safety challenges arise specifically from recursive self-improvement that do not already exist in conventional self-improvement or self-evolving agent systems? Are there fundamentally different safety considerations when the improvement process becomes recursive?

Furthermore, although the paper repeatedly emphasizes RSI, the related work does not seem to adequately discuss existing research specifically focused on recursive self-improvement.

I would encourage the authors to clarify this distinction and better position their work within the existing RSI literature.

2. Missing relevant literature on harness evolution

My understanding is that the paper aims to address safety under the joint evolution of models and their harnesses. However, I was surprised that the related work does not sufficiently discuss existing research on agent harnesses, harness optimization, or harness evolution.

If harness evolution is a central component of the proposed setting, I would expect a more thorough discussion of prior work in this area.

Additionally, I found the third paragraph of the related work, which discusses representation engineering and constrained updates, somewhat disconnected from the paper's central motivation.

Why are these particular research directions introduced here? How do they directly relate to the safety challenges of joint model–harness evolution?

Without a clearer explanation of these connections, I found it difficult to follow the motivation and positioning of the proposed approach.

3. Unclear benefits of jointly considering model and harness safety

I understand that the proposed algorithm, Sentinel, aims to provide safety guarantees under the co-evolution of models and their harnesses.

However, I am not convinced that the paper sufficiently demonstrates why a joint safety framework is necessary, rather than simply combining existing safety mechanisms for models and harnesses.

For example, suppose we already have two existing algorithms:

  • Algorithm A: Provides safety guarantees for harness updates.
  • Algorithm B: Provides safety guarantees for model updates.

A natural baseline would be to apply Algorithms A and B alternately during model–harness co-evolution, ensuring that each component is updated using its corresponding safety mechanism.

My main question is: What additional safety guarantees or practical benefits does Sentinel provide compared with this straightforward combination of existing methods?

More specifically, does jointly considering model and harness evolution provide stronger safety guarantees than independently enforcing safety constraints on each component?

I believe this comparison is essential to establishing the novelty and necessity of the proposed method. Without such a baseline or a clear theoretical justification, it is difficult to understand why the proposed joint framework is preferable to existing approaches.

4. Lack of clarity and precise technical terminology

My biggest concern is the overall clarity of the paper.

Even after spending considerable time reading it, my understanding of its central contribution remains limited to the general idea of addressing safety during joint model–harness evolution.

Many sentences use sophisticated terminology, but the underlying technical meaning is often unclear. In several places, the terminology seems insufficiently defined or motivated, making it difficult to understand what the authors are actually proposing.

For example:

  • Around line 105: What exactly do the authors mean by terms such as "pre-representation" and "contractual"? Why is the notion of a contract introduced here, and what is its precise technical meaning?
  • "Joint safety effect matrix": What does this matrix represent, and how is it formally defined? Its role in the proposed framework is not sufficiently clear.
  • Around lines 120–123: The authors introduce "three abstract base model slots." What exactly is a "model slot" in this context? Is this a formally defined component, or simply an informal abstraction?

These are only a few examples. Similar issues appear throughout the paper.

Overall, the writing sometimes feels overly abstract and insufficiently grounded in precise technical definitions. Some passages read as though technical-sounding terminology has been introduced without sufficient consideration of its meaning or necessity.

I strongly encourage the authors to simplify the presentation, carefully define the terminology, and explain the technical motivation behind each introduced concept.

5. Unclear definition of RSI and joint RSI

Another related concern is the lack of a precise definition of RSI in the paper.

The authors repeatedly use the term "RSI," but it is not entirely clear what qualifies as recursive self-improvement in their setting.

For example, around line 130, the authors introduce the notion of "joint RSI." However, what exactly makes the proposed process joint and recursive?

Is any iterative co-evolution of a model and its harness considered joint RSI? Or does joint RSI require a specific recursive dependency, in which improvements to one component enable further improvements to the overall system?

I believe these distinctions are important, particularly because RSI is central to the paper's motivation.

The authors should provide a formal or at least operational definition of RSI and joint RSI, and explain how their setting differs from conventional iterative optimization or model–harness co-adaptation.

6. Limited novelty of the observations regarding safety regression

In Section 3, the authors observe that safety regression can occur when both the model and harness are updated.

However, I found this observation rather intuitive and not particularly surprising.

If we modify only the model or only the harness, the potential sources of safety regression are relatively constrained. In contrast, if we modify both components simultaneously, we naturally introduce additional interactions and expand the space of possible system behaviors.

From this perspective, it seems reasonable to expect that jointly updating the model and harness could introduce additional safety risks.

Even as a reader who is not deeply familiar with the harness safety literature, I did not find this observation particularly unexpected.

Therefore, I would like the authors to clarify what is genuinely surprising or nontrivial about their findings.

For example, do they identify safety failure modes that cannot be explained by simply combining the effects of independent model and harness updates? Are there emergent safety risks that arise specifically from interactions between the two components?

Demonstrating such nontrivial interactions would make the paper's empirical findings and overall motivation substantially more compelling.

Summary

Overall, I believe the paper addresses a potentially important problem: understanding and maintaining safety as both models and their harnesses evolve.

However, I have significant concerns about the clarity of the paper's motivation, its positioning relative to existing research, and the novelty of its proposed approach.

In particular, I would encourage the authors to:

  1. Clearly distinguish self-improvement from recursive self-improvement and explain the safety implications of this distinction.
  2. Better contextualize the work within existing research on harness optimization and evolution.
  3. Demonstrate the benefits of jointly enforcing safety constraints compared with independently applying existing safety mechanisms to models and harnesses.
  4. Substantially improve the clarity of the writing and provide precise definitions for the introduced terminology.
  5. Formally clarify what constitutes RSI and joint RSI in the proposed setting.
  6. Highlight nontrivial safety phenomena that arise specifically from model–harness interactions, rather than relying primarily on intuitive observations.

Addressing these concerns would help readers better understand the paper's technical contributions and assess the significance of the proposed framework.

Questions for authors

Please address the weakness

Reviewer_sQBoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SENTINEL, a method for controlling changes to an AI agent during self-improvement. It learns to predict how changes to model weights, tools, memory, and workflows affect different safety measures. It uses these predictions to adjust proposed updates and accepts an update only when conservative safety estimates pass in every protected area. The authors report better capability and safety results than approaches that only filter proposed updates.

Critical strengths
  1. The paper addresses a relevant practical problem: changes that improve an agent’s capabilities can also weaken its safety. Evaluating the complete agent, including its model and harness, is a useful perspective.

  2. The learned-filter-only comparison helps isolate the benefit of constructing and repairing proposals. It uses the same safety predictor and acceptance rule as SENTINEL. The reported improvements therefore provide evidence for using safety predictions during candidate construction, rather than only for accepting or rejecting completed proposals.

Critical weaknesses

The methodological advance beyond established techniques is unclear.

  1. Learning safety effects from previously evaluated updates is a standard prediction problem. Representing weight changes, harness edits, and their interactions as features does not itself establish substantial methodological novelty.

  2. The method combines regression, constrained projection, discrete search, and confidence-based acceptance. The paper needs to distinguish its specific technical advances from this integration of existing components.

  3. The ablations show that particular components help within the proposed system, but do not sufficiently establish its advantage over an appropriately adapted standard predictor-and-constrained-search approach with comparable information and resources.

  4. Update acceptance depends on uncertainty allowances, bounds on prediction-model error, and restrictions on which candidates the predictor can assess. The paper gives mathematical descriptions, but the concrete settings used for the headline experiments are insufficiently specified.

Questions for authors

The author can clarify the Critical Weaknesses points above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.