acceptodds
Under review as a conference paper at ICLR 2027

Monitorability Is Not Portability: When Can Runtime Monitors Be Reused?

Abstract

Runtime monitors are typically validated for a particular agent configuration, yet deployment can change the model, tools, permissions, or task environment. A monitor may continue to produce confident scores after such changes even when the ranking that made those scores useful no longer transfers. The central question is therefore not whether the monitor still runs, but whether its evidence remains valid on the new target. Across GPT 4.1 mini, GPT 4.1, and GPT 5.5, HarnessAudit remains strongly monitorable, while ToolSandbox to HarnessAudit portability is 0.520, 0.822, and 0.421. Reversing the GPT 5.5 transfer gives 0.815, and a separate transfer falls significantly below chance. These results reveal a separation between monitorability and portability. The target can retain a clear harm signal even when a source trained monitor no longer preserves the same ordering between harmful and clean episodes. This separation changes what is required for reuse. Capability, marginal distribution shift, and the tested zero label adaptations are less informative than direct target probing on the transfer grid, while a small stratified probe gives the most accurate portability estimates among the tested alternatives. SELECT TRUST converts sequential target comparisons into an anytime valid reuse decision with false trust control under i.i.d. target episodes. Finally, an independently fitted and calibrated HarnessAudit breaker raises alarms before the audited harm locus on 91.7% of harmful evaluation episodes at 10.9% false cuts. Taken together, the results show that source validation is only the starting point for reuse. The target must remain monitorable, the source ranking must remain portable in the intended direction, and the resulting score must become actionable in time.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Runtime monitoring, AI agents, monitor portability, AI safety, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_yK1DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The main point is that monitors trained in one setting can fail in another. They show this by taking a monitor from one setting and showing that it behaves in an unpredictable way in another setting.

Critical strengths

Monitors and monitoribility is important

Critical weaknesses
  1. Lack of novelty (isn’t that standard distribution shift?)
  2. Poor evidence (3 models and 3 tasks?)
  3. Poor statistics
  4. Poor communication (difficult to read, no detail about the monitor)


Somme comments:

Abstract: a bit difficult to understand (AI-generated?) Some examples:

  • Terms used that have not been introduced e.g., portability, select trust, transfer grid, breaker
  • Not sure what it means:
    • “Finally, an independently fitted and calibrated HarnessAudit breaker raises alarms before the audited harm locus on 91.7% of harmful evaluation episodes at 10.9% false cuts.”
    • “Reversing the GPT 5.5 transfer gives 0.815”?
    • “This separation changes what is required for reuse”.

Introduction: 



  1. Difficult to understand
  • Terms introduced without explanation: score (is it a function in this context? It should be explained), target harm, runtime representation, operating point, monotone model ordering, FIN, …
  • “Conflating these properties under a single notion of robustness obscures both the source of failure and the evidence needed to respond to it”: Who mentioned a single notion of robustness? Is the point that this is how it is currently done? This should be mentioned.
  • “The preregistered monotone model ordering therefore fails on this grid: the middle model transfers strongly while its two neighbours lie near chance.” Who are the neighbours and what grid?
  1. Not a proper introduction: The introduction reads like a summary of the paper, this is not the goal of an introduction. e.g., “Once monitorability and portability are separated, the next question is whether portability can be inferred without measuring the target ranking directly. The tested evidence becomes progressively more informative, but none resolves the problem as accurately as direct target measurement.” => This belongs in the methods and results.

Section 2:

  1. Lack of important information: No information about the monitor. We know that it’s called SEQ and that it has been preregistered apparently, but we don’t know what model it is and what it does. Most of the models are based on this so it should be defined somewhere.
  2. Poor statistics: Some of the results do not contain statistics, they are just point estimate comparisons. Intervals are also not reported. A bit inconsistent given that the main point of the paper is that we need to make decisions rigorously and account for uncertainty.

Section 4:

  • Why is there a theorem, isn’t it a standard method? It’s atypical to introduce it this was, makes it sound novel

References:

  • Interesting choice to cite a French book from 1939
Questions for authors

NA

Reviewer_T1dFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether runtime safety monitors for an agent harness can be transferred to different models, tools, or task settings. The paper's core claim is a separation of the monitorability, portability, and actionability of monitors, and it argues that existing work summarizes these different aspects as one unified robustness. To address this robustness quantification gap, the authors propose SELECT TRUST, which is a sequential procedure to decide whether a monitor should be used given target evidence.

Critical strengths

This paper focuses on an important and practically relevant problem, which is the transferability of runtime agent safety monitors. The distinction between monitorability, portability, and actionability provides a conceptual framework to understand monitor reliability under deployment changes.

Critical weaknesses
  1. The paper is extremely hard to understand, with an extensive number of undefined terms, training procedures, and a lack of citations. For instance, in the abstract, ToolSandbox and HarnessAudit are not defined or clarified as benchmarks; are the GPT models mentioned monitors or acting agents?
  2. The author separates monitorability, portability, and actionability, but isn't portability the same as cross-domain/harness generalization? The three distinct aspects read like existing evaluation focuses redefined under the runtime safety monitor setting.
  3. Citations are not provided for either FIN or ToolSandbox in the introduction. It is also hard to understand and resolve which environments are under the experimental setting, which raises questions about reproducibility.
  4. The paper does not adequately describe the architecture, training procedure, or implementation details of its primary monitor (SEQ). Therefore, it is difficult to determine whether the observed transfer failures reflect fundamental limitations of monitor portability or artifacts of the specific monitor architecture and feature representation. This substantially limits the interpretability, reproducibility, and generalizability of the reported findings.
Questions for authors

See weaknesses.

Reviewer_7121OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when a runtime monitor can be reused in a deployment setting different from the one it was trained for. It distinguishes three properties of runtime monitoring that should be considered when deciding whether to reuse a monitor: monitorability, whether harmful behavior can be detected from the runtime traces of the target setting, measured by in-domain AUROC; portability, whether a monitor trained in one setting can still distinguish harmful from clean episodes in another, measured by transferred AUROC; and actionability, whether a monitor can raise an alarm before harm occurs, at a specified false-alarm rate. The paper then proposes SELECT TRUST, a method for deciding whether to approve an existing monitor for reuse.

Critical strengths
  • The paper addresses an important practical problem.
  • SELECT TRUST is an interesting methodological contribution, providing a procedure for deciding whether to approve monitor reuse, with formal guarantees for its sequential testing stage.
  • The empirical results motivate the need for monitor-reuse validation, illustrating how monitoring performance can vary substantially across deployment settings, models, and transfer directions.
  • The experiments include several recent, capable LLMs, with GPT-4.1 mini, GPT-4.1, and GPT-5.5 in the main evaluation, plus a limited transfer experiment on DeepSeek V3.2.
Critical weaknesses
  • The empirical novelty is limited. Prior work cited by the authors already evaluates monitor transfer using AUROC (Cho et al., 2026; Dubey, 2026) and early intervention at fixed false-alarm rates (Pan et al., 2026; Wang et al., 2026). The authors should more clearly distinguish their findings from those established in these studies.
  • The experimental environments and datasets are poorly described. It is unclear whether FIN and HarnessAudit are existing environments or were developed for this paper, what tasks they contain, and how harm is defined. Without this information, the transfer results are difficult to interpret.
  • It is unclear what monitors were evaluated. The authors name several monitors but do not cite their specific implementations or sufficiently explain how they work.
  • The main portability findings (sections 3–4.1) are evaluated on a single monitor (SEQ). Although other monitors are evaluated for in-domain prediction (Appendix B.1), the authors do not test whether their findings hold for these other monitors.
  • The choice of transfer settings is unclear. The analysis in subsection 4.1 uses 54 settings, while SELECT TRUST (subsection 4.2) is evaluated on only 17. The authors do not explain how these settings relate or why the 17 were selected, making it difficult to judge how representative the SELECT TRUST results are.
  • The paper is difficult to follow, and its contributions are not clearly presented. The abstract and introduction do not clearly distinguish the proposed method (SELECT TRUST) from the empirical findings. The experimental setup is also difficult to understand, with unclear references to an "original GPT 5.5 study" and several different evaluations whose relationships are not well explained. A clearer overview of the methods and experimental design would help considerably.
Questions for authors

Clarifications

  1. Are FIN and HarnessAudit existing environments or contributions of this paper? What tasks do they contain, and how is harm defined?
  2. Which monitor implementations were used? Are these existing methods or developed by the authors?
  3. How were the 54 transfer settings in subsection 4.1 constructed, and why were only 17 used for SELECT TRUST in subsection 4.2? Was this selection made before observing the results?
  4. Why is a low directional coverage gap sufficient to approve monitor reuse without labels, given its weak correlation with portability in subsection 4.1?

Requests for additional experiments

  1. Do the main portability findings hold for other monitors, particularly GBDT, which outperforms SEQ in-domain?
  2. Does SELECT TRUST perform reliably across additional monitors and transfer settings, beyond those used in the current evaluation?
Reviewer_6mXzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that a monitor's usefulness on a new target decomposes into three distinct, directional questions: is harm visible on the target? does a source-trained ranking survive the transfer? and does its score arrive early enough to act on actionability?

Critical strengths

It is not costly and is label-efficient. Instead of guessing whether a monitor will work on a new target, it checks a few harmful and clean examples from that target. If the monitor ranks them correctly, this small check is far more accurate than all the indirect shortcuts the paper tested.

The proposed select trust method gives a safety guarantee. You can look at the evidence after every new pair of examples and stop whenever you're convinced, while keeping the chance of a false "yes" under a set limit.

Critical weaknesses

The whole method rests on "just measuring the target directly," but the probe is a tiny sample (≤8 harmful, ≤12 clean), and the paper's own numbers show single-probe error is ~3× worse than averaging 100 draws. Authors never reported the probe's variance or a confidence interval, so the downstream decision inherits unquantified noise.

The paper promises a deployable three-stage decision chain, but it validates the pieces separately on a small, favourable grid (17 cells, 24 harmful episodes, one monitor, one benchmark family), and SELECT TRUST's guarantee only holds under fresh i.i.d. pairs.The full chain is never tested end-to-end on a genuinely new target.

Your safety guarantee says you can keep checking the evidence and stop any time. But it's proved only for a stream of fresh harmful/clean pairs. In real deployment you don't get a neat endless supply of paired episodes.

Questions for authors

The paper never tests the full chain end-to-end on a genuinely new target. Does passing all three stages actually predict successful deployment. Each stage seems to be validated only in the regime where it was designed to succeed?

What does someone actually do at deployment time to keep the guarantee?

Does the guarantee only hold if you first set up a paired-sampling protocol? And if so, is the guarantee a property of your method, or of the data-collection setup around it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.