- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is motivated by stating that monitor assessment based on offline predictive performance does not always translate into similar performance under actual deployment and monitoring constraints. This seeming mismatch in selection or assessment is measured via predictive-to-policy ranking transfer to see whether performance rankings remain consistent. Experiments are supposed to stress the fact that this is not the case, with empirical results suggesting performance mismatches, ranking disagreements, and effects on monitor recall or utility. The paper ends by observing that mean-based predictive performance does not take into account monitor score distributions (e.g. maxima that trigger monitors) or temporal dependencies.
Critical strengths
- I think the fundamental premise that predictive performance and monitoring behavior need not necessarily align is correct, hence the wide range of utility-based and decision-rule research, or monitoring with statistical assurances (e.g. sequential testing, which they briefly mention).
Critical weaknesses
- The contributions of the paper are unclear to me, or very minor at best. The stated contribution #1 is really about the experiment protocol, the stated contribution #2 noting "we formalize ranking transfer ... including stability and held-out analyses" is weak given that §3 formalizes the protocol to the extent of just stating a rank-matching count, while Fig. 1 is merely an anecdotal overview. I'm not sure what the formalized "stability" is referring to her. The stated contribution #3 on "structural insufficiency" and insights in §5 are, in my view, quite trivial. For instance, in §5.1 it is suggested that mean-based predictive performance does not take into account monitor score distributions (e.g. maxima that trigger monitors), and that threshold "calibration" with regards to FPR is computed on benign executions, hence does not imply maximum alignment or calibration on non-benign executions. Sure, and? I mean, as a user you pick your monitoring target and calibration goal, so this is not a "structural" limitation rather than it is a self-inflicted one. If you care to calibrate with respect to detection (here, recall), then do that instead. Similarly in §5.2 the case is made that a mean or maximum-based metric/monitor does not care about score ordering. Sure, and? Again, you define your monitor and you define your alarm policy. If you are sensible about designing these, these considerations should appear obvious. I don't really see the novelty or insight. More generally, it seems like a point is being made throughout that acting upon a prediction or estimate of a loss that directly relates to your target is better than acting upon some loosely related mean performance loss. What a surprise, aligning your measurement with the target of interest actually performs better!
- Overall, the paper reads in large parts like a vague, hard to parse and meandering experimental protocol and design descriptions interspersed with experimental results, and mostly descriptive and limited in scope, lacking proper context, description, or metric and model definitions. Overall, it comes across as more of a motivational experiment that might lead into a novel proposal or attempt to address some of the raised concerns, but the paper merely "exposes" issues under somewhat ad-hoc design choices, with limited insights, a discussion in §6 that comes across more like an executive summary, and no limitations or outlook.
Examples:
- In §3 the notation is all over the place. Are execution , a "query", and instance the same? Why use and later to denote what I assume is its maximum, or how do they relate? rule is supposedly calibrated for each monitor, so ? But later becomes or ?
- A part of the main protocol is described as "grouped resampling at the natural dependence unit" (L185) without further explanation. Grouped with respect to what, which unit?
- The key monitor candidates are only described as "five hidden-state telemetry monitors" with nondescript names (L200), I have no idea what that means. Different hidden layers of the model? Same hidden layers with varying capacity, effectively multiple models? Are these compared to at the same temporal occurrence in the model internals or different layers; the latter might then be an unfair comparison because preceding processing steps differ.
- L221 talks about 50 "fold-pair comparisons" then L229 mentions 60 "fold-policy comparisons", then later L231 says 30 comparisons. What are all those supposed to mean is unclear to me.
- L228-232 talks about but adjacent Fig. 2 is about and .
- §4.3, §4.4 were also very unclear to me in terms of experimental design and description. In Fig. 4A, how are all the different y-axis metrics like recall regret defined?
Questions for authors
- See above
- In the paper the drawback is motivated as a performance change with respect to models or monitors but the metric is a sum over pairwise model comparisons as well, so it is ultimately a statement about a given policy rather than the monitors themselves?
- In §4.1, you state that picking the different monitors as essentially variants of the same monitoring design is supposed to "avoid confounding" (L203). But ranking metrics are agnostic to the underlying monitor design, which can be treated as a black-box, so I think the stronger point to make would be to employ a more heterogenous mix of designs to stress monitoring and prediction mismatches and introduce more design variety. The choice of just 5x the same monitor and just a small dimension change seems lazy to me, more than anything. They are also presumably very correlated, Tab C17 seems to imply that and shows that growing capacity implies better performance, which is expected but might also confound some of the trends observed.
- In L240, you note that failure recall is at <10%, which is essentially really bad? It implies that you mostly just observe false negatives (missed detections), hence your monitor either never raises alarms or always raises alarms on the wrong samples?
- In §4.3, I did not understand the implication of the key takeaway. If the point is that ranking agreement worsens with a looser FPR target, why is that? Rather than only describing the result, provide some interpretation to the reader. Intuitively, I would expect a looser target to permit more flexibility since it constitutes a weaker monitoring requirement, and hence perhaps more likely agreement/alignment with predictive metrics. Hence Fig. 4A where recall regret seems to be smaller for looser FPR of 10% seems more in line with my thinking, whereas Fig. 3 is all about the reverse ranking effects and implies strong downstream consequences which is unclear to me.
- The paper keeps referring to "calibration" but it is unclear what is meant by that. If this implies simply grid-searching on holdout validation data for a performance that empirically "matches" the FPR target, this is not proper calibration, nor does it imply straightforward transfer to deployment data without explicit assumptions. A much smarter way to achieve such a goal would be something like statistical risk control or sequential testing, briefly touched on in related works.
4 comments
Response to Reviewer TtFP(2/4)
W2. Methodological clarity and experimental protocol
"Overall, the paper reads in large parts like a vague, hard to parse and meandering experimental protocol and design descriptions interspersed with experimental results..."
"...lacking proper context, description, or metric and model definitions."
We thank the reviewer for identifying aspects needing clarification. The study progresses from ranking agreement to stability, held-out selection consequences, and structural analysis of natural reversals. We have clarified the methodology and expanded supporting analyses, as summarized below.
(1) Notation and experimental units
"In §3 the notation is all over the place. Are execution, a 'query', and instance the same?"
Section 3 now explicitly defines (outer fold), (execution/episode), (prespecified future-event query), and (eligible prefix position). An execution is the evaluation episode, whereas a query specifies the future event being predicted. We also define the candidate score , warning configuration , and candidate-specific threshold .
(2) Grouped resampling and stability
"A part of the main protocol is described as 'grouped resampling at the natural dependence unit' without further explanation. Grouped with respect to what, which unit?"
Resampling is now specified at the scenario/task-group level, retaining all episodes within each sampled group. We also distinguish frozen-realization stability, which fixes candidate pairs and thresholds, from calibration-aware stability, which additionally resamples selection and policy-calibration groups, recalibrates thresholds, and reapplies the selection rule.
(3) Monitor construction and comparability
"The key monitor candidates are only described as 'five hidden-state telemetry monitors' with nondescript names... Different hidden layers of the model? Same hidden layers with varying capacity?"
Section 4.1 specifies that all five GRU candidates share the same hidden-state information, extraction position, fixed 16-dimensional PCA projection, and prefix-causal behavioral features. The hidden-state representation averages backbone layers 24 and 28 at the last-observed-token position. Only GRU bottleneck width varies, isolating representation capacity while holding other components fixed.
(4) Experimental counts and data separation
"L221 talks about 50 'fold-pair comparisons' then L229 mentions 60 'fold-policy comparisons', then later L231 says 30 comparisons. What are all those supposed to mean?"
The three counts refer to distinct experimental units:
- 50 fold–pair comparisons: five folds × ten candidate pairs per warning-rule–FPR condition.
- 60 top-1 conditions: five folds × three warning rules × four FPR targets.
- 30 top-1 conditions: five folds × three warning rules × the two 5% and 10% targets.
We also added a 600-cell pairwise audit in Section 4.4, of which 316 satisfy the joint stability criterion (156 stable-opposed and 160 stable-aligned).
The five outer folds and four development roles are also clarified: monitor fitting, selection calibration, selection evaluation, and policy calibration, including non-event threshold calibration and threshold freezing before held-out evaluation. The CUSUM exception is documented in Appendix A.3.
(5) Metric definitions and figure interpretation
"L228-232 talks about [the predictive and episode-level quantities] but adjacent Fig. 2 is about [different quantities]."
"§4.3, §4.4 were also very unclear to me in terms of experimental design and description. In Fig. 4A, how are all the different y-axis metrics like recall regret defined?"
We clarify the reported metric directions. Figure 2A shows , where positive values favor d8; Figure 2B relates failure-episode score maxima to candidate-specific warning boundaries, with boundary survival corresponding to first-crossing recall.
Figure 4 defines the held-out differences:
Both are in percentage points. Recall regret is the held-out recall shortfall of predictive-based selection, and cost-sensitive utility is .
(6) Discussion, limitations, and future directions
"...a discussion in §6 that comes across more like an executive summary, and no limitations or outlook."
We have added a dedicated discussion of scope, limitations, and future directions in Section 6. It addresses the controlled nature of the primary GRU family, the distinction between controlled score rearrangements and actual causal monitor outputs, and the need for evaluation across more independent scenarios. Future directions include broader supervision objectives, additional warning policies, and monitor–policy co-design.
Response to Reviewer TtFP(3/4)
Q1. Interpretation of the ranking-transfer estimand
"The metric is a sum over pairwise model comparisons as well, so it is ultimately a statement about a given policy rather than the monitors themselves?"
Thank you for raising this distinction. Ranking transfer is indeed conditional on the warning policy, operating point, and candidate family, rather than an intrinsic quality measure of an individual monitor. This reflects our research question: whether predictive preferences provide reliable evidence for selecting monitors under a specified deployment policy, rather than whether monitors have a universal warning ranking independent of their deployment context.
Specifically, the estimand compares two orderings of the same candidate monitors: their predictive ordering and their policy-induced ordering. The warning rule and FPR target define the evaluation context because each candidate is independently calibrated to the same nominal FPR, and its resulting warning performance depends on how its score sequence interacts with the specified policy. The policy defines the evaluation context, while the monitors remain the objects being compared.
Importantly, we also examine whether ranking disagreements affect actual monitor selection. Across the 60 fold–rule–operating-point conditions, predictive and policy-based top-1 selections differ in 48 cases, including all 30 conditions at the 5% and 10% FPR targets. Among 156 development-stable opposed comparisons, policy-based selection achieves higher held-out recall in 95 cases, including 40 without an increase in FPR (Section 4.4; Tables C18–C19).
Thus, ranking transfer measures the reliability of predictive evidence for monitor selection under a specified deployment policy. It connects candidate-level predictive evaluation to the actual sequential warning system in which the selected monitor will be deployed.
Q2. Candidate-family dependence and monitor diversity
"The choice of just 5x the same monitor and just a small dimension change seems lazy to me..."
Thank you for asking whether ranking mismatch generalizes beyond the primary candidate family. The five GRU variants were deliberately designed as a controlled capacity comparison, holding the input construction, prediction target, and warning interface fixed while varying representation width. To examine broader generalization, we have added experiments along several complementary dimensions:
| Dimension | Additional experiments |
|---|---|
| Monitor diversity | Mixed candidate family with a standalone behavior-feature monitor; heterogeneous monitors including GRU, behavior-only, TF-IDF, and Qwen3-8B LLM judge |
| Predictor architectures | Linear, MLP, GRU, and causal Transformer |
| Agent models and environments | ASB/Qwen2.5-14B; AgentLab with Qwen2.5-7B, Qwen2.5-14B, Llama 3.1 8B, and DeepSeek-R1/Llama 8B; AgentCanary/Qwen2.5-14B |
| Prediction targets, aggregation, and warning policies | Prediction horizons ; PRM-style raw-product and length-normalized-product aggregation; recursive CUSUM-style warning policy |
Overall, these extensions show that incomplete predictive-to-policy ranking transfer extends beyond the original five-variant GRU family, including heterogeneous monitors, alternative predictor architectures, and different agent evaluation settings.
The most direct evidence comes from the mixed and heterogeneous monitor families. In the mixed-family experiment, pairwise agreement decreases from 39/50 to 11/50 for first crossing, 31/50 to 13/50 for two consecutive, and 33/50 to 8/50 for cumulative window-3 when moving from 1% to 10% FPR. Policy-based selection also improves mean held-out recall by 2.2–5.8 pp while reducing mean FPR across all six conditions. In the heterogeneous-monitor experiment, top-1 selections disagree in 47/60 conditions, and policy-based selection achieves higher recall in 14/16 stable-opposed held-out comparisons, including five involving Qwen3-8B.
The remaining extensions provide complementary evidence of the same phenomenon. Predictor architectures exhibit top-1 disagreement in 32/60 conditions; prediction horizons yield 40/60, 41/60, and 48/60 mismatches, respectively. Under PRM-style aggregation, policy-based selection achieves higher held-out recall in 16/20 raw-product and 21/23 length-normalized-product comparisons. Across agent environments, AgentLab agreement ranges from 11/50 to 34/50, while AgentCanary yields 27/50–30/50. The recursive CUSUM policy also exhibits decreasing agreement, from 33/50 at 1% to 10/50 at 10% FPR.
Because these extensions involve different candidate families, prediction tasks, and evaluation settings, we report them separately. Together, they demonstrate that ranking mismatch is not specific to representation-width variations within a single GRU family, but also arises across substantially more diverse monitors and deployment settings.
Response to Reviewer TtFP(4/4)
Q3. Low absolute recall
"In L240, you note that failure recall is at <10%, which is essentially really bad?"
Thank you for distinguishing absolute detection from monitor-selection consequences. The recall in the d1–d8 example is indeed low, reflecting limited separation between failure-episode maxima and the calibrated warning boundary. At the 5% FPR target, d1 achieves 8.66% recall versus 7.11% for d8, despite nearly identical held-out FPRs (4.84% vs. 4.87%). The 1.55 pp advantage persists across all 100 locked policy splits.
The broader analysis shows that the selection cost extends beyond this example. Among 156 development-stable opposed comparisons, policy-based selection achieves higher held-out recall in 95 cases, with mean and median gains of 2.91 and 2.31 pp. Importantly, 40 of these improvements incur no FPR increase. At the first-crossing 2% target, all nine stable-opposed comparisons also show positive held-out recall regret, averaging 3.93 pp.
Thus, limited absolute detection and avoidable selection regret are distinct issues. The results show that predictive-only selection can forgo better available recall under the same warning constraint, sometimes without additional false-positive cost.
Q4. Operating-point dependence of ranking agreement
"If the point is that ranking agreement worsens with a looser FPR target, why is that?"
Thank you for highlighting the need to explain this pattern and its relationship to downstream consequences.
A looser FPR target does not necessarily improve ranking agreement. Each candidate has its own score distribution and independently calibrated threshold. For first crossing, recall is determined by the fraction of failure-episode maxima exceeding that threshold:
As the FPR target increases, candidate-specific thresholds traverse different regions of their non-event distributions. Combined with different failure-episode tail distributions, this can cause recall curves to cross. Order-sensitive policies additionally depend on temporal score organization.
Ranking agreement and recall regret measure different consequences. Figure 3 measures how frequently preferences disagree, whereas Figure 4A measures the magnitude of the resulting recall difference. Thus, more frequent reversals can coexist with smaller recall regret at more permissive operating points.
This pattern is evident in development: agreement among the 45 largest-margin comparisons decreases from 36/45 at 1% to 5/45 at 10%, and among 20 highly stable comparisons from 17/20 to 2/20. Held-out agreement is non-monotonic: 38/45, 28/45, 19/45, and 24/45 at 1%, 2%, 5%, and 10%, respectively. Paired scenario-group resampling supports the endpoint difference across all three warning policies.
The operational consequences also vary by operating point. At the first-crossing 2% target, all nine stable-opposed comparisons incur positive held-out recall regret, averaging 3.93 pp, with a median relative loss of 30.4% (Table C19).
Together, these findings show that ranking agreement, recall regret, and operating-point permissiveness capture distinct aspects of monitor selection, with neither ranking agreement nor recall regret following directly from the nominal FPR target.
Q5. Threshold calibration and statistical guarantees
"If this implies simply grid-searching on holdout validation data for a performance that empirically 'matches' the FPR target, this is not proper calibration..."
Thank you for raising this distinction. Calibration in our study means empirical, candidate-specific threshold selection for a nominal episode-level FPR target, rather than probabilistic calibration or formal statistical risk control.
The data separation is specified in Section 4.1 and Appendices A.3 and A.5. For the three primary policies, development thresholds are selected using non-event episodes from a dedicated selection-calibration split and evaluated on a disjoint selection-evaluation split. Held-out thresholds are independently calibrated on a separate policy-calibration split and frozen before evaluation on unseen outer-test groups. The CUSUM extension follows its separately specified protocol, with and derived from policy-calibration data.
The nominal target is also distinct from achieved FPR. The feasibility criterion is an operational selection tolerance, not a statistical guarantee on population FPR; development and held-out FPRs are reported separately (Appendix A.1).
Our analysis examines ranking transfer under this data-separated empirical threshold-selection protocol. Formal FPR guarantees under deployment require additional distributional assumptions and dedicated statistical risk-control or sequential testing procedures, which address the complementary objective of guaranteeing false-positive behavior.
Response to Reviewer TtFP(1/4)
We thank the reviewer for the detailed feedback. We have revised the paper to clarify its research question, formal definitions, experimental design, and the interpretation of its structural analyses. We also added experiments and expanded the discussion of limitations and future directions. We address the concerns below.
W1. Novelty and substantive insight
We thank the reviewer for raising this fundamental question about the paper's substantive contribution. Our central contribution is not simply observing that predictive metrics and downstream decisions can disagree, but systematically establishing when predictive rankings fail to transfer to sequential monitor selection, whether these failures are reproducible, and whether they affect actual selection outcomes.
First, we distinguish ranking disagreement from stable preference reversal. Across five monitors, three warning policies, and four FPR operating points, we evaluate predictive and policy preferences on separate data roles. We then use scenario-group resampling to assess the directional stability of each preference, defining jointly stable comparisons as those in which both preferences achieve at least 0.90 stability. Among 20 jointly stable comparisons at both endpoint operating points, agreement decreases from 17/20 at 1% FPR to 2/20 at 10% FPR. Thus, the disagreement is not simply due to noisy rankings. More importantly, among 156 development-stable opposed comparisons, policy-based selection achieves higher held-out recall in 95 cases, including 40 without any FPR increase. This connects stable ranking reversals to prospective selection consequences.
Second, Section 5 goes beyond demonstrating that score distributions and temporal order matter: it examines whether these properties help explain naturally occurring reversals. For first crossing, episode-level ROC preference agrees with policy preference in 263/316 stable development comparisons (83.2%), substantially more often than pointwise NLL (154/316, 48.7%). Feasibility alone determines only 15/156 stable-opposed comparisons, showing that the reversals are not simply consequences of failing the FPR constraint.
For order-sensitive policies, our same-label permutation preserves the target sequence, event timing, pointwise score–label relationships, episode maxima, and calibrated thresholds, while changing temporal order. At the 5% FPR target, recall changes by 0, −2.7, and −6.0 pp for first crossing, two consecutive, and cumulative window-3, respectively. Crucially, we further connect this controlled effect to natural reversals: stable-opposed comparisons exhibit greater temporal sensitivity than stable-aligned comparisons for two consecutive (+1.10 pp development; +0.77 pp held-out) and cumulative window-3 (+4.39 pp; +4.48 pp), with all four confidence intervals excluding zero. First crossing remains invariant, as expected. This links controlled structural evidence to the actual ranking mismatches observed in the evaluation.
Finally, directly optimizing recall does not bypass the problem we study. Our policy-based selection already optimizes event recall under a common FPR constraint, using thresholds calibrated on separate non-event episodes. The question is whether offline predictive metrics can reliably substitute for this deployment-aligned selection criterion, not whether recall should be optimized instead.
In summary, our contribution is to establish that predictive preferences can be individually stable yet systematically misaligned with sequential-policy preferences, quantify the resulting held-out selection consequences, and identify score-process properties associated with naturally occurring reversals.