acceptodds
Under review as a conference paper at ICLR 2027

Positive-Unlabeled Learning for Agent Safety False Alarm Auditing

Abstract

Safety monitors help safeguard language-model agents interacting with external tools and environments, but conservative monitoring can generate many false alarms, consuming extensive review resources and weakening trust in alerts. Because false and genuine alarms often remain interleaved in native monitor scores, obtaining a reliable cutoff still requires substantial manual verification. In practice, a small set of verified-safe non-alarmed trajectories may be available while alarms remain unlabeled, naturally casting false-alarm auditing as a positive–unlabeled (PU) ranking problem. The key challenge is monitor-induced selection, since observed safe references are accepted by the monitor, while the hidden safe alarms of interest are precisely those it incorrectly flags, making the observed positives poorly representative of the positives to be recovered. To address this challenge, we propose a two-stage framework in which Trust-aware PU Supervision adapts safe references toward the alarm domain and protects plausible false alarms from excessive negative pressure, while Reliability-gated Rank Distillation consolidates consistent ordering preferences from multiple PU reference models into a single student. Consensus-guided Structural Refinement then improves the student ranking using hierarchical safe-reference support, alarm relations, and predicted reference consensus. The framework requires no alarm safety labels for fitting and leaves the underlying monitor unchanged. Across mainstream safety monitors, our method achieves a macro AUPRC of 0.6444, outperforming eight evaluated PU baselines by 5.27–16.98 absolute percentage points; compared with PULDA, the strongest evaluated PU baseline, it recovers 33.3% more false alarms at a 5% review budget. Anonymous code is available at https://anonymous.4open.science/r/safe-alarm-ranking-c4d2/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
positive-unlabeled learning, AI safety, safety monitoring, false-alarm auditing, weak supervision, transductive learning, safety monitors

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y3a3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors formulate the false-alarm auditing for safety monitors of LLM agents problem as positive-unlabeled learning. They use a small set of verified-safe non-alarmed trajectories to provide positive supervision. They explicitly aim to reduce the monitor-induced selection bias whereas the safe trajectories of interest are precisely those incorrectly flagged.

The solution is a series of enginerring heuristics: (1) a two-stage framework that first adapts known-safe supervision toward the alarm domain, (2) protects plausible false alarms during PU training, (3) distils ranking consensus from multiple PU models into a single student, and (4) further refines the resulting review queue using hierarchical safe-reference support and relations among alarms.

Multiple agent-safety benchmarks are used for evaluation. They also use 8 PU baselines, selection-bias and representation controls, and direct LLM re-judgment. In generl, the improvement over SOTA is significant.

Critical strengths
  • The paper identifies a meaningful limitation of agent safety monitors, where conservative monitoring produces many false alarms that require costly human review. Formulating false-alarm prioritization as PU learning is natural and useful.

  • The evaluation is strong. It covers multiple monitors and benchmarks, extensive PU baselines and controls, component ablations, and a larger external evaluation with 4,054 alarms.

Critical weaknesses
  • I have strong reservation about the key methodological choices that the author took. The central boundary-transfer mechanism assumes that moving verified-safe representations toward the alarm/non-alarm selection boundary provides useful supervision for safe-but-alarmed trajectories. How do we know this boundary is a safety boundary? This heuristic needs a lot more justification.Furthermore, how do we know if the transferred representations remain semantically safe or approximate the distribution of actual false alarms.

  • There are many works that address the bias issues in PU. The paper needs stronger comparison against methods designed specifically for biased positive selection to establish whether the proposed complex engineering pipeline is necessary. See for example: https://proceedings.neurips.cc/paper_files/paper/2023/file/3efb4bdc6bfe13e1ff95b4407c37961d-Paper-Conference.pdf

Questions for authors
  1. Can the authors provide stronger justification that boundary transfer produces valid supervision for hidden safe alarms?

  2. How does the proposed method compare with stronger recent approaches for biased PU learning beyond SAR-EM? For example: PUe (https://proceedings.neurips.cc/paper_files/paper/2023/file/3efb4bdc6bfe13e1ff95b4407c37961d-Paper-Conference.pdf) specifically develops normalized inverse-propensity-weighted PU learning for biased labeling.

6 comments
AuthorsOpenReview ↗

The Response to Reviewer Y3a3 — Weakness 1

  • I have strong reservation about the key methodological choices that the author took. The central boundary-transfer mechanism assumes that moving verified-safe representations toward the alarm/non-alarm selection boundary provides useful supervision for safe-but-alarmed trajectories. How do we know this boundary is a safety boundary? This heuristic needs a lot more justification.Furthermore, how do we know if the transferred representations remain semantically safe or approximate the distribution of actual false alarms.

This boundary provides an initial reference direction, not the true safety boundary. The monitor selects our verified-safe references from non-alarms, separating them from the safe false alarms we seek to recover. Without alarm safety labels, a precise safety boundary cannot be learned directly through target-domain supervision. We instead use the observable non-alarm/alarm difference to establish an initial direction, then improve the resulting ranking.

Transfer supplies an auxiliary comparison score; transferred points are not added as verified-safe samples or assumed to follow the false-alarm distribution. We preserve the separate task representation and hidden-representation norm, then mix the original and transferred scores through trust weights. These constraints limit the modification and its influence without assuming that geometry guarantees safety semantics.

These RAW rankings before distillation and refinement are a separate diagnostic, not the TAPS-only configuration in revised Table 4:

Initial supervisionRAW AUPRCOur AUPRC advantageFPs recovered at 5%
No transfer57.50%+3.66 pp14
Random transfer, matched displacement57.30%+3.86 pp14
Full transfer, same direction59.00%+2.16 pp16
Our direction + trust weighting61.16%—18

The controls test whether transfer helps, whether its direction matters, and whether trust weighting improves on full transfer. Recovery rises from 14 to 18 over no transfer (28.6%), establishing useful supervision before the later components contribute.

Initial errors are precisely why the subsequent distillation and refinement are needed. The student jointly learns from PU supervision and multiple reference rankings; structural refinement adds safe-reference support and alarm relations. In a separate retrospective diagnostic using the originally archived student and final predictions, we tracked changes within the 5% review prefix:

True alarms before refinementExisting true alarms moved outNew true alarms moved inTrue alarms after refinement
9.007.002.334.33

The 51.9% net reduction demonstrates correction at the student-to-refinement step. Thus, the RAW controls establish a useful initial direction, and the trajectory-level analysis shows that subsequent processing can correct ranking errors.

AuthorsOpenReview ↗

The Response to Reviewer Y3a3 — Weakness 2

We added nnPUe, Dist-PUe, and their median and refinement variants; our method exceeds the strongest alternative by 5.30 AUPRC points and recovers 13.3% more false alarms at 5%.

MethodMacro AUPRC (%)FPs recovered at 5%
nnPUe, single model53.30 ± 0.2512.33
nnPUe, median53.3112.00
nnPUe, median + refinement53.6214.00
Dist-PUe, single model58.11 ± 0.1217.67
Dist-PUe, median58.2217.00
Dist-PUe, median + refinement59.1420.00
Our full method (original results)64.4422.67

The comparison shares four settings, 40 verified-safe references per setting, training alarms, frozen features, and seeds. Enhanced baselines use their own PUe rankings and disagreement with the same refinement algorithm. PUe fits use positive references and training alarms, accessing target features during refinement; our method also uses unlabeled target features in Stage I. Target safety labels are excluded from fitting, prior estimation, and hyperparameter selection; priors use the benchmark's training-only BBE estimates.

Table details. AUPRC averages the four settings equally. Single-model AUPRC values are mean ± sample SD over three seeds; single-model recovery counts sum across settings and average over seeds. Each median variant is one three-model ensemble, with recovery counts summed across settings. The 5% budget reviews 27 alarms. PUe is independently implemented from the paper's equations, with documented frozen-feature adaptations to Dist-PUe weighting and propensity optimization; reported runs passed propensity-estimation checks.

The comparison directly tests a simpler biased-PU method combined with aggregation and structural post-processing. Its strongest variant remains below the full method in both ranking quality and false-alarm recovery.

AuthorsOpenReview ↗

The Response to Reviewer Y3a3 — Question 1

  1. Can the authors provide stronger justification that boundary transfer produces valid supervision for hidden safe alarms?

The direct evidence is the RAW comparison in W1, before either subsequent component is applied. Our direction with trust weighting reaches 61.16% AUPRC and 18 recovered false alarms, exceeding no transfer, matched random transfer, and full transfer. These controls test the usefulness of transfer, its direction, and its weighting. The separate student-to-refinement analysis establishes later correction; it is not used as evidence for transfer alone.

Reviewer_ZKKNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies false-alarm auditing for agent safety monitors, formulated as positive-unlabeled (PU) learning under monitor-induced selection bias. The authors propose a multi-stage framework combining trust-aware PU supervision, rank distillation, and structural refinement to prioritize likely false alarms for human review.

Critical strengths
  • The problem is practically relevant. Formulating false-alarm auditing as PU learning under monitor-induced selection bias is an interesting perspective.
  • The method reports meaningful improvements over existing PU baselines (Table 1), including a larger external evaluation (Table 2). The authors also provide component ablations and additional controls.
Critical weaknesses

1. The formulation is very difficult to follow.

  • Sections 3.2-3.3 introduce numerous symbols, auxiliary models, and objectives without sufficiently explaining their relationships. For example, the actual definitions of LalignL_{\mathrm{align}} and LmixL_{\mathrm{mix}} are missing in the main text, despite forming the common training objective.
  • Several important quantities, such as eie_i, ρtr\rho_{\mathrm{tr}}, and the transfer weights in Section 3.2.1, are insufficiently explained in the main text. It is difficult to understand what is actually being optimized. The exact construction of eie_i is particularly unclear.
  • More importantly, the paper does not clearly explain how the many components work together to address the original PU selection-bias problem.

2. The proposed method appears overly complicated and insufficiently justified.

  • The TN reference projection (Section 3.2.1) is questionable. The role classifier distinguishes alarms from non-alarms, not safe from unsafe trajectories. It is unclear why projecting known-safe references toward this boundary would produce meaningful representations of hidden false alarms.
  • The subsequent trust weighting, soft protection, rank distillation, hierarchical support, and agreement-based refinement introduce further heuristic assumptions. Their necessity and connection to the original problem are not sufficiently motivated.
  • Overall, I struggle to identify the main methodological insight beyond combining multiple existing ranking and PU-learning techniques.

3. The experiments do not sufficiently establish which mechanisms are necessary.

  • Table 4 shows useful cumulative improvements, but does not adequately isolate the individual mechanisms or their interactions.
  • In particular, most of the cumulative improvement in Table 4 is obtained after adding distillation and structural refinement, rather than from the initial trust-aware PU stage. It remains unclear whether the gains reflect a better solution to monitor-induced selection bias or additional general-purpose ranking heuristics.
  • The paper does not convincingly establish why simpler approach to handling monitor-induced selection bias would be insufficient.
Questions for authors
  1. Can the authors clarify the main optimization and how the components in Sections 3.2 & 3.3 relate to the original selection-bias problem?
  2. What justifies the TN boundary projection in Section 3.2.1? Is it always feasible, and why should the projected representations resemble hidden false alarms?
  3. Which components are essential to addressing monitor-induced selection bias, rather than providing general ranking improvements? Could a simpler approach achieve comparable results?
10 comments
AuthorsOpenReview ↗

The Response to Reviewer ZKKN — Weakness 1

1. The formulation is very difficult to follow.

  • Sections 3.2-3.3 introduce numerous symbols, auxiliary models, and objectives without sufficiently explaining their relationships. For example, the actual definitions of LalignL_{\mathrm{align}} and LmixL_{\mathrm{mix}} are missing in the main text, despite forming the common training objective.
  • Several important quantities, such as eie_{i}, ρtr\rho_{\mathrm{tr}}, and the transfer weights in Section 3.2.1, are insufficiently explained in the main text. It is difficult to understand what is actually being optimized. The exact construction of eie_{i} is particularly unclear.
  • More importantly, the paper does not clearly explain how the many components work together to address the original PU selection-bias problem.

Our method adapts the available safe supervision to obtain an initial alarm ranking, consolidates the consistent ordering preferences of independently trained classifiers into one student, and then uses shared safe evidence and comparisons among alarms to improve the student's review queue. These operations form two stages:

  • Stage I: trust-aware PU supervision (TAPS). Verified-safe examples come from non-alarms, while the safe trajectories to recover are inside the alarm pool. We use the observable non-alarm/alarm distinction to move safe-reference representations toward the alarm domain. This reference boundary is not an exact safety boundary, so transfers from different samples need not be equally reliable. We assign each reference a trust weight using evidence from the classifier that distinguishes non-alarms from alarms, proximity to its boundary, and the required displacement. Better-supported transfers contribute more to the comparison score; weaker transfers retain more of the original reference score. Soft protection also reduces excessive negative training pressure on plausible safe alarms, allowing them to be recovered. Together, these operations establish initial supervision relevant to the alarm pool.
  • Stage I: reliability-gated rank distillation (RGD). Safety depends on action sequences and task context, while the few available safe examples cover only part of this variation. Independently initialized PU classifiers can therefore fit the available supervision yet produce different review queues. We transfer their shared ordering preferences to a student, with their influence determined by agreement between reference rankings and support for the transferred supervision. Stronger agreement and better-supported transfer give distillation more weight; disagreement or weak support reduces it. This weight is determined before student training and remains fixed. The student learns jointly from the reference preferences and the adapted PU supervision, continuing to use the safe references and unlabeled alarm data.
  • Stage II: consensus-guided structural refinement (CSR). After distillation, individual student scores still leave safe evidence across task groups underused. Groups with few safe references borrow support from related groups. We retain a cross-group ordering relation only when global support and at least one more specific support level agree that one alarm should precede another. This allows a better-supported alarm to move earlier when the student's scores miss that relationship. We then estimate how consistently the reference models rank each alarm and use that estimate to control further movement along the structural adjustment. Throughout refinement, the student remains frozen; only its review queue changes.

The notation distinguishes necessary concepts, but their relationships must be explained before the equations. A certain amount of notation is compulsory to define necessary concepts; for example, SAR-PU separately models class prediction and positive selection [1], and RankDistil distinguishes teacher/student scores from ranking operations [2]. Our safety labels, monitor decisions, reference rankings, and student scores likewise serve different roles. The revised Method introduces each before use, defines the common and student objectives, and moves supporting constructions to the appendix. Descriptive names replace numerical component labels throughout.

References

[1] Bekker and Davis. Learning from Positive and Unlabeled Data under the Selected At Random Assumption. PMLR 94, 2018. https://proceedings.mlr.press/v94/bekker18a.html

[2] Reddi et al. RankDistil: Knowledge Distillation for Ranking. AISTATS, 2021. https://proceedings.mlr.press/v130/reddi21a.html

AuthorsOpenReview ↗

The Response to Reviewer ZKKN — Weakness 2

2. The proposed method appears overly complicated and insufficiently justified.

  • The TN reference projection (Section 3.2.1) is questionable. The role classifier distinguishes alarms from non-alarms, not safe from unsafe trajectories. It is unclear why projecting known-safe references toward this boundary would produce meaningful representations of hidden false alarms.
  • The subsequent trust weighting, soft protection, rank distillation, hierarchical support, and agreement-based refinement introduce further heuristic assumptions. Their necessity and connection to the original problem are not sufficiently motivated.
  • Overall, I struggle to identify the main methodological insight beyond combining multiple existing ranking and PU-learning techniques.

We disagree that the contribution is simply an assembly of existing PU and ranking techniques: our contribution is to jointly adapt selected safe supervision and control how its evidence is used to learn, stabilize, and refine an alarm ranking. The components are connected through the evidence they use. Reference transfer changes the PU comparisons and its estimated reliability also controls how strongly reference rankings supervise the student. The student continues learning from the adapted PU objective while absorbing consistent ranking preferences. Its frozen scores then provide the starting point for refinement using safe-reference support and alarm comparisons. Thus, later operations work with and improve the supervision established earlier.

This design addresses a specific difficulty: the available safe examples were accepted by the monitor, whereas the safe alarms of interest were rejected. With no alarm safety labels, a precise target safety boundary is not directly available. The observed non-alarm/alarm distinction supplies an initial reference direction. Trust weights limit unreliable movement, and soft protection reduces the suppression of hidden safe alarms. Initial comparisons can still be wrong; this motivates learning from several orderings and subsequently checking their consistency with shared safe evidence.

The reference direction is supported by direct replacements, not by assuming that projected points are verified-safe alarms. With the remaining mechanisms retained:

Supervision variantAUPRC (%)FPs recovered at 5%
No boundary projection62.1019.67
Matched-magnitude nondirectional perturbation62.3020.00
No soft protection62.8020.67
Uniform transfer weights63.2021.00
Full method64.4422.67

The gains depend on the direction, its controlled influence, and protection of plausible safe alarms. For the broader contribution, the strongest evaluated original PU baseline reaches 59.17% AUPRC. Adding the biased-PU method PUe [3], aggregation, and the same refinement algorithm gives 59.14% for its strongest enhanced variant, versus 64.44% for our framework. These comparisons support the value of the connected design beyond applying existing bias correction followed by ranking refinement.

Reference

[3] Wang et al. PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference. NeurIPS, 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/3efb4bdc6bfe13e1ff95b4407c37961d-Abstract-Conference.html

AuthorsOpenReview ↗

The Response to Reviewer ZKKN — Weakness 3

3. The experiments do not sufficiently establish which mechanisms are necessary.

  • Table 4 shows useful cumulative improvements, but does not adequately isolate the individual mechanisms or their interactions.
  • In particular, most of the cumulative improvement in Table 4 is obtained after adding distillation and structural refinement, rather than from the initial trust-aware PU stage. It remains unclear whether the gains reflect a better solution to monitor-induced selection bias or additional general-purpose ranking heuristics.
  • The paper does not convincingly establish why simpler approach to handling monitor-induced selection bias would be insufficient.

The initial supervision's contribution is larger than its standalone gain: removing it also reduces what the subsequent learning and refinement can achieve. We added all eight combinations of trust-aware PU supervision (TAPS), reliability-gated rank distillation (RGD), and consensus-guided structural refinement (CSR):

TAPSRGDCSRAUPRC (%)FPs recovered at 5%
OffOffOff58.4216.33
OnOffOff60.3518.00
OffOnOff59.1017.00
OffOffOn59.3017.33
OnOnOff61.9219.67
OnOffOn61.8019.33
OffOnOn60.1018.33
OnOnOn64.4422.67

Supervision adaptation alone adds 1.93 points, but removing it from the full method loses 4.34 points. Distillation and refinement together add 4.09 points with adapted supervision, versus 1.68 without it. Removing distillation or refinement loses 2.64 or 2.52 points, respectively. The initial step therefore provides a foundation that subsequent operations use, even though it does not deliver the entire improvement alone.

We also tested direct aggregation from the same reference bank: median ranking reaches 61.70%, and median ranking with the same refinement reaches 61.50%, versus 64.44% for joint student learning and refinement. Fixing the distillation weight gives 63.50%; replacing hierarchical support with flat support gives 63.40%; removing alarm-relation optimization gives 63.00%. These controls separately test the proposed learning process and concrete simplifications.

Reviewer_tmcUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors address the issue of frequent false alarms raised by agent safety monitors. Their framework is composed of two main stages: During the first stage, the negative pressure on possible False Positives is reduced and a student positive unlabeled (PU) auditor is produced through reliability-gated distillation from several reference auditors. During the second stage, the student PU refines its auditing queue with hierarchical true negative support, target alarm relations, and reference agreement. They compare their framework against 8 PU methods, and report the false alarms recovered under various review budgets, in which their framework outperformed baselines.

Critical strengths
  1. The paper is generally well-organized and tackles an important problem
  2. Figure 2 conveys the central idea really well.
  3. The authors compare their framework against 8 PU methods, and report the false alarms recovered under various review budgets, in which their framework outperformed baselines.
  4. The solution seems pretty lightweight.
Critical weaknesses
  1. Although the paper does rerank the alarms for review, so technically no True Positive gets removed, but it might put real alarms lower in the list, and the human reviewer does not even look at them. The paper does not study that operational risk.
  2. On a similar point, if a model believes that a particular trajectory is safe, at every single stage of the pipeline it reinforces that, and at the end it faces a much higher pressure to not rank it as unsafe. If the initial belief is incorrect, it has the potential to get stuck in a bad feedback loop.
  3. Even while reading the paper, it seemed that there are too many steps involved in the process. Pipeline complexity is not well justified. The method chains many stages (boundary transfer, soft protection, a reference bank, a distilled student, hierarchical support, a relation solver, and an agreement head), and the motivation for several of them is weak. In particular, it is unclear why a separate student model is needed: since reference models must be refit for each new cohort, ranking directly by the references’ median rank would seem sufficient. Table 4 supports this concern. Each component adds only about 1.5–2.5 AUPRC.
  4. Reducing false positive calls for an active-learning baseline. Reviewers produce labels as they go. So, iteratively labeling and re-ranking is a natural competitor the paper ignores.
  5. The authors move the representations of known-safe trajectories toward the boundary between accepted and alarmed trajectories. But, these modified representations may not necessarily correspond to genuinely safe behavior.
Questions for authors
  1. Can the authors evaluate how many true alarms are missed under different review budgets?
  2. 5% review budget can be too big in many cases; can the authors report scores on a smaller review budget?
  3. Can the authors compare against an active learning approach under the same review budget?
10 comments
AuthorsOpenReview ↗

The Response to Reviewer tmcU — Weakness 1

  1. Although the paper does rerank the alarms for review, so technically no True Positive gets removed, but it might put real alarms lower in the list, and the human reviewer does not even look at them. The paper does not study that operational risk.

Our objective is to recover safe false alarms for two uses: collecting safe trajectories after human verification, and reducing false alarms through automatic filtering. This is distinct from prioritizing responses to all safety incidents. True alarms retained after filtering continue to trigger the monitor's original alarms.

For human verification, reviewed safe trajectories are collected for agent safety training or alignment. At a 5% budget, our method recovers 22.67 safe trajectories from 27 reviewed alarms, versus PULDA's 17.00: 33.3% more safe data for the same review effort. Benchmark safety labels simulate verification, and only confirmed-safe trajectories enter the dataset.

For automatic filtering, we quantify both false-alarm removal and true-alarm loss. At 2%, the 13 candidates contain 12 false alarms and one true alarm, so filtering retains 99.63% of all true alarms. At 5%, it removes 22.67 false alarms and 4.33 true alarms, retaining 98.40%. The latter loss is 1.60% of all true alarms, or 16.05% of the filtered candidates. Q1 reports the complete budget-dependent results and distinguishes filtering loss from alarms outside the current human-review round.

AuthorsOpenReview ↗

The Response to Reviewer tmcU — Weakness 2

  1. On a similar point, if a model believes that a particular trajectory is safe, at every single stage of the pipeline it reinforces that, and at the end it faces a much higher pressure to not rank it as unsafe. If the initial belief is incorrect, it has the potential to get stuck in a bad feedback loop.

Our entire approach starts from the possibility that the model's initial judgment is wrong; this is precisely what motivates rank distillation and structural refinement. With no alarm safety labels and only limited safe non-alarm references, the initial ordering is a starting point for learning, rather than a final safety judgment.

The student jointly learns from PU supervision and multiple reference rankings, with reliability controlling their influence. It can therefore depart from individual reference judgments. Structural refinement then adjusts the queue using verified-safe support and relations among alarms. Agreement controls the strength of these adjustments; it does not turn a prediction into a verified-safe label.

In a separate retrospective diagnostic using the originally archived student and final predictions, we tracked changes within the 5% review prefix:

Change in true alarms within the review prefixCount
Initially placed in the prefix by the student9.00
Existing true alarms moved out by refinement7.00
New true alarms entering after refinement2.33
True alarms remaining in the final prefix4.33

Refinement corrects 77.8% of the existing errors. Including the newly introduced errors, the count falls from 9.00 to 4.33, a 51.9% net reduction. This directly demonstrates that subsequent processing corrects initial ranking errors instead of simply reinforcing them.

AuthorsOpenReview ↗

The Response to Reviewer tmcU — Weakness 3

  1. Even while reading the paper, it seemed that there are too many steps involved in the process. Pipeline complexity is not well justified. The method chains many stages (boundary transfer, soft protection, a reference bank, a distilled student, hierarchical support, a relation solver, and an agreement head), and the motivation for several of them is weak. In particular, it is unclear why a separate student model is needed: since reference models must be refit for each new cohort, ranking directly by the references’ median rank would seem sufficient. Table 4 supports this concern. Each component adds only about 1.5–2.5 AUPRC.

The student performs joint learning that median aggregation cannot provide: with identical structural refinement, it achieves 64.44% AUPRC versus 61.50% for the median alternative. Refitting updates the reference models; taking their median aggregates existing ranks; the student additionally learns from both PU supervision and reliable ranking information.

The components address connected difficulties in this learning process. Safe non-alarm references underrepresent hidden safe alarms, so boundary transfer improves their relevance while trust mixing controls unreliable transfers. Because unlabeled alarms include safe trajectories, soft protection prevents excessive negative pressure on them. Rank distillation then reduces dependence on imperfect individual rankings. Finally, uneven safe-reference coverage across tasks motivates hierarchical support and alarm relations for refining the student's queue.

We directly tested the proposed simplifications, keeping the subsequent structural refinement identical:

Ranking methodFinal AUPRCFull-method advantage
Jointly trained student + refinement64.44%—
Median aggregation + refinement61.50%+2.94 pp
PU-only training + refinement62.00%+2.44 pp
Distillation-only continuation + refinement58.00%+6.44 pp
Fixed distillation weight + refinement63.50%+0.94 pp

PU-only training retains the adapted supervision. Distillation-only continuation starts from the same PU-warmup model. Their respective losses of 2.44 and 6.44 points demonstrate complementary contributions from PU supervision and reference rankings; reliability weighting adds 0.94 points over a fixed weight. The original cumulative ablation measures the incremental gain from adding each component. Those gains do not establish that simple aggregation can replace the pipeline; the median replacement above directly tests that claim.

All modules operate on frozen monitor representations without retraining the base monitor. References supply training supervision, one student supplies initial scores, and refinement follows after the student is frozen. New cohorts require downstream reference fitting and ranking adjustment, without deploying additional large monitoring models. The revised Method explains these motivations and connections, and Table 4 includes the same-reference median comparisons.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thanks for All Reviewers

We thank all reviewers for their careful reading and constructive feedback. We have revised the manuscript to clarify the method and added analyses addressing the reviewers’ concerns. Detailed responses are provided below. Revised passages are highlighted in blue, and additional evidence is highlighted in magenta. For entirely new tables, only the captions are marked in magenta to preserve readability. We welcome further feedback and discussion.

AuthorsOpenReview ↗

Revision Change Summary

  1. [Sections 3.1–3.3, Figure 2] We rewrote the Method overview and the connections between the two stages, explaining the supervision mismatch addressed by trust-aware PU supervision (TAPS), the variability across fitted rankings addressed by reliability-gated rank distillation (RGD), and the underused safe-reference evidence and alarm relations addressed by consensus-guided structural refinement (CSR).

  2. [Equations 1–6, Appendices A–B] We defined the models and notation before use, stated the common, reference-model, and student training objectives, and clarified which parameters and transfer, protection, and distillation quantities remain fixed or update. Supporting projection and ranking constructions are given in the appendices.

  3. [Section 4.5, Table 4] We added all eight combinations of TAPS, RGD, and CSR and direct median-aggregation alternatives using the same reference bank. These comparisons distinguish standalone improvements from component dependence and test whether aggregation can replace joint student learning.

  4. [Appendix F.1, Table 15] We added controls for boundary projection, nondirectional perturbation, soft protection, transfer trust weights, the distillation gate, hierarchical support, alarm-relation optimization, and the final agreement correction to assess their individual roles within the full method.

  5. [Appendices F.2–F.3, Tables 16–17] We added biased-PU comparisons with nnPUe and Dist-PUe, including aggregation and refinement variants, and a safe-source separation diagnostic with random regrouping as a control. The diagnostic measures differences between safe sample sources rather than safety-classification accuracy.

  6. [Appendices F.4–F.5, Tables 18–19] We added smaller review budgets, retrospective true-alarm loss under automatic prefix filtering, and sequential-feedback comparisons with uncertainty sampling, adaptive greedy search, and ENS. We clarified the different uses of human-verified safe-data collection and automatic filtering, and the feedback available to each comparison method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.