acceptodds
Under review as a conference paper at ICLR 2027

Refine, Don't Merge: Certified Repair of Blind Conformal Detectors

Abstract

Sequential conformal detectors can control false alarms yet miss changes that preserve their score distribution. This matters for score-based monitoring, including prospective monitoring of language-model agents. We formalize these rank-fiber collisions and introduce CoFi-E, which refines inherited score groups and independently certifies each retained split's information gain. We prove information retention and a geometric rate for population greedy refinement under a weak-witness condition. Freezing the repair preserves runtime false-alarm control under conditional exchangeability. In a controlled joint collision, CoFi-E detects 84.1% of streams versus 1.4% for marginal merging. Experiments span image, time-series, and intrusion data. Across 290 matched tasks, selected partitions reduce restricted delay by 34.8% against the inherited detector and 18.8% against a fixed grid; audit-gated prefixes yield 26.7% and 24.6%, respectively. These results support certified repair of specified detector blind spots.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal detection, e-processes, rank-fiber collisions, representation learning, distribution shift

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YyPWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper argues that a detector can have perfect false-alarm validity while being completely blind to a distribution shift that leaves its score distribution unchanged. It proposes an approach that recursively splits existing score groups rather than replacing them, with the objective of exposing specified within-group discrepancies.

Critical strengths
  • The work points out that a valid conformal score need not be a sufficient representation for every relevant alternative, i.e., the distribution of the scores can remain unchanged under a target distribution.
  • Refinement is a natural approach to tackle the problem. This approach also yields interpretable theoretical results.
Critical weaknesses
  1. QuantTree constructs a recursive binary partition of the observation space specifically for multivariate change detection. The distinction between the proposed approach and QuantTree from an algorithmic and mathematical perspective has not been clarified.

  2. The Algorithm 1 details and the descriptions in the main text are not sufficient to provide a good understanding for reproducibility. Section 4 is too sparse for a good understanding.

  3. The approach "searches supplied target changes; it does not generate new attacks during refinement". In realistic anomaly detection, the changes are typically unknown in advance.

Questions for authors
  1. How is the "logistic-halfspace dictionary with axis-aligned fallbacks" done? What is the computational complexity?

  2. What is the precise novelty relative to multicalibrated partition refinement and recursive partition in QuantTree?

  3. The method explicitly searches supplied target changes. What happens when the true future alternative Q is outside the declared portfolio?

  4. In contrast to the theory, the Pearson vs Gini improvement does not meet the target criterion and the results do not establish that the proposed Pearson split criterion is better than a standard supervised partition criterion.

  5. The paper concludes that positive design-family certificates did not establish competitive unseen-family ranking. Does this mean that certifying that a partition captures information about declared/training alternatives does not imply that the partition is a good anomaly representation for unseen alternatives? How can the method repair a detector in a practical application?

  6. The difference between CoFi-E (0.58 AUROC) and Isolation Forest (0.75) is large. Why?

  7. There should be comparisons to other baseline approaches like sequential predictive two-sample testing, sequential kernelized testing and nonparametric e-detectors.

  8. The current audit guarantee appears to depend critically on the target/search specification being fixed independently of the audit sample. The precise adaptive-data-selection boundary should be stated more explicitly.

  9. Conformal systems commonly operate with a fixed calibration set rather than continuously obtaining fresh samples. How to ensure the runtime guarantee?

3 comments
AuthorsOpenReview ↗

Response (1/3) – presentation, implementation, and novelty

Thank you for the detailed criticism of presentation, reproducibility, and novelty. We revised the presentation without replacing the submitted primary method.

Presentation and Q1 (candidate construction/complexity). Figure 1 now separates (a) the inherited score's blind spot, (b) joint refinement versus marginal merging, and (c) propose–select–audit–freeze. The Introduction adds a concrete ZIP-code/street-level example. Section 4.1 and the expanded 13-step Algorithm 1 identify data roles explicitly; Sec. 4.1 defines the round cap T, minimum child support, and FitAndFreeze, while Appendices C.1–C.3 specify implementation and measured cost.

In the matched study: balanced logistic regression on proposal normal/target examples supplies one direction per target; we add min(15,d) seeded coordinate directions. For each direction and current group, candidate thresholds are the seven normal-selection quantiles 1/8,...,7/8. A greedy selection-data search starts from four inherited groups, adds at most 12 splits (K ∈ {8,16}), checks at least six normal rows per child, and caps evaluations at 16,384. The complete path is fixed before auditing. The stated complexity is O(sDn(d + log n)) for search, O(n_w d) per logistic-fit iteration, and four exact count intervals per proposed split (App. C.3). Median proposal/search/audit times are 6.15/57.21/3.42 ms (App. C.3).

Q2 (QuantTree and multicalibrated partition refinement). Our contribution is an audited repair of an inherited detector, with an explicit retention constraint and per-split gain certificates. Recursive splitting, nesting, and data processing are shared ingredients. The analysis (1) characterizes score-law collisions and why fixed-weight marginal merging remains blind when every component score law is unchanged (Sec. 3); (2) quantifies information recovered by nested splits and proves retention and conditional greedy progress (Theorems 1–2; Corollary 1); and (3) connects independent certification to valid rank monitoring after freezing (Secs. 4–5).

Appendix E.1 (Table 19) now compares the original formulations: QuantTree builds normal-data histograms for change detection; multicalibrated partitions target importance-weight accuracy. We do not claim these approaches cannot be extended with inherited groups, auditing, or conformal monitoring. In a partition-only comparison using common KT evidence, CoFi-E's delay is 53.09 versus 103.81 for QuantTree partitions (App. D.9); this is not a comparison with QuantTree's native test and CoFi-E uses target examples. The novelty claim is the integrated repair objective plus independently audited guarantees, not binary partitioning itself.

AuthorsOpenReview ↗

Response (2/3) – unknown changes, Pearson, and AUROC

Q3 (future change outside the declared target portfolio). Nested refinement still preserves the inherited grouping's population rank information for an unlisted alternative (Corollary 1). If CoFi-E+ includes the inherited evidence process with weight w, its evidence is at least w times that component's evidence at every time (Sec. 4.5). These safeguards do not ensure equal finite-reference delay or competitive unseen-family rankings. Theorem 3 (App. B.11) gives conditional transfer results requiring target-law coverage. No fixed finite grouping can generally expose every unknown change; our guarantee is retention, with additional certified gain for the supplied targets.

Q4 (Pearson versus Gini). Pearson is the default because selection and certification use the same information gain (Theorem 1), and Theorem 2 proves the stated Pearson-greedy rate. This does not establish a detection advantage: the corrected exact-K contrast is 0.25% [−1.66, 2.07], below the prespecified 10% criterion (Sec. 6.3). Gini is an interpretable alternative criterion and receives the same independent certificates (Remark 2; App. D.5). The contribution remains applicable with either criterion.

Q5 (do target certificates predict unseen-anomaly rankings? how is repair used?). Certificates establish gain for specified changes, not generic unseen-family ranking. The practical use is to repair a missed behavior identified through incident review, red-teaming, or recurring false negatives. Representative examples are split into proposal, selection, and audit roles; accepted refinements are certified and frozen before monitoring. In the clean panel, Pearson maps improve target AUROC to 0.821 versus 0.733 for the original Isolation Forest score, while other-family AUROC is 0.643 (App. D.10). This supports repairing an identified blind spot alongside the existing detector.

Q6 (historical 0.582 versus 0.754 AUROC). The historical evaluation excluded each test family from repair; the gap across its 605 folds remains a negative result (App. D.12, Table 18). Two diagnostics identify relevant factors:

  • Ranking: our new intervention holds repaired groups fixed and refits only their ranking using additional future-family selection labels. Pearson other-family AUROC rises 0.643 → 0.742; with the same extra supervision, repaired groups add 0.034 [0.010, 0.063] over the original four groups (App. D.10).
  • Coverage: the audit already reported in the submission finds that including the held-out family among the targets raises AUROC 0.5722 → 0.6137 across 13 datasets (App. D.12).

The ranking intervention uses extra labels on a separate panel, so it is not an unsupervised improvement or a full explanation of the historical gap. Together, the diagnostics show why target coverage and the fitted ranking matter when assessing transfer from repaired groups.

AuthorsOpenReview ↗

Response (3/3) – baselines, audit adaptivity, and calibration

Q7 (sequential baselines). Section 6.5 and Appendix D.9 (Table 16) add all requested families on 145 clean fits with paired evaluation, the same nominal 5% anytime false-alarm criterion, and a common restricted-delay endpoint:

ProcedureDelay ↓DetectionFATarget examplesRefs/step
QuantTree partition + common KT103.810.4790.034no63
Kernel betting63.770.8040.003no1
Certified CoFi-E53.090.8120.034yes63
Predictive test, target-initialized42.900.8840.025yes1
Predictive test, no target initialization150.050.0570.000no1
E-detector, adapted to anytime PFA32.470.9240.006yes1

CoFi-E reduces restricted delay versus QuantTree partitions by 48.86% [36.54, 62.36] and kernel betting by 16.75% [8.62, 26.10] (approximate Bonferroni-adjusted 98.75% four-dataset-cluster intervals). The e-detector is faster; the difference from the target-initialized predictive test is inconclusive. QuantTree uses common KT evidence rather than its native test. The faster direct methods use target examples and full features; this panel does not measure earlier-family retention.

Retention motivates repair: the separate historical panel reports earlier-family detection of 97.1% for CoFi-E versus 25.3%/36.0% for logistic/HGB replacement, under the qualifications disclosed for Table 1. This does not establish retention superiority over the new sequential baselines. Together with population-information retention (Corollary 1), it supports repair as a useful option when preserving an existing detector's distinctions matters.

Q8 (audit adaptive-selection boundary). Before audit, the method fixes preprocessing, targets, directions, thresholds, complete paths, and the entire family of parent/child comparisons for which simultaneous bounds are allocated (Sec. 4.3; Apps. B.7–B.8). Audit outcomes may select a certified prefix or a parent-consistent branch subset within that fixed family. They cannot create new cuts, refit features, change targets, or move thresholds without fresh independent audit data or another valid adaptive argument. This is the boundary required for the claimed simultaneous error control.

Q9 (fixed calibration rather than fresh references). Appendix B.12 (new Proposition 4) supplies a separate online conformal construction requiring only one initial calibration set. Freeze the representation first, then rank each new score among the calibration scores, previous monitored scores, and itself, with randomized ties. Under joint exchangeability conditional on the frozen design, these sequential p-values are iid uniform. Nonnegative betting functions with integral at most one, chosen using only the design and past p-values, give anytime false-alarm control. The guarantee is marginal over the calibration sample, not conditional on its realized values. The evaluated score-group KT update still requires fresh normal references; the online construction's detection power is not evaluated here.

Reviewer_gDb5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies distribution changes that remain invisible to a conformal detector because they preserve its score distribution. CoFi-E repairs such blind spots by splitting existing score groups and independently auditing the information gained by each retained split. The theory establishes population information retention and gives a refinement rate under a witness assumption. Controlled experiments demonstrate the mechanism, while benchmark comparisons show gains over fixed partitions.

Critical strengths
  1. The paper draws a useful distinction between false-alarm control and the ability to observe a change. Its analysis explains why combining marginal detectors can remain blind to changes in their joint distribution.

  2. The independent audit gives each accepted split a specific statistical interpretation, and the paper carefully separates this guarantee from finite-reference detection power.

Critical weaknesses
  1. Some matched experiments use historical feature projections that predate the separation of data roles. The paper consequently treats their audit results as development diagnostics, leaving incomplete empirical validation of the independent certification procedure.

  2. Pearson search does not show a clear advantage over Gini search under matched conditions, and the registered success criterion was not met. Since Gini-selected splits can undergo the same audit, the additional practical value of the proposed search criterion remains unclear.

  3. The real-feature collision experiment does not demonstrate useful detection, while the intrusion comparison also changes the evidence rule. The practical advantage of joint refinement over marginal merging is less well supported than the controlled example suggests.

Questions for authors
  1. Do the matched-study conclusions hold when feature projections are fitted without access to any observations later assigned to audit?

  2. Which benefit of the proposed method depends on Pearson search rather than nested refinement with the same audit applied to Gini-selected splits?

  3. With the evidence rule held fixed, which real-data result demonstrates useful detection of a joint change that marginal merging misses?

3 comments
AuthorsOpenReview ↗

Response (1/2) – independent rerun and audit calibration

Thank you for identifying the data-role concern. We addressed it with a corrected matched rerun and complementary checks of the audit implementation. The submitted results remain labeled as development diagnostics and are reported beside the rerun.

Q1. Do the conclusions hold with preprocessing and projections fitted independently of audit? Yes for the main targeted-refinement comparisons. We reran the same 290-task / four-dataset / five-seed / K ∈ {8,16} design with (i) scalers, PCA, and learned scores fitted on proposal data only, (ii) 142 duplicated ElectricDevices records removed, (iii) source variants assigned to the same data role, and (iv) the full family of candidate paths and audit comparisons fixed before audit (Sec. 6.3; App. D.3). We report submitted and corrected results side by side (App. D.4, Table 12).

At the primary proposed exact-K endpoint, Pearson partitions reduce restricted delay by 35.96% [26.10, 43.19] against the unrepaired detector and 18.59% [8.23, 27.02] against the fixed grid. Separately, certified prefixes achieve 27.54% [22.83, 34.05] and 26.43% [22.02, 34.05], respectively (Table 10). CoFi-E's advantage over the grid holds on all four datasets in both views (Table 11). The audited grid keeps the original groups on three datasets; the primary exact-K comparison additionally controls the proposed group count. Random cuts do not improve on the inherited detector (delay 74.70 versus 73.26). Intervals are approximate, based on four dataset clusters; other historical panels were not rerun.

Independent audit validation. Appendix B.8 adds a check on known iid distributions: each of eight null settings yielded 0 false certificates in 2,000 repetitions, with an exact 95% upper bound of 0.0018 per setting (Table 3). Certificate power increases with sample size in the tested alternatives. An independent verifier also reconstructs audit counts, bounds, and alarm paths for the corrected matched panel (App. D.3). These are complementary checks: the null study assesses implementation under its specified model, and the clean rerun addresses independence on the real-data panel.

The corrected primary comparison therefore supports targeted refinement over untargeted partitions, with the certified results now based on independently fitted features and verified audit calculations.

AuthorsOpenReview ↗

Response (2/2) – Pearson/Gini and fixed-evidence joint changes

Q2. What specifically depends on Pearson rather than audited Gini search? Pearson aligns selection with the gain being certified: its score is exactly the decrease in hidden χ² information (Theorem 1), and Theorem 2 proves a population rate for greedy selection by that score under the weak-witness condition. These are its theoretical roles.

The broader repair guarantees follow from nestedness, independent certification, and valid frozen-map monitoring. Retention holds for every alternative (Corollary 1), and the audit can certify paths chosen by Gini or another criterion if fixed before audit (Sec. 4.3; Remark 2). Gini is also interpretable through triangular discrimination (App. D.5).

Empirically, the corrected Pearson–Gini delay contrast is 0.25% [−1.66, 2.07] at exact K and −0.44% [−4.02, 1.54] after audit. Neither meets the prespecified 10% criterion, and the additional gain-to-held-out-evidence diagnostic also fails. We retain these findings and now state explicitly that the split criterion is replaceable: the evidence supports certified targeted repair with either criterion.

Q3. Which fixed-evidence real-record result demonstrates joint detection missed by marginal merging? New Sec. 6.4 and App. D.8 construct 80 changes from UNSW-NB15 records that preserve both individual score-group marginals while changing their joint table. A separate target-informed repair study supplies examples of those constructed alternatives. With the same KT evidence rule, B = 63 references, paired draws, and onset 25, the results are:

MethodDetectionNull alarmsUtility cases
Marginal merging1.6%3.7%—
Audited Pearson58.2%3.4%47/80
Audited Gini63.3%3.4%55/80
Full product70.1%3.5%61/80

Utility requires detection of at least 50%, at least 20 percentage points above merging, and a positive paired delay improvement. Null alarms are measured over 249 steps.

The full product performs best and is construction-aware: these shifts were defined on its joint table. Audited refinement recovers joint information missed by the individually blind marginal scores, with a certificate for each retained cut. The comparison concerns reweighted empirical laws, not naturally occurring attacks, and does not establish superiority over the product or a target-trained standalone detector.

We retain the negative results: maps trained for natural intrusion families detect only 0.9%/3.2% of these constructed shifts (App. D.7), and the submitted real-feature collision study remains negative. Section 6.2 now labels the different-evidence intrusion replay a component diagnostic. Sections 6.3–6.4 provide the requested evidence that targeted refinement can help with the evidence rule held fixed.

Thank you for the response and revisions. I have also considered the other reviews and the authors’ replies. The additional evidence is helpful, but my concern about the practical detection benefit remains only partly addressed. I therefore maintain my original rating.

Reviewer_DvtDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of blind spots existed in sequential conformal detectors when facing anomalies that keep the score distribution unchanged, and proposes a repair method named CoFi-E. In this method, CoFi-E searches for discriminative splits within the original score groups and uses independent audit data to verify the information gain from these splits, thereby enhancing the ability to recognize potential distribution changes while retaining the original detection information. At the same time, authors theoretically analyze properties like information retention, split gain, and false positive control, and conducts experiments on image, time series, and network intrusion detection datasets. Experimental results show that this method can effectively improve the detector's performance in detecting specific hidden distribution changes.

Critical strengths

The research question has a certain degree of novelty, and the method design is relatively reasonable. The paper focuses on the problem that even if existing detectors meet the false alarm control requirements, they may still fail to identify anomalous changes due to loss of information in score representations, which provides a clear research motivation. Unlike directly replacing detectors or combining multiple detection results, the paper gradually refines the original score grouping to discover new distribution differences while retaining existing detection information. Additionally, by introducing an independent auditing mechanism to verify the information gain from each division, the method not only aims to improve detection performance but also emphasizes the reliability of the correction process.

Critical weaknesses

There's still room to further analyze the computational efficiency of the method. Although the paper mentions that partitioned search and statistical validation increase computational costs, it doesn't specifically show how much time and resources these steps require.

Questions for authors

The paper uses independent audits to filter out effective groupings, but if an earlier grouping isn't approved, even if a later grouping could help improve detection, it can't be kept. Did the authors consider this situation? Could this filtering method be affected by the order of the groupings, potentially limiting information recovery and the final detection performance?

2 comments
AuthorsOpenReview ↗

Response (1/1) – runtime cost and split-order dependence

Thank you for recognizing the importance of repairing blind spots while retaining the original detector's distinctions. We address both questions directly.

Computational efficiency (App. C.3, Tables 6–7). On 145 clean fits using Intel Xeon Platinum 8462Y+ CPUs and one numerical-library thread per worker, the median proposal-direction, Pearson-search, and audit times are 6.15, 57.21, and 3.42 ms, respectively. Mapping a query plus 63 references takes 7.62 ms. For D directions, n selection rows, d features, and s splits, the stated search bound is O(sDn(d + log n)), with at most 16,384 candidate evaluations in the matched study. Peak memory is 526–904 MiB per complete worker; this includes data and monitoring rather than isolating search memory.

The important deployment qualification is that the primary anytime-valid runtime protocol draws 63 fresh normal references per step (15,687 over 249 steps). Timings exclude feature extraction and reference collection. Appendix B.12 supplies a separate valid online-rank construction from one initial calibration set, but its detection power and compatibility with the experimentally used KT update are not claimed.

Can an early audit failure block a useful later split? Yes. The primary algorithm retains the longest passing prefix, so a failure can block a later split on another branch. We added a secondary branch-aware rule (App. B.8, Table 4): retain a certified split if all splits needed to create its parent were retained; skip failed splits and their descendants, without blocking independent branches. It introduces no audit-selected new cuts, and the existing simultaneous certificate event covers the accepted splits, with the same error control.

On the clean matched fits, mean accepted Pearson splits increase 1.659 → 2.261, with a further 7.07% [5.17, 9.56] delay reduction relative to prefix retention. For Gini, accepted splits increase 2.216 → 2.570, with 4.77% [0.87, 6.87] further reduction. These are approximate four-dataset-cluster intervals. The primary prespecified prefix protocol remains unchanged; the branch rule is reported separately. The greedy search still depends on split order, which can affect power; fixing the path before audit preserves the simultaneous audit error bound.

The revision therefore quantifies the repair and reference costs, and supplies a measured way to recover useful later splits under the same audit error bound.

Thank you for your response. Having reviewed the feedback, my concerns have been addressed, and I will maintain my rating.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Summary of revisions and evidence

We thank all reviewers for the constructive feedback. CoFi-E addresses a specific need: repairing an identified blind spot while preserving the distinctions made by an existing detector. The revision retains the submitted problem, primary prefix algorithm, and core theoretical results, and adds the requested validation and comparisons. Appendix A.2 maps each concern to the revised evidence.

Targeted repair remains beneficial after the independence correction. We reran the 290-task matched study with preprocessing and learned scores fitted only on proposal data, 142 duplicated ElectricDevices records removed, source variants kept within the same data role, and paths fixed before audit. Under a common KT evidence rule, proposed exact-K partitions reduce restricted delay by 35.96% versus the inherited detector and 18.59% [8.23, 27.02] versus a fixed grid. Separately, certified prefixes yield 27.54% and 26.43% [22.02, 34.05], respectively. CoFi-E outperforms the grid on all four datasets in both views (Sec. 6.3; Apps. D.3–D.4). Intervals are approximate, based on four dataset clusters. Pearson does not outperform Gini; both criteria can be independently audited.

Refinement exposes joint changes that individually blind scores miss. On 80 constructed changes from UNSW-NB15 records, both marginal score-group distributions remain fixed while their joint table changes. With identical KT evidence, reference counts, and paired draws, marginal merging detects 1.6% at onset 25, versus 58.2%/63.3% for target-informed audited Pearson/Gini refinement. The construction-aware product partition remains strongest at 70.1% (Sec. 6.4; App. D.8). These reweighted empirical distributions test the existing joint-change result; they are not naturally occurring attacks.

Further additions include null-certificate calibration; measured computation and reference costs; a secondary branch-aware acceptance analysis; comparisons with QuantTree partitions, sequential predictive testing, kernel betting, and an e-detector; explicit audit-adaptation boundaries; and an online conformal construction using one initial calibration set (Apps. B.8, B.12, C.3, D.9). Figure 1, Algorithm 1, and Appendix E.1 clarify the method and its relationship to prior partitioning work.

What this establishes: nested refinement retains inherited population information, independent audits certify gains for specified targets, and freezing permits valid monitoring under the stated sampling assumptions. These guarantees support targeted repair of deployed detectors; they do not guarantee finite-reference power or competitive rankings for arbitrary unseen anomalies. Our replies below explain the work completed for each concern.

Area_Chair_sL6ZOpenReview ↗

Regarding the paper scope

Dear authors,

Thanks for your submission. I quickly read the reviews and paper and may have several scope questions for clarifications. I read in a quick rhythm so I may omit important details or contexts.

  • Basically detection distribution shift (regardless of whatever methods) should consider two properties, the precision and the recall. That implies avoid false alarm and precise detecting the true shift.
  • This will inevitably create the trade-off between the precision and recall.
  • How the problem setup (like conformal detection, e-processes, rank-fiber collisions, representation learning) considered this context
  • What experiments want to justify? for what kind of conclusion?

If possible, can you explain these in plain language? Thank you

AuthorsOpenReview ↗

Response (1/2): Detection Scope, Trade-offs, and Problem Setup

Dear Area Chair,

Thank you for reading our paper and for these helpful questions. We are glad to explain our work in plain words.

Question 1: Basically detection distribution shift (regardless of whatever methods) should consider two properties, the precision and the recall. That implies avoid false alarm and precise detecting the true shift.

Response: We agree, and both matter to us. One small note on terms: we control false alarms rather than precision itself. Precision also depends on how often real changes happen, which no detector can control. So we focus on avoiding false alarms, meaning rarely raising an alarm when nothing has changed. The recall side means noticing a real change, and noticing it quickly. Most monitoring systems turn each new observation into a single score. They raise an alarm when the scores start to look unusual. Our paper studies a case where this breaks down. If a change only shuffles data among inputs that get the same score, the scores look just as they did before. As a result, the system cannot see the change at all.

Question 2: This will inevitably create the trade-off between the precision and recall.

Response: You are right that this trade-off is usually there. A more sensitive alarm catches more changes, but it also raises more false alarms. In our setting, we fix the false-alarm limit in advance. Under the paper's assumptions, the chance of ever raising a false alarm during normal operation is at most 5%. Within that limit, we try to catch real changes as early as we can. However, the blind spot we study cannot be fixed by moving the threshold. For a hidden change, the scores look exactly as they do when nothing has changed. A lower threshold raises alarms equally in both cases, so it still cannot tell them apart. What is missing is information, not tuning. Our repair adds that missing information for the targeted change, and it keeps the same false-alarm guarantee.

Question 3: How the problem setup (like conformal detection, e-processes, rank-fiber collisions, representation learning) considered this context

Response: Each part has a simple role. Conformal detection keeps false alarms in check by comparing each new score with scores from normal data. E-processes keep a running total of evidence, so we can check it at any time without adding false alarms. A rank-fiber collision is simply our name for the hidden change described above. Here, the representation is just what the monitor can see. We do not replace the existing score; we refine it. The score is a short summary of the data, and a collision is information that this summary threw away. Our method brings some of that information back. Using examples of the missed change, it splits the score's groups into finer ones. It keeps every distinction the system already made, so the information represented by the original score is preserved. Next, it runs a statistical check on separate data to confirm that each split adds real information about the targeted change. Finally, it locks the result in before monitoring starts, so the false-alarm guarantee stays the same.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.