acceptodds
Under review as a conference paper at ICLR 2027

When Coarse Annotations Conceal Novel States: TC-REJECT for Auditing Time-Series Labels

Abstract

Coarse interval labels can conceal changes in behavior or physiology, contaminating scientific analyses and downstream models. Auditing such labels is difficult because an unseen state may appear typical of the training distribution while conflicting with its assigned label. In controlled simulations, global atypicality and label agreement fail in opposite conditions: density scores invert for coherent lower-variance novelty, while large scale shifts reverse their relative utility. We therefore introduce Typicality-Corrected Rejection (TC-REJECT), which calibrates complementary multichannel representations and combines them using e-value summation or rank averaging without requiring examples of the target novel state. Across wearable activity, naturalistic electrocorticography, and clinical local field potential recordings from deep-brain nuclei in two participants with deep-brain stimulation, TC-REJECT detects withheld or concealed states and supports selective label review. On prespecified previously unused neural recordings, rank-based fusion is more robust than e-value summation and achieves the highest mean performance among independent comparison methods on both evaluated novelty targets. TC-REJECT provides a practical framework for auditing and conservatively refining partially labeled time series.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Novelty detection, Multivariate time series, Label auditing, Conformal prediction, Neural time series

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G55DOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

TC-REJECT scores multichannel windows for conflict with coarse time-series labels so they can be reviewed, dropped, or relabeled. Five evidence channels (amplitude and covariance, normalized temporal shape, spectral power, a classifier's softmax, and a self-supervised embedding) each give an empirical conformal p-value against the best-fitting named class. The p-values are fused by a weighted e-value sum with a data-dependent gate or by rank averaging. A synthetic benchmark shows a density score and a label-agreement score failing in opposite conditions. Validation covers withheld activities in PAMAP2, out-of-vocabulary and compound annotations in AJILE12 (with 12 prespecified held-out sessions), and reviewer-identified movements inside clinical Rest in two DBS recordings. The paper reports competitive detection, selective high-precision relabeling, and mixed downstream benefits.

Critical strengths

The prespecified test on 12 unused AJILE12 sessions (Appendix H) is a genuine confirmatory experiment: rank fusion beats e-value fusion in 11 of 12 sessions, with mean paired AUROC difference 0.086 and subject-bootstrap interval [0.049, 0.123]. It tells me the evaluation is careful. My rating rests on the size of the contribution, which this strength does not change. The negative results are reported in full. Label refinement on AJILE12 beats matched random removal by 0.007 balanced accuracy (Table K.1), PAMAP2 trimming helps at one training budget and nowhere else (Table K.2), and the adaptive e-sum score is stated to forfeit the e-value guarantee (Section 3.1). This is why I trust the numbers I cite below, including the ones that argue against the paper.

Critical weaknesses

The method is a recombination of known components, and the experiments do not show that the combination adds anything. Class-conditional Mahalanobis scores (Lee et al., 2018) and e-value averaging (Vovk and Wang, 2021) are cited. Per-score conformal p-values for OOD detection (Bates et al.; Magesh et al.), anomaly-ensemble selection (SELECT), and label-noise detection such as confident learning, the obvious comparison for label auditing, are neither cited nor compared. What is new is a particular five-channel bank, a hand-set gate and weight cap, and a 0.70/0.30 rank split. None of it is ablated. There is no table showing the fusion beats its best single channel, and the evidence that exists says it barely does. An OpenMax-style score on spectral features alone, essentially one of the five channels, reaches 0.929 mean AUROC on PAMAP2 against 0.933 for the full method (Table I.2). PCA residual reaches 0.798 against 0.813 on the fresh AJILE12 structural target (Table H.1). The baselines receive different inputs from TC-REJECT (Appendix F), so nothing can be attributed to the fusion. In two-class settings the rank route replaces the conformal p-values of three channels with raw scores (p. 16); AJILE12 and DBS are both two-class, so the headline real-data numbers come from an uncalibrated variant of the method Section 3.1 describes. The two fusion rules also disagree about which is better, e-value sum on every synthetic condition (Table B.1) and rank averaging on both real datasets, and the paper's answer is to report both. This is the reason for my rating. A method paper whose method is not shown to beat its own parts does not clear the bar for ICLR, and no rebuttal experiment changes the size of the idea. The claims outrun the evidence. The motivating result (Section 4.1; Discussion) is that auditing needs both global atypicality and agreement with the assigned label. The score evaluated everywhere compares each window with whichever named class fits best and, in the authors' words, "does not force the comparison to use the window's available annotation" (Section 3.1). The discriminative channel is max-softmax, and own-label agreement enters only the relabeling step, so the detection results test a plain any-class novelty score and the motivating experiment has no bearing on them. The dissociation itself is thin: label-agreement AUROC of 0.063 on scale-inflated novelty (Appendix B.1) is far enough below chance to look like a sign flip in the classifier's confidence, and the authors' own temporal comparator does not reproduce the label-relative half (template readout 0.474, Appendix B.2). The title promises concealed states, but PAMAP2 and AJILE12 structural novelty are withheld known classes. The AJILE12 blend target is built by the SCALE channel's own refinement rule (Appendix C), and that is where the margin over baselines is widest (0.658 vs. 0.611, Table H.1). The only genuinely concealed states are in two DBS recordings with four and five reviewed segments, an S1 interval of [0.257, 0.947] (Appendix J), several baselines above TC-REJECT on S1 (Figure 5C), and no description of how the reference intervals were reviewed (Appendix E). Downstream gains are 0.007 on AJILE12 (Table K.1) and one clear cell out of 32 on PAMAP2 (Table K.2). The pipeline is also transductive: standardization, reliability probes, and the gate are fit on the scored run, novel windows included (Appendix F), and no inductive variant is reported. If the authors show the fusion beating its best single channel on identical inputs across all three real datasets with paired uncertainty, and reframe the paper around withheld-class OOD detection, I would move to weak reject and no further in this cycle.

Questions for authors

Does the fusion beat its best single channel on identical inputs? Please report rank and e-value averaging of the same three representations the baselines receive, then drop one channel at a time, with paired intervals against OpenMax on spectral features and PCA residual. A clear win here is the one result that would move my score(to 2). Why does the evaluated score omit label-relative evidence after Section 4.1 argues it is necessary, and what happens when

1 - P(y_avail | x)

is added as a channel? Why is label-agreement AUROC 0.063 on scale-inflated novelty instead of near 0.5? Who reviewed the DBS Rest intervals, from what signal, and with what agreement?

Reviewer_3fp4OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper proposes Typicality-Corrected Rejection, a framework that addresses the time-series label auditing problem. This paper proposes Typicality-Corrected Rejection, a calibrated multi-channel fusion framework. TC-REJECT introduces e-value summation fusion module and a rank averaging fusion module.

Critical strengths
  1. This paper addresses a critical challege of label quality auditing in coarsely annotated physiological and behavioral time series.
  2. The e-sum route preserves the magnitude of the calibrated evidence, while the rank-avg route discards it in exchange for scale robustness.
  3. Ablation studies systematically evaluate the two fusion routes, e-value summation and rank averaging, providing strong empirical insights.
  4. experiments span synthetic, wearable, ECoG and clinical DBS data, and include a prespecified evaluation on 12 previously unused AJILE12 sessions with a paired interval of [0.049, 0.123].
  5. the paper is overall well written and clearly structured.
  6. the DBS recordings provide a measured test of concealed label contamination, with independently reviewed Extra-Movement inside clinically annotated Rest rather than synthetically inserted novelty.
Critical weaknesses
  1. The novelty-score construction in the fusion module is a simple nearest-named-class Mahalanobis distance converted to an empirical conformal p-value per channel, then combined by a fixed-weight e-value sum or an unweighted rank average. 2.The method introduces several sensitive hyperparameters, e.g., the five-channel composition, the discriminative weight 0.30 in the rank-average fusion, the operating quantile α = 0.10, and the e-value calibrator κ = 0.4, none of which is varied.
  2. A major concern lies in the lack of essential baselines, e.g. learned time-series OOD representations, in particular HyperTF.
  3. this experiments are confined to a single model (a single 1D-CNN contrastive encoder for the LEARNED channel), without comparisons against stronger or larger model.
  4. Some parts of the fusion are not clear: how is the base weight w̄_k computed
  5. The prespecified 12-session result covers two targets, but the blend-type target is constructed by the SCALE channel’s own named-class fit rule, which the paper acknowledges makes it descriptive. The claim “highest on both targets” should be restricted to structural novelty.
Questions for authors

Can you give an a priori rule for choosing e-sum vs rank-avg on a new recording, or is the choice undeterminable without target labels? Please restate the 12-session confirmatory claim for structural novelty alone, given the blend-type target’s dependence on SCALE. Please report a leave-one-channel-out ablation and one fully inductive run. How is the base weight w̄_k computed when only the mixture probe is available? S2’s downstream balanced accuracy rises from 0.319 to 0.360 — both far below chance for two classes. What does an improvement on a failing decoder establish?

Reviewer_tuTnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes TC-REJECT, a framework for identifying novel or mislabeled states concealed within coarsely annotated time-series data. The authors highlight the distinction between global atypicality and label disagreement, arguing that conventional out-of-distribution detection may fail to identify certain hidden states. To address this issue, TC-REJECT combines five evidence channels derived from statistical, spectral, discriminative, and learned representations through e-value summation or rank averaging. Experiments are conducted on synthetic data, wearable activity recordings, ECoG, and clinical DBS-LFP signals to evaluate novelty detection and label refinement.

Critical strengths

The motivation is interesting and practically relevant. The paper addresses an important but often overlooked problem in time-series analysis: a sample may appear typical of the overall training distribution while being inconsistent with its assigned label. Distinguishing global atypicality from label disagreement is an interesting perspective, particularly for medical and physiological signals where coarse annotations and hidden state transitions are common. The controlled experiments also provide useful insights into why conventional novelty detection methods may fail under different conditions.

Critical weaknesses
  1. The overall presentation needs substantial improvement. The manuscript would benefit from further polishing, particularly in its visual presentation. Several figures contain excessively small text, making them difficult to read and interpret. For example, Figures 1-5 include numerous annotations, plots, and legends that are difficult to distinguish. I recommend simplifying the figures, increasing the font sizes, and improving their overall readability. The authors should also make the main methodological pipeline easier to follow.

  2. Limited methodological novelty. The proposed framework mainly combines existing techniques, including Mahalanobis distance, spectral features, contrastive representations, conformal-style calibration, and score fusion. While their integration is reasonable, the individual components are relatively standard, and the methodological contribution appears incremental. More systematic ablation studies would clarify the necessity of each component and whether the complex fusion framework offers meaningful advantages over simpler alternatives.

  3. Distinction between novelty detection and label disagreement. The main novelty score primarily evaluates whether a sample can be explained by any known class, while label compatibility is handled separately. The paper could better establish the relationship between these mechanisms and the central motivation. More importantly, the downstream benefits of label refinement are relatively limited. For example, on AJILE12, the improvement over size-matched random removal is marginal. In addition, some novelty targets lack independent ground-truth annotations, and the clinical DBS evaluation involves only two participants. More comprehensive validation is needed to demonstrate that the framework can reliably identify mislabeled states and meaningfully improve downstream performance.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.