- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
TC-REJECT scores multichannel windows for conflict with coarse time-series labels so they can be reviewed, dropped, or relabeled. Five evidence channels (amplitude and covariance, normalized temporal shape, spectral power, a classifier's softmax, and a self-supervised embedding) each give an empirical conformal p-value against the best-fitting named class. The p-values are fused by a weighted e-value sum with a data-dependent gate or by rank averaging. A synthetic benchmark shows a density score and a label-agreement score failing in opposite conditions. Validation covers withheld activities in PAMAP2, out-of-vocabulary and compound annotations in AJILE12 (with 12 prespecified held-out sessions), and reviewer-identified movements inside clinical Rest in two DBS recordings. The paper reports competitive detection, selective high-precision relabeling, and mixed downstream benefits.
Critical strengths
The prespecified test on 12 unused AJILE12 sessions (Appendix H) is a genuine confirmatory experiment: rank fusion beats e-value fusion in 11 of 12 sessions, with mean paired AUROC difference 0.086 and subject-bootstrap interval [0.049, 0.123]. It tells me the evaluation is careful. My rating rests on the size of the contribution, which this strength does not change. The negative results are reported in full. Label refinement on AJILE12 beats matched random removal by 0.007 balanced accuracy (Table K.1), PAMAP2 trimming helps at one training budget and nowhere else (Table K.2), and the adaptive e-sum score is stated to forfeit the e-value guarantee (Section 3.1). This is why I trust the numbers I cite below, including the ones that argue against the paper.
Critical weaknesses
The method is a recombination of known components, and the experiments do not show that the combination adds anything. Class-conditional Mahalanobis scores (Lee et al., 2018) and e-value averaging (Vovk and Wang, 2021) are cited. Per-score conformal p-values for OOD detection (Bates et al.; Magesh et al.), anomaly-ensemble selection (SELECT), and label-noise detection such as confident learning, the obvious comparison for label auditing, are neither cited nor compared. What is new is a particular five-channel bank, a hand-set gate and weight cap, and a 0.70/0.30 rank split. None of it is ablated. There is no table showing the fusion beats its best single channel, and the evidence that exists says it barely does. An OpenMax-style score on spectral features alone, essentially one of the five channels, reaches 0.929 mean AUROC on PAMAP2 against 0.933 for the full method (Table I.2). PCA residual reaches 0.798 against 0.813 on the fresh AJILE12 structural target (Table H.1). The baselines receive different inputs from TC-REJECT (Appendix F), so nothing can be attributed to the fusion. In two-class settings the rank route replaces the conformal p-values of three channels with raw scores (p. 16); AJILE12 and DBS are both two-class, so the headline real-data numbers come from an uncalibrated variant of the method Section 3.1 describes. The two fusion rules also disagree about which is better, e-value sum on every synthetic condition (Table B.1) and rank averaging on both real datasets, and the paper's answer is to report both. This is the reason for my rating. A method paper whose method is not shown to beat its own parts does not clear the bar for ICLR, and no rebuttal experiment changes the size of the idea. The claims outrun the evidence. The motivating result (Section 4.1; Discussion) is that auditing needs both global atypicality and agreement with the assigned label. The score evaluated everywhere compares each window with whichever named class fits best and, in the authors' words, "does not force the comparison to use the window's available annotation" (Section 3.1). The discriminative channel is max-softmax, and own-label agreement enters only the relabeling step, so the detection results test a plain any-class novelty score and the motivating experiment has no bearing on them. The dissociation itself is thin: label-agreement AUROC of 0.063 on scale-inflated novelty (Appendix B.1) is far enough below chance to look like a sign flip in the classifier's confidence, and the authors' own temporal comparator does not reproduce the label-relative half (template readout 0.474, Appendix B.2). The title promises concealed states, but PAMAP2 and AJILE12 structural novelty are withheld known classes. The AJILE12 blend target is built by the SCALE channel's own refinement rule (Appendix C), and that is where the margin over baselines is widest (0.658 vs. 0.611, Table H.1). The only genuinely concealed states are in two DBS recordings with four and five reviewed segments, an S1 interval of [0.257, 0.947] (Appendix J), several baselines above TC-REJECT on S1 (Figure 5C), and no description of how the reference intervals were reviewed (Appendix E). Downstream gains are 0.007 on AJILE12 (Table K.1) and one clear cell out of 32 on PAMAP2 (Table K.2). The pipeline is also transductive: standardization, reliability probes, and the gate are fit on the scored run, novel windows included (Appendix F), and no inductive variant is reported. If the authors show the fusion beating its best single channel on identical inputs across all three real datasets with paired uncertainty, and reframe the paper around withheld-class OOD detection, I would move to weak reject and no further in this cycle.
Questions for authors
Does the fusion beat its best single channel on identical inputs? Please report rank and e-value averaging of the same three representations the baselines receive, then drop one channel at a time, with paired intervals against OpenMax on spectral features and PCA residual. A clear win here is the one result that would move my score(to 2). Why does the evaluated score omit label-relative evidence after Section 4.1 argues it is necessary, and what happens when
1 - P(y_avail | x)
is added as a channel? Why is label-agreement AUROC 0.063 on scale-inflated novelty instead of near 0.5? Who reviewed the DBS Rest intervals, from what signal, and with what agreement?