acceptodds
Under review as a conference paper at ICLR 2027

Auditing Subject-Level Conformal Reliability in Medical Time Series

Abstract

Medical time-series (MedTS) classification often uses subject-independent evaluation, where models generalize to unseen patients who each contribute multiple potentially dependent observations. This creates a mismatch between the subject as the unit of deployment and the window as the unit commonly used for conformal calibration and evaluation. We conduct a systematic audit of subject-level conformal reliability in this setting. Across four EEG and ECG benchmarks, pooled evaluation can conceal substantial under coverage for individual unseen subjects; on ADFTD, near-nominal pooled and class-stratified coverage conceal the same failure. We show that pooled coverage forms a window-count-weighted average of subject-level coverage, while pooled calibration targets a quantile of a mixture of heterogeneous subject-specific score distributions. Across our diagnostic probes, training-time conformal optimization shows inconsistent effects on classification behavior, while similarity-weighted recalibration and selection-aware inference do not consistently resolve subject-level miscoverage. Established error-driven quantile adaptation can substantially improve reliability in sequential deployment when feedback is sufficient and representative, but its effectiveness depends on within-subject feedback volume and, for a shared-state controller, stream organization, and can reverse under prediction-dependent outcome observation. These findings concern continuous online adaptation rather than a threshold calibrated once and then fixed, and show that nominal pooled reliability need not reflect the reliability experienced by individual unseen subjects.

Primary area:
learning on time series and dynamical systems
Keywords:
Conformal Reliability, Medical Time Series

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jVYWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper discusses several concepts related to subject-level reliability and temporal adaptation, which are important problems in (clinical) time-series analysis. However, the paper itself does not clearly establish a novel research gap or propose a new algorithm. In fact the work reads more like an exploratory evaluation of existing methods than a substantial scientific research contribution.

Given the limited novelty and methodological contributions, I do not believe the paper currently meets the standard expected for acceptance at a leading AI conference or journal.

Critical strengths

The problem itself is relevant and important in patient level time-series analysis, particularly when accounting for subject-level heterogeneity and ensuring reliable predictions for future unseen patients.

Critical weaknesses
  1. As mentioned in the summary, the biggest issue is that the paper does not clearly identify a specific research gap or propose a new method. As a result, the work reads more like an undergraduate-level student project or technical report than a substantial scientific research paper.

  2. The existing literature contains many studies that have already addressed related problems using data-driven machine learning and statistical learning approaches. For example:

(1) Dunn, Robin, Larry Wasserman, and Aaditya Ramdas. "Distribution-free prediction sets for two-layer hierarchical models." Journal of the American Statistical Association 118.544 (2023): 2491-2502.

(2) Gibbs, Isaac, and Emmanuel Candes. "Adaptive conformal inference under distribution shift." Advances in Neural Information Processing Systems 34 (2021): 1660-1672.

(3) Isaac Gibbs, John J Cherian, Emmanuel J Candès, Conformal prediction with conditional guarantees, Journal of the Royal Statistical Society Series B: Statistical Methodology, Volume 87, Issue 4, September 2025, Pages 1100–1126, https://doi.org/10.1093/jrsssb/qkaf008

(4) Yan, Mengying, et al. "TRACER: Transfer Learning based Real-time Adaptation for Clinical Evolving Risk." arXiv preprint arXiv:2512.12795 (2025).

(5) Bharti, Beepul, et al. "Parameter-Free and Group Conditional Online Conformal Prediction." arXiv preprint arXiv:2606.00419 (2026).

Given these existing contributions, the paper does not sufficiently establish its novelty or significance as a research contribution.

Questions for authors

See above.

Reviewer_QYgvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Medical time-series models are calibrated and evaluated on pooled windows but deployed per patient. The paper audits what that means for conformal prediction, where pooled coverage is high but individual user can fail. Four existing methods are tested and only online threshold tracking with complete label feedback helps which needs many windows per patient and unbiased feedback.

Critical strengths

The problem setting is valuable in medical scenario, where models can seem high performing overall but disadvantaging certain subjects.

Critical weaknesses
  1. The novel contribution is limited. It is known that overall coverage does not guarantee coverage for each group or individual. The challenge identified here is an instance of this known problem.
  2. The evaluation outcome is also predictable. None of the first three probes uses any information specific to the test patient. Only the fourth uses that patient’s own labels, so it’s the only one that can work. The paper neither proposes a solution nor compares against methods designed for this problem: (e.g., hierarchical conformal prediction, group-conditional coverage, few-shot conformal calibration for new tasks).
  3. The paper focuses on coverage as the metric for evaluation. But coverage is easy to cheat. Always output every class and you get 100% coverage with no useful information. It's hard to tell whether the fairness conclusion or comparison based on this is meaningful enough.
  4. The abstract and intro is not clearly written and hard to follow full of undefined jargons. The motivation, problem setting, challenge and contribution should be better clarified.
Questions for authors
  1. Two of the four tasks are binary, and the paper doesn’t state the class counts up front. How is conformal prediction defined or meaningful in the setting of binary classification?
  2. How do subject-aware methods compare under the same protocol?
  3. Please describe the backbones used for each dataset, and report accuracy, train/calibration/test subject counts, windows per subject and average set size.
  4. What finding / contribution of the paper is specific to medical time series, or just application and evaluation of existing methods?
Reviewer_bhmAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper calibrates one pooled split-conformal threshold over all windows of the calibration subjects and then reports coverage per test subject on four EEG/ECG datasets (ADFTD, APAVA, PTB, PTB-XL). Pooled coverage is near 0.90, but some subjects are far below it, e.g. ADFTD subject 31 at 0.496. Four existing methods are then used as "probes": ConfTr, similarity-weighted recalibration, CAP, and online quantile tracking (ACI-style P-control). Quantile tracking with true labels after every window cuts the spread of per-subject coverage a lot (23.5× on ADFTD); the paper then studies when this gain shrinks (few windows per subject, interleaved streams, selective feedback, frozen state). Appendix C gives six propositions about pooled coverage and pooled quantiles. No new method and no new guarantee are proposed.

Critical strengths
  1. The question matters in practice: subject-independent splits are standard in MedTS, and reporting coverage per patient is a useful habit.
  2. The controlled interventions are well designed: capping windows per subject, blocked vs. interleaved streams, per-subject controller state, and frozen held-out halves.
  3. The paper reports results that go against its own positive control (Table 3), and checks the main effect across target levels, the APS score, and five trained models on ADFTD.
Critical weaknesses

The main finding is what "marginal coverage" means. Pooled coverage is an average over windows, so some subjects can fall far below it; Vovk (2013), cited here, already made this point. Propositions 1 to 4 are one-line algebra on a weighted average, Proposition 5 restates the definition of a quantile, and Proposition 6 the definition of a covariance. Dunn et al. (2023) give prediction sets for exactly this two-level structure; the paper cites it but does not run it. There is no new method and no new guarantee.

The failures look like a wrong classifier plus sampling noise, on very few subjects. The motivating case is 1 of 19 ADFTD subjects, and APAVA has 4 test subjects. Subject 31 has coverage 0.496 with set size 1.69, i.e. the model mostly gets this patient's diagnosis wrong, yet classifier accuracy is never reported and the "CoTAR-based backbone" is never described or cited. What does this CoTAR actually means... On PTB-XL (about 11 windows per subject), a subject with true coverage 0.90 drops below 0.80 about 9% of the time by binomial chance alone, against the reported 14.8%, and neither R s u b j nor the dispersion metric corrects for n m . The main results use only one seed, which might not be robust enough.

The positive result needs the answer it is meant to find. In ADFTD, APAVA and PTB every window of a subject carries the same diagnosis, so "exogenous feedback" keeps revealing the diagnosis after every window, and it is scored on the same windows that fed the controller. Frozen and tested on held-out windows, the gain drops to 1.5 to 1.7× on ADFTD and 1.0 to 1.3× on PTB, while a plain per-subject split-CP threshold from the same labels gets 6.8×, 7.4× and 5.6× (Table 3). With interleaved subjects, the shared controller gives 1.0× on ADFTD (Table 2).

The other probes cannot answer the question, and the text hedges everything. ConfTr is scored by hardest-class recall on a different backbone, and CAP on a different subset, where its PTB gain is "mechanical" by the authors' own account. Reweighting does help (ADFTD standard deviation 0.1200 to 0.0781, minimum coverage 0.4957 to 0.7215) but is reported as a failure. The "endogenous" rule hides feedback for confident singletons, which is sure to break the controller; on PTB it keeps 0.06% of the feedback. Disclaimers fill the text ("deliberately non-novel positive control", "not an algorithmic leaderboard", a seed checkpoint "not retained"), while split sizes and the number of calibration subjects are missing.

Questions for authors

See Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.