acceptodds
Under review as a conference paper at ICLR 2027

Beyond Accuracy: Conformal Risk Control for Medical Time-Series Classification

Abstract

Medical time-series classifiers are typically evaluated by predictive accuracy, yet accuracy alone provides no finite-sample control of clinically relevant deployment risks. We study conformal risk control (CRC) for medical time-series classification and show that its direct application is complicated by the hierarchical structure of medical time series: medical time series have a patient–window hierarchy in which each patient contributes many correlated windows. Treating windows as exchangeable calibration units creates a nominal sample-size illusion and may control a window-weighted risk rather than the risk for a newly sampled patient. Patient-level calibration corrects this mismatch but substantially reduces the effective calibration size, yielding a small-sample regime in which post-hoc CRC can become conservative. Separately, end-to-end conformal risk training requires careful model selection to preserve the target risk constraint. We develop classification-specific CRC for binary false-negative-rate control and multiclass coverage, together with a subject-aware formulation that calibrates directly over patients. We further extend the framework to subject-level conditional value-at-risk (CVaR) to control tail risk across patients. Across 12 architectures and three EEG/ECG datasets, experimental results reveal a validity–efficiency–stability trade-off: post-hoc CRC provides finite-sample expected-risk control under exchangeability, whereas end-to-end training can improve efficiency on binary tasks but is less efficient than closed-form fine-tuning on the multiclass ADFTD task once model selection correctly enforces the coverage constraint. Reliable deployment risk control therefore requires careful choice of both the calibration unit and the model-selection criterion used during optimization.

Primary area:
learning on time series and dynamical systems
Keywords:
Medical time-series, Conformal Risk Control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WbtCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the application of conformal risk control to medical time-series classification. The authors consider binary false-negative-rate control and multiclass coverage, compare post-hoc calibration with end-to-end conformal risk training, and extend the framework to subject-level risk and conditional value-at-risk.

Critical strengths

The authors highlight important challenges in real-world medical time-series classification. These challenges are relevant to the practical application of conformal risk control in medical settings.

Critical weaknesses

The contribution is limited, as the proposed framework largely consists of straightforward applications and adaptations of existing conformal risk control and conformal risk training methods to medical time-series classification. The theoretical formulations primarily follow from established results, without introducing substantial new methodological or theoretical advances.

Although the paper presents extensive experimental results, the analysis does not provide sufficiently clear conclusions or insights into when and why the proposed approaches are beneficial. Consequently, the broader implications of the empirical findings remain unclear.

Questions for authors
  1. A substantial portion of the methodology appears to be a direct application of existing Conformal Risk Control (CRC) and Conformal Risk Training (CRT) methods. In particular, several derivations in Section 4 follow directly from established results or straightforward algebraic manipulations. Could the authors clarify the specific contributions that distinguish this work from these existing approaches?

  2. Alignment between the stated challenges and contributions. The challenges introduced in the Introduction are meaningful, but their connections to the proposed contributions are not sufficiently clear. The authors should clarify how each proposed component directly addresses the challenges identified in the Introduction.

  3. Medical time-series windows from the same patient are often highly correlated, which may violate the exchangeability assumption when individual windows are treated as separate calibration units. Although the authors acknowledge this issue and propose subject-aware CRC to address it, the experimental analysis does not sufficiently investigate how the degree of within-patient correlation affects calibration validity and predictive efficiency. I believe more dedicated controlled experiments varying the strength of within-patient correlation would help isolate and quantify its impact.

  4. Insufficient discussion of experimental findings. Much of the Results section focuses on describing numerical outcomes rather than providing deeper explanations of the observed patterns. For example, simply concluding that "the effect of E2E is task dependent" or counting the number of architectures achieving smaller prediction sets provides limited insight without explaining why these differences arise. A more detailed analysis of the factors driving these results, including dataset characteristics, model architectures, and calibration sample sizes, would significantly strengthen the empirical contribution.

  5. According to Table 4, all evaluated classifiers already achieve FNR below the target level of α=0.05\alpha=0.05 before calibration. Post-hoc CRC reduces FPR at the cost of FNR to increase toward the prescribed threshold. However, the clinical significance of this trade-off is not sufficiently discussed. In medical diagnosis, false negatives may have substantially more severe consequences than false positives, although their relative importance depends on the clinical context. The authors should provide stronger justification for the selected risk thresholds and discuss whether the resulting improvements in FPR are clinically meaningful given the corresponding increases in FNR.

  6. The CVaR experiments primarily demonstrate an anticipated trade-off: stronger protection against high-risk patients leads to more conservative predictions and reduced predictive efficiency. In some settings, this results in nearly trivial prediction rules. While these observations are consistent with the underlying CVaR formulation, it is unclear what additional insights they provide beyond established theoretical expectations. The authors should clarify the practical benefits of subject-level CVaR control and investigate whether meaningful improvements in tail-risk protection can be achieved?

6 comments
AuthorsOpenReview ↗

Response to Reviewer WbtC

We thank the reviewer for the detailed and constructive feedback. We clarified our contributions and expanded the analyses. All revisions are highlighted in blue in the paper.

R2W1: Building on established CRC and CRT methods, we examine their limitations in hierarchical medical time series. Within-patient dependence raises window-level CRC's risk-exceedance probability to 0.261, versus 0.054 for subject-level CRC (Appendix D.11, Table 25). Larger calibration cohorts reduce subject-level conservativeness to approximately 2% additional set size at n=3,500n=3{,}500, while window-level CRC remains miscalibrated (Appendix D.12, Table 26).

R2Q1: Our contributions distinguish window-weighted from patient-level risk, formulate subject-aware CRC and CVaR, and assess E2E efficiency and stability. On PTB-XL (12 architectures), subject-level CRC achieves near-nominal coverage with only 2–3% larger prediction sets, whereas window-level CRC systematically undercovers (Appendix D.8, Table 21). On ADFTD, subject-level CRC reduces risk-exceedance frequency under photic stimulation and resting-to-photic transfer (COMET: 0.56 to 0.07; Appendix D.9, Table 22). These experiments extend validation to a larger clinical cohort and a different recording condition without retraining.

R2Q2: The revised Introduction links classification-specific CRC and E2E to risk formulation and efficiency (Challenge I), subject-aware CRC to patient dependence (II), and calibration cohort size and E2E stability to limited independent data (III). CVaR additionally addresses tail-sensitive risk.

R2Q3: We vary within-patient correlation while fixing marginal distributions and sample sizes (2,000 trials; Table 1; Appendix D.11, Table 25). As correlation increases from 0 to 0.9, effective sample size falls from 750 to 26. At N=30N=30, window-level risk exceedance rises to 0.261, versus 0.054 for subject-level CRC; at N=100N=100, the corresponding values are 0.105 and 0.040. Window-level CRC has lower FPR, illustrating the efficiency–reliability trade-off. Importantly, the effective sample size is a design-effect approximation, not the number of independent calibration patients.

Table 1. Effect of within-patient dependence on CRC calibration (risk exceedance / FPR).

ρICCn_effWindow N=30Subject N=30Window N=100Subject N=100
007500/.1130/.2250/.1120/.134
.1.04273.009/.1130/.2240/.1120/.134
.25.10132.053/.1130/.222.004/.1120/.133
.5.2263.159/.113.003/.221.029/.113.003/.135
.75.4037.232/.117.028/.222.066/.114.018/.136
.9.5826.261/.120.054/.232.105/.115.040/.138

R2Q4: We added analyses of label noise (Appendix D.5, Table 16), calibration size (D.12, Table 26), architecture and risk level (D.6, Tables 18–19), and checkpoint selection (D.7, Table 20). On PTB, the E2E–CE FPR difference changes from +0.030 without noise to −0.296 at 20% label noise. Window-E2E outperforms CE on 6/12 PTB architectures but only 3/12 ADFTD architectures. Risk-aware selection improves ADFTD E2E set size over last-epoch selection (2.206 vs. 2.364), but worsens PTB FPR relative to efficiency-only selection (0.520 vs. 0.426). These findings show task-dependent effects without establishing a single causal explanation. Label quality and calibration size are identifiable contributing factors, while architecture and optimization effects remain dataset-dependent.

R2Q5: Lower FPR does not necessarily imply clinical benefit; α=0.05\alpha=0.05 is illustrative. On PTB, subject-level CRC reduces missed diagnoses from 3.7 to 1.7 per 100 diseased patients but adds 10.2 referrals per 100 healthy patients (Appendix D.13, Table 27). Window-level CRC can violate patient-level FNR targets, while subject-level CRC becomes trivial at α≤0.05\alpha\leq0.05 with only 19 calibration patients (Appendix D.13, Table 28). Insufficient disease-positive calibration units can therefore prevent nontrivial certified decisions. Clinical thresholds require disease-specific justification.

R2Q6: CVaR closely matches mean-risk control at budget βα\beta\alpha for sparse losses; on PTB, calibration is trivial in 79% of resplits at α=0.1,β=0.5\alpha=0.1,\beta=0.5 (Appendix D.14, Table 29). Direct tail-exceedance control, P(Li>τ)≤αP(L_i>\tau)\leq\alpha, achieves a high-loss patient fraction of 0.067 and FPR of 0.395 on PTB, versus FPRs of 0.448 for mean-risk CRC and 0.959 for CVaR CRC (Table 30). Exceedance control targets prevalence rather than severity of high losses. Formal guarantees remain marginal and require subject-level exchangeability and independent calibration. Neither CVaR nor exceedance control guarantees that every individual patient satisfies the prescribed risk level.

AuthorsOpenReview ↗

Additional Experimental Results

Appendix D.6, Table 18 Matched comparison of post-hoc CRC, CE, window-E2E, and subject-E2E.

MethodPTB (α = 0.1)ADFTD (α = 0.2)
Post-hoc subject CRC0.373 / 0.30 / 0.8312.285 / 0.13 / 0.535
CE + subject CRC0.401 / 0.29 / 0.8392.276 / 0.11 / 0.518
Window-E2E + subject CRC0.428 / 0.30 / 0.8602.415 / 0.11 / 0.416
Subject-E2E + subject CRC0.442 / 0.29 / 0.8482.395 / 0.12 / 0.418
Window-E2E better than CE6 of 123 of 12
Subject-E2E better than Window-E2E7 of 125 of 12

Appendix D.13, Table 27 Clinical operating points under subject-level CRC on PTB.

Decision ruleMissed diagnoses / 100 diseasedUnnecessary referrals / 100 healthy
Default (P(disease) ≥ 0.5), no guarantee3.7 ± 1.165.2 ± 5.4
Subject-level CRC, α = 0.020.0100.0 (refers everyone)
Subject-level CRC, α = 0.051.7 ± 2.075.4 ± 7.8
Subject-level CRC, α = 0.107.1 ± 4.553.9 ± 11.4
Subject-level CRC, α = 0.1512.0 ± 5.939.8 ± 11.3
Subject-level CRC, α = 0.2016.9 ± 6.828.7 ± 9.9

R2Q1

Our contributions distinguish window-weighted from patient-level risk, formulate subject-aware CRC and CVaR, and assess E2E efficiency and stability. I believe this is technically direct application of conformal risk control and conformal risk training [1] to different granularity based on the hierarchy of patient data. Section 4.1 - 4.4 are largely just simple re-writing of definitions or existing methodology from [1,2]. Although I understand the application to important domain has contribution, that contribution is also not clear, which means entire study's contribution is very limited. Could authors clarify what is novel in section 4.1 - 4.4 compared to [1]?

[1] Conformal risk control.

[2] Conformal risk training: End-to-end optimization of conformal risk control

R2Q4

While my point was insuffcient discussion about the "experimental findings", the additional experiments that authors conducted was more towards sensitivity analysis in controlled setting. I am convinced the need for the additional experiments. Additionally, all the additional experiment results look very mixed, less consistent trend, and simply presented numbers without summarized and organized interpretation, which makes it very difficult to understand and interpret.

D.5: label noise and number of training samples. I believe those are less related to the authors research hypothesis. Additionally, the result trends are not consistent, not sufficiently showing the proposed methods benefit.

D.6: how can we interpret "Window-E2E better than CE" and "Subject-E2E better than Window-E2E" in Table 18 and "Window-E2E better" in Table 19?

D.7: I believe checkpoint selection is not a ablation. Rather it is necessary details to be included.

R2Q5

I believe the author and I pointing out the same thing. As author stated, clinical benefit and thresholds require disease-specific justification. Therefore, I believe the authors statement calibration mainly converts excess conservativeness in the raw decision rule into lower FPR while moving FNR toward the prescribed boundary does not necessarily desirable and benefitable in all cases without detailed justification.

Reviewer_nkUPOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Conformal Risk Control (CRC) for medical time-series classification, with a particular focus on the patient–window hierarchy. The authors investigate four related approaches, progressing from post-hoc risk calibration and end-to-end conformal risk training to subject-aware calibration and subject-level tail-risk control using CVaR. The framework aims to provide statistical control over clinically relevant risks, such as false-negative rates and multiclass miscoverage, rather than focusing solely on conventional classification metrics. Experiments are conducted on three EEG/ECG datasets using 12 classification backbones. The results demonstrate the importance of selecting appropriate calibration units and reveal trade-offs among statistical validity, prediction efficiency, and calibration stability.

Critical strengths
  1. Interesting motivation with practical relevance. The paper addresses an important problem in medical time-series classification. Controlling clinically relevant risks beyond conventional classification accuracy is meaningful for real-world deployment. I find the overall research direction interesting and worth investigating.

  2. Clear and progressive methodological organization. I appreciate how the paper develops its methodology through four related perspectives, from post-hoc calibration to end-to-end optimization, subject-aware calibration, and finally tail-risk control for the worst-performing patients. This progression is logical and helps connect the different aspects of risk control.

  3. Appropriate consideration of patient-level data characteristics. The authors recognize that different patients may contribute substantially different numbers of windows, which can bias window-level calibration and exaggerate the effective sample size. This is an important and often overlooked issue in medical time-series analysis.

  4. Informative experimental comparisons. The paper evaluates both post-hoc and end-to-end approaches across multiple classification backbones. In particular, the comparisons between conventional fine-tuning and end-to-end risk training are informative, including cases where the more complicated approach does not outperform simpler alternatives.

Critical weaknesses
  1. The overall presentation could be improved. Although the general motivation is interesting, the Introduction is somewhat difficult to follow, and the main contributions are not sufficiently emphasized. I suggest reorganizing the Introduction to state the core problem and its practical significance more directly, then clearly explain how the four methodological components address different aspects of it.

  2. The experimental evaluation is limited by the datasets. The paper evaluates only three datasets, and some have very few independent subjects. For example, APAVA contains only 23 subjects, which may not be sufficient to demonstrate robust patient-level risk control. I strongly recommend expanding the experimental evaluation with larger, more diverse clinical EEG datasets. I know it may be traditional to follow Medformer and use this dataset, but I'm fine with you not using it in your manuscript. Several relevant datasets could be considered: ADFTD photic stimulation extension (OpenNeuro ds006036): This provides complementary recording conditions for the same cohort and could help evaluate robustness across different acquisition conditions. You could combine it with the original ADFTD (resting-state) for more robust evaluation. TDBRAIN V3.1: The updated dataset includes additional healthy controls and could support further evaluation of Parkinson's disease detection. CAUEEG: A relatively large clinical EEG dataset for dementia-related classification, although access requires an application. These additional datasets would help demonstrate whether the proposed framework generalizes across different diseases, cohorts, and recording conditions. You could also choose another large MedTS-based disease detection dataset available online. These suggestions are meant to improve your paper; if you don't have enough time to finish them in this revision, that's ok.

  3. Missing subject-level classification performance. Although the paper introduces subject-aware calibration and reports subject-level risk metrics, these results are obtained by averaging window-level losses within each patient. This is different from actual subject-level classification, where window predictions are aggregated into a single patient-level decision. I recommend additionally reporting subject-level Accuracy, F1, AUROC, and relevant risk metrics based on aggregated patient-level predictions. For example, the LEAD paper provides a subject-level evaluation protocol using majority voting across EEG samples. Such an evaluation would better demonstrate the clinical relevance of the proposed framework and complement the current subject-aware risk analysis. Also, if you have a better way to obtain subject-level performance that suits your method, that would also be good.

  4. The relationship between end-to-end training and subject-aware calibration needs further investigation. The paper emphasizes the importance of patient-level risk control, but the end-to-end CRT procedure still performs pseudo-calibration and optimization at the window level. Subject-aware calibration is applied separately rather than being integrated into the end-to-end optimization objective. This leaves a gap between the training objective and the intended patient-level deployment risk. Investigating whether to incorporate subject-aware risk into end-to-end training would make the framework more complete and strengthen its central motivation.

Questions for authors

Q1. Given that subject-aware calibration is an important contribution, why do the authors not incorporate subject-level risk directly into end-to-end conformal risk training? Would a subject-grouped mini-batch strategy make this feasible? For example, LEAD [1] introduces a subject-regularized training strategy that groups samples from the same subject within each mini-batch to compute a subject-level loss. Although its objective differs from conformal risk control, this strategy might provide a useful reference for incorporating subject-level information into end-to-end optimization.

[1] LEAD: An EEG Foundation Model for Alzheimer's Disease Detection

4 comments
AuthorsOpenReview ↗

Response to Reviewer nkUP

We thank the reviewer for the feedback. We clarified the contributions and expanded the experiments, with new results in the appendix and revisions highlighted in blue.

R2W1: We thank the reviewer for this suggestion. We have reorganized the Introduction to clarify the central challenge and practical significance of patient-level risk control in medical time-series classification. We now explicitly connect the four methodological components—classification-specific CRC, E2E risk training, subject-aware calibration, and patient-level CVaR—to risk formulation, predictive efficiency, patient-level validity, and tail-risk control, respectively. We have also refined the contribution summary to better emphasize the methodological and empirical contributions.

R2W2: We thank the reviewer for these suggestions. We extend our evaluation to PTB-XL (17,596 patients, 12 architectures, 100 resplits), where subject-level CRC achieves near-nominal coverage (0.900/0.800/0.700 at (α=0.1/0.2/0.3)\alpha=0.1/0.2/0.3)) with only 2–3% larger prediction sets for most architectures, while window-level CRC systematically undercovers (Appendix D.8, Table 21). We also evaluate ADFTD under photic stimulation and resting-to-photic transfer without retraining. Window-level CRC violates the risk target in 20–58% of photic resplits, versus at most 16% for subject-level CRC; under transfer, COMET's violation frequency decreases from 56% to 7% (Appendix D.9, Table 22). These results support subject-aware calibration across cohort sizes and recording conditions, while APAVA remains a small-sample boundary case and broader clinical validation remains necessary.

R2W3: We thank the reviewer for this suggestion. We distinguish patient-averaged window risk from risk based on aggregated patient-level predictions. Using Medformer, majority voting and mean-probability aggregation improve subject-level accuracy on PTB, ADFTD, and PTB-XL, with the largest gain on ADFTD (+14.4 percentage points; Appendix D.10, Table 23). We further apply subject-level CRC to aggregated predictions (Appendix D.10, Table 24), achieving empirical risk below the target and improved efficiency over patient-averaged window-loss calibration (ADFTD CE: 1.69 vs. 1.93; PTB E2E: 0.292 vs. 0.411). The formal guarantee requires subject-level exchangeability and independent calibration. E2E also improves PTB accuracy (0.896 vs. 0.852) and AUROC (0.939 vs. 0.917) relative to CE, but underperforms on ADFTD. These results demonstrate the task-dependent benefits of patient-level aggregation and E2E training.

R2W4: We thank the reviewer for this observation. We introduce subject-grouped E2E CRT, which uses patient-aggregated losses for pseudo-calibration and optimizes subject-averaged efficiency. Across 12 architectures and five seeds (Appendix D.6, Table 18), subject-grouped E2E achieves efficiency comparable to window-level E2E under subject-level CRC (PTB FPR: 0.442 vs. 0.428; ADFTD set size: 2.395 vs. 2.415), but neither consistently outperforms post-hoc CRC. These findings support the feasibility of subject-aware E2E optimization while reinforcing post-hoc subject-aware CRC with independent calibration as our preferred deployment strategy.

R2Q1: We thank the reviewer for this suggestion. Inspired by subject-grouped training strategies such as LEAD, we implement subject-grouped E2E CRT using patient-level pseudo-calibration and differentiable risk control. Across 12 architectures and five seeds (Table 1 in this response; Appendix D.6, Table 18), subject-grouped E2E achieves efficiency comparable to window-level E2E (PTB FPR: 0.442 vs. 0.428; ADFTD set size: 2.395 vs. 2.415), without consistently outperforming post-hoc CRC. On APAVA, the limited number of patients causes the training threshold to default to zero in all subject-grouped steps. These results demonstrate feasibility but highlight limited efficiency gains and small-cohort sensitivity, supporting post-hoc subject-aware CRC with independent calibration as our preferred deployment strategy.

Table 6. Comparison of subject-grouped and window-level E2E training, including in-batch calibration stability.

MethodPTB (α = 0.1)ADFTD (α = 0.2)APAVA (α = 0.1)
Post-hoc subject CRC0.373 / 0.30 / 0.8312.285 / 0.13 / 0.535trivial
CE + subject CRC0.401 / 0.29 / 0.8392.276 / 0.11 / 0.518trivial
Window-E2E + subject CRC0.428 / 0.30 / 0.8602.415 / 0.11 / 0.416trivial
Subject-E2E + subject CRC0.442 / 0.29 / 0.8482.395 / 0.12 / 0.418trivial
Subject-E2E better than Window-E2E7 of 125 of 12—
In-batch λ = 0 (Window / Subject-E2E)0.00 / 0.020.00 / 0.000.00 / 1.00
AuthorsOpenReview ↗

Additional Experimental Results

Appendix D.8 Table 21 Evaluation of window-level and subject-level CRC on PTB-XL (12 architectures, five classes, 100 resplits, 3,519 calibration patients per resplit). Results report patient-level coverage and average prediction-set size.

ArchitectureWindow acc.Window-level CRCSubject-level CRC
FEDformer0.3750.897/3.51 · 0.795/2.78 · 0.693/2.230.900/3.53 · 0.800/2.81 · 0.700/2.26
TeCh0.7350.895/1.66 · 0.792/1.16 · 0.693/0.900.900/1.71 · 0.800/1.19 · 0.700/0.92
iTransformer0.6870.896/2.04 · 0.795/1.37 · 0.695/1.010.900/2.08 · 0.801/1.39 · 0.700/1.03
Informer0.7110.895/1.83 · 0.794/1.25 · 0.694/0.950.900/1.88 · 0.800/1.27 · 0.699/0.96
Medformer0.7170.895/1.75 · 0.793/1.22 · 0.694/0.930.900/1.79 · 0.800/1.24 · 0.700/0.95
Reformer0.7190.895/1.78 · 0.794/1.22 · 0.694/0.930.900/1.83 · 0.800/1.24 · 0.700/0.95
TCN0.7390.894/1.65 · 0.792/1.14 · 0.693/0.890.900/1.70 · 0.800/1.17 · 0.699/0.90
Transformer0.7170.895/1.79 · 0.794/1.23 · 0.694/0.940.900/1.84 · 0.800/1.25 · 0.699/0.95
PatchTST0.7260.895/1.81 · 0.793/1.21 · 0.693/0.920.900/1.86 · 0.800/1.24 · 0.699/0.94
Non-stationary Tr.0.7160.895/1.82 · 0.793/1.22 · 0.693/0.930.900/1.87 · 0.799/1.25 · 0.699/0.95
COMET0.7310.894/1.78 · 0.791/1.18 · 0.692/0.910.900/1.84 · 0.799/1.21 · 0.699/0.92
MedGNN0.7140.896/1.82 · 0.794/1.24 · 0.693/0.940.900/1.86 · 0.801/1.27 · 0.699/0.96

Thanks for the rebuttal and hard work addressing my concerns and suggestions. The manuscript looks much better than the original version. The newly added experiments and the use of subject-grouped E2E CRT look good. Again, I think this paper tries to address important challenges in MedTS, rather than simply modifying or combining modules as in many other works. I believe it deserves acceptance, so I have increased my score.

We sincerely thank the reviewer for the encouraging feedback and for recognizing our efforts to address the concerns and improve the manuscript. We greatly appreciate the reviewer's positive assessment of our contributions and support for the paper.

Reviewer_QM8AOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at the problem of medical time series classification. While the majority of the papers in this area use standard predictive metrics such as AUROC and AUPRC, this paper instead considers controlling the expected loss a model may incur through conformal risk control (Angelopoulos et al., 2024). They achieve this by applying conformal risk control specialised to medical classification losses and using the conformal risk training approach of Yeh et al. (2025) to fine-tune models specifically for the conformal risk objective. Additionally, the paper applies the approach of Yeh et al. (2025) for controlling the Conditional Value at Risk (CVaR, Rockafellar and Uryasev, 2000), as opposed to just the average risk.

One of the main challenges addressed in this paper is that patient data is recorded at different time periods (windows), whereas the goal is to control the risk for newly seen patients at test time. Moreover, the temporal nature of the data-generating process creates dependencies between the data points observed at different windows, breaking exchangeability and preventing a direct application of the standard conformal risk approach. To address this, they average the window-level losses for each patient into a single patient-level loss and then run conformal risk control across patients. In their experiments, they demonstrate that while conformal risk control reliably controls the target risk, calibrating over patients rather than windows yields overly conservative prediction sets. Additionally, they show that end-to-end conformal risk training improves efficiency on binary tasks but not on multiclass tasks, often performing no better than ordinary fine-tuning.

References:

Angelopoulos, Anastasios, et al. "Conformal risk control." International conference on learning representations. Vol. 2024. 2024. Rockafellar, R. Tyrrell and Stanislav P. Uryasev. “Optimization of conditional value-at risk.” Journal of Risk 3 (2000): 21-41. Yeh, Christopher, et al. "Conformal risk training: End-to-end optimization of conformal risk control." Advances in Neural Information Processing Systems 38 (2026): 70056-70092.

Critical strengths

While the paper does not contribute any new methodology or theory, I believe there is still strength in identifying an important, practical problem and coming up with a statistically sound approach to solve it.

Critical weaknesses

While I believe the paper addresses an important, practical problem, I believe there are significant issues when it comes to 1) how the paper is presented, and 2) the suitability of the paper for ICLR. Below, I comment on each of these at a high level and then provide further evidence for them at a lower level.

High-level:

  1. The paper is poorly presented on several fronts:

    • a. There is a lot of notation and terminology used without being properly introduced throughout the paper, from the abstract onwards. Moreover, there is not enough detail in Section 3 to understand the problem setting and what kind of guarantee we are looking for. This problem also repeats elsewhere in the paper (e.g., throughout Section 5).
    • b. It is not actually clear what overall method the paper is proposing (Algorithm 2 is referenced but not included).
  2. Given that the paper's approach is mainly a combination of existing methods, I believe the quality of the paper rests heavily on how convincing the results are and the quality of the corresponding analysis. However, in its current state, I do not believe the quality of the analysis is high enough, nor are the results convincing enough to meet the bar for acceptance at ICLR (see below for more details).

Low-Level:

  1. Presentation:

    • a.i. What is the input and output space of fθf_\theta? What makes it a time-series classifer? Is pθp_\theta derived from fθf_\theta?
    • a.ii. What exactly is a window? What is the expectation in (1) with respect to (I know that it is over the randomness of the calibration set and test point, but I believe it is essential to explicitly state this).
    • a.iii. λmin\lambda_\mathrm{min} is not defined anywhere and used in (2)
    • a.iv. I am guessing window-level calibration means using (19)? If so, I would state it explicitly, as equation (19) is never used or mentioned anywhere except on line 283.
    • b.i. What is the overall procedure that the paper is recommending? Is it E2E? Or is it just comparing different methods?
    • b.ii. What is Algorithm 2 - it is mentioned on line 991 but not used anywhere.
  2. Suitability:

    • i. Again, as stated in 1), it is not clear what approach is actually being recommended. As it stands, Section 5 just serves as a comparison of different approaches.
    • ii. What is the motivation behind comparing window-level to patient-level calibration? If I understood correctly, window-level calibration does not come with the marginal guarantee of conformal risk control (unless you perform weighted conformal risk prediction), while patient-level calibration does. Given that the goal of the paper is to guarantee risk control, I do not believe it makes sense to perform window-level calibration in the first place.
    • iii. In Section 5.3, you mention that the performance of E2E is task-dependent; however, there is no further insight or analysis on this. Are there particular dataset characteristics (e.g., the amount of noise in the dataset) that influence this dependency?
    • iv. From Section 5.3, it seems like there is no real, clear-cut advantage to using E2E empirically, and that the straightforward approaches of post-hoc conformal risk control and standard fine-tuning are best. This seems like a negative result.
    • v. Near the end of Section 5.3, you mention the effect that model selection can have on E2E, and also mention the importance of model selection in the introduction and abstract. However, there is not enough detail or information about what model selection is actually performed or what issue it is resolving. This issue repeats in Section 5.4.
Questions for authors

I do not have any questions that could materially affect my assessment. I believe the paper tackles an important problem; however, as it is, I do not believe the paper meets the bar for acceptance.

I recommend that the authors improve the presentation and writing of the paper (see my points above), provide more analysis and discussion of their results, and better position their paper. As it stands, it is not clear whether the authors are providing a comparison of different methods (in which case, there would need to be more baselines presented) or recommending E2E.

2 comments
AuthorsOpenReview ↗

Response to Reviewer QM8A

We thank the reviewer for the detailed and constructive feedback. Below, we address all identified weaknesses and questions. All new experimental results are included in the appendix of the revised paper, with all revisions highlighted in blue.

R2W1: We thank the reviewer for these comments. (a) Notation and experimental clarity. We have revised Section 3 to clarify the classifier, patient–window hierarchy, calibration units, and assumptions underlying finite-sample guarantees, and Section 5.1 to specify evaluation protocols. (b) Methodological clarity and missing algorithm. Section 4 now identifies independently calibrated post-hoc subject-aware CRC as our recommended deployment strategy, with E2E training as an optional efficiency extension. Algorithm 1 in Appendix C.1 summarizes training, checkpoint selection, and independent final calibration.

R2W2(1-a.i-iv): We thank the reviewer for identifying these ambiguities. Section 3 now defines the classifier (fθ:RC×T→ΔK−1f_\theta:\mathbb{R}^{C\times T}\to\Delta^{K-1}), its probability output (pθ(x)p_\theta(x)), and windows as fixed-length physiological segments sharing a patient-level diagnosis. We clarify that Eq. (1) takes expectation over calibration and test units, with the classifier fixed, and that subject-aware CRC requires patient-level exchangeability and independent final calibration. We also define (λmin⁡\lambda_{\min}) as the most conservative threshold (Eq. 2) and clarify that Eq. (19) represents pooled window-level empirical risk, whereas window-level CRC applies Eq. (2) to individual window losses without a general patient-level guarantee.

R2W2(1-b.i–ii): We have revised Section 4 to clarify our recommended deployment strategy: a fixed classifier followed by independent subject-aware CRC for patient-level risk control, with E2E training as an optional efficiency extension. Algorithm 1 in Appendix C.1 details the complete procedure, including differentiable calibration, risk-aware checkpoint selection, early stopping, and independent final calibration.

R2W2(2-i–ii): We have clarified in Sections 4 and 5.3 that our recommended deployment strategy is post-hoc subject-aware CRC with independent calibration. E2E training is an optional, task-dependent efficiency extension, while CVaR addresses tail-sensitive risk. Window-level CRC serves only as a diagnostic baseline, illustrating how correlated windows inflate nominal calibration size and target window-weighted rather than patient-level risk. Section 4.3 clarifies that patient-level guarantees require subject-aware calibration under the relevant exchangeability and independence assumptions.

R2W2(2-iii–iv): We have expanded Section 5.3 to investigate E2E's task-dependent efficiency. Controlled PTB experiments show that E2E is more robust than CE to training-label noise (FPR reductions of 0.151–0.296), while training-cohort effects are non-monotonic (Tables 16–17 in Appendix D.5). Matched comparisons across 12 architectures and five seeds (Table 18 in Appendix D.6) show that post-hoc CRC achieves the lowest median FPR on PTB (0.373), compared with CE (0.401), window-E2E (0.428), and subject-E2E (0.442). Window-E2E outperforms CE on 6/12 architectures on PTB and 3/12 on ADFTD. Further risk–efficiency analyses (Table 19 in Appendix D.6) confirm the absence of consistent E2E gains, supporting post-hoc subject-aware CRC as the preferred deployment strategy.

R2W2(2-v): We have clarified checkpoint selection in Sections 4.2 and 5.4 and Appendix D.7. Checkpoints minimize (s=E^+wmax⁡(0,R^−α−τ)s=\hat E+w\max(0,\hat R-\alpha-\tau)) over five validation resplits, with ((w,τ)=(100,0.01)(w,\tau)=(100,0.01)) for binary FNR and ((10,0.02)) for multiclass coverage, using early-stopping patience of 10. The ablation (Table 2 in this response; Table 20 in Appendix D.7) shows that risk-aware selection achieves the lowest set sizes on ADFTD for CE (2.111) and E2E (2.206), but offers no consistent advantage on PTB. These results highlight task-dependent optimization effects; formal guarantees require independent final calibration.

R2Q1: We thank the reviewer's feedback on the paper's clarity, positioning, and empirical contributions. We have clarified the paper's contribution as a systematic study of patient-level risk control in hierarchical medical time-series classification, rather than proposing E2E as universally superior. Section 4 and Algorithm 1 (Appendix C.1) recommend post-hoc subject-aware CRC with independent calibration, positioning window-level CRC as a diagnostic baseline and E2E as an optional efficiency extension. Analyses of label noise and cohort size (Appendix D.5), matched comparisons across 12 architectures and five seeds (Appendix D.6), and checkpoint selection (Appendix D.7) reveal task-dependent E2E benefits without consistent superiority over simpler alternatives. We have also revised the Introduction and Results discussion to clarify the methodological positioning, limitations, and practical implications.

AuthorsOpenReview ↗

Additional Experimental Results

Appendix D.5, Table 16 Effect of training-label noise on CE and E2E performance on PTB.

Noise pCE: FPRE2E: FPRE2E − CEFNR (CE / E2E)Subject acc. (CE / E2E)
00.482 ± 0.0610.512 ± 0.091+0.0300.079 / 0.0770.858 / 0.831
0.10.699 ± 0.0730.548 ± 0.099−0.1510.072 / 0.0740.803 / 0.825
0.20.799 ± 0.0880.503 ± 0.117−0.2960.072 / 0.0760.749 / 0.858
0.30.845 ± 0.0720.572 ± 0.034−0.2730.072 / 0.0740.656 / 0.787
Post-hoc (clean)0.601——0.0830.787

Appendix D.6, Table 18 Matched comparison of post-hoc CRC, CE, window-E2E, and subject-E2E.

MethodPTB (α = 0.1)ADFTD (α = 0.2)
Post-hoc subject CRC0.373 / 0.30 / 0.8312.285 / 0.13 / 0.535
CE + subject CRC0.401 / 0.29 / 0.8392.276 / 0.11 / 0.518
Window-E2E + subject CRC0.428 / 0.30 / 0.8602.415 / 0.11 / 0.416
Subject-E2E + subject CRC0.442 / 0.29 / 0.8482.395 / 0.12 / 0.418
Window-E2E better than CE6 of 123 of 12
Subject-E2E better than Window-E2E7 of 125 of 12

Appendix D.7, Table 20 Checkpoint-selection ablation for CE and E2E.

RuleMethodPTB (α=0.1\alpha=0.1)ADFTD (α=0.2\alpha=0.2)
Risk-aware (ours)CE0.502 ± 0.044 / 0.077 / 0.8362.111 ± 0.040 / 0.129 / 0.614
Risk-aware (ours)E2E0.520 ± 0.034 / 0.080 / 0.8422.206 ± 0.026 / 0.130 / 0.579
Efficiency onlyCE0.527 ± 0.055 / 0.078 / 0.8472.162 ± 0.027 / 0.131 / 0.526
Efficiency onlyE2E0.426 ± 0.012 / 0.076 / 0.8912.276 ± 0.044 / 0.138 / 0.526
Validation lossCE0.535 ± 0.048 / 0.079 / 0.8252.167 ± 0.016 / 0.130 / 0.526
Validation lossE2E0.508 ± 0.069 / 0.079 / 0.8422.261 ± 0.043 / 0.149 / 0.474
Last epochCE0.425 ± 0.052 / 0.081 / 0.8472.123 ± 0.004 / 0.135 / 0.526
Last epochE2E0.435 ± 0.020 / 0.078 / 0.8632.364 ± 0.030 / 0.153 / 0.316

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

We thank the Area Chair and all reviewers for their evaluations and constructive discussions. We have substantially revised the manuscript to clarify its methodological positioning, strengthen the empirical evidence, and acknowledge its theoretical and clinical limitations. All revisions are highlighted in blue, with new experimental results in the appendix.

Reviewer QM8A: Methodological clarity and empirical justification

The reviewer questioned the presentation, recommended procedure, and lack of consistent E2E advantages. We revised Sections 3–5 to clarify the patient–window hierarchy, calibration units, exchangeability assumptions, and evaluation protocols. We added Algorithm 1 (Appendix C.1) and explicitly recommend post-hoc subject-aware CRC with independent calibration, positioning E2E training as an optional efficiency extension and window-level CRC as a diagnostic baseline.

We conducted controlled label-noise and training-cohort experiments (Appendix D.5), matched comparisons across 12 architectures and five seeds (Appendix D.6), and checkpoint-selection analyses (Appendix D.7). These reveal task-dependent E2E benefits but no consistent superiority over post-hoc CRC. We emphasize the practical implications rather than claiming universal E2E improvements.

Reviewer nkUP: Broader validation and patient-level evaluation

The reviewer requested larger datasets, aggregated patient-level classification, and subject-aware E2E optimization. We addressed these suggestions through four additions:

PTB-XL evaluation across 12 architectures and 17,596 patients, demonstrating near-nominal subject-level coverage with modest prediction-set enlargement (Appendix D.8).

ADFTD photic-stimulation and resting-to-photic evaluations, supporting subject-aware calibration across recording conditions (Appendix D.9).

Patient-level classification through majority voting and probability aggregation, followed by aggregate-then-calibrate CRC (Appendix D.10).

Subject-grouped E2E CRT, evaluating the feasibility and limitations of patient-aware optimization (Appendix D.6).

In the follow-up discussion, the reviewer explicitly acknowledged the improvements, recognized the importance of patient-level challenges in medical time series, increased their score, and expressed support for acceptance.

Reviewer WbtC: Novelty, interpretation, and clinical implications

The reviewer remained concerned about methodological novelty, mixed empirical findings, and whether statistical risk–efficiency improvements imply clinical benefit.

We acknowledge that the paper builds on established CRC and CRT principles rather than introducing a new general-purpose conformal theorem. Our contribution is to investigate their behavior under hierarchical patient–window dependence, including the mismatch between window-weighted and patient-level risk, within-patient correlation, and validity–efficiency–feasibility trade-offs arising from limited independent calibration patients.

We added controlled dependence simulations (Appendix D.11), calibration-cohort analyses (Appendix D.12), expanded architecture and risk-level comparisons (Appendix D.5–D.7), clinical operating-point analyses (Appendix D.13), and comparisons of CVaR, stricter mean-risk, and direct tail-exceedance control (Appendix D.14). We clarified that mixed E2E results support post-hoc subject-aware CRC as the preferred deployment strategy, while FNR/FPR comparisons measure statistical efficiency rather than validated clinical utility.

The reviewer maintained their rejection recommendation, primarily because they consider the application and empirical characterization of existing conformal methods insufficiently novel. We respect this assessment but emphasize that our contribution extends beyond changing the training split to systematically characterizing patient-level calibration under dependence and limited sample sizes.

Overall Assessment

The revised paper provides a more comprehensive study of patient-level risk control in hierarchical medical time-series classification. New experiments extend validation to larger cohorts and different recording conditions, quantify within-patient dependence, and identify limitations of E2E and tail-risk objectives.

We do not claim new general-purpose conformal guarantees or demonstrated clinical utility. Our contribution is an empirically grounded understanding of how established conformal methods should be applied and evaluated when the deployment unit is a patient rather than a time-series window.

We hope the Area Chair will consider these revisions, the additional evidence, and the positive reviewer reassessment alongside the remaining novelty concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.