- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the application of conformal risk control to medical time-series classification. The authors consider binary false-negative-rate control and multiclass coverage, compare post-hoc calibration with end-to-end conformal risk training, and extend the framework to subject-level risk and conditional value-at-risk.
Critical strengths
The authors highlight important challenges in real-world medical time-series classification. These challenges are relevant to the practical application of conformal risk control in medical settings.
Critical weaknesses
The contribution is limited, as the proposed framework largely consists of straightforward applications and adaptations of existing conformal risk control and conformal risk training methods to medical time-series classification. The theoretical formulations primarily follow from established results, without introducing substantial new methodological or theoretical advances.
Although the paper presents extensive experimental results, the analysis does not provide sufficiently clear conclusions or insights into when and why the proposed approaches are beneficial. Consequently, the broader implications of the empirical findings remain unclear.
Questions for authors
-
A substantial portion of the methodology appears to be a direct application of existing Conformal Risk Control (CRC) and Conformal Risk Training (CRT) methods. In particular, several derivations in Section 4 follow directly from established results or straightforward algebraic manipulations. Could the authors clarify the specific contributions that distinguish this work from these existing approaches?
-
Alignment between the stated challenges and contributions. The challenges introduced in the Introduction are meaningful, but their connections to the proposed contributions are not sufficiently clear. The authors should clarify how each proposed component directly addresses the challenges identified in the Introduction.
-
Medical time-series windows from the same patient are often highly correlated, which may violate the exchangeability assumption when individual windows are treated as separate calibration units. Although the authors acknowledge this issue and propose subject-aware CRC to address it, the experimental analysis does not sufficiently investigate how the degree of within-patient correlation affects calibration validity and predictive efficiency. I believe more dedicated controlled experiments varying the strength of within-patient correlation would help isolate and quantify its impact.
-
Insufficient discussion of experimental findings. Much of the Results section focuses on describing numerical outcomes rather than providing deeper explanations of the observed patterns. For example, simply concluding that "the effect of E2E is task dependent" or counting the number of architectures achieving smaller prediction sets provides limited insight without explaining why these differences arise. A more detailed analysis of the factors driving these results, including dataset characteristics, model architectures, and calibration sample sizes, would significantly strengthen the empirical contribution.
-
According to Table 4, all evaluated classifiers already achieve FNR below the target level of before calibration. Post-hoc CRC reduces FPR at the cost of FNR to increase toward the prescribed threshold. However, the clinical significance of this trade-off is not sufficiently discussed. In medical diagnosis, false negatives may have substantially more severe consequences than false positives, although their relative importance depends on the clinical context. The authors should provide stronger justification for the selected risk thresholds and discuss whether the resulting improvements in FPR are clinically meaningful given the corresponding increases in FNR.
-
The CVaR experiments primarily demonstrate an anticipated trade-off: stronger protection against high-risk patients leads to more conservative predictions and reduced predictive efficiency. In some settings, this results in nearly trivial prediction rules. While these observations are consistent with the underlying CVaR formulation, it is unclear what additional insights they provide beyond established theoretical expectations. The authors should clarify the practical benefits of subject-level CVaR control and investigate whether meaningful improvements in tail-risk protection can be achieved?
6 comments
Additional Experimental Results
Appendix D.6, Table 18 Matched comparison of post-hoc CRC, CE, window-E2E, and subject-E2E.
| Method | PTB (α = 0.1) | ADFTD (α = 0.2) |
|---|---|---|
| Post-hoc subject CRC | 0.373 / 0.30 / 0.831 | 2.285 / 0.13 / 0.535 |
| CE + subject CRC | 0.401 / 0.29 / 0.839 | 2.276 / 0.11 / 0.518 |
| Window-E2E + subject CRC | 0.428 / 0.30 / 0.860 | 2.415 / 0.11 / 0.416 |
| Subject-E2E + subject CRC | 0.442 / 0.29 / 0.848 | 2.395 / 0.12 / 0.418 |
| Window-E2E better than CE | 6 of 12 | 3 of 12 |
| Subject-E2E better than Window-E2E | 7 of 12 | 5 of 12 |
Appendix D.13, Table 27 Clinical operating points under subject-level CRC on PTB.
| Decision rule | Missed diagnoses / 100 diseased | Unnecessary referrals / 100 healthy |
|---|---|---|
| Default (P(disease) ≥ 0.5), no guarantee | 3.7 ± 1.1 | 65.2 ± 5.4 |
| Subject-level CRC, α = 0.02 | 0.0 | 100.0 (refers everyone) |
| Subject-level CRC, α = 0.05 | 1.7 ± 2.0 | 75.4 ± 7.8 |
| Subject-level CRC, α = 0.10 | 7.1 ± 4.5 | 53.9 ± 11.4 |
| Subject-level CRC, α = 0.15 | 12.0 ± 5.9 | 39.8 ± 11.3 |
| Subject-level CRC, α = 0.20 | 16.9 ± 6.8 | 28.7 ± 9.9 |
R2Q1
Our contributions distinguish window-weighted from patient-level risk, formulate subject-aware CRC and CVaR, and assess E2E efficiency and stability. I believe this is technically direct application of conformal risk control and conformal risk training [1] to different granularity based on the hierarchy of patient data. Section 4.1 - 4.4 are largely just simple re-writing of definitions or existing methodology from [1,2]. Although I understand the application to important domain has contribution, that contribution is also not clear, which means entire study's contribution is very limited. Could authors clarify what is novel in section 4.1 - 4.4 compared to [1]?
[1] Conformal risk control.
[2] Conformal risk training: End-to-end optimization of conformal risk control
R2Q4
While my point was insuffcient discussion about the "experimental findings", the additional experiments that authors conducted was more towards sensitivity analysis in controlled setting. I am convinced the need for the additional experiments. Additionally, all the additional experiment results look very mixed, less consistent trend, and simply presented numbers without summarized and organized interpretation, which makes it very difficult to understand and interpret.
D.5: label noise and number of training samples. I believe those are less related to the authors research hypothesis. Additionally, the result trends are not consistent, not sufficiently showing the proposed methods benefit.
D.6: how can we interpret "Window-E2E better than CE" and "Subject-E2E better than Window-E2E" in Table 18 and "Window-E2E better" in Table 19?
D.7: I believe checkpoint selection is not a ablation. Rather it is necessary details to be included.
R2Q5
I believe the author and I pointing out the same thing. As author stated, clinical benefit and thresholds require disease-specific justification. Therefore, I believe the authors statement calibration mainly converts excess conservativeness in the raw decision rule into lower FPR while moving FNR toward the prescribed boundary does not necessarily desirable and benefitable in all cases without detailed justification.
Response to Reviewer WbtC
We thank the reviewer for the detailed and constructive feedback. We clarified our contributions and expanded the analyses. All revisions are highlighted in blue in the paper.
R2W1: Building on established CRC and CRT methods, we examine their limitations in hierarchical medical time series. Within-patient dependence raises window-level CRC's risk-exceedance probability to 0.261, versus 0.054 for subject-level CRC (Appendix D.11, Table 25). Larger calibration cohorts reduce subject-level conservativeness to approximately 2% additional set size at n=3,500, while window-level CRC remains miscalibrated (Appendix D.12, Table 26).
R2Q1: Our contributions distinguish window-weighted from patient-level risk, formulate subject-aware CRC and CVaR, and assess E2E efficiency and stability. On PTB-XL (12 architectures), subject-level CRC achieves near-nominal coverage with only 2–3% larger prediction sets, whereas window-level CRC systematically undercovers (Appendix D.8, Table 21). On ADFTD, subject-level CRC reduces risk-exceedance frequency under photic stimulation and resting-to-photic transfer (COMET: 0.56 to 0.07; Appendix D.9, Table 22). These experiments extend validation to a larger clinical cohort and a different recording condition without retraining.
R2Q2: The revised Introduction links classification-specific CRC and E2E to risk formulation and efficiency (Challenge I), subject-aware CRC to patient dependence (II), and calibration cohort size and E2E stability to limited independent data (III). CVaR additionally addresses tail-sensitive risk.
R2Q3: We vary within-patient correlation while fixing marginal distributions and sample sizes (2,000 trials; Table 1; Appendix D.11, Table 25). As correlation increases from 0 to 0.9, effective sample size falls from 750 to 26. At N=30, window-level risk exceedance rises to 0.261, versus 0.054 for subject-level CRC; at N=100, the corresponding values are 0.105 and 0.040. Window-level CRC has lower FPR, illustrating the efficiency–reliability trade-off. Importantly, the effective sample size is a design-effect approximation, not the number of independent calibration patients.
Table 1. Effect of within-patient dependence on CRC calibration (risk exceedance / FPR).
R2Q4: We added analyses of label noise (Appendix D.5, Table 16), calibration size (D.12, Table 26), architecture and risk level (D.6, Tables 18–19), and checkpoint selection (D.7, Table 20). On PTB, the E2E–CE FPR difference changes from +0.030 without noise to −0.296 at 20% label noise. Window-E2E outperforms CE on 6/12 PTB architectures but only 3/12 ADFTD architectures. Risk-aware selection improves ADFTD E2E set size over last-epoch selection (2.206 vs. 2.364), but worsens PTB FPR relative to efficiency-only selection (0.520 vs. 0.426). These findings show task-dependent effects without establishing a single causal explanation. Label quality and calibration size are identifiable contributing factors, while architecture and optimization effects remain dataset-dependent.
R2Q5: Lower FPR does not necessarily imply clinical benefit; α=0.05 is illustrative. On PTB, subject-level CRC reduces missed diagnoses from 3.7 to 1.7 per 100 diseased patients but adds 10.2 referrals per 100 healthy patients (Appendix D.13, Table 27). Window-level CRC can violate patient-level FNR targets, while subject-level CRC becomes trivial at α≤0.05 with only 19 calibration patients (Appendix D.13, Table 28). Insufficient disease-positive calibration units can therefore prevent nontrivial certified decisions. Clinical thresholds require disease-specific justification.
R2Q6: CVaR closely matches mean-risk control at budget βα for sparse losses; on PTB, calibration is trivial in 79% of resplits at α=0.1,β=0.5 (Appendix D.14, Table 29). Direct tail-exceedance control, P(Li>τ)≤α, achieves a high-loss patient fraction of 0.067 and FPR of 0.395 on PTB, versus FPRs of 0.448 for mean-risk CRC and 0.959 for CVaR CRC (Table 30). Exceedance control targets prevalence rather than severity of high losses. Formal guarantees remain marginal and require subject-level exchangeability and independent calibration. Neither CVaR nor exceedance control guarantees that every individual patient satisfies the prescribed risk level.