- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Isotonic Survival Regression (ISR), a post-hoc calibration framework for Cox and Deep Cox survival models under right censoring. The primary method, DR-ISR, constructs doubly robust survival-probability pseudo-outcomes using augmented inverse probability of censoring weighting (AIPCW), followed by two-dimensional isotonic regression to enforce monotonicity in both time and the predicted risk score. The authors establish theoretical results on double robustness, survival-function consistency, and threshold calibration under risk-score misspecification. Experiments on six synthetic settings and three TCGA-based real-data modalities (clinical text reports, RNA-seq, and histopathology images) compare ISR against existing post-hoc calibration methods.
However, the paper has two major weaknesses: (1) the theoretical consistency guarantees rely on a sufficiently informative fixed risk score and do not directly extend to the learned Deep Cox models used in practice, and (2) the empirical evidence does not consistently support the claimed improvements, with contradictory statistical-significance statements and incomplete comparisons against relevant post-hoc calibration methods.
Critical strengths
1. A principled post-hoc calibration method with meaningful theoretical contributions.
The combination of doubly robust AIPCW pseudo-outcomes and two-dimensional isotonic regression provides a principled approach to calibrating survival distributions under right censoring. In particular, the threshold-calibration result under risk-score misspecification (Theorem 3) is an interesting contribution, as it establishes a meaningful calibration property even when the risk-ordering assumption fails.
2. A practically relevant problem with encouraging empirical results.
Improving survival-probability calibration without retraining the underlying Deep Cox model is practically valuable, particularly for applications involving high-dimensional biomedical data. The experiments demonstrate encouraging improvements in PIT calibration, and the additional sensitivity analyses across calibration sample sizes, censoring rates, and crossing survival curves help characterize the method's behavior.
Critical weaknesses
1. The theoretical guarantees do not establish end-to-end consistency for learned Deep Cox risk scores.
The strongest theoretical result, Theorem 2, relies on Assumption A2, which requires the scalar risk score to correctly order the true conditional survival functions at every time point. As established in Appendix A (Lemma 1), A2 also implies that the risk score is sufficient for the conditional survival distribution, i.e., .
This assumption is substantially stronger than ordinary risk discrimination. In practice, the risk score is learned from finite data using a Deep Cox model, and may not preserve all survival-relevant information in the original covariates. The theoretical analysis treats the risk mapping as fixed and does not establish that the learned Deep Cox score satisfies A2 or converges to a sufficiently informative score. The authors acknowledge this limitation in Section 7, but do not provide a corresponding end-to-end theoretical analysis.
Although Theorem 3 establishes threshold calibration when A2 fails, this is weaker than the full conditional survival-function consistency guaranteed by Theorem 2. Double robustness addresses misspecification of the survival or censoring nuisance models, but cannot recover prognostic information already discarded by the scalar risk representation.
Consequently, the paper does not establish that its strongest individualized consistency guarantee applies to the actual learned-risk-score pipeline, substantially limiting the practical interpretation of its theoretical contribution.
2. Insufficient and partly inconsistent empirical evidence for the claimed calibration and predictive improvements.
The real-data results do not consistently support the paper's performance claims. In Section 6.2 (lines 467–469), the authors state that both ISR variants achieve statistically significant IBS improvements over the baselines on text reports and RNA-seq. However, Table 2 reports IBS p-values of 0.68 and 0.32 for RW-ISR and DR-ISR on text reports, and 1.00 for DR-ISR on RNA-seq. These results do not support the stated conclusion.
Furthermore, Appendix Table 14 shows that RW+-ISR achieves lower IBS than the primary DR-ISR method on all three real-data modalities (0.111 vs. 0.122 for text, 0.088 vs. 0.098 for RNA-seq, and 0.094 vs. 0.110 for H&E). While DR-ISR consistently achieves the best AUPIT, its advantage does not extend consistently to overall predictive accuracy. This important trade-off is insufficiently emphasized in the main discussion.
The benchmark also omits KSP [1], a directly relevant post-hoc survival calibration method. In addition, despite emphasizing preservation of discriminative performance, the paper does not report pre- and post-calibration C-index results. Since isotonic pooling can introduce ties, preserving the original risk ordering does not necessarily imply exactly preserving the C-index.
Finally, all three real-data modalities are evaluated on the heterogeneous TCGA cancer population, without cancer-type-stratified calibration analyses. This limits the evidence that improvements in aggregate calibration translate into reliable predictions within clinically meaningful patient subgroups.
Overall, the results support improvements in PIT calibration, but do not convincingly establish consistent gains in distributional accuracy or adequately substantiate the broader claims regarding predictive reliability and discrimination preservation.
References
[1] Park J, Kim D, Kim C, et al. KSP: Kolmogorov-Smirnov metric-based Post-Hoc Calibration for Survival Analysis[J]. Advances in Neural Information Processing Systems, 2026, 38: 24506-24568.
Questions for authors
Q1. Theoretical guarantees and learned risk-score sufficiency.
Assumption A2 implies that the scalar risk score is sufficient for the complete conditional survival distribution, whereas the practical algorithm uses a risk score learned from finite data.
Can the authors clarify what theoretical guarantees extend to the learned Deep Cox risk score, rather than a fixed score satisfying A2? In particular, what additional assumptions on score estimation, convergence, or ordering stability would be needed for an end-to-end consistency result?
When A2 fails, Theorem 3 provides threshold calibration rather than full conditional consistency. Can the authors further clarify how this distinction affects individualized survival predictions when relevant prognostic information is not captured by the learned score?
Q2. Statistical significance and predictive performance.
Section 6.2 states that both ISR variants significantly outperform the baselines on IBS for text reports and RNA-seq. However, Table 2 reports nonsignificant p-values for several of these comparisons, including DR-ISR on both datasets.
Can the authors reconcile this statement with the reported p-values and clarify which improvements are statistically significant?
Furthermore, Appendix Table 14 shows that RW+-ISR outperforms DR-ISR on IBS across all three real-data modalities, despite DR-ISR achieving better AUPIT. What explains this consistent trade-off, and under which conditions should DR-ISR be preferred over RW+-ISR?
Q3. Missing baselines and discrimination preservation.
The benchmark compares against CSD and CiPOT but omits KSP, a directly relevant post-hoc survival calibration method.
Can the authors include KSP under the same evaluation protocol and report AUPIT, IBS, and calibration performance?
Additionally, although ISR preserves the risk ordering up to ties, isotonic pooling may change the C-index. Can the authors report pre- and post-calibration C-index values for the proposed methods on all three real-data modalities?
Q4. Direct validation of the theoretical calibration guarantees.
Theorem 3 establishes threshold calibration under risk-score misspecification, but the empirical evaluation primarily focuses on AUPIT, which measures marginal PIT calibration. These calibration properties are not equivalent.
Can the authors directly evaluate threshold calibration using time-dependent reliability diagrams or quantitative calibration errors across predicted-risk groups? In particular, how does calibration vary across clinically relevant low- and high-risk subgroups, where the fixed-score theoretical assumptions may be most consequential?
Q5. Clinical endpoint definition and reproducibility.
The real-data experiments use a TCGA survival endpoint referred to as progression-free survival (PFS). However, the cited TCGA clinical resource distinguishes conventional PFS from progression-free interval (PFI).
Can the authors clarify the exact endpoint variable, time origin, event definition, and censoring rules used in the experiments? Given the heterogeneity of the pooled TCGA population, can the authors provide cancer-type-stratified calibration results or results within a clinically homogeneous cancer cohort?
Finally, can the authors clarify whether the calibration set is strictly independent of all Deep Cox training, validation, early-stopping, and model-selection procedures, and provide the complete implementation corresponding to the reported results?
9 comments
Rebuttal (2/3)
Question 3.
We thank the reviewer for pointing out KSP. We evaluate its performance on real data and report results below. While it does operate in the same setting as us, it does not empirically beat any of the other methods. We will add discussion around KSP to the related work section and extend the paper with these empirical comparisons.
Table 2 reproduced from the paper, with KSP as an added comparison
| Data Type | Method | AUPIT | IBS | 20% Quantile | 50% Quantile | 80% Quantile |
|---|---|---|---|---|---|---|
| Text | Deep Cox | 0.0970 (1.00) | 0.125 (1.00) | 371 (1.00) | 700 (1.00) | 596 (1.00) |
| CSD | 0.1256 (1.00) | 0.125 (1.00) | 370 (1.00) | 704 (1.00) | 598 (1.00) | |
| CiPOT | 0.0948 (1.00) | 0.125 (1.00) | 363 (1.00) | 696 (0.86) | 590 (0.01) | |
| KSP | 0.0915 (1.00) | 0.125 (1.00) | 365 (1.00) | 704 (1.00) | 592 (0.99) | |
| RW-ISR (ours) | 0.1394 (1.00) | 0.123 (0.68) | 357 (0.72) | 700 (1.00) | 621 (1.00) | |
| DR-ISR (ours) | 0.0819 (<0.001) | 0.122 (0.32) | 357 (0.28) | 692 (0.14) | 613 (1.00) | |
| RNA | Deep Cox | 0.0546 (1.00) | 0.102 (1.00) | 375 (0.99) | 740 (1.00) | 603 (1.00) |
| CSD | 0.0826 (1.00) | 0.102 (1.00) | 376 (1.00) | 735 (1.00) | 608 (1.00) | |
| CiPOT | 0.0544 (1.00) | 0.102 (1.00) | 375 (1.00) | 732 (1.00) | 601 (0.01) | |
| KSP | 0.0542 (1.00) | 0.102 (1.00) | 375 (0.98) | 739 (1.00) | 603 (0.99) | |
| RW-ISR (ours) | 0.0923 (1.00) | 0.092 (<0.001) | 375 (0.99) | 699 (1.00) | 619 (1.00) | |
| DR-ISR (ours) | 0.0517 (<0.001) | 0.098 (1.00) | 373 (0.02) | 682 (<0.001) | 625 (1.00) | |
| HE | Deep Cox | 0.1007 (1.00) | 0.113 (1.00) | 441 (1.00) | 781 (1.00) | 661 (1.00) |
| CSD | 0.0787 (1.00) | 0.110 (1.00) | 429 (1.00) | 775 (1.00) | 628 (0.01) | |
| CiPOT | 0.0800 (1.00) | 0.111 (1.00) | 396 (1.00) | 771 (1.00) | 653 (1.00) | |
| KSP | 0.0921 (1.00) | 0.112 (1.00) | 428 (1.00) | 784 (1.00) | 654 (1.00) | |
| RW-ISR (ours) | 0.0889 (1.00) | 0.096 (<0.001) | 371 (<0.001) | 729 (<0.001) | 651 (0.99) | |
| DR-ISR (ours) | 0.0568 (<0.001) | 0.110 (1.00) | 380 (0.99) | 761 (1.00) | 752 (1.00) |
As discussed in the paper, our method preserves C-index exactly if ties are broken by assigned risk. However, to more explicitly investigate the effect of ties caused by isotonic regression, we now compute the C-index using the predicted mean survival time derived from the calibrated curves. We empirically find that the resulting C-index decreases only very slightly (<0.5% of original). We will add this explicit investigation in the main paper.
Difference between C-indices before and after calibration (mean 2 s.e. over 100 seeds)
| Setting | Base | Calibrated | |Base - Calibrated| | Base - Calibrated |
|---|---|---|---|---|
| Setting1 | 0.61423 ± 0.00095 | 0.61419 ± 0.00095 | 0.00008 ± 0.00001 | 0.00004 ± 0.00002 |
| Setting2 | 0.81355 ± 0.00053 | 0.81353 ± 0.00054 | 0.00011 ± 0.00003 | 0.00002 ± 0.00004 |
| Setting3 | 0.74956 ± 0.00074 | 0.74954 ± 0.00074 | 0.00012 ± 0.00002 | 0.00002 ± 0.00003 |
| Setting4 | 0.78778 ± 0.00078 | 0.78779 ± 0.00078 | 0.00013 ± 0.00002 | -0.00001 ± 0.00003 |
| Setting5 | 0.55205 ± 0.00099 | 0.55197 ± 0.00098 | 0.00014 ± 0.00004 | 0.00008 ± 0.00004 |
| Setting6 | 0.55280 ± 0.00114 | 0.55273 ± 0.00114 | 0.00011 ± 0.00006 | 0.00007 ± 0.00006 |
| Text | 0.68166 ± 0.00244 | 0.67865 ± 0.00407 | 0.00322 ± 0.00301 | 0.00254 ± 0.00255 |
| RNA | 0.70327 ± 0.00199 | 0.70460 ± 0.00455 | 0.00286 ± 0.00273 | 0.00384 ± 0.00384 |
| H&E | 0.68789 ± 0.00240 | 0.68661 ± 0.00269 | 0.00149 ± 0.00128 | 0.00167 ± 0.00167 |
Question 4.
We thank the reviewer for this question. Threshold calibration can be assessed by expected calibration error (ECE) at different times since threshold calibration requires that among patients with survival probability prediction at time , a fraction will suffer the event after time . DR-ISR overwhelmingly performs the best on ECE across all times and datasets. For numerical results, please refer to our response to reviewer JMJP (Weakness 3).
To examine calibration across clinically meaningful subgroups, we evaluate on each cancer type (“cohort”) separately. We only include cohorts with at least 100 patients (~1% of the overall data) in our evaluation. We report the number of cancer cohorts for which each method has statistically significant gains on the AUPIT and IBS metrics compared against all others (“wins”). DR-ISR achieves wins in nearly a third of the instances overall, far more than any other baseline. Thus, DR-ISR achieves the best calibration conditional on clinically relevant subgroups too. For numerical results, please refer to our response to reviewer JMJP (Weakness 3).
Rebuttal (3/3)
Question 5.
Our real data experiments do in fact use the PFS endpoint provided in the supplemental data in [3]. However, as noted in [3] (a peer reviewed publication with ~4000 citations), PFS does not have a standard definition. In particular, [3] defines PFS as the PFI which it measures by including based on deaths with tumor but not due to other causes. Note the relevant excerpt from [3] discussing PFS below.
We would also like to point out that there is another often used endpoint, PFS. The definition of PFS is not standard and often relies on the clinical study at hand. In our definition of PFI, the events included deaths with tumor, but do not include deaths from other causes. We hold the opinion that for cancer studies, deaths with tumor are more relevant thus we choose to use our definition here which we term as PFI to tell its difference from commonly used term of PFS which, in some definitions, contains death from other causes. For convenience of users who prefer to use this definition of PFS, we included this endpoint in Table S1 Tab ExtraEndpoints.
For our experiments, we choose to use the term PFS for this endpoint since this is better known and hence may be accessible to readers who are not clinicians. We defer to the judgement of the clinical experts in [3] on further details of endpoint calculation and encourage the reviewer to refer to [3] for further details. For cancer-specific evaluations, see our response to Question 4.
Our calibration set is completely independent of all Deep Cox training, validation, early-stopping, and model-selection procedures. Hyperparameters were selected using grid search and are available in Appendix E.
[1] Kolmogorov-Smirnov metric-based Post-Hoc Calibration for Survival Analysis. Park et al. NeurIPS, 2026.
[2] Deep Extended Hazard Models for Survival Analysis. Zhong et al. NeurIPS 2021.
[3] An Integrated TCGA Pan-cancer Clinical Data Resource to Drive High-quality Survival Outcome Analytics. Liu et al. Cell, 2018.
Rebuttal (1/3)
We thank the reviewer for their detailed comments and positive remarks on the practical relevance and encouraging results. Below, we respond to the weaknesses and questions.
Question 1.
We have now updated the paper PDF with Appendix section H with complete proofs extending Theorems 2 and 3 to the settings where risks are learned rather than fixed; these are stated as Theorems 4 and 5, respectively.
At a high level, if the estimated risk function converges to some risk function r∗, Theorem 3 still holds under a mild smoothness assumption on r∗. If we additionally assume that r∗ orders patients correctly and the convergence to r∗ is uniform, then Theorem 2 also holds. Additionally, note that since isotonic regression only cares about the risk ordering, the aforementioned assumptions on r∗ are up to monotonic transformations; this is rigorously developed in the proofs. Thus, our extended proofs extend end-to-end consistency and calibration guarantees under appropriate assumptions on the limiting risk function. We make no theoretical claims on the convergence of Deep Cox models itself, which is beyond the scope of this paper and an active area of research (e.g., see [2] and subsequent follow-ups). Hence, our extended theorems are stated directly with assumptions on the convergence of the learned risk score to an appropriate limit. We will include this discussion more formally in the appendix after polishing the new proofs and improving their presentation.
Regarding consistency vs threshold calibration, we first note that our algorithm cannot recover information lost by the risk score because we do not want to change the discrimination of the baseline model, and hence operate directly on top of the predicted risks. Theorem 2 (or 4) should be seen as a minimal desirable property of any post-hoc calibration algorithm in the sense that if the learned risk function is correct, then the calibration algorithm should yield exactly correct predictions. Theorem 3 (or 5) is the more practical calibration guarantee ensuring threshold calibration even when the risk function is incorrect. Thus, even without risk-correctness, while predictions at an individual level may not be exactly correct, they will still be calibrated on average across other patients with similar survival predictions. Note that the Cox model offers no such guarantee when the PH assumption is violated.
Question 2.
The reported p-values in the paper are based on comparing each method with all others. Our statement, however, addresses the statistical significance of our methods when compared only against the baselines and not each other. Both RW and DR are better than the baselines, but not always significantly better than each other. When comparing against only the three baselines, DR-ISR improvements on IBS for both text and RNA datasets are statistically significant. We apologize for this ambiguity and will add this clarification to the paper.
Largest p-value for DR-ISR when comparing pairwise against other baselines
Overall, we find that DR-ISR performs best and enjoys the strongest theoretical guarantees. On synthetic data, it achieves the best calibration in the majority of settings. However, the reviewer correctly notes that results across our proposed ISR variants are more mixed on real data. These evaluations are noisier, since the true event times are unobserved and the metrics must rely on IPCW. Even so, DR-ISR achieves the best AUPIT, ECE, conditional calibration, and remains competitive on IBS (see JMJP rebuttal). Hence, we recommend DR-ISR as the default choice in practice. When the censoring mechanism is known, RW$^+$-ISR is likely the better method since it depends only on the quality of the censoring estimate while making more efficient use of data than RW-ISR.