acceptodds
Under review as a conference paper at ICLR 2027

Isotonic Survival Regression: Calibrated Survival Distributions from Deep Cox Models

Abstract

Time-to-event data is widespread across the life sciences and engineering, but it is typically encountered together with censoring, which complicates the application of standard machine learning methods. Deep Cox models have emerged as a popular method for analyzing time-to-event data because they gracefully handle censoring and can be used with unstructured data such as clinical text reports, genomic sequences, and pathology images. However, their predicted survival probabilities are often poorly calibrated, thus limiting their practical utility. In this paper, we propose a novel post hoc calibration method for Deep Cox models that uses isotonic regression to refine predicted survival probabilities without affecting discriminative power. We establish favorable theoretical guarantees, including a double-robustness property and asymptotic calibration. Experiments on synthetic and real-world clinical data demonstrate the empirical effectiveness of our method.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
survival analysis, calibration, Deep Cox models, isotonic regression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uiGVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Isotonic Survival Regression (ISR), a post-hoc calibration framework for Cox and Deep Cox survival models under right censoring. The primary method, DR-ISR, constructs doubly robust survival-probability pseudo-outcomes using augmented inverse probability of censoring weighting (AIPCW), followed by two-dimensional isotonic regression to enforce monotonicity in both time and the predicted risk score. The authors establish theoretical results on double robustness, survival-function consistency, and threshold calibration under risk-score misspecification. Experiments on six synthetic settings and three TCGA-based real-data modalities (clinical text reports, RNA-seq, and histopathology images) compare ISR against existing post-hoc calibration methods.

However, the paper has two major weaknesses: (1) the theoretical consistency guarantees rely on a sufficiently informative fixed risk score and do not directly extend to the learned Deep Cox models used in practice, and (2) the empirical evidence does not consistently support the claimed improvements, with contradictory statistical-significance statements and incomplete comparisons against relevant post-hoc calibration methods.

Critical strengths

1. A principled post-hoc calibration method with meaningful theoretical contributions.

The combination of doubly robust AIPCW pseudo-outcomes and two-dimensional isotonic regression provides a principled approach to calibrating survival distributions under right censoring. In particular, the threshold-calibration result under risk-score misspecification (Theorem 3) is an interesting contribution, as it establishes a meaningful calibration property even when the risk-ordering assumption fails.

2. A practically relevant problem with encouraging empirical results.

Improving survival-probability calibration without retraining the underlying Deep Cox model is practically valuable, particularly for applications involving high-dimensional biomedical data. The experiments demonstrate encouraging improvements in PIT calibration, and the additional sensitivity analyses across calibration sample sizes, censoring rates, and crossing survival curves help characterize the method's behavior.

Critical weaknesses

1. The theoretical guarantees do not establish end-to-end consistency for learned Deep Cox risk scores.

The strongest theoretical result, Theorem 2, relies on Assumption A2, which requires the scalar risk score to correctly order the true conditional survival functions at every time point. As established in Appendix A (Lemma 1), A2 also implies that the risk score is sufficient for the conditional survival distribution, i.e., P(T>t∣X)=P(T>t∣r(X))P(T>t\mid X)=P(T>t\mid r(X)).

This assumption is substantially stronger than ordinary risk discrimination. In practice, the risk score is learned from finite data using a Deep Cox model, and may not preserve all survival-relevant information in the original covariates. The theoretical analysis treats the risk mapping as fixed and does not establish that the learned Deep Cox score satisfies A2 or converges to a sufficiently informative score. The authors acknowledge this limitation in Section 7, but do not provide a corresponding end-to-end theoretical analysis.

Although Theorem 3 establishes threshold calibration when A2 fails, this is weaker than the full conditional survival-function consistency guaranteed by Theorem 2. Double robustness addresses misspecification of the survival or censoring nuisance models, but cannot recover prognostic information already discarded by the scalar risk representation.

Consequently, the paper does not establish that its strongest individualized consistency guarantee applies to the actual learned-risk-score pipeline, substantially limiting the practical interpretation of its theoretical contribution.

2. Insufficient and partly inconsistent empirical evidence for the claimed calibration and predictive improvements.

The real-data results do not consistently support the paper's performance claims. In Section 6.2 (lines 467–469), the authors state that both ISR variants achieve statistically significant IBS improvements over the baselines on text reports and RNA-seq. However, Table 2 reports IBS p-values of 0.68 and 0.32 for RW-ISR and DR-ISR on text reports, and 1.00 for DR-ISR on RNA-seq. These results do not support the stated conclusion.

Furthermore, Appendix Table 14 shows that RW+-ISR achieves lower IBS than the primary DR-ISR method on all three real-data modalities (0.111 vs. 0.122 for text, 0.088 vs. 0.098 for RNA-seq, and 0.094 vs. 0.110 for H&E). While DR-ISR consistently achieves the best AUPIT, its advantage does not extend consistently to overall predictive accuracy. This important trade-off is insufficiently emphasized in the main discussion.

The benchmark also omits KSP [1], a directly relevant post-hoc survival calibration method. In addition, despite emphasizing preservation of discriminative performance, the paper does not report pre- and post-calibration C-index results. Since isotonic pooling can introduce ties, preserving the original risk ordering does not necessarily imply exactly preserving the C-index.

Finally, all three real-data modalities are evaluated on the heterogeneous TCGA cancer population, without cancer-type-stratified calibration analyses. This limits the evidence that improvements in aggregate calibration translate into reliable predictions within clinically meaningful patient subgroups.

Overall, the results support improvements in PIT calibration, but do not convincingly establish consistent gains in distributional accuracy or adequately substantiate the broader claims regarding predictive reliability and discrimination preservation.


References

[1] Park J, Kim D, Kim C, et al. KSP: Kolmogorov-Smirnov metric-based Post-Hoc Calibration for Survival Analysis[J]. Advances in Neural Information Processing Systems, 2026, 38: 24506-24568.

Questions for authors

Q1. Theoretical guarantees and learned risk-score sufficiency.

Assumption A2 implies that the scalar risk score is sufficient for the complete conditional survival distribution, whereas the practical algorithm uses a risk score learned from finite data.

Can the authors clarify what theoretical guarantees extend to the learned Deep Cox risk score, rather than a fixed score satisfying A2? In particular, what additional assumptions on score estimation, convergence, or ordering stability would be needed for an end-to-end consistency result?

When A2 fails, Theorem 3 provides threshold calibration rather than full conditional consistency. Can the authors further clarify how this distinction affects individualized survival predictions when relevant prognostic information is not captured by the learned score?

Q2. Statistical significance and predictive performance.

Section 6.2 states that both ISR variants significantly outperform the baselines on IBS for text reports and RNA-seq. However, Table 2 reports nonsignificant p-values for several of these comparisons, including DR-ISR on both datasets.

Can the authors reconcile this statement with the reported p-values and clarify which improvements are statistically significant?

Furthermore, Appendix Table 14 shows that RW+-ISR outperforms DR-ISR on IBS across all three real-data modalities, despite DR-ISR achieving better AUPIT. What explains this consistent trade-off, and under which conditions should DR-ISR be preferred over RW+-ISR?

Q3. Missing baselines and discrimination preservation.

The benchmark compares against CSD and CiPOT but omits KSP, a directly relevant post-hoc survival calibration method.

Can the authors include KSP under the same evaluation protocol and report AUPIT, IBS, and calibration performance?

Additionally, although ISR preserves the risk ordering up to ties, isotonic pooling may change the C-index. Can the authors report pre- and post-calibration C-index values for the proposed methods on all three real-data modalities?

Q4. Direct validation of the theoretical calibration guarantees.

Theorem 3 establishes threshold calibration under risk-score misspecification, but the empirical evaluation primarily focuses on AUPIT, which measures marginal PIT calibration. These calibration properties are not equivalent.

Can the authors directly evaluate threshold calibration using time-dependent reliability diagrams or quantitative calibration errors across predicted-risk groups? In particular, how does calibration vary across clinically relevant low- and high-risk subgroups, where the fixed-score theoretical assumptions may be most consequential?

Q5. Clinical endpoint definition and reproducibility.

The real-data experiments use a TCGA survival endpoint referred to as progression-free survival (PFS). However, the cited TCGA clinical resource distinguishes conventional PFS from progression-free interval (PFI).

Can the authors clarify the exact endpoint variable, time origin, event definition, and censoring rules used in the experiments? Given the heterogeneity of the pooled TCGA population, can the authors provide cancer-type-stratified calibration results or results within a clinically homogeneous cancer cohort?

Finally, can the authors clarify whether the calibration set is strictly independent of all Deep Cox training, validation, early-stopping, and model-selection procedures, and provide the complete implementation corresponding to the reported results?

9 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for their detailed comments and positive remarks on the practical relevance and encouraging results. Below, we respond to the weaknesses and questions.

Question 1.

We have now updated the paper PDF with Appendix section H with complete proofs extending Theorems 2 and 3 to the settings where risks are learned rather than fixed; these are stated as Theorems 4 and 5, respectively.

At a high level, if the estimated risk function converges to some risk function r∗r^*, Theorem 3 still holds under a mild smoothness assumption on r∗r^*. If we additionally assume that r∗r^* orders patients correctly and the convergence to r∗r^* is uniform, then Theorem 2 also holds. Additionally, note that since isotonic regression only cares about the risk ordering, the aforementioned assumptions on r∗r^* are up to monotonic transformations; this is rigorously developed in the proofs. Thus, our extended proofs extend end-to-end consistency and calibration guarantees under appropriate assumptions on the limiting risk function. We make no theoretical claims on the convergence of Deep Cox models itself, which is beyond the scope of this paper and an active area of research (e.g., see [2] and subsequent follow-ups). Hence, our extended theorems are stated directly with assumptions on the convergence of the learned risk score to an appropriate limit. We will include this discussion more formally in the appendix after polishing the new proofs and improving their presentation.

Regarding consistency vs threshold calibration, we first note that our algorithm cannot recover information lost by the risk score because we do not want to change the discrimination of the baseline model, and hence operate directly on top of the predicted risks. Theorem 2 (or 4) should be seen as a minimal desirable property of any post-hoc calibration algorithm in the sense that if the learned risk function is correct, then the calibration algorithm should yield exactly correct predictions. Theorem 3 (or 5) is the more practical calibration guarantee ensuring threshold calibration even when the risk function is incorrect. Thus, even without risk-correctness, while predictions at an individual level may not be exactly correct, they will still be calibrated on average across other patients with similar survival predictions. Note that the Cox model offers no such guarantee when the PH assumption is violated.

Question 2.

The reported p-values in the paper are based on comparing each method with all others. Our statement, however, addresses the statistical significance of our methods when compared only against the baselines and not each other. Both RW and DR are better than the baselines, but not always significantly better than each other. When comparing against only the three baselines, DR-ISR improvements on IBS for both text and RNA datasets are statistically significant. We apologize for this ambiguity and will add this clarification to the paper.

Largest p-value for DR-ISR when comparing pairwise against other baselines

DatasetAUPITIBS
Text<0.001<0.001
RNA<0.001<0.001
H&E<0.0010.42

Overall, we find that DR-ISR performs best and enjoys the strongest theoretical guarantees. On synthetic data, it achieves the best calibration in the majority of settings. However, the reviewer correctly notes that results across our proposed ISR variants are more mixed on real data. These evaluations are noisier, since the true event times are unobserved and the metrics must rely on IPCW. Even so, DR-ISR achieves the best AUPIT, ECE, conditional calibration, and remains competitive on IBS (see JMJP rebuttal). Hence, we recommend DR-ISR as the default choice in practice. When the censoring mechanism is known, RW$^+$-ISR is likely the better method since it depends only on the quality of the censoring estimate while making more efficient use of data than RW-ISR.

 

AuthorsOpenReview ↗

Rebuttal (2/3)

Question 3.

We thank the reviewer for pointing out KSP. We evaluate its performance on real data and report results below. While it does operate in the same setting as us, it does not empirically beat any of the other methods. We will add discussion around KSP to the related work section and extend the paper with these empirical comparisons.

Table 2 reproduced from the paper, with KSP as an added comparison

Data TypeMethodAUPITIBS20% Quantile50% Quantile80% Quantile
TextDeep Cox0.0970 (1.00)0.125 (1.00)371 (1.00)700 (1.00)596 (1.00)
CSD0.1256 (1.00)0.125 (1.00)370 (1.00)704 (1.00)598 (1.00)
CiPOT0.0948 (1.00)0.125 (1.00)363 (1.00)696 (0.86)590 (0.01)
KSP0.0915 (1.00)0.125 (1.00)365 (1.00)704 (1.00)592 (0.99)
RW-ISR (ours)0.1394 (1.00)0.123 (0.68)357 (0.72)700 (1.00)621 (1.00)
DR-ISR (ours)0.0819 (<0.001)0.122 (0.32)357 (0.28)692 (0.14)613 (1.00)
RNADeep Cox0.0546 (1.00)0.102 (1.00)375 (0.99)740 (1.00)603 (1.00)
CSD0.0826 (1.00)0.102 (1.00)376 (1.00)735 (1.00)608 (1.00)
CiPOT0.0544 (1.00)0.102 (1.00)375 (1.00)732 (1.00)601 (0.01)
KSP0.0542 (1.00)0.102 (1.00)375 (0.98)739 (1.00)603 (0.99)
RW-ISR (ours)0.0923 (1.00)0.092 (<0.001)375 (0.99)699 (1.00)619 (1.00)
DR-ISR (ours)0.0517 (<0.001)0.098 (1.00)373 (0.02)682 (<0.001)625 (1.00)
HEDeep Cox0.1007 (1.00)0.113 (1.00)441 (1.00)781 (1.00)661 (1.00)
CSD0.0787 (1.00)0.110 (1.00)429 (1.00)775 (1.00)628 (0.01)
CiPOT0.0800 (1.00)0.111 (1.00)396 (1.00)771 (1.00)653 (1.00)
KSP0.0921 (1.00)0.112 (1.00)428 (1.00)784 (1.00)654 (1.00)
RW-ISR (ours)0.0889 (1.00)0.096 (<0.001)371 (<0.001)729 (<0.001)651 (0.99)
DR-ISR (ours)0.0568 (<0.001)0.110 (1.00)380 (0.99)761 (1.00)752 (1.00)

As discussed in the paper, our method preserves C-index exactly if ties are broken by assigned risk. However, to more explicitly investigate the effect of ties caused by isotonic regression, we now compute the C-index using the predicted mean survival time derived from the calibrated curves. We empirically find that the resulting C-index decreases only very slightly (<0.5% of original). We will add this explicit investigation in the main paper.

Difference between C-indices before and after calibration (mean ±\pm 2 s.e. over 100 seeds)

SettingBaseCalibrated|Base - Calibrated|Base - Calibrated
Setting10.61423 ± 0.000950.61419 ± 0.000950.00008 ± 0.000010.00004 ± 0.00002
Setting20.81355 ± 0.000530.81353 ± 0.000540.00011 ± 0.000030.00002 ± 0.00004
Setting30.74956 ± 0.000740.74954 ± 0.000740.00012 ± 0.000020.00002 ± 0.00003
Setting40.78778 ± 0.000780.78779 ± 0.000780.00013 ± 0.00002-0.00001 ± 0.00003
Setting50.55205 ± 0.000990.55197 ± 0.000980.00014 ± 0.000040.00008 ± 0.00004
Setting60.55280 ± 0.001140.55273 ± 0.001140.00011 ± 0.000060.00007 ± 0.00006
Text0.68166 ± 0.002440.67865 ± 0.004070.00322 ± 0.003010.00254 ± 0.00255
RNA0.70327 ± 0.001990.70460 ± 0.004550.00286 ± 0.002730.00384 ± 0.00384
H&E0.68789 ± 0.002400.68661 ± 0.002690.00149 ± 0.001280.00167 ± 0.00167

 

Question 4.

We thank the reviewer for this question. Threshold calibration can be assessed by expected calibration error (ECE) at different times since threshold calibration requires that among patients with survival probability prediction pp at time tt, a fraction pp will suffer the event after time tt. DR-ISR overwhelmingly performs the best on ECE across all times and datasets. For numerical results, please refer to our response to reviewer JMJP (Weakness 3).

To examine calibration across clinically meaningful subgroups, we evaluate on each cancer type (“cohort”) separately. We only include cohorts with at least 100 patients (~1% of the overall data) in our evaluation. We report the number of cancer cohorts for which each method has statistically significant gains on the AUPIT and IBS metrics compared against all others (“wins”). DR-ISR achieves wins in nearly a third of the instances overall, far more than any other baseline. Thus, DR-ISR achieves the best calibration conditional on clinically relevant subgroups too. For numerical results, please refer to our response to reviewer JMJP (Weakness 3).

AuthorsOpenReview ↗

Rebuttal (3/3)

Question 5.

Our real data experiments do in fact use the PFS endpoint provided in the supplemental data in [3]. However, as noted in [3] (a peer reviewed publication with ~4000 citations), PFS does not have a standard definition. In particular, [3] defines PFS as the PFI which it measures by including based on deaths with tumor but not due to other causes. Note the relevant excerpt from [3] discussing PFS below.

We would also like to point out that there is another often used endpoint, PFS. The definition of PFS is not standard and often relies on the clinical study at hand. In our definition of PFI, the events included deaths with tumor, but do not include deaths from other causes. We hold the opinion that for cancer studies, deaths with tumor are more relevant thus we choose to use our definition here which we term as PFI to tell its difference from commonly used term of PFS which, in some definitions, contains death from other causes. For convenience of users who prefer to use this definition of PFS, we included this endpoint in Table S1 Tab ExtraEndpoints.

For our experiments, we choose to use the term PFS for this endpoint since this is better known and hence may be accessible to readers who are not clinicians. We defer to the judgement of the clinical experts in [3] on further details of endpoint calculation and encourage the reviewer to refer to [3] for further details. For cancer-specific evaluations, see our response to Question 4.

Our calibration set is completely independent of all Deep Cox training, validation, early-stopping, and model-selection procedures. Hyperparameters were selected using grid search and are available in Appendix E.

 

[1] Kolmogorov-Smirnov metric-based Post-Hoc Calibration for Survival Analysis. Park et al. NeurIPS, 2026.

[2] Deep Extended Hazard Models for Survival Analysis. Zhong et al. NeurIPS 2021.

[3] An Integrated TCGA Pan-cancer Clinical Data Resource to Drive High-quality Survival Outcome Analytics. Liu et al. Cell, 2018.

Reviewer_JMJPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes post hoc calibration of Cox and Deep Cox predictions using isotonic regression. The proposed method (DR-ISR) builds doubly robust survival pseudo-outcomes and fits survival curves that are non-increasing in time and the original risk score. Consistency and calibration results are provided, with simulations and applications to three TCGA data.

Critical strengths
  1. Improving survival probabilities is useful.
  2. The double-robustness results are a strength.
  3. The distinction between consistency and calibration under risk-score misspecification is helpful.
  4. The experiments show improvements.
Critical weaknesses
  1. The method requires a conditional censoring model, which can be difficult to estimate when censoring comes from several sources. Double robustness helps with model misspecification, but does not ensure stable finite-sample predictions. The implementation bounds estimated survival and censoring probabilities at 0.02 for real-data calibration, while 10^-4 is used in the simulations. Please examine sensitivity.
  2. Cox models are not limited to discrimination. Cox plus Breslow already estimates survival probabilities, as the manuscript acknowledges. The question is whether the proposed method improves these estimates enough to justify the additional censoring model and calibration step. Please compare with simpler updates. Comparisons with discrete-time, piecewise-exponential, and semiparametric models using the same features and development data would help. The current comparisons mainly address performance against other post hoc calibration methods, rather than the broader choice of survival modeling.
  3. Consistency requires the score to contain sufficient information about survival and to order patients correctly at every time. The method cannot recover information lost by the score or represent crossing survival curves. Without this assumption, the theorem establishes threshold calibration, not recovery of each patient’s conditional survival distribution. Please report subgroup and time-specific calibration.
  4. The empirical results and tradeoff seem less clearer, e.g. different ISR achieve the best performance under various scenarios; recommendation about which methods to be used in practice?
  5. The derivation defines cumulative hazard through -log S, but the algorithm uses 1-S(t_j)/S(t_{j-1}) for increments. These are not identical. Please provide the corresponding discrete-time formulation, explain how tied event and censoring times are handled, clarify which guarantees apply to the implemented estimator.
Questions for authors

please see above

7 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for their thoughtful questions and for highlighting the strengths of our theoretical results, including double-robustness, consistency, and calibration. We comment on the weaknesses below.

Weakness 1.

We vary the clipping probability for the synthetic and real data and report the mean ±\pm 2 s.e. over 100 random seeds below. In general, we find that our algorithm is insensitive to the clipping choice on the synthetic datasets. Performance on real-world datasets is more sensitive, with larger clipping probabilities giving better calibration. We believe this is because the models trained on real data are less accurate, so clipping is more important. In practice, we recommend 0.010.01 as a reasonably good first choice. We will add this discussion to the appendix.

AUPIT and IBS of DR-ISR for different clipping thresholds

Setting10−510^{-5}10−410^{-4}10−310^{-3}10−210^{-2}
AUPIT
10.016 ± 0.0010.016 ± 0.0010.016 ± 0.0010.015 ± 0.001
20.018 ± 0.0010.018 ± 0.0010.018 ± 0.0010.017 ± 0.001
30.024 ± 0.0020.024 ± 0.0020.024 ± 0.0020.023 ± 0.002
40.042 ± 0.0030.042 ± 0.0030.042 ± 0.0030.037 ± 0.003
50.017 ± 0.0030.017 ± 0.0030.016 ± 0.0020.014 ± 0.001
60.019 ± 0.0030.019 ± 0.0030.017 ± 0.0010.015 ± 0.001
IBS
10.135 ± 0.0020.135 ± 0.0020.135 ± 0.0020.134 ± 0.002
20.075 ± 0.0010.075 ± 0.0010.075 ± 0.0010.074 ± 0.001
30.100 ± 0.0010.100 ± 0.0010.099 ± 0.0010.099 ± 0.001
40.109 ± 0.0010.109 ± 0.0010.109 ± 0.0010.108 ± 0.001
50.093 ± 0.0050.093 ± 0.0050.091 ± 0.0040.089 ± 0.004
60.090 ± 0.0050.090 ± 0.0050.088 ± 0.0040.086 ± 0.003
Dataset10−410^{-4}3×10−43\times10^{-4}10−310^{-3}3×10−33\times10^{-3}10−210^{-2}2×10−22\times10^{-2}5×10−25\times10^{-2}
AUPIT
Text0.170940.155530.114670.088560.082620.081940.08101
RNA0.053940.053800.052930.052610.052190.051670.05060
HE0.217940.205750.172150.109310.065850.056810.05323
IBS
Text0.282390.245010.183980.142310.126040.122270.12067
RNA0.105580.105390.103600.100910.098170.097590.09817
HE0.330200.307310.258730.171820.120330.109950.10432

 

Weakness 2.

Cox models with the Breslow estimator do produce survival probabilities, but as shown by our base models and prior work, these predictions are often poorly calibrated. This is precisely the motivation for our work.

We emphasize that ISR is a post hoc calibration method, not an alternative survival model. As with post hoc calibration more generally (not only in survival analysis), the appropriate comparisons are with other post hoc calibration methods applied to the same base model. This is exactly what we report, using CSD and CiPOT as baselines. A broader comparison across survival model families is valuable, but addresses a different question from the primary objective of our paper.

To assess whether the calibration step is justified, we additionally compare against training the Cox model on the combined training and calibration data, as well as applying a simple linear transformation to the risk score based on the calibration data. Both alternatives yield only marginal improvements in calibration.

SettingCoxCox (combined)Cox (linear cal)DR-ISR
AUPIT
10.0165 ± 0.00140.0146 ± 0.00110.0166 ± 0.00120.0157 ± 0.0010
20.0588 ± 0.00180.0579 ± 0.00140.0585 ± 0.00190.0178 ± 0.0012
30.0696 ± 0.00270.0677 ± 0.00190.0683 ± 0.00270.0237 ± 0.0019
40.1154 ± 0.00490.1113 ± 0.00340.1174 ± 0.00410.0421 ± 0.0035
50.0108 ± 0.00110.0076 ± 0.00090.0100 ± 0.00100.0173 ± 0.0026
60.0105 ± 0.00090.0082 ± 0.00080.0104 ± 0.00090.0189 ± 0.0030
IBS
10.1326 ± 0.00200.1323 ± 0.00200.1325 ± 0.00200.1346 ± 0.0020
20.1142 ± 0.00170.1139 ± 0.00120.1138 ± 0.00150.0746 ± 0.0013
30.1330 ± 0.00190.1321 ± 0.00150.1328 ± 0.00170.0995 ± 0.0015
40.1499 ± 0.00250.1492 ± 0.00230.1503 ± 0.00250.1094 ± 0.0012
50.0875 ± 0.00360.0870 ± 0.00360.0872 ± 0.00360.0927 ± 0.0049
60.0850 ± 0.00330.0847 ± 0.00330.0847 ± 0.00330.0898 ± 0.0053
AuthorsOpenReview ↗

Rebuttal (2/3)

Weakness 2 (contd.)

Remark on broader survival modeling. Selecting baseline survival models to deploy in practice requires accounting for multiple objectives, such as discrimination, calibration, ease of training, scalability, generalization, data efficiency, etc. Rather than systematically analyzing these factors ourselves, we focus on Cox-style models since these are incredibly popular with practitioners (see references in introduction and experimental section).

 

Weakness 3.

We agree with the reviewer and additionally note that the reason our algorithm cannot recover information lost by the risk score is because we do not want to change the discrimination of the baseline model, and hence operate directly on top of the predicted risks. Theorem 2 should be seen as a minimal desirable property of any post-hoc calibration algorithm in the sense that if the learned risk function is correct, then the calibration algorithm should yield exactly correct predictions. Theorem 3 is the more practical calibration guarantee ensuring threshold calibration even when the risk function is incorrect. Note that the Cox model offers no such guarantee when the PH assumption is violated.

Recall that threshold calibration requires that among patients with survival probability prediction pp at time tt, a fraction pp will suffer the event after time tt. This can be measured by the expected calibration error (ECE) and we report the IPCW estimate of ECE values at 10 times below. To ensure complete evaluation across time, we choose 10 evenly spaced quantiles of the empirically observed times. DR-ISR overwhelmingly performs the best on ECE across all times and datasets.

IPCW estimate of ECE across 10 times. We report mean across 100 seeds and p-value compared against all other methods.

DatasetMethodTime 1Time 2Time 3Time 4Time 5Time 6Time 7Time 8Time 9Time 10
TextDeep Cox0.015 (1.000)0.028 (1.000)0.038 (1.000)0.045 (1.000)0.048 (1.000)0.053 (1.000)0.061 (1.000)0.068 (1.000)0.081 (1.000)0.100 (1.000)
CSD0.078 (1.000)0.092 (1.000)0.091 (1.000)0.084 (1.000)0.075 (1.000)0.067 (1.000)0.067 (1.000)0.066 (1.000)0.076 (1.000)0.093 (1.000)
CiPOT0.018 (1.000)0.029 (1.000)0.036 (1.000)0.041 (1.000)0.043 (1.000)0.048 (1.000)0.056 (1.000)0.062 (1.000)0.076 (1.000)0.098 (1.000)
DR-ISR0.012 (<0.001)0.020 (<0.001)0.027 (<0.001)0.031 (<0.001)0.034 (<0.001)0.036 (<0.001)0.041 (<0.001)0.049 (<0.001)0.056 (<0.001)0.069 (<0.001)
RNADeep Cox0.013 (1.000)0.021 (0.997)0.027 (0.981)0.030 (0.971)0.032 (0.986)0.035 (0.993)0.038 (0.961)0.043 (0.997)0.051 (1.000)0.063 (0.999)
CSD0.038 (1.000)0.068 (1.000)0.076 (1.000)0.072 (1.000)0.066 (1.000)0.057 (1.000)0.055 (1.000)0.056 (1.000)0.059 (1.000)0.065 (1.000)
CiPOT0.014 (1.000)0.022 (1.000)0.027 (0.996)0.030 (0.984)0.032 (0.993)0.035 (0.955)0.038 (0.937)0.044 (0.998)0.052 (1.000)0.066 (1.000)
DR-ISR0.012 (<0.001)0.019 (0.003)0.025 (0.019)0.028 (0.029)0.030 (0.014)0.033 (0.045)0.036 (0.063)0.040 (0.003)0.046 (<0.001)0.057 (0.001)
HEDeep Cox0.029 (1.000)0.055 (1.000)0.078 (1.000)0.091 (1.000)0.098 (1.000)0.107 (1.000)0.114 (1.000)0.123 (1.000)0.126 (1.000)0.120 (1.000)
CSD0.123 (1.000)0.104 (1.000)0.084 (1.000)0.075 (1.000)0.069 (1.000)0.070 (1.000)0.074 (1.000)0.082 (1.000)0.087 (1.000)0.096 (0.936)
CiPOT0.032 (1.000)0.046 (1.000)0.057 (1.000)0.065 (1.000)0.070 (1.000)0.079 (1.000)0.086 (1.000)0.096 (1.000)0.103 (1.000)0.106 (1.000)
DR-ISR0.016 (<0.001)0.026 (<0.001)0.033 (<0.001)0.036 (<0.001)0.039 (<0.001)0.046 (<0.001)0.050 (<0.001)0.062 (<0.001)0.074 (<0.001)0.091 (0.064)
AuthorsOpenReview ↗

Rebuttal (3/3)

Weakness 3 (contd.)

To examine calibration across clinically meaningful subgroups, we evaluate on each cancer type (“cohort”) separately. We only include cohorts with at least 100 patients (~1% of the overall data) in our evaluation. We report the number of cancer cohorts for which each method has statistically significant gains on the AUPIT and IBS metrics compared against all others (“wins”). DR-ISR achieves wins in nearly a third of the instances overall, far more than any other baseline. Thus, DR-ISR achieves the best calibration conditional on clinically relevant subgroups too.

Frequency of statistically significant gains over all competing approaches across cancer cohorts on text (N=26), RNA (N=26), and H&E (N=19) datasets.

DatasetMethodAUPIT WinsIBS Wins
TextDeep Cox0 / 260 / 26
CSD3 / 264 / 26
CiPOT3 / 261 / 26
DR8 / 2611 / 26
RNADeep Cox2 / 262 / 26
CSD5 / 261 / 26
CiPOT4 / 263 / 26
DR2 / 267 / 26
HEDeep Cox0 / 192 / 19
CSD4 / 195 / 19
CiPOT2 / 192 / 19
DR8 / 195 / 19

 

Weakness 4.

Overall, we find that DR-ISR performs best and enjoys the strongest theoretical guarantees. On synthetic data, it achieves the best calibration in the majority of settings. However, the reviewer correctly notes that results across our proposed ISR variants are more mixed on real data. These evaluations are noisier, since the true event times are unobserved and the metrics must rely on IPCW. Even so, DR-ISR achieves the best AUPIT, ECE, conditional calibration, and remains competitive on IBS. Hence, we recommend DR-ISR as the default choice in practice. When the censoring mechanism is known, RW$^+$-ISR is likely the better method since it depends only on the quality of the censoring estimate while making more efficient use of data than RW-ISR.

 

Weakness 5.

The pseudo-outcome in Equation 2 is written in continuous time, but computing it in practice requires discretizing time and numerically integrating over a grid. Hence, we use the discrete-time hazard. This is simply a choice of how the integral is computed, not a fundamentally different estimator. Since the time grid must be fine enough to compute the integral accurately, the difference between the two approximations is negligible.

To confirm that this choice has no practical effect, we compute DR-ISR with discrete and continuous increments on the synthetic settings. For each, we report the absolute difference in calibration metrics, along with the maximum difference between the two calibrated survival functions, max⁡t,x∣ΔS^(t∣x)∣\max_{t,x} |\Delta \widehat S(t \mid x)| over 5 seeds. As expected, there is little to no difference in the results.

Settingmax⁡t,xΔS^ISRDR(t∣x)\max_{t,x}\Delta \widehat{S}^{\mathrm{DR}}_{\mathrm{ISR}}(t \mid x)AUPITIBSMedian QL
Setting 10.00034 ± 0.000060.00005 ± 0.000030.00001 ± 0.000010.00001 ± 0.00001
Setting 20.00277 ± 0.001280.00010 ± 0.000070.00001 ± 0.000010.00091 ± 0.00178
Setting 30.00213 ± 0.000940.00010 ± 0.000070.00002 ± 0.000010.00001 ± 0.00001
Setting 40.00487 ± 0.001960.00023 ± 0.000180.00002 ± 0.000020.00008 ± 0.00005
Setting 50.00151 ± 0.002160.00006 ± 0.000040.00001 ± 0.000010.00002 ± 0.00002
Setting 60.00062 ± 0.000280.00009 ± 0.000060.00003 ± 0.000050.00022 ± 0.00042

Our guarantees therefore apply to the implemented estimator, up to approximation from numerical integration. Tied times require no special handling, since the time grid consists of the unique calibration times.

Reviewer_oVEgOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors presented a theoretically grounded calibration method for survival analysis (DR-ISR) with empirical and theoretical contributions. The clear articulation of threshold calibration guarantees, preservation of discrimination (C-index), and systematic comparison with recent baselines (CSD, CiPOT) make this a compelling contribution.

Critical strengths
  • The authors made a significant effort to present the paper with sufficient mathematical rigor. For example Theorem 3 is a clean and valuable guarantee,
  • Overall the results are promising and shows a clear grounding with respect to calibration. The comparison suite is strong and accurately highlights the contribution of the paper
  • The paper is well presented, well motivated, and clearly articulated
Critical weaknesses

There are a few aspects where the paper can be improved upon

  • While the theorems are well presented, please ensure all theorem assumptions are explicitly stated, proofs are self-contained (or clearly referenced to the appendix), and any regularity conditions are verified. For example The exact role of Assumption A2 should be highlighted. On the isotonic projection step, the proofs should explicility address measurability, uniform convergenece rates, and how finiti-sample deviations affect the calibration guarantee. It would be useful to add a remark on the finite-sample behavior of the framework.
  • The experimental results highlight calibration accuracy. Calibration ensures well-calibrated probabilities, but does not guarantee optimal discrimination or overall predictive accuracy. It would be important to explicility report accuracy aligned metrics in addition to the present calibration aligned ones
  • The method seems to be computationally complex. Please consider adding a concise complexity analysis covering key aspects such as time complexity per training/calibration step. memory footprint (especially for large cohorts or high-dimensional covariates), and scalability notes
Questions for authors

The current framework targets standard right-censored survival. The isotonic projection and DR structure are conceptually modular, which suggests potential for broader application. Would this be applicable to other settings such as competing risks, left-truncation, and recurrent events?

2 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their encouraging feedback and for recognizing our strong and systematic comparison suite! We briefly address the weaknesses and questions below.

 

Weakness 1.

We will take these suggestions into account to further strengthen the presentation of our theoretical results, and in particular, clarify the role of Assumption A2 in Section 4. Regarding finite-sample behavior, while non-asymptotic results exist for doubly robust estimators, extending such guarantees for calibration is less straightforward, because isotonic regression introduces finite-sample bias. We will add a remark discussing these limitations in the paper.

 

Weakness 2.

Because our method targets calibration of the survival distribution, our primary evaluation uses calibration-oriented metrics (AUPIT, IBS, quantile loss). Nonetheless, we agree that direct summaries of estimation error are a useful complement. In the table below we report the RMSE of the estimated mean event time for each method across all six synthetic settings.

RMSE of the mean event time on the six synthetic settings (mean ± 2 s.e. over 20 seeds)

MethodSetting 1Setting 2Setting 3Setting 4Setting 5Setting 6
Cox228.10 ± 54.7712.06 ± 0.634.38 ± 0.1810.72 ± 0.449.87 ± 0.3916.69 ± 2.92
CSD228.29 ± 54.7611.85 ± 0.614.25 ± 0.1610.76 ± 0.329.91 ± 0.3916.76 ± 2.92
CiPOT228.26 ± 54.7712.05 ± 0.624.36 ± 0.1710.73 ± 0.439.91 ± 0.3816.76 ± 2.92
RW-ISR229.10 ± 54.749.22 ± 0.193.70 ± 0.0910.40 ± 0.2010.05 ± 0.3816.92 ± 2.90
DR-ISR228.18 ± 54.769.00 ± 0.123.62 ± 0.069.61 ± 0.1810.00 ± 0.3816.90 ± 2.94

The ranking of methods by RMSE closely tracks their ranking by IBS. On Settings 2-4, DR-ISR substantially improves estimation error and is only marginally worse than the baselines on the remaining settings. In every setting, either DR-ISR or the uncalibrated Cox model is the best method.

 

Weakness 3.

We will add a short discussion of computational complexity to the Appendix. In summary, constructing the DR pseudo-outcomes costs O(n∣T∣)O(n |\mathcal{T}|), Dykstra’s algorithm costs O(TDyk⋅n∣T∣)O(T_{\mathrm{Dyk}} \cdot n |\mathcal{T}|), and memory usage is O(n∣T∣)O(n |\mathcal{T}|). Empirically, we find that the number of iterations necessary for Dykstra’s algorithm to converge (TDykT_{\mathrm{Dyk}}) is typically 10.

Since our calibration algorithm operates directly on the risk scores, the runtime remains the same even with high-dimensional covariates. In the table below, we report runtimes and peak memory usage at various calibration sample sizes. Note that the runtimes are mostly smaller than those currently reported in the paper since we have now improved the code implementation.

Calibration runtime in seconds (mean ± 2 s.e. over 5 seeds)

ncaln_{\mathrm{cal}}CSDCiPOTRW-ISRDR-ISR
1000.023 ± 0.0000.018 ± 0.0000.004 ± 0.0000.105 ± 0.001
2500.051 ± 0.0010.039 ± 0.0000.005 ± 0.0000.289 ± 0.002
5000.107 ± 0.0010.075 ± 0.0000.007 ± 0.0000.602 ± 0.003
10000.220 ± 0.0010.146 ± 0.0010.014 ± 0.0001.259 ± 0.018
25000.562 ± 0.0010.369 ± 0.0020.040 ± 0.0013.492 ± 0.035
50001.167 ± 0.0050.731 ± 0.0030.115 ± 0.0028.514 ± 0.348
100002.371 ± 0.0161.473 ± 0.0080.727 ± 0.00621.788 ± 0.380

Calibration peak memory during calibration in GB (mean ± 2 s.e. over 5 seeds)

ncaln_{\mathrm{cal}}CSDCiPOTRW-ISRDR-ISR
1000.012 ± 0.0000.003 ± 0.0000.008 ± 0.0010.141 ± 0.000
2500.035 ± 0.0000.011 ± 0.0000.020 ± 0.0010.360 ± 0.000
5000.072 ± 0.0000.025 ± 0.0000.039 ± 0.0010.736 ± 0.000
10000.146 ± 0.0000.053 ± 0.0000.084 ± 0.0021.533 ± 0.000
25000.370 ± 0.0000.137 ± 0.0000.238 ± 0.0024.270 ± 0.000
50000.745 ± 0.0000.278 ± 0.0000.562 ± 0.0049.986 ± 0.001
100001.491 ± 0.0010.559 ± 0.0001.446 ± 0.00025.584 ± 0.302

At our time grid resolution, the memory usage of DR-ISR becomes substantial for large calibration sets. Since memory and runtime are linear in ∣T∣|\mathcal{T}|, they can be reduced by using a coarser time grid or more efficient memory operations.

AuthorsOpenReview ↗

Rebuttal (2/2)

Question.

Our general framework consists of (i) constructing pointwise pseudo-outcomes, followed by (ii) a smoothing step. We believe this approach can be extended to other settings where such pseudo-outcomes are available, including competing risks and left-truncation or even recurrent events, as suggested by the reviewer. When the data have a natural ordering (e.g., risk score in our setting), isotonic regression can be used for the smoothing step to potentially yield favorable calibration properties. Identifying an appropriate ordering for isotonic regression in these different settings is perhaps less natural and would require deeper investigation, though related ideas have been explored for CATE estimation [1]. Developing algorithms and establishing theoretical guarantees for these settings are certainly interesting directions for future work.

 

[1] Causal Isotonic Calibration for Heterogeneous Treatment Effects. van der Laan et al. ICML, 2023.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewer Concerns and Rebuttals

We thank the reviewers for their helpful feedback and comments. We summarize below the main concerns raised and our responses.

Reviewer uiGV

  1. Lack of end-to-end theoretical guarantees.

    Our theorems were stated for a fixed risk score function rather than one learned using data. We have now added a stand-alone Appendix H to our submitted PDF which now extends our theorems to provide the same guarantees under analogous assumptions on the learned risk function.

  2. Soundness of the experimental results.

    We have now thoroughly investigated each point raised by the reviewer through the following experiments.

    • We clarified the difference between the reported p-values (which compare against all methods) versus p-values that compare only against baselines to establish that DR-ISR achieves statistically significant gains over the baselines.
    • We compared against the additional KSP baseline.
    • We verified that there is little to no change in the C-index post-calibration.
    • We investigated performance on different cancer types separately.
    • We investigated calibration performance using the expected calibration error (ECE) metric across various times.

Overall, we find that DR-ISR continues to perform the best on each experimental setting and metric. We have also clarified minor points raised regarding discrete-vs-continuous time implementations, experimental procedures and the definition of the endpoint (PFS) used for our real-data experiments. We respectfully disagree with reviewer uiGV that our added results do not satisfactorily address the limitations of practically learned risk scores. Our new theoretical guarantees extend to learned risk scores under assumptions that previous works ([1] and subsequent follow-ups) have thoroughly investigated for Deep Cox models. No other competing method offers such a calibration under misspecification guarantee. Our extended experimental results also show the practical effectiveness of our methods. Since the author-reviewer discussion period has drawn to an end, we are unable to seek further clarification but emphasize that our rebuttal meticulously addressed all their initial concerns to the best of our knowledge.

[1] Deep Extended Hazard Models for Survival Analysis. Zhong et al. NeurIPS 2021.

 

Reviewer JMJP

  1. Sensitivity analysis around the clipping threshold.

    Like many importance weighting procedures, our algorithm is sensitive to the clipping threshold on real data. We will acknowledge this in the paper.

  2. Different survival modeling choices.

    We emphasize that we compare against other post-hoc calibration methods since these are direct alternatives and are hence appropriate comparisons. We focus on Cox models since these are overwhelmingly the most popular among practitioners. Nonetheless, we conducted additional experiments against discrete time methods and piece-wise exponential methods (as explicitly requested by the reviewer). Different methods perform the best on calibration metrics on different settings, but any margins over DR-ISR are small. Further, DR-ISR also retains the C-index of the Cox model, which is the best in all settings.

  3. Extended experiments.

    As requested by the reviewer, we investigated performance on clinically relevant subgroups and time-specific calibration (as measured by ECE). DR-ISR achieves the best calibration conditional on clinically relevant subgroups and is significantly better on ECE across nearly all times and datasets.

  4. Best ISR method.

    We clarified that DR-ISR enjoys the strongest theoretical guarantees and overall empirical performance, but RW$+$-ISR may be more suitable if the censoring mechanism is known.

  5. Continuous versus discrete time implementation.

    We clarified the two are equivalent theoretically and also experimentally verified that any differences are negligible.

  6. Censoring model choice.

    The reviewer raised an additional point regarding sensitivity to the estimated censoring model, which we have thoroughly investigated.

To our knowledge, all the reviewer’s concerns have been thoroughly addressed.

 

Reviewer oVEg suggested improvements in the presentation of proofs which we will incorporate. We also provided performance on accuracy based metrics (RMSE) to show that performance qualitatively mimics that on the IBS metric. We investigated the computational and memory complexities of our method with suggestions on improving scalability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.