acceptodds
Under review as a conference paper at ICLR 2027

The loss or the architecture? What matters in survival analysis with competing risks

Abstract

Often, eg in healthcare and predictive maintenance, what matters is which of several competing causes will strike first, and with what probability: survival analysis with competing risks. Corresponding deep learning methods are developed as bundles: a new architecture together with a new training loss, tested on one to five datasets against a handful of baselines. Thus, a state-of-the-art result cannot be attributed to the architecture, the loss, or the evaluation protocol. The coupling is not accidental: with one incidence curve per cause, a network’s head is tied to the losses expressible on it. We vary the two axes independently, three losses, two of them strictly proper, on four backbones, across two modalities that tease the axes apart: 20 tabular and 3 pathology-image datasets. One loss wins in both: modelling cumulative incidence directly under censoring weighting. It brings twice as much as benefits a good choice of backbone on competing-risks data, and no more than it on single-event data: the loss is decisive for competing risks. The backbone matters too, and must match the modality: tabular networks lead on tables, pretrained vision encoders on images. The two choices compound rather than substitute. Proper scoring rules reveal these dynamics; the concordance index (C-index), which is not proper, does not separate models: neither axis moves it, not even tuning on the loss built to improve it. This is an avenue of progress: composed, the two give a model nobody has tried, RealMLP or TabM trained with the cumulative-incidence loss, which takes the best Brier score in our benchmark.

Primary area:
datasets and benchmarks
Keywords:
Competing-risks, Survival Analysis, Tabular Data, Benchmark, Scoring rules

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HH9EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In fields like healthcare and predictive maintenance, survival analysis with competing risks predicts which of several mutually exclusive events will happen first and when, despite right-censored data. Deep competing-risks methods usually introduce a new architecture and a new loss together, so it’s unclear which one drives their gains. This paper separates the two by training three losses on four backbones across 20 tabular and 3 pathology-image datasets. The CRNLL (Competing Risks Negative Log-Likelihood) loss, which models the cumulative incidence function (CIF) directly, wins on every backbone, and on competing-risks data it matters about twice as much as the backbone choice; the backbone still has to match the modality. The C-index barely separates any configuration, so the authors recommend strictly proper scoring rules like the Brier score, where RealMLP or TabM trained with CRNLL performs best.

Critical strengths
  1. Disentangled experimental design Each loss is paired with each backbone under an identical search space and tuning budget, which isolates the contribution of the loss from that of the architecture, something no prior competing-risks study has done.

  2. Extensive empirical coverage The evaluation spans 10 competing-risks and 10 single-event tabular datasets along with 3 imaging datasets, roughly ten times the scale of earlier method papers, and assesses accuracy, discrimination and calibration.

Critical weaknesses
  1. Study design cannot answer “loss or architecture” in general The architecture axis is narrow by construction. Three of four tabular backbones are MLP variants, and the imaging encoders are frozen, so the image comparison reduces to pretrained features versus an MLP on raw inputs. The loss-to-backbone ratio thus reflects this backbone selection rather than a general finding. Without synthetic data of known cumulative incidence, the claim of better probability estimates rests only on IPCW-based metrics that share the winning loss’s weighting.

  2. Central claim lacks statistical support The central “twice as much” claim (Table D.1) compares a median loss spread of 0.053 with a median backbone spread of 0.027, but their interquartile ranges ([0.020, 0.084] vs [0.021, 0.062]) almost fully overlap, and no confidence interval or test is given. The single-event reversal (0.072 vs 0.086) includes a backbone spread of 3.136 on NWTCO (Table D.2), which implies a failed run with IBS about three times the Aalen-Johansen reference. The claim that the C-index is flat is drawn from overlapping boxes (Figs. D.3, D.9), while IBS gets a paired sign test; Table D.3 itself shows the C-index varies across losses about 60% as much as IBS. The imaging conclusions in Fig. 2b rest on three datasets, with three points per box.

Questions for authors

Q1. The loss axis changes several things at once: CRNLL adds IPCW weighting and a per-horizon softmax output, while the hazard-based losses use a joint softmax over causes and time. Model selection is also done on IBS, which shares CRNLL’s weighting. Since only the extra survival head is ablated (Appendix D.6), how can the observed gains be attributed to the loss itself rather than to these co-varying choices, and would the ordering hold if hyperparameters were selected with a criterion not aligned with CRNLL?

Q2. Does CRNLL still come out ahead on synthetic competing-risks data with known cumulative incidence functions, or under an evaluation metric that does not use IPCW weights?

Q3. Would the loss-to-backbone ratio hold with architecturally distinct backbones, such as tree ensembles, fine-tuned encoders or pathology foundation models, instead of three MLP variants and frozen ImageNet encoders?

Q4. Can bootstrap confidence intervals be reported for the effect sizes in Table D.1, with failed runs such as the 3.136 NWTCO spread either excluded or explicitly accounted for?

Q5. If the C-index is evaluated with the same paired within-dataset sign tests used for IBS, does it still show no separation between losses or backbones?

Q6. Given that MIMIC and eICU patients are censored at their last hospital discharge, how is the independent-censoring assumption behind IBS justified on these datasets, and do the conclusions change if they are excluded?

8 comments

We thank the reviewer for a thorough and demanding review, and for highlighting the two features the paper is built on: a disentangled design, in which each loss is paired with each backbone under an identical search space and budget, and a scale about ten times that of earlier method papers. The questions tested exactly this design, and answering them made the evidence substantially stronger, including where it narrows our original claims. We address each point below:

W1, Q3: scope of the backbone axis. We agree, and the answer to Q3 is no: we do not expect the ratio to hold in general.

  • The ratio is a property of the backbones compared (three MLP-family networks and a transformer on tables, pretrained encoders on images). We removed it from the abstract and conclusion.
  • Our remaining, narrower claim: within each backbone tried, the loss ordering is largely the same, including on two new vision encoders, Phikon-v2 (a pathology foundation model) and DINOv2.

W1, Q2: evidence that does not rely on IPCW.

  1. Synthetic data with known CIFs. 5 configurations × 2 training sizes × 5 seeds, with a cause-specific Weibull generator: independent censoring (50 % or 85 %) or covariate-dependent censoring (50 %), and 1 to 3 causes.

We measure the integrated squared distance between predicted and true CIFs, which uses no censoring weights. These are the results:

  • Full follow-up. CRNLL is closer to the truth than HazardLikelihood in 28/40 (configuration, size, backbone) cells.
  • Covariate-dependent censoring. The censoring time is drawn at random from a Weibull model whose hazard depends on the covariates, so that low-risk subjects are lost to follow-up sooner. Every subject keeps a positive probability of remaining uncensored. The marginal censoring weights are then misspecified, both in CRNLL's training loss and in the IBS. Ranked by distance to the truth, HLRanking comes first (mean rank 1.6 of 3), then CRNLL (2.1) and HazardLikelihood (2.3); the differences are not significant (Friedman p = 0.42).
  • First half of follow-up. The hazard-based losses are closer to the truth: HazardLikelihood beats CRNLL in 36/40 cells, and putting CRNLL on a monotone PMF head does not change this (20/24).
  • IBS vs oracle ranking. Under 50 % independent censoring, the IBS ranks models much like the oracle (τ = 0.78–0.96), but less well with covariate-dependent censoring (τ = 0.03) or 85 % censoring (τ = 0.16).
  1. Right-censored log-likelihood. This score is proper without a censoring model and is HazardLikelihood's own training objective. We re-scored the stored predictions of all 320 cells and Aalen–Johansen on 10 time intervals (per dataset, median over backbones). The score needs non-decreasing incidence curves, which CRNLL's per-horizon head does not enforce, so we sort each predicted curve over time (monotone rearrangement); this leaves the other models' curves unchanged. CRNLL ties both hazard losses: better than HazardLikelihood on 9/20 and then HLRanking on 10/20 (Holm p = 1). CRBS comes last (loss asked by reviewer BcaT, changing the NLL to the Brier Loss).

We have rewritten the claim accordingly: losses that score the CIF directly with censoring weights (CRNLL and the new IPCW Brier loss) give the most accurate probabilities consistently across backbones. For short horizons and the event-time density, likelihood-based losses are as good.

W2, Q4: statistics. Dataset-bootstrap 95 % CIs on the corrected runs:

  • Competing risks (10 datasets): ratio 1.28 [0.42, 3.54], paired Wilcoxon p = 0.62; with equal-size axes, 2.46 [0.51, 4.19], p = 0.11.
  • Single event (10): ratio 1.30 [0.79, 1.59].
  • Pooled (20): ratio 1.48 [0.83, 2.12]; without ICU cohorts (17), 1.57 [1.05, 2.10].
  • With CRBS as a fourth loss (4 × 4, so equal-size axes):
    • competing risks: loss spread 0.039 [0.010, 0.081], backbone spread 0.022 [0.011, 0.035], ratio 1.77 [0.40, 4.31], paired Wilcoxon p = 0.23;
  • single event: ratio 1.94 [1.28, 2.32];
  • pooled: ratio 1.77 [1.26, 2.53] (paired Wilcoxon p = 0.03), or 1.90 [1.40, 2.61] without ICU cohorts.

The reviewer is right that "twice" is not established; we withdraw it.

We fixed the failed fit, and re-run the affected cells, now the HazardLikelihood-RealMLP's IBS on NWTCO goes from 0.490 to 0.098 (Aalen–Johansen: 0.112).

The loss ordering, by contrast, is statistically solid (per dataset, median over backbones):

  • among the three original losses, CRNLL is first on 20/20 datasets, and on 17/17 without ICU cohorts;
  • among all four, it is first on 15/20 and CRBS on the other 5 (Friedman p = 5 × 10⁻¹⁰);
  • Holm-corrected Wilcoxon p = 10⁻⁵ against HazardLikelihood and against HLRanking (20/20 each).

For imaging, we now show per-cohort points and make no statistical claim from three cohorts.

Q1: co-varying choices, and selection on the IBS. Because of time, we did the experiments on 17 datasets, the other ones are still running. We added one ablation per choice the reviewer lists, each changing one factor at a time on all four backbones, median over backbones.

  • Output head. CRNLL on HazardLikelihood's own joint-PMF head (CIF by cumulative sum) beats HazardLikelihood on that identical head on 16/17 datasets (median IBS-gain difference +0.038, Holm p = 2 × 10⁻⁴), and is only 0.005 behind CRNLL with its own head. The effect comes from the loss, not the per-horizon softmax.
  • Scoring rule. The IPCW Brier version (CRBS) is a close second (see our reply to BcaT): although trained on the evaluation metric itself, it does not beat CRNLL on it.
  • Selection criterion. With hyper-parameters selected on the C-index, which does not share CRNLL's weighting, CRNLL still has the best IBS on 16/17 non-ICU datasets (Appendix E).

Q5: C-index with the same paired tests. We ran them on the three original losses.

  • The C-index barely separates the losses (Friedman p = 0.06). The only pair close to significance favours HLRanking, ahead of CRNLL on 14/17 datasets, by a median of about 0.02; Appendix E.

We withdraw "the C-index does not separate models". Revised statement: the C-index differs little between losses, and where it does, it favours the ranking loss built for it, while the proper scores separate the losses more strongly and in the opposite direction.

Q6: MIMIC and eICU censoring. The reviewer is right: censoring at the last hospital contact depends on hospital use and is informative, so the independent-censoring assumption behind the IBS fails for MIMIC-III/IV. In the revision:

  1. eICU now has a genuine competing-risks outcome (in-hospital death vs discharge alive), with only administrative censoring at 30 days, so the KM weights are valid.
  2. MIMIC-III/IV cannot be rebuilt this way. We state the limitation and report every result with and without the ICU cohorts. Without them, CRNLL is first of the three original losses on 17/17 datasets and above HazardLikelihood on 17/17 (p = 2 × 10⁻⁵), with the effect sizes given above.
  3. We re-scored all 20 datasets with covariate-dependent censoring weights (stratified KM and a Cox censoring model). The ordering is unchanged: CRNLL is first on 16/20 and 17/20 datasets respectively, above both hazard-based losses on all 20, with median Kendall τ of 0.98 and 0.93 against the marginal-KM ranking.

Auditing these cohorts, we also removed a MIMIC-IV covariate that encoded in-hospital death and enforced the 24 h time origin. All models were re-run on the corrected cohorts:

  • CRNLL keeps the lead over the hazard-based losses there (0.1–0.2 points of IBS gain, averaged over backbones);
  • it is the best loss in 8 of the 12 (cohort, backbone) pairs;

With these cohorts included, CRNLL is first of the three original losses on 20/20 datasets.

Overall, the review led us to add what the submission lacked: synthetic data with known CIFs, a score without censoring weights, confidence intervals and paired tests, and one ablation per co-varying choice. On the IBS, the ordering of the losses holds throughout: CRNLL is the best of the three original losses on all 20 datasets. The new analyses also mark its limits, which the paper now states: the advantage is concentrated at medium-to-long horizons, it shrinks under covariate-dependent censoring, and on the right-censored log-likelihood CRNLL ties the hazard-based losses. We hope the clearer presentation and the new analyses address the reviewer's concerns, and would be grateful if the reviewer would consider strengthening their support.

I thank the authors for their detailed response and the honest withdrawal on multiple points, e.g.

  1. That the losses matters twice (W1)
  2. C-index does not separate models (Q5)

Having read the revised manuscript, I feel it is now a significantly new paper. Many of the cohort results are new, and the comparison doesn't stand. Some of the withdrawn claims are still in the abstract and conclusion, e.g., saying the loss matters more than backbone, although the ratio is (1.77, 95% CI [0.40, 4.31]). Moreover, the central claim has now been withdrawn, it undermines the study desing and faithfulness of the resutls reported in the original manuscript.

This paper will need a full-review. Hence, I keep my rating the same.

I suggest the authors to fix the study design issues, qualify the generality of the claims and consider submitting the revised work in future venue.

Reviewer_BcaTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper deals with survival analysis with competing risks, i.e. where distinct events can happen. The papers starts with the observation that in many survival analysis papers, multiple factors that could individually explain performance gains are conflated, making it hard to say whether the evaluation protocol, architecture or loss function (e.g.) are responsible for the gains. They propose to disentangle those effects, focusing on loss / architecture, across many datasets: 3 losses (competing risk NLL, DeepHit loss, CRNLL. Key conclusions are that: (1) only the Brier score and LL consistently separate models, (2) CRNLL leads for competing risks compared to other loss functions.

Critical strengths
  • The paper provides a robust ablation study / controlled study of different loss functions and backbones. Every loss is trained on every backbone with the same search space and budget. They validate that the CRNLL advantage does not come from its extra survival head. T

  • The results point in a clear direction: CRNLL appears to consistently outperform on the basis of the analysis.

Critical weaknesses
  • The losses considered are somewhat limited, which limits the conclusions somewhat. For losses, the IPCW Brier loss on the cumulative incidence, or adapting classical survival losses such as the Cox proportional log likelihood and variations (such as Efron's approximation) to the use-case could be beneficial to strengthening the analysis. For the last set, while the authors mention excluding losses involving a strong parametric prior, they might want to consider an exception given how extensively the Cox-family losses are used in practice. Similarly backbone choices are limited as well. For tabular data, dedicated tabular foundation models (e.g. TabPFN) are excluded. For pathology, one could imaging a medical encoder, or a more recent visual encoder.

  • Few conclusions on the impact of dataset properties (dataset size, number of features or number of causes whereas censoring percentage is already addressed) and relative model performance.

Questions for authors
  • Would the conclusions hold with more losses or encoders considered (see weaknesses above)? I would prioritize the Brier loss.
2 comments

We thank the reviewer for the positive assessment of the design, a controlled study in which every loss is trained on every backbone with the same search space and budget, and for concrete suggestions. We implemented both prioritised suggestions, and they make the comparison substantially broader: a fourth loss in the matrix, a cause-specific Cox loss, and two new image encoders.

IPCW Brier loss on the cumulative incidence. We added it as a fourth loss, CRBS, on all four backbones and 20 datasets (80/80 runs), and define it in the paper next to the other three. CRBS is CRNLL with the log score replaced by the multiclass Brier score. The head, censoring weights and time integration are unchanged, so it isolates the scoring rule. Like CRNLL, it is strictly proper at each horizon when the censoring weights are correct.

Per dataset (median over backbones):

  • CRBS is a strong second. In IBS it beats HazardLikelihood on 18/20 datasets and HLRanking on 19/20 (Holm p ≤ 6 × 10⁻⁵); in INLL it beats HazardLikelihood on 15/20 (Holm p = 3 × 10⁻⁴).
  • CRNLL beats CRBS on 15/20 in IBS (Holm p = 0.01) and on 19/20 in INLL (Holm p = 8 × 10⁻⁵). Among the four losses, CRNLL is first on 15/20 datasets and CRBS on the other 5. Per (dataset, backbone) cell, the best loss is CRNLL in 48/80 and CRBS in 22/80 (definition: Section 2.2, eq. 4; results: Section 6, Appendix D.5 and Table D.4).

The gain thus comes from scoring the CIF directly with censoring weights, whether the score is logarithmic or quadratic.

This also addresses HH9E's circularity concern. We select and evaluate on the IBS, so CRBS is trained on the evaluation metric itself, yet it does not beat CRNLL on it (Appendix D.5). CRNLL's lead therefore does not come from sharing the metric's weighting. On synthetic data with known CIFs, CRBS is also further from the truth than CRNLL (22/32 cells, p = 0.04) (Appendix D.8).

Cox-family losses. We implemented a cause-specific Cox partial likelihood on the four backbones: one output per cause, Efron's handling of ties (frequent with day-level times), mini-batch risk sets, and Breslow baseline hazards with the CIF obtained by product integration. It matches the Efron likelihood of lifelines and statsmodels exactly.

We had excluded it because it breaks the assumptions for the CR setting but, given how widely Cox models are used, we agree the comparison is informative.

The results on 16 datasets are in Appendix D.6; the four datasets re-run on their full files are still in progress. Per dataset (median over backbones), the Cox loss ranks third, after the two CIF losses and ahead of the two hazard-based ones: it beats HazardLikelihood and HLRanking on 15/16 datasets (Holm p = 0.01); CRNLL beats it on 13/16 (Holm p = 0.01), and CRBS on 12/16 (Holm p = 0.02); the exception is SurvTRACE, with which it is behind HazardLikelihood on 12/16 datasets.

Backbones

  • Pathology. We added a pathology foundation model (Phikon-v2, a ViT-L pretrained with DINOv2 on 460M pathology tiles) and DINOv2 ViT-B/14, with the same head, losses and budget as the CNNs. Phikon-v2 is a publicly available encoder pretrained without labels on a large multi-organ collection of public whole-slide images; it represents the pathology foundation models designed for this kind of data (Appendix C.4). We keep every encoder frozen and train an attention-based multiple-instance aggregator over 20× tiles, instead of using one thumbnail per slide.

    • Phikon-v2 + CRNLL is the best image model on all three cohorts. Across 8 encoders × 3 cohorts, CRNLL is the best loss in 21/24 cases; the three exceptions are on BRCA.
    • We now present the pathology study only as a check that the loss ordering holds on another modality.
  • Tabular foundation models (TabPFN). They answer a different question than our design:

    • they do not learn a dataset-specific predictor by minimising a training loss, but predict by in-context inference, with weights pretrained on a synthetic classification/regression prior;
    • putting them in the matrix would mean fine-tuning a foundation model with each survival loss on each dataset, confounding the loss with the pretraining objective; We now discuss them as future work in the Scope paragraph.

Dataset properties. We grouped the 20 datasets by each property and repeated the paired comparison within each group (per dataset, median over backbones; the lead is in IBS gain over Aalen–Johansen):

DatasetsnCRNLL first of the 4 lossesCRNLL > both hazard lossesMedian lead over the better hazard loss
< 5k training rows9890.041
≥ 5k training rows117110.067
≤ 10 features4340.033
11–40 features7570.069
> 40 features9790.047
1 cause106100.073
2–3 causes109100.039
< 30 % censored2220.046
30–60 % censored117110.025
≥ 60 % censored7670.086
  • In every group, CRNLL beats both hazard-based losses on every dataset, and CRBS is first wherever CRNLL is not. In each group of at least 6 datasets this is significant (Wilcoxon p ≤ 0.02); the smaller groups are too small for a test.
  • The lead does not depend on the training size (Mann–Whitney p = 0.94), the number of features (Kruskal–Wallis p = 0.65) or the number of causes (1 vs 2-3: p = 0.10). It is larger on the datasets censored at 60 % or more than at 30–60 % (0.086 vs 0.025, p = 0.006). The smallest leads are on the three ICU cohorts (≤ 0.004). (Appendix D.3)

We also made a learning curve on SEER Breast and Loan (1k to 50k training rows, MLP and TabM). CRNLL has the lowest IBS in 16/16 settings. Its margin shrinks with n for Loan-MLP (0.012 → 0.005) and stays at 0.003–0.006 elsewhere. (Appendix D.11, Table D.7)

Would the conclusions hold with more losses or encoders?

  • The ordering of loss families holds.
    • With CRBS, a CIF-based loss is best in 70/80 (dataset, backbone) cells: CRNLL in 48, CRBS in 22.
  • With the Cox loss (64 cells on 16 datasets), the order is CRNLL, CRBS, Cox, then the two hazard-based losses.
    • On images, the ordering holds with the two new encoders.
  • The size of the loss effect relative to the backbone effect does not generalise, and we no longer present it as a general ratio: the pooled ratio is 1.77 [1.26, 2.53] with CRBS, against 1.48 [0.83, 2.12] with the original three losses.
  • A better loss alone improves a given model. On a fixed backbone, changing only the loss improves the probabilistic accuracy: on DeepHit's own MLP, CRNLL raises the median IBS gain over Aalen–Johansen from 0.134 to 0.176 on the competing-risks datasets, and it beats both hazard-based losses on every backbone.

These results answer the reviewer's requests one by one, and they sharpen the paper's message: for a fixed backbone, the choice of loss consistently improves the predicted probabilities, and a CIF-based loss is the best loss in 70 of the 80 (dataset, backbone) cells. We thank the reviewer for suggestions that made the benchmark broader, and would be grateful if the reviewer would reconsider the score.

Reviewer_SrPUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the relative importance of training loss and network architecture in competing-risks survival prediction. It compares four losses across four tabular backbones on ten competing-risks and ten single-event datasets, with additional experiments on three pathology-image datasets. The results show that CRNLL generally achieves better probabilistic scores than hazard-based losses, while the choice of backbone remains important across data modalities. In contrast, ranking metrics such as the C-index reveal much smaller differences. The paper provides a controlled empirical study of these factors and offers practical guidance for survival model selection.

Critical strengths
  1. The controlled comparison addresses an important gap in existing survival models. By varying the training loss and backbone separately, the paper makes it possible to assess their contributions rather than comparing models that change both. The evaluation covers 20 tabular datasets and three pathology-image datasets, providing a broader empirical basis than many prior competing-risks studies.

  2. The experimental analysis is extensive and provides useful evidence for the main findings. The authors examine the effects of the additional survival head, censoring weights, and model-selection criteria. The consistent advantage of CRNLL over hazard-based losses across tabular backbones is particularly informative. The comparison between probabilistic and ranking metrics also highlights why the choice of evaluation metric matters.

Critical weaknesses
  1. The relative importance of loss and architecture depends on the comparison setup. The reported 1.77 ratio is based on the spread across four selected losses and backbones, rather than a general decomposition of their contributions. On the ten competing-risks datasets alone, its confidence interval is wide ([0.40, 4.31]), and the loss spread exceeds the backbone spread on only five datasets. This makes the broader claim that loss matters more than architecture less convincing for competing-risks prediction specifically.

  2. The conclusions remain sensitive to the censoring assumptions. Although the paper evaluates alternative IPCW estimators, the synthetic experiments show that CRNLL does not consistently achieve lower true prediction error under covariate-dependent censoring, even when IBS still ranks it first (Appendix D.8). The paper also acknowledges potentially informative censoring in the ICU cohorts. These results raise questions about how broadly the reported advantage of CRNLL applies beyond settings where the censoring assumptions hold.

  3. The pathology-image experiments provide limited evidence for the broader cross-modality conclusions. The image evaluation covers only three cohorts and does not include external competing-risks baselines. Moreover, Appendix D.10 reports that none of the image models improves on the covariate-free Aalen–Johansen estimator. Although CRNLL performs better than the other tested losses within most backbones, the practical benefit of these models on pathology images remains unclear.

Questions for authors
  1. How robust is the reported loss-to-backbone effect ratio when restricting the analysis to competing-risks datasets? Given the wide confidence interval and the smaller effect with only the three original losses, what evidence supports the broader conclusion that loss matters more than architecture?

  2. How should the reported CRNLL advantage be interpreted under covariate-dependent censoring? In particular, the synthetic experiments show cases where IBS favors CRNLL without a corresponding reduction in true prediction error. Which conclusions remain reliable in these settings?

  3. What conclusions can be drawn from the pathology-image experiments when none of the tested models outperforms Aalen–Johansen? Could the authors clarify whether the cross-modality results primarily establish the relative ordering of losses rather than an improvement in absolute predictive performance?

2 comments

We thank the reviewer for a precise and constructive review, and for recognising what the paper sets out to do: decouple the loss from the backbone, on a broad evaluation with controls for the survival head, the selection metric and the censoring estimate. Each point below led to a change, and the data-validity questions led us to re-audit the whole pipeline. We have updated the manuscript on OpenReview during the rebuttal.

Counts. The reviewer’s right. The different counts came from subsets of datasets and because one big dataset (Loan) was missing for a few experiments, this caused the inconsistency. We review all numbers here: unless a subset is named, counts are over the 20 datasets of Table 2. This correction did not impact the overall results, and all conclusions hold. The matrix is now complete on all of them: the 3 original losses × 4 backbones (240/240 runs), plus a fourth loss, CRBS (80/80) (loss asked by reviewer BcaT, changing the NLL to the Brier Loss). A dataset's value is the median over backbones. The only other denominators are:

  • 17 datasets: the 20 without the ICU cohorts (MIMIC-III, MIMIC-IV, eICU);
  • (dataset, backbone) pairs: at most 80.

Preprocessing within folds. Appendix A.1 described the submitted code accurately: vectorisation and mean imputation were fitted on all rows before the split. All preprocessing is now part of each model's pipeline and is refitted on every training fold, in both the search and the final fits. We compared against the submitted runs:

  • the median relative IBS change is 0.9 % (90th percentile 7.0 %);
  • the CRNLL-vs-HazardLikelihood winner is unchanged in 48/51 pairs. (Appendices A.1 and C.5 in the updated version.)

Missing data. On the six datasets with missing values, i.e. 24 (dataset, backbone) cells for each of the three original losses, CRNLL has the lowest IBS in: 20/24 cells with mean imputation; 18/24 with complete-case analysis; 20/24 with in-fold iterative (MICE-style) imputation, which changes the IBS by a median of 0.0 % (IQR −0.7 % to +1.0 %). The best loss is the same as with mean imputation in 20/24 and 18/24 cells, so the conclusion does not depend on how missing values are handled (Appendix D.11).

PBC. The reviewer is right: rows are visits and the split was by row. PBC is now split by patient, in both the outer split and the search CV. Every model loses accuracy, the most flexible ones most (SurvivalBoost IBS 0.058 → 0.102). CRNLL remains the best loss on all four backbones (Appendix C.5; PBC row of Table D.2).

ICU time origin. The covariates are first-24 h measurements, so the time origin is admission + 24 h, but this was not enforced. The revised cohorts:

  • apply a 24 h landmark (only subjects still followed at 24 h are kept);
  • drop two MIMIC-IV covariates: hospital_expire_flag, which encodes in-hospital death, and a provider identifier;
  • redefine the eICU outcome as in-hospital death vs discharge alive (true competing events), with administrative censoring at 30 days.

After re-running on the corrected cohorts and averaged over the four backbones it is still the best loss on each cohort,by 0.1–0.2 points of IBS gain, and it is best in 8/12 (cohort, backbone) pairs (cohort definitions in Appendix A.2; results in the MIMIC-III, MIMIC-IV and eICU rows of Table D.2).

With the corrected cohorts, CRNLL has the best IBS of the three original losses on 20/20 datasets and is above HazardLikelihood on 20/20 (sign test p = 2 × 10⁻⁶). As the sensitivity analysis the reviewer asked for, every aggregate is also reported without the ICU cohorts: CRNLL is then first on 17/17 (p = 2 × 10⁻⁵ against HazardLikelihood) (Section 6 and Appendix D.1).

"Twice as much". A ratio of median max − min spreads is a ballpark figure, not a decomposition, and it depends on the backbones compared. We now report it as such, with:

  • dataset-bootstrap 95 % CIs;
  • paired within-dataset tests;
  • equal-size axes (the 3 original losses on every 3-of-4 backbone subset; 4 × 4 with CRBS);
  • a two-way variance decomposition.

Loss / backbone spread ratio [95 % CI]:

Datasets3 original losses4 losses (with CRBS)
Competing risks (10)1.28 [0.42, 3.54]1.77 [0.40, 4.31]
Single event (10)1.30 [0.79, 1.59]1.94 [1.28, 2.32]
All (20)1.48 [0.83, 2.12]1.77 [1.26, 2.53]
All without ICU cohorts (17)1.57 [1.05, 2.10]1.90 [1.40, 2.61]

In the variance decomposition (20 datasets), the loss explains 53 % [28, 67] of the within-dataset variance and the backbone 28 % [23, 45] with the three original losses; with four losses, 60 % [35, 74] and 19 % [16, 29].

We therefore replace "twice as much" by this ballpark comparison: on our backbones the loss moves the IBS more than the backbone does, by a factor of about 1.5 to 1.9, but the intervals include 1 on competing risks alone, and we no longer present it as a general ratio (Section 6, Appendix D.1). What is robust is the ordering of the losses.

Common complete set. Since the matrix is complete, the 20 datasets are the common complete set, with the same datasets and backbones for every loss. On it: among the 3 original losses, CRNLL is first on 20/20 datasets and in 69/80 pairs; among all 4, CRNLL is first on 15/20 datasets and CRBS on the other 5 (Friedman p = 1 × 10⁻⁹), and one of the two is best in 70/80 pairs; with dataset-bootstrap 95 % CIs, the average ranks are CRNLL 1.25 [1.10, 1.45], CRBS 1.90 [1.60, 2.25], HazardLikelihood 3.25 [3.00, 3.50], HLRanking 3.60 [3.30, 3.85], and CRNLL's median lead over HazardLikelihood is 0.051 [0.023, 0.078];

  • CRNLL is above HazardLikelihood and HLRanking on 20/20 (Holm p = 10⁻⁵ each).

Where the odd counts came from.

  • "22 datasets" (Table E.2): our figure loader listed MIMIC III and IV under two keys (diff nb of samples).
  • "94 cells" (C.3): these counted CRNLL runs per (dataset, training size, backbone).

In the revision, every count is over the 20 datasets of Table 2, and Appendix D lists the datasets behind each count.

Evaluation-weight sensitivity. We re-scored the stored test predictions of all 320 runs (4 losses × 4 backbones × 20 datasets) without retraining. The IBS was recomputed with two kinds of covariate-dependent censoring weights:

  • Kaplan–Meier within covariate strata;
  • a penalised Cox censoring model.

IBS values move, by up to 25 % with stratified weights and 63 % with Cox weights, but the ordering does not:

  • CRNLL is the best of the four losses on 16/20 datasets with marginal and with stratified weights, and on 17/20 with Cox weights; CRBS is first on the rest;
  • CRNLL is above HazardLikelihood and HLRanking on 20/20 under all three weightings;
  • per dataset, the rankings of the 16 runs agree with the marginal-KM ranking at median Kendall τ = 0.98 (stratified) and 0.93 (Cox) (Appendix C.3, Table C.5).

Synthetic data with covariate-dependent censoring is a stricter test, because the training and evaluation weights are then both misspecified. There, CRNLL is closer to the true CIF than HazardLikelihood on 3/4 backbones at n = 2k, but on only 1/4 at n = 10k. We report this in the new synthetic appendix (Appendix D.8).

Taken together, these checks make the benchmark more solid than the submitted version: preprocessing is fitted within folds, PBC is split by patient, and the ICU cohorts follow their stated time origin. On this corrected basis, CRNLL remains the best of the three original losses on all 20 datasets, and on all 17 without the ICU cohorts. We thank the reviewer for pushing us there, and hope these answers resolve both weaknesses, which the reviewer rated as fixable.

Comments on the paper

1 comment

We thank the AC and the three reviewers for their constructive feedback. All three recognized the value of our controlled design, which evaluates every loss on every backbone under identical tuning budgets. We updated the manuscript to address their comments. The new version on OpenReview has a few focused changes, all highlighted in red.

1. New baselines strengthen the conclusion

  • IPCW Brier loss (CRBS): Added across all 80 cells (20 datasets x 4 backbones). CRNLL still outperforms CRBS on 15/20 datasets (Holm p = 0.01). Together, these two CIF-based losses win 70/80 cells (BcaT).
  • Cause-specific Cox loss: Evaluated on 16 datasets, ranking third, behind both CIF-based losses (BcaT).
  • Two new image encoders: Phikon-v2 and DINOv2 with attention-MIL. CRNLL wins in 21/24 encoder–cohort combinations, with Phikon-v2 + CRNLL leading on all three cohorts (BcaT, HH9E).

2. CRNLL beats other losses on IBS in the whole benchmark

Following SrPU and HH9E, we re-audited the entire pipeline: preprocessing is now fitted within folds (SrPU), PBC is split by patient (SrPU), ICU cohorts use a consistent 24 h landmark (SrPU), outcome leakage was removed (SrPU), and failed runs were corrected (HH9E). The original benchmark is now complete (240/240 runs) (SrPU).

CRNLL remains the best of the three original losses on 20/20 datasets, including 17/17 when excluding the ICU cohorts (SrPU, HH9E), and wins in 69/80 dataset–backbone pairs. It beats HazardLikelihood on 20/20 datasets (p = 2 × 10⁻⁶).

3. Ablations rule out alternative explanations

  • Output head: On HazardLikelihood's identical head, CRNLL still wins on 16/17 datasets (p = 2 × 10⁻⁴) (HH9E).
  • Model selection: Selecting hyperparameters using the C-index instead of IBS preserves CRNLL's IBS lead on 16/17 datasets (HH9E).
  • Evaluation weights: With stratified-KM and Cox-based censoring weights, CRNLL beats both hazard-based losses on 20/20 datasets (SrPU, HH9E).
  • Oracle synthetic evaluation: CRNLL is closer to the true CIF than HazardLikelihood in 28/40 settings without IPCW evaluation (HH9E).
  • Additional checks: Missing-data strategies (SrPU), dataset characteristics (BcaT), and learning curves (BcaT) support the same overall IBS ordering.

4. We narrowed our claims

We withdraw the general "twice as much" loss-versus-backbone claim and replace it with a variance decomposition and confidence intervals (SrPU, HH9E). Paired tests show weaker separation between losses on the C-index (Friedman p = 0.06) (HH9E).

We also clarify the limitations (HH9E): CRNLL's advantage is concentrated at medium-to-long horizons, synthetic results are mixed under covariate-dependent censoring, and CRNLL ties hazard-based losses on right-censored log-likelihood. MIMIC censoring remains a limitation (HH9E), addressed through analyses excluding the ICU cohorts (SrPU).

The revised evidence supports a clear conclusion: across the evaluated datasets and backbones, directly scoring the CIF yields consistently stronger integrated probabilistic performance than the hazard-based alternatives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.