- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In fields like healthcare and predictive maintenance, survival analysis with competing risks predicts which of several mutually exclusive events will happen first and when, despite right-censored data. Deep competing-risks methods usually introduce a new architecture and a new loss together, so it’s unclear which one drives their gains. This paper separates the two by training three losses on four backbones across 20 tabular and 3 pathology-image datasets. The CRNLL (Competing Risks Negative Log-Likelihood) loss, which models the cumulative incidence function (CIF) directly, wins on every backbone, and on competing-risks data it matters about twice as much as the backbone choice; the backbone still has to match the modality. The C-index barely separates any configuration, so the authors recommend strictly proper scoring rules like the Brier score, where RealMLP or TabM trained with CRNLL performs best.
Critical strengths
-
Disentangled experimental design Each loss is paired with each backbone under an identical search space and tuning budget, which isolates the contribution of the loss from that of the architecture, something no prior competing-risks study has done.
-
Extensive empirical coverage The evaluation spans 10 competing-risks and 10 single-event tabular datasets along with 3 imaging datasets, roughly ten times the scale of earlier method papers, and assesses accuracy, discrimination and calibration.
Critical weaknesses
-
Study design cannot answer “loss or architecture” in general The architecture axis is narrow by construction. Three of four tabular backbones are MLP variants, and the imaging encoders are frozen, so the image comparison reduces to pretrained features versus an MLP on raw inputs. The loss-to-backbone ratio thus reflects this backbone selection rather than a general finding. Without synthetic data of known cumulative incidence, the claim of better probability estimates rests only on IPCW-based metrics that share the winning loss’s weighting.
-
Central claim lacks statistical support The central “twice as much” claim (Table D.1) compares a median loss spread of 0.053 with a median backbone spread of 0.027, but their interquartile ranges ([0.020, 0.084] vs [0.021, 0.062]) almost fully overlap, and no confidence interval or test is given. The single-event reversal (0.072 vs 0.086) includes a backbone spread of 3.136 on NWTCO (Table D.2), which implies a failed run with IBS about three times the Aalen-Johansen reference. The claim that the C-index is flat is drawn from overlapping boxes (Figs. D.3, D.9), while IBS gets a paired sign test; Table D.3 itself shows the C-index varies across losses about 60% as much as IBS. The imaging conclusions in Fig. 2b rest on three datasets, with three points per box.
Questions for authors
Q1. The loss axis changes several things at once: CRNLL adds IPCW weighting and a per-horizon softmax output, while the hazard-based losses use a joint softmax over causes and time. Model selection is also done on IBS, which shares CRNLL’s weighting. Since only the extra survival head is ablated (Appendix D.6), how can the observed gains be attributed to the loss itself rather than to these co-varying choices, and would the ordering hold if hyperparameters were selected with a criterion not aligned with CRNLL?
Q2. Does CRNLL still come out ahead on synthetic competing-risks data with known cumulative incidence functions, or under an evaluation metric that does not use IPCW weights?
Q3. Would the loss-to-backbone ratio hold with architecturally distinct backbones, such as tree ensembles, fine-tuned encoders or pathology foundation models, instead of three MLP variants and frozen ImageNet encoders?
Q4. Can bootstrap confidence intervals be reported for the effect sizes in Table D.1, with failed runs such as the 3.136 NWTCO spread either excluded or explicitly accounted for?
Q5. If the C-index is evaluated with the same paired within-dataset sign tests used for IBS, does it still show no separation between losses or backbones?
Q6. Given that MIMIC and eICU patients are censored at their last hospital discharge, how is the independent-censoring assumption behind IBS justified on these datasets, and do the conclusions change if they are excluded?
8 comments
Q1: co-varying choices, and selection on the IBS. Because of time, we did the experiments on 17 datasets, the other ones are still running. We added one ablation per choice the reviewer lists, each changing one factor at a time on all four backbones, median over backbones.
- Output head. CRNLL on HazardLikelihood's own joint-PMF head (CIF by cumulative sum) beats HazardLikelihood on that identical head on 16/17 datasets (median IBS-gain difference +0.038, Holm p = 2 × 10⁻⁴), and is only 0.005 behind CRNLL with its own head. The effect comes from the loss, not the per-horizon softmax.
- Scoring rule. The IPCW Brier version (CRBS) is a close second (see our reply to BcaT): although trained on the evaluation metric itself, it does not beat CRNLL on it.
- Selection criterion. With hyper-parameters selected on the C-index, which does not share CRNLL's weighting, CRNLL still has the best IBS on 16/17 non-ICU datasets (Appendix E).
Q5: C-index with the same paired tests. We ran them on the three original losses.
- The C-index barely separates the losses (Friedman p = 0.06). The only pair close to significance favours HLRanking, ahead of CRNLL on 14/17 datasets, by a median of about 0.02; Appendix E.
We withdraw "the C-index does not separate models". Revised statement: the C-index differs little between losses, and where it does, it favours the ranking loss built for it, while the proper scores separate the losses more strongly and in the opposite direction.
Q6: MIMIC and eICU censoring. The reviewer is right: censoring at the last hospital contact depends on hospital use and is informative, so the independent-censoring assumption behind the IBS fails for MIMIC-III/IV. In the revision:
- eICU now has a genuine competing-risks outcome (in-hospital death vs discharge alive), with only administrative censoring at 30 days, so the KM weights are valid.
- MIMIC-III/IV cannot be rebuilt this way. We state the limitation and report every result with and without the ICU cohorts. Without them, CRNLL is first of the three original losses on 17/17 datasets and above HazardLikelihood on 17/17 (p = 2 × 10⁻⁵), with the effect sizes given above.
- We re-scored all 20 datasets with covariate-dependent censoring weights (stratified KM and a Cox censoring model). The ordering is unchanged: CRNLL is first on 16/20 and 17/20 datasets respectively, above both hazard-based losses on all 20, with median Kendall τ of 0.98 and 0.93 against the marginal-KM ranking.
Auditing these cohorts, we also removed a MIMIC-IV covariate that encoded in-hospital death and enforced the 24 h time origin. All models were re-run on the corrected cohorts:
- CRNLL keeps the lead over the hazard-based losses there (0.1–0.2 points of IBS gain, averaged over backbones);
- it is the best loss in 8 of the 12 (cohort, backbone) pairs;
With these cohorts included, CRNLL is first of the three original losses on 20/20 datasets.
Overall, the review led us to add what the submission lacked: synthetic data with known CIFs, a score without censoring weights, confidence intervals and paired tests, and one ablation per co-varying choice. On the IBS, the ordering of the losses holds throughout: CRNLL is the best of the three original losses on all 20 datasets. The new analyses also mark its limits, which the paper now states: the advantage is concentrated at medium-to-long horizons, it shrinks under covariate-dependent censoring, and on the right-censored log-likelihood CRNLL ties the hazard-based losses. We hope the clearer presentation and the new analyses address the reviewer's concerns, and would be grateful if the reviewer would consider strengthening their support.
I thank the authors for their detailed response and the honest withdrawal on multiple points, e.g.
- That the losses matters twice (W1)
- C-index does not separate models (Q5)
Having read the revised manuscript, I feel it is now a significantly new paper. Many of the cohort results are new, and the comparison doesn't stand. Some of the withdrawn claims are still in the abstract and conclusion, e.g., saying the loss matters more than backbone, although the ratio is (1.77, 95% CI [0.40, 4.31]). Moreover, the central claim has now been withdrawn, it undermines the study desing and faithfulness of the resutls reported in the original manuscript.
This paper will need a full-review. Hence, I keep my rating the same.
I suggest the authors to fix the study design issues, qualify the generality of the claims and consider submitting the revised work in future venue.
We thank the reviewer for a thorough and demanding review, and for highlighting the two features the paper is built on: a disentangled design, in which each loss is paired with each backbone under an identical search space and budget, and a scale about ten times that of earlier method papers. The questions tested exactly this design, and answering them made the evidence substantially stronger, including where it narrows our original claims. We address each point below:
W1, Q3: scope of the backbone axis. We agree, and the answer to Q3 is no: we do not expect the ratio to hold in general.
W1, Q2: evidence that does not rely on IPCW.
We measure the integrated squared distance between predicted and true CIFs, which uses no censoring weights. These are the results:
We have rewritten the claim accordingly: losses that score the CIF directly with censoring weights (CRNLL and the new IPCW Brier loss) give the most accurate probabilities consistently across backbones. For short horizons and the event-time density, likelihood-based losses are as good.
W2, Q4: statistics. Dataset-bootstrap 95 % CIs on the corrected runs:
The reviewer is right that "twice" is not established; we withdraw it.
We fixed the failed fit, and re-run the affected cells, now the HazardLikelihood-RealMLP's IBS on NWTCO goes from 0.490 to 0.098 (Aalen–Johansen: 0.112).
The loss ordering, by contrast, is statistically solid (per dataset, median over backbones):
For imaging, we now show per-cohort points and make no statistical claim from three cohorts.