acceptodds
Withdrawn from ICLR 2027

When the Loss Stops Tracking the Metric: Value-Monotonicity in Deep Survival Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A validation loss is routinely interpreted as if a lower value implied a better model. Deep survival prediction makes the mismatch stark, since models there are commonly trained with likelihood objectives, evaluated by the concordance index (C-index), and selected on the numerical value of the validation loss. Standard surrogate theory does not justify that interpretation. Fisher consistency constrains the minimizer, not the surrogate value at the intermediate points where early stopping and checkpoint selection occur. We formalize the missing property as value-monotonicity, requiring the loss to fall as the metric rises, to within a stated tolerance, along a training trajectory rather than only at the optimum. We prove that the Cox partial likelihood admits no finite such tolerance on a score class closed under rescaling, because rescaling leaves the C-index unchanged while the likelihood varies without bound, and the discrete-time likelihoods decouple by the same mechanism. The sigmoid concordance loss (SCL), in contrast, approximates one minus the C-index with a two-sided, computable tolerance, so a decrease beyond twice that tolerance certifies a concordance gain. Across eighteen datasets and four modalities the absolute rank correlation between validation SCL and validation C-index averages 0.97 and never falls below 0.91, against 0.37 to 0.46 for the likelihood losses. Replacing C-index-based checkpointing by loss-based checkpointing moves the SCL’s test C-index by at most 0.006, against up to 0.064 for the baselines. The SCL also retains competitive discrimination, with the best mean within-dataset rank, 2.00, among the six losses run on every dataset, although the leading methods are not statistically separated. The contribution is therefore not improved discrimination, but a loss whose numerical value remains interpretable as a proxy for the evaluation metric during training.

Primary area:
optimization
Keywords:
survival analysis, concordance index, loss function, surrogate loss

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qzd4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether numerical changes in a survival training loss reliably track changes in the C-index, rather than merely sharing an optimum. It introduces value-monotonicity, gives a Cox rescaling counterexample, and proposes a corresponding argument for discrete-time likelihoods. For sigmoid concordance loss (SCL), it derives a two-sided approximation bound whose usefulness depends on the mass of near-tied score pairs. Across 18 datasets and four modalities, SCL shows stronger loss-metric coupling and closer agreement between loss-based and C-index-based checkpoint selection than the baselines, with comparable discrimination. The claimed contribution is with regards to the interpretation of loss values during training, not improved predictive accuracy.

Critical strengths
  1. The distinction between agreement at the optimum and agreement between loss and metric values during training is clearly stated. The Cox rescaling example makes the issue concrete, while the SCL approximation bound identifies a condition under which the loss value can be interpreted. I also appreciate that the authors explicitly discuss where this interpretation fails, rather than claiming unconditional alignment.

    1. The empirical study covers a broad set of survival datasets and includes a useful control separating the training objective from the monitoring signal. The near-tie diagnostic connects the analysis to a quantity that can be checked on validation predictions. The authors are also transparent about preprocessing leakage, pooling bias and implementation corrections, although documenting these issues does not eliminate their effects on the reported results.
Critical weaknesses
  1. The practical benefit beyond monitoring the validation C-index remains limited. Appendix B.12 shows that evaluating SCL on Cox-trained predictions raises coupling from 0.51 to 0.96, and the authors acknowledge that monitoring SCL is not preferable to computing the C-index directly. Table A2 also shows that closer agreement between the two selection rules does not establish better test performance. I recognize that the paper claims interpretability rather than improved discrimination, so I do not view this as a contradiction. My concern is whether making the training objective and monitoring signal coincide provides enough additional value beyond that interpretability. Evidence that the proposed error bounds are informative at the scale of actual checkpoint differences would help establish this value.

    1. The fixed-risk fibre argument in Proposition A2 does not hold as stated. A positive-dimensional fibre need not approach the boundary where the observed bin has zero probability. For example, with weights w=(0,1,2)w=(0,1,2) and a predicted distribution p=(0.8,0.1,0.1)p=(0.8,0.1,0.1), fixing the risk at 0.30.3 forces every distribution qq in the fibre to satisfy q1=0.7+q3≥0.7q_1=0.7+q_3 \geq 0.7. If the event occurs in bin 1, its negative log-likelihood cannot diverge along this fibre. The required feasibility condition is that the fixed risk lie in the convex hull of the remaining bin weights. Censored observations require an analogous condition for the total surviving mass. This affects the stated discrete-time extension, not the Cox rescaling result or the SCL approximation bound, but it requires a corrected statement or a different construction.
Questions for authors
  1. Can Proposition A2 be restated with the necessary feasibility conditions, or established through a different construction? Which discrete-time models in the experiments satisfy the corrected assumptions?

    1. Along the measured training trajectories, how often does the loss decrease between checkpoints exceed the sum of their computable approximation-error bounds? This would clarify whether the improvement certificate is informative in practice, beyond the observed loss-metric correlation.
Reviewer_JBh6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether the numerical value of a training loss reliably tracks the evaluation metric during optimization. It introduces value-monotonicity, shows that Cox partial likelihood can change under risk-score rescaling while the C-index remains unchanged, and analyzes a sigmoid concordance loss (SCL) that approximates (1-C). Experiments across 18 survival datasets show stronger loss-metric correlation and greater agreement between loss-based and C-index-based checkpoint selection for SCL, but no clear improvement in predictive performance.

Critical strengths

The paper clearly distinguishes alignment at the optimum from alignment along the training trajectory. The rescaling argument is straightforward, and the experiments cover multiple data modalities with relatively extensive comparisons.

Critical weaknesses

My main concern is the motivation. Training losses and evaluation metrics measure different properties in many machine learning problems, and this is not necessarily a problem that needs to be fixed. In survival analysis, Cox partial likelihood captures information about relative-risk magnitudes, while the C-index only measures discrimination. A change in likelihood without a change in C-index is therefore expected and does not imply that the loss has become uninformative. Moreover, C-index is only one of several relevant survival metrics, alongside IBS, calibration, and predictive deviance. The paper does not convincingly explain why alignment with this particular metric should be a desirable property of the training objective.

The practical contribution is also unclear. If C-index is the target metric, it can be computed directly on the validation set for checkpoint selection. The experiments in Appendix B.12 further suggest that much of the reported benefit comes from the monitoring signal rather than the training objective. Since SCL is essentially an existing smooth approximation of the concordance indicator, its high correlation with C-index is not surprising. The paper does not establish a compelling advantage over simply training with an appropriate likelihood and selecting checkpoints using the desired evaluation metric.

Questions for authors
  1. What practical setting motivates requiring validation loss to track C-index when C-index itself can be directly monitored? Can the authors demonstrate a concrete benefit beyond agreement between checkpoint-selection rules?

  2. What is the substantive novelty over existing smooth concordance surrogates? Also, how often does the proposed (2\epsilon) certificate actually certify improvements during training? Reporting the empirical bounds relative to typical loss changes would help establish whether the theoretical guarantee is useful.

  3. Proposition A2 appears to require stronger assumptions. For example, with (w=(0,1,2)) and (w^\top p=1.8), every distribution on the corresponding fiber satisfies (p_3\geq0.8). If the observed event is in bin 3, its negative log-likelihood cannot diverge along this fiber. Could the authors clarify the conditions under which the claimed unboundedness holds?

Reviewer_bcbFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors state that the procedure consisting in using the c-index on a validation set for the optimization procedure (early-stopping etc) leads to sub-optimal results, and propose a new loss, the sigmoid concordance loss as a better criterion. They provide theoretical results around the concept of value-monotonicity, and experimental results showing that this optimization procedure leads to better performance.

Critical strengths

The paper seems strong and consistent.

Critical weaknesses

The starting point of this article, that the c-index is not a good metric, is well-known and documented (see Blanche, P., Kattan, M. W., & Gerds, T. A. (2019). The c-index is not proper for the evaluation of-year predicted risks. Biostatistics, 20(2), 347-357. for the fixed horizon case, and the ICML 2026 position paper "Stop Chasing the C-index when Evaluating Survival Analysis Models" (Lillelund et al.)). A good notion to define whether a metric or loss or score is good for learning and for evaluation is the notion of proper scoring rule, which the authors mention in section A5 of the appendices. However, it is not clear to me when reading how the authors position their candidate loss, the SCL, in terms of properness. Also, there are several key references about proper scoring rules for survival analysis that are not mentioned, and should be considered as baselines, before proposing a new loss (Rindt, D., Hu, R., Steinsaltz, D., & Sejdinovic, D. (2022, May). Survival regression with proper scoring rules and monotonic neural networks. In International conference on artificial intelligence and statistics (pp. 1190-1205). PMLR., Yanagisawa, H. (2023, July). Proper scoring rules for survival analysis. In International Conference on Machine Learning (pp. 39165-39182). PMLR., Survival Models: Proper Scoring Rule and Stochastic Optimization with Competing Risks Julie Alberge, Vincent Maladiere, Olivier Grisel, Judith Abécassis, Gael Varoquaux Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:3619-3627, 2025.)

Questions for authors
  • is the proposed loss a proper scoring rule?
  • can the authors compare the performances with other existing proper scoring rules to establish the advantage of their loss?
  • why is there no IBS computed for the RSF method on tabular data? it seems feasible, as in alberge 2025 cited above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.