- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies the mismatch between conventional survival-prediction metrics and the utility of selecting a high-value transplant recipient. It gives worst-case examples showing that a high C-index alone need not imply good single-recipient allocation, connects NDCG to a position-weighted selection policy, and develops expected-years imputation and inverse-probability-of-censoring weighting (IPCW) for estimating discounted gains from censored outcomes. A second-stage gradient-boosted model combines regression on imputed survival labels with an NDCG-weighted ranking objective. Experiments use the UNOS heart-transplant registry, first with artificially censored observed-event records and then with the full registry.
The motivating problem is important. However, the current evidence more directly supports improved retrospective top-of-list survival ranking than improved organ allocation. The statistical interpretation of the estimated NDCG comparisons and the contribution of the ranking objective also need stronger justification.
Critical strengths
S1. The paper identifies a consequential mismatch between evaluation and use. Section 3.1 gives a simple, effective construction in which almost all pairwise comparisons are correct while the selected candidate is the worst one. This is an important warning against treating a global ranking metric as sufficient evidence for a top-selection decision. The focus on right-censored outcomes makes this more relevant than a generic learning-to-rank example.
S2. The empirical study includes useful controls and acknowledges nuisance-model limitations. The use of several survival-model families, two relevance estimators, cross-fitting, and the additional covariate-dependent censoring experiments in Appendix I is valuable. Appendix H.1 also provides the necessary regression-only and ranking-only controls. These components make it possible to assess which parts of the proposed framework are actually responsible for the gains, even though the interpretation needs revision.
Critical weaknesses
W1. Unbiased DCG estimation does not establish the claimed reliability of NDCG-based model comparisons, and the censoring experiments need to be reconciled with the assumptions. Section 4.2 correctly observes that two estimated NDCG values share a normalizer on a fixed dataset. This proves that their ordering equals the ordering of their estimated DCGs. It does not prove that this ordering matches the true DCGs or true NDCGs. Likewise, an upward bias in estimated IDCG does not by itself establish a downward bias in its ratio with an estimated numerator. The numerator and denominator are dependent, so a ratio argument needs separate justification. A twofold ratio of estimated gains is also not automatically a twofold ratio of true allocation gains.
This distinction matters most at the small values of k emphasized by the paper. In Table A10, IPCW model-selection accuracy at k=1 is approximately 55% at gamma=0, rather than uniformly reliable; EY is better, but remains imperfect. The positive EY biases in Table A9 also show that a generally negative NDCG bias is not an adequate description of the implemented estimators. Appendix I should therefore qualify the main-text claims, rather than be summarized as unconditional preservation of model rankings.
There is a further mismatch in Appendix F: for selected patients, the censoring time is drawn uniformly from [0, T*]. This construction depends on the event time and does not generally satisfy conditional independent censoring given the baseline covariates. Appendix I uses an appropriate alternative construction and is an important partial response, but it primarily reports estimator diagnostics. The headline bootstrapping gains should also be established under that assumption-compatible construction. Restricting the starting cohort to patients with observed events additionally changes the population and should not be treated as validation on a representative fully observed transplant cohort.
W2. The experiments do not yet isolate a decision-focused ranking advance or validate the intended allocation use. Theorem 1 concerns one allocation with known nonnegative utilities and a specified true NDCG bound. At k=1, Corollary 1 is essentially the identity between selected-to-optimal utility and NDCG@1. It does not establish sequential allocation performance, transplant benefit, or how an estimated validation metric supplies a bound for a new donor-specific candidate set. Section 6 evaluates historical outcomes of recipients who actually received transplants, rather than alternatives for the same arriving organ. The relationship between graft survival, expected benefit relative to no transplant, candidate compatibility, and future allocation opportunities consequently remains untested. The claim that graft survival dominates life-years gained does not resolve these identification and decision-design issues.
The algorithmic attribution is also weaker than the headline improvements suggest. In Table A8, regression-only training achieves NDCG@1 of 0.468, compared with 0.494 for the hybrid loss and 0.288 for the original Cox model. Regression-only training is better than the hybrid at k=5, 10, and 50. Thus, much of the improvement may come from the imputed labels and the added boosted-tree model, rather than the ranking component. Learning to rank for allocation and NDCG-weighted boosted-tree training already have substantial precedents, including Kamran et al. (2024) and Burges (2010). Both are acknowledged by the manuscript; the issue is isolating the additional contribution here, not missing citations.
At minimum, the paper needs a broader matched regression-only comparison and a controlled donor-specific allocation experiment with known utilities. Without such evidence, the conclusion should be restricted to retrospective ranking. Section 6.2 should also avoid claiming statistically significant improvement across every survival model: Table 3 contains nonsignificant NDCG@1 comparisons, including the IPCW results for AFT and DeepSurv.
Questions for authors
Q1 — W1, theoretical clarification. What result, beyond cancellation of the common estimated denominator, supports preservation of true model rankings in Section 4.2? Can you state the estimand, distinguish expected from realized rankings, and provide either a comparison-error bound under explicit assumptions or a narrower empirical claim? Please also justify or remove the asserted general direction of NDCG bias.
Q2 — W1, essential experimental evidence. Can you report the baseline-versus-bootstrapped ranking gains under the conditionally independent censoring construction in Appendix I, alongside the Appendix F results? Please clarify the artificial-censoring fraction, nuisance-model fitting, and the evaluation population induced by selecting observed-event patients. This would determine whether the central gains persist in the setting covered by the assumptions.
Q3 — W2, contribution attribution. With the same pseudo-labels, boosted-tree architecture, tuning budget, and splits, how does the hybrid objective compare with regression-only training across all base models? Are its incremental NDCG@1 gains stable, and how should the stronger regression-only results at larger k in Table A8 affect the claimed benefit of ranking optimization?
Q4 — W2, decision relevance. What exactly forms a query/candidate set in training and evaluation, and how does it correspond to compatible recipients for one donor? Can a controlled allocation setting with known donor–recipient utilities establish the proposed metric's decision benefit? If that evidence is unavailable, would you limit the claims to survival ranking rather than improved transplant allocation?
3 comments
Author response (2/2)
With the same pseudo-labels, boosted-tree architecture, tuning budget, and splits, how does the hybrid objective compare with regression-only training across all base models? Are its incremental NDCG@1 gains stable, and how should the stronger regression-only results at larger k in Table A8 affect the claimed benefit of ranking optimization?
We already have the results of this ablation and provide them for the reviewer here. Our decision-focused approach is specifically to optimize NDCG; regression may unintentionally perform well at this task by minimizing predictive errors at the top of the ranking. We provide the ablation for regression only training across all base models below and hope this clarifies the reviewer’s concern. We will add this to the paper.
| Predictor | alpha = 0.3 | alpha = 1 |
|---|---|---|
| KM | 0.423 | 0.412 |
| AFT | 0.514 | 0.550 |
| DeepHit | 0.598 | 0.522 |
| DeepSurv | 0.593 | 0.447 |
What exactly forms a query/candidate set in training and evaluation, and how does it correspond to compatible recipients for one donor? Can a controlled allocation setting with known donor–recipient utilities establish the proposed metric's decision benefit? If that evidence is unavailable, would you limit the claims to survival ranking rather than improved transplant allocation?
We appreciate the reviewer's question and believe it is important to distinguish the optimization of the predictive component from the optimization of the entire sequential allocation process. We do discuss this distinction in the paper, and can elaborate on this challenge.
Our objective is not to provide an end-to-end guarantee for the entire sequential allocation process, but rather to establish the correct optimization objective for the predictive component used within allocation algorithms. Every sequential allocation policy ultimately relies on comparing feasible candidates at each donor arrival. Our results show that optimizing aggregate survival objectives such as C-index is fundamentally misaligned with this decision, whereas NDCG@1 directly aligns the predictor with the allocation objective.
The training queries are resampled groups of 256 historical transplant recipients, which is stated already in Section 5, Appendix E.1. We evaluate the resulting predictors on their ability to identify patients with high graft-survival outcomes. Our experiments establish improvements in survival ranking, while our theoretical results establish why this ranking objective is directly relevant to allocation.
In practice, the single allocation problem is a concrete component of sequential heart allocation. The waitlist is highly dynamic and patients continuously enter and leave the system, and other papers have already observed that the distribution of patient characteristics remains relatively stable over time (e.g., “Near-optimal dynamic matching via coarsening and application to heart transplantation”, ICML 2026). At each donor arrival, the allocation algorithm repeatedly encounters a similar prediction problem: identifying the highest-utility candidate from the current feasible set of patients, which is exactly what we address.
Thank you for the clarifications and additional experiments. Concerns remain about the claimed direction of NDCG bias, the completeness and stability of the regression-only ablation, and the link to allocation benefits, so I will maintain my score.
Author response (1/2)
We thank reviewer Loby for their review.
This does permit a comparison-error guarantee. For two fixed model rankings with true DCG difference Δ>0 and an IPCW difference estimator with conditional variance bounded by V, Cantelli's inequality yields that the probability that Δ≤0∣T∗,X is less than V+Δ2V. Thus, under conditional independent censoring, the probability of incorrectly comparing two models decreases as their true DCG difference becomes large relative to estimation uncertainty. We will add this clarification and distinguish realized from expected model comparisons explicitly.
Our empirical evidence directly evaluates the preservation of true model rankings. Table 2 and Appendix I compare estimated model rankings against rankings computed from known outcomes, validating the accuracy of the estimators at identifying the correct ranking.
Yes, of course, this is an addition we can immediately add. The artificial censoring fraction is explicitly reported in Table A4 and is 49.47%. The results from Appendix I are intended to validate the nuisance model estimates, and the results for the NDCG gain are nearly identical to the Section F setup. As described in Appendix F, we cross-fit nuisance models to separate nuisance estimation from the evaluation of predictive performance. We will include those results in the final version of the paper and we provide them here. They reinforce the benefit of our decision-focused approach.