- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents a learning and optimization problem, designed for decision support in online heart transplant programmes.
The work builds over literature about kidney transplant programs, and in particular on the idea that cascade effect from decisions can be efficiently approximated by adjusting the cost function of the optimization problem solved at each step, rather than by introducing a more expensive look ahead component.
The adjustments take the form or "potential", which are modeled as black-box functions depending on a variety of terms (patient, donor, and queue features) and implemented as ML models (linear, NNs).
The models are trained by imitation learning, trying to replicate the ranking performed by a clairvoyant optimization model, which needs to solve a large-scale assignment problem with perfect information. Data augmentation techniques, based on the idea of subsampling and readjusting the arrival times for real world time series, are used to improve the robustness of the fit.
The results show that the family of approaches introduced in this paper provides very significant improvements w.r.t. both the current organ assignment rule and its proposed replacement; however, some details of the empirical evaluation protocol may have been underestimated in the interpretation of its outcome.
Critical strengths
The proposed approach is both clever and sound: I was already familiar with the related literature in kidney exchanges, and I always felt that the idea of using potential as a cheap replacement for look-ahead steps in online optimization has been underappreciated.
The proposed training technique is indeed very efficient: heart transplantation optimization problems have a much simpler structure compared to kidney transplantation ones, thus making it viable to run a clairvoyant solver even on large scale problems.
Imitation learning is also much more efficient than using black-box optimization over a simulator. In theory, using imitation learning may lead to suboptimal results (due to the possible misalignment between the optimal decisions made under perfect and imperfect information), but the fact that most of the gap w.r.t. the clairvoyant solution is closed by the proposed strategy suggests that the degree of misalignment is limited in this case.
The results are also very good, though with a critical caveat that I'll discuss in the weaknesses section.
Critical weaknesses
While the paper is sound and well presented, I am afraid it suffers from a critical weaknesses that prevents me from recommending acceptance, at least in its current state.
The issue concerns the underlying simulation model and its interaction with the optimization problem formulation. Specifically, the authors are relying on a Cox model for estimating survival (a reasonable choice), but they never seem to account for: 1) epistemic uncertainty on the fitted model itself; and 2) epistemic uncertainty in the estimates used by the optimization problem.
The former point (no epistemic uncertainty on the Cox model) is a minor issue in my opinion: on the one hand, it makes the simulation result unrealistically accurate, which might lead to unrealistically good ML models; on the other hand, however, the available dataset is large enough w.r.t. the size of the model input that I would not expect overfitting to be a significant issues (even without data augmentation).
The latter point (no epistemic uncertainty in the optimizer) is unfortunately critical detail: the current evaluation does not allow one to claim that the main contribution (the learned potential) is driving the gain over the baselines. In fact, the results suggest the opposite: most of the gain over the baselines is already achieved by a myopic approach (using the "perfect" utility functions, but not potential); adding potentials on top of that only marginally improves the results.
In summary, the current evidence does not allow one to infer that the proposed contribution is significant enough.
I think this problem could definitely be fixed, but that would require time. The peculiarities of the ICLR reviewing process (which traditionally allows for submitting revised versions of a paper) provide a chance to address the issue, but that remains to be seen.
My main suggestion to improve the paper is to include a full ablation study. By doing that:
- Both the effectiveness of data augmentation and the risk of using an overly accurate simulator would be tested, to some degree, by running experiments with no data augmentation.
- The impact of the potentials could be assessed separately from the utility function, by replacing the utility function with one derived by the CAS score or from the current rule; in fact, it may well be the case that potentials have a more significant impact in such a scenario
Ideally, the authors should run experiments where the reliability of the Cox model and of the utility estimates is reduced (for example via noise injection); however, it is difficult to do that in a way that does not trigger additional concerns (e.g. favorable bias in the injected noise, or unbiased noise). Using the baselines to obtain approximate utilities seems a more realistic approach to achieve a similar goal.
Questions for authors
Have you tried running an ablation experiments to separately assess the impact of the data augmentation procedure?
Have you tried running an ablation experiments to separately assess the impact of the optimization based on the (very accurate) utility function? Alternatively, have you tried to introduce noise in the utility estimates?
I am willing to substantially revise my score based on this two key points.
AFTER THE RESPONSE
The authors have clarified some of my concerns, but the main critical point (impact of the potentials) remains not adequately addressed. Since a considerable part of the contribution depends on that, I am not willing to recommend acceptance, though I won't object if someone provides strong reasons to champion the paper
4 comments
Author response (2/2)
The issue concerns the underlying simulation model and its interaction with the optimization problem formulation. Specifically, the authors are relying on a Cox model for estimating survival (a reasonable choice), but they never seem to account for: 1) epistemic uncertainty on the fitted model itself; and 2) epistemic uncertainty in the estimates used by the optimization problem. The former point (no epistemic uncertainty on the Cox model) is a minor issue in my opinion: on the one hand, it makes the simulation result unrealistically accurate, which might lead to unrealistically good ML models; on the other hand, however, the available dataset is large enough w.r.t. the size of the model input that I would not expect overfitting to be a significant issues (even without data augmentation).
The latter point (no epistemic uncertainty in the optimizer) is unfortunately critical detail: the current evaluation does not allow one to claim that the main contribution (the learned potential) is driving the gain over the baselines. In fact, the results suggest the opposite: most of the gain over the baselines is already achieved by a myopic approach (using the "perfect" utility functions, but not potential); adding potentials on top of that only marginally improves the results.
The impact of the potentials could be assessed separately from the utility function, by replacing the utility function with one derived by the CAS score or from the current rule; in fact, it may well be the case that potentials have a more significant impact in such a scenario
We do include several experiments to highlight the effectiveness of the potential learning framework in varying settings. In Appendix K.1 (Table A4), we compare imitation learning against black-box optimization while holding the CAS policy class and feature representation fixed (i.e., learning the weights of CAS using our method and the black-box optimization method). Our imitation learning framework achieves a 39.4% improvement over the black-box optimization when optimizing CAS. This provides evidence that our learning procedure offers substantial advantages independent of the expressiveness of the potential functions and the myopic utility term.
We also already include an experiment where we replace the PLYG utility function with a utility function that is a weighted sum of PLYG and the transplant rate for biologically disadvantaged patients (Section J). We not only show that the potential learning framework is still 96.5% competitive with the hindsight optimal policy, but we also show that it is effective at trading off PLYG to improve the fairness for biologically disadvantaged patients. This shows that the potentials are effective not just for the PLYG utility function, but are adaptable and effective for other utility functions.
The improvement over myopic matching should be interpreted in the context of a strong baseline. Our learned potentials recover a statistically significant fraction of the remaining gap to the optimal solution by accounting for future matching opportunities. Put differently, the learned potentials recover roughly 11% of the remaining gap between myopic matching and the omniscient upper bound. Although the relative percentage appears modest, the absolute impact is very meaningful in this application. The improvement corresponds to approximately 22 additional life-years gained per month, or over 260 additional life-years annually. To put these numbers into context, our policy is extending (on average) the life of each transplanted patient by multiple months. Extending every transplanted patient’s life by several months is invaluable.
To provide more perspective on that, we would like to point out that Eli Lilly developed a cancer drug, ALIMTA, that extends life in certain cancer types by 2-5 months on average, and had costs of hundreds of millions of dollars to develop and significant lethal toxicity risks for the patients. This 2-5-month life extension was considered so significant that they were selected as one of the six finalist teams for the 2005 Edelman Award, an INFORMS award that is the leading award in optimization with measured real-world impact.
Thanks for the clarification, but...
Thanks for the clarification.
I had indeed missed the table from Appendix K.2, which addresses my concern about data augmentation.
I am afraid, however, that the comment about the perfect utility remains not adequately addressed:
- Centered (and hence unbiased) noise is exactly one of the examples that I made when pointing out that noise injection is difficult to perform without inducing further concerns. That's the main reason why I was suggesting using the other scores as baselines. That said, this specific issue is not critical
- The gap between the myopic and the potential-based approach remains very small, in all noise regimes. Again, this points out that the main factor behind the improved results is the utility estimate, rather than the potentials. In turn, that reduces by a large margin the practical significance of several of the paper contributions.
The experiments from K.1 does not appear to match the one I was suggesting (just using classically trained CAS scores and relying on the potentials to fix the mismatch).
About the Eli Lilly example: one thing is achieving this results in a controlled experiment with real patients, another is one on static historical data. Moving from the latter setting to the former may easily eat on most of the reported statistical margin, especially since unbiased estimates (like those assumed in the noise injection experiment) are very difficult to obtain in practice.
All in all, it seems to me that the main conclusion I can draw from the work is about the effectiveness of the utility estimates, not of the potentials.
I will update the my score, but my opinion remains slightly on the negative side.
We thank the reviewer for their follow-up comment, and for raising their score. We also now understand what the reviewer was looking for to show the effectiveness of the potentials. Here we provide new evidence to highlight this using the reviewer’s suggested experiment. We are grateful to the reviewer for suggesting this experiment as we feel the results strengthen the contribution of our paper.
In this experiment, we replace the estimated PLYG utility with the CAS score itself as the immediate utility function, using uniform weights across all features. We then compare greedy allocation, which selects the patient maximizing CAS, against our potential-based policy, which learns non-myopic adjustments to the same CAS utility through imitation of the hindsight-optimal matching.
We use the same setup as the main experiment, training on January to March 2019 and evaluating on April to December 2019. Our learned potentials improve the total CAS utility by 15.9% over greedy allocation, closing 73.1% of the remaining gap to the hindsight optimum and achieving 95.2% of its utility.
We hope these results illustrate not only the effectiveness, but also the flexibility and computational efficiency of our framework which enabled us to conduct these experiments so quickly. Changing the oracle’s objective requires no modification to the imitation learning methodology. We hope this addresses the reviewer's primary concern and we thank the reviewer again for their engagement in this discussion! We will include these experiments in the paper.
| Policy | Mean CAS Utility | Percent improvement over greedy |
|---|---|---|
| Greedy (argmax CAS) | 447.9 | — |
| Learned potentials | 519.3 | +15.9% |
| Omniscient | 545.6 | +21.8% |
Author response (1/2)
We thank Reviewer sogr for their thoughtful review and particularly appreciate their recognition that our approach is clever and sound, computationally efficient, and achieves strong empirical results. We believe that some of the reviewer's central concerns are addressed by ablations that are already in the submitted manuscript and the rest are addressed by additional experiments we conducted in response to these reviews that we received from ICLR. We clarify these results below and encourage the reviewer to reconsider their assessment in light of this evidence. We are also happy to rapidly conduct additional experiments should the reviewer want further evidence.
The requested ablation is included in Appendix K.2, Table A5. We explicitly compare training on the original historical trajectory (January–March 2019) against training on semi-synthetic augmented trajectories, while evaluating both on the same held-out historical event trajectories (April–December 2019). One important clarification is that the data augmentation is used exclusively for training and we always evaluate on the real, unseen trajectory of events (not augmented data).
The results of this ablation show that training with data augmentation improves the learned neural network policies. Table A5 also reports the corresponding results for CAS, linear potentials, and the smaller neural network, including configurations where augmentation does not improve performance.
To address the reviewer's concern, we conducted an additional sensitivity analysis introducing increasing levels of noise (ϵ) into the predicted match utilities. Given an ϵ, we randomly sample a value in [−ϵ,ϵ] and add it to the utility estimate supplied to the algorithms. The evaluation uses the prediction without noise to determine the PLYG. We provide the average PLYG across 9 simulations per ϵ in the table below.
The results show that learned linear potentials can retain their advantage over myopic allocation even with substantial noise in the utility estimates. Our SVM potentials outperforms myopic matching at every tested noise level. The more expressive NN(34) performs best without noise but is indeed more sensitive to larger perturbations. Note that above ϵ=20, the utility estimate is practically uniform random since graft survival is typically no longer than 20-30 years.
These results give additional evidence that the benefits of learned potentials are not exclusively dependent on perfectly accurate utility estimates and degrade gracefully as noise is injected.
See also the responses to the other reviewers for additional comments and new experiments.