- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Survival Policy Learning as Probabilistic Inference, a framework for learning treatment policies that maximize long-term survival from longitudinal observational healthcare data. The problem is important, and connecting hazard modeling with offline reinforcement learning reward design is potentially interesting. However, the paper treats the control-as-inference objective with a KL regularization term (Eq. (12)) as if it were equivalent to optimizing counterfactual survival, but these objectives are not equivalent. Without the KL term, the expected cumulative log-survival reward is only a Jensen lower bound on . Once the behavior-policy KL penalty is added, the objective is no longer a lower bound on the survival quantity; it becomes an additional behavior-regularized objective. Moreover, the causal interpretation relies on sequential ignorability and learned state sufficiency assumptions that are not sufficiently justified. The experiments are also primarily evaluated in a simulator learned from the authors' data, which raises concerns about model exploitation and distribution leakage.
Critical strengths
- The problem is important and relevant. Long-term survival optimization, dynamic treatment regimes, and time-varying confounding are central challenges in healthcare reinforcement learning.
- The behavior-policy anchoring mechanism is well motivated. Using a KL penalty to constrain the learned policy toward the behavior policy is a reasonable and important consideration for offline healthcare policy learning.
Critical weaknesses
- The paper conflates survival optimization with the KL-regularized control-as-inference objective.
The paper first defines the objective as maximizing final counterfactual survival:. It then uses Jensen's inequality to derive the lower bound on . However, the actual objective optimized by the method is defined in Eq. (12). After adding the KL term, optimizing is not equivalent to maximizing , changes the policy objective itself, not merely the stability of offline learning, and the paper should not directly describe the method as optimizing counterfactual survival.
In addition, the current version often treats the following three objectives as interchangeable, which is an important issue and should be distinguished carefully: (1) maximizing $ S^{\pi}(\tau)$; (2) maximizing ; maximizing .
- The handling of right censoring is incomplete in the main algorithm, and the IPCW extension is not rigorously connected to Stage 2.
Section 3.7 states that uncensored at-risk transitions are used and censored transitions are masked. However, if censoring depends on future health status, simply masking transitions does not recover the full data distribution. The IPCW method is introduced only in Appendix D and is not clearly incorporated into Algorithm 1. The IPCW formulation also requires clarification, e.g., does denote censoring at time , or censoring between and ? After applying IPCW, does the Bellman target still correspond to Equation (12)?
Questions for authors
- What is the actual objective optimized by the method: final survival, expected cumulative log-survival, or the KL-regularized ELBO?
- In Proposition 1, is the survival evidence defined under the behavior policy as a fixed evidence term, or under the candidate policy as a counterfactual survival?
- What value of is used in the experiments? In proposition 4, it seems like the results only hold when . If , what causal target does the resulting objective correspond to?
- Why should the learned state satisfy sequential ignorability? Are there theoretical conditions or diagnostic experiments supporting representation sufficiency?
- Is IPCW used in the main experiments? If not, are the censoring-related identification claims limited to conditionally independent censoring? 6.What is the exact direction of the KL divergence, and what propensity clipping procedure is used?
- Why does log reward not consistently outperform linear reward in the CIRCS experiments? Is the main contribution intended to be dense reward design rather than a performance advantage of the logarithmic reward?
- Can the authors report the true simulator value, estimated value, and policy-value estimation error, rather than only the final survival rate?
2 comments
Q4. Why should the learned state satisfy sequential ignorability? Are there theoretical conditions or diagnostic experiments supporting representation sufficiency?
We need the representation sufficiency for on top of untestable identifiability condition on the observed data:
- Hazard sufficiency: ; and
- State sufficiency: ; and
- Balancing: ,
where is the history up to time . Identification under these assumptions are described in the Appendix F of the revised manuscript.
We tested these assumptions with the CIRCS semi-synthetic data by evaluating the predictive performance gain by adding , to or .
| Target | Metric | Gain | Verdict |
|---|---|---|---|
| log-loss | −0.0016 to +0.0007 | holds | |
| nMSE | +0.0080 to +0.0094 | supported | |
| log-loss | −0.0016 to +0.0011 | holds |
Diagnostics support hazard sufficiency and balancing. Adding history reduces next-state prediction error by only 4–5%, supporting approximate state sufficiency for the tested 20 leading PCs.
Q5. Is IPCW used in the main experiments? If not, are the censoring-related identification claims limited to conditionally independent censoring?
No, IPCW is not used in any experiment, and yes, the claims are limited to conditionally independent censoring.
Q6. What is the exact direction of the KL divergence, and what propensity clipping procedure is used?
We use the reverse KL, . Training uses neither importance sampling nor IPCW, so no propensity clipping is involved.
Q7. Why does log reward not consistently outperform linear reward in the CIRCS experiments? Is the main contribution intended to be dense reward design rather than a performance advantage of the logarithmic reward?
Yes, the empirical contribution is dense reward design. We added theoretical analysis on the first-order equivalence between the log and linear rewards in Appendix H of the revised manuscript.
Q8. Can the authors report the true simulator value, estimated value, and policy-value estimation error, rather than only the final survival rate?
Yes. Table 1 is the true simulator value. We also perform 4 off-policy evaluations (OPEs): doubly robust (DR; Jiang & Li, 2016; Thomas & Brunskill, 2016), self-normalized doubly robust (SNDR; Kallus & Uehara, 2019), the direct method (DM) via fitted Q iteration (Ernst et al., 2005; Riedmiller, 2005), and weighted importance sampling (WIS; Precup et al., 2000). The following table shows bias / MAE over cells from the true simulator values.
| KL | Cells | DR | SNDR | DM | WIS |
|---|---|---|---|---|---|
| < 2 | 6 | +0.002 / 0.044 | +0.021 / 0.050 | −0.003 / 0.007 | +0.028 / 0.059 |
| 2 – 4 | 5 | +0.068 / 0.068 | +0.113 / 0.113 | +0.036 / 0.036 | +0.096 / 0.208 |
| ≥ 4 | 7 | +0.089 / 0.096 | +0.168 / 0.168 | +0.064 / 0.074 | +0.289 / 0.289 |
We thank the reviewer for a detailed and technically precise review.
Yes, we will clarify that maximizing Jα(π) solves the dual problem of maximizing ESt,At∼qπ[∑tlog(1−λt)], a lower bound of logSπ(τ) with a first-order difference in λ, within a constrained policy class Πδ={π:ESt,At∼qπ[∑tKL(π(⋅∣St)∥πβ(⋅∣St))]≤δ}, where δ=δ(α) is the KL budget attained by the maximizer πα of Jα (Lagrangian sufficiency). We clarify this in the revised manuscript (Figure 2).
Our claim relies on conditionally independent censoring:
Tg⊥Cg∣St,Atfor all t and regime g,
where Tg and Cg are counterfactual time-to-event and time-to-censoring. We will clarify this in the revised manuscript.
Yes, Algorithm 1 is for dataset without right-censoring. We will write the complete algorithm with right censoring in Appendix D as Alogorithm 2 in the revised manuscript.
The method optimizes the KL-regularized objective Jα(π), i.e., the ELBO with tempered optimality variables. Equivalently, it maximizes the expected cumulative log-survival (a lower bound on logSπ(τ)) within a KL budget δ(α) around the behavior policy.
The evidence is defined under the behavior prior πβ and is a fixed constant, independent of the candidate policy. The candidate policy π is the variational distribution. For every π,
logpπβ(O1:τ=1)=J(π)+DKL(qπ∥pπβ(X∣O1:τ=1)),
so maximizing J(π) over π minimizes the KL to the posterior, and the maximizer is the posterior policy π⋆.
Thank you for catching this. γ is the discount factor in the critic target (Eq. 23). We used γ=1.0 in all experiments. The reviewer is correct that Proposition 4, and the objective of Eqs. (10) and (13), hold only for γ=1: the objective is an undiscounted sum over a finite horizon.