acceptodds
Under review as a conference paper at ICLR 2027

Survival Policy Learning as Probabilistic Inference

Abstract

Learning optimal treatment policies that maximize survival probabilities from longitudinal observational data is a central problem in causal inference for medicine. Existing applications of reinforcement learning (RL) to longitudinal medical data typically adapt standard RL algorithms under a Markov decision process formulation and rely on heuristic rewards specified by clinicians or auxiliary models, thereby weakening the scientific validity and interpretability of the learned policies. In this study, we formulate causal survival policy learning within the framework of control as inference. This formulation yields a principled dense reward which connects reward design directly to survival quantities and induces a behavior-anchored, KL-regularized policy learning objective for offline observational data. We further use an autoregressive model to learn state representations that approximate the Markov property of the sequential decision process. We evaluate the proposed method using synthetic data and semi-synthetic data based on real-world chronic disease epidemiological study, tracking individual health trajectories over a 30-year horizon. Across experiments, the proposed survival-derived reward provides an effective alternative to sparse survival rewards for survival policy learning.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, survival analysis, causal inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7Ny8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Survival Policy Learning as Probabilistic Inference, a framework for learning treatment policies that maximize long-term survival from longitudinal observational healthcare data. The problem is important, and connecting hazard modeling with offline reinforcement learning reward design is potentially interesting. However, the paper treats the control-as-inference objective with a KL regularization term (Eq. (12)) as if it were equivalent to optimizing counterfactual survival, but these objectives are not equivalent. Without the KL term, the expected cumulative log-survival reward is only a Jensen lower bound on log⁡Sπ(τ)\log S^{\pi} (\tau). Once the behavior-policy KL penalty is added, the objective is no longer a lower bound on the survival quantity; it becomes an additional behavior-regularized objective. Moreover, the causal interpretation relies on sequential ignorability and learned state sufficiency assumptions that are not sufficiently justified. The experiments are also primarily evaluated in a simulator learned from the authors' data, which raises concerns about model exploitation and distribution leakage.

Critical strengths
  1. The problem is important and relevant. Long-term survival optimization, dynamic treatment regimes, and time-varying confounding are central challenges in healthcare reinforcement learning.
  2. The behavior-policy anchoring mechanism is well motivated. Using a KL penalty to constrain the learned policy toward the behavior policy is a reasonable and important consideration for offline healthcare policy learning.
Critical weaknesses
  1. The paper conflates survival optimization with the KL-regularized control-as-inference objective.

The paper first defines the objective as maximizing final counterfactual survival:π⋆=arg⁡max⁡Sπ(tau)\pi^{\star} = \arg\max S^{\pi}(tau). It then uses Jensen's inequality to derive the lower bound on log⁡Sπ(τ)\log S^{\pi}(\tau). However, the actual objective optimized by the method is Jα(π)J_\alpha (\pi) defined in Eq. (12). After adding the KL term, optimizing Jα(π)J_\alpha (\pi) is not equivalent to maximizing log⁡Sπ(τ)\log S^{\pi}(\tau), α\alpha changes the policy objective itself, not merely the stability of offline learning, and the paper should not directly describe the method as optimizing counterfactual survival.

In addition, the current version often treats the following three objectives as interchangeable, which is an important issue and should be distinguished carefully: (1) maximizing $ S^{\pi}(\tau)$; (2) maximizing Eq∗[∑tlog⁡(1−λt)]E_{q^*}[\sum_t \log(1-\lambda_t)]; maximizing Jα(π)J_\alpha (\pi).

  1. The handling of right censoring is incomplete in the main algorithm, and the IPCW extension is not rigorously connected to Stage 2.

Section 3.7 states that uncensored at-risk transitions are used and censored transitions are masked. However, if censoring depends on future health status, simply masking transitions does not recover the full data distribution. The IPCW method is introduced only in Appendix D and is not clearly incorporated into Algorithm 1. The IPCW formulation also requires clarification, e.g., does CtC_t denote censoring at time tt, or censoring between tt and t+1t+1? After applying IPCW, does the Bellman target still correspond to Equation (12)?

Questions for authors
  1. What is the actual objective optimized by the method: final survival, expected cumulative log-survival, or the KL-regularized ELBO?
  2. In Proposition 1, is the survival evidence defined under the behavior policy πβ\pi_\beta as a fixed evidence term, or under the candidate policy π\pi as a counterfactual survival?
  3. What value of γ\gamma is used in the experiments? In proposition 4, it seems like the results only hold when γ=1\gamma =1. If γ<1\gamma<1, what causal target does the resulting objective correspond to?
  4. Why should the learned state StS_t satisfy sequential ignorability? Are there theoretical conditions or diagnostic experiments supporting representation sufficiency?
  5. Is IPCW used in the main experiments? If not, are the censoring-related identification claims limited to conditionally independent censoring? 6.What is the exact direction of the KL divergence, and what propensity clipping procedure is used?
  6. Why does log reward not consistently outperform linear reward in the CIRCS experiments? Is the main contribution intended to be dense reward design rather than a performance advantage of the logarithmic reward?
  7. Can the authors report the true simulator value, estimated value, and policy-value estimation error, rather than only the final survival rate?
2 comments

We thank the reviewer for a detailed and technically precise review.

W1. The paper first defines the objective as maximizing final counterfactual survival: π⋆=arg⁡max⁡Sπ(τ)\pi^\star = \arg\max S^\pi(\tau). It then uses Jensen's inequality to derive the lower bound on log⁡Sπ(τ)\log S^\pi(\tau). However, the actual objective optimized by the method is Jα(π)J_\alpha(\pi) defined in Eq. (12). After adding the KL term, optimizing Jα(π)J_\alpha(\pi) is not equivalent to maximizing log⁡Sπ(τ)\log S^\pi(\tau), α\alpha changes the policy objective itself, not merely the stability of offline learning, and the paper should not directly describe the method as optimizing counterfactual survival. In addition, the current version often treats the following three objectives as interchangeable, which is an important issue and should be distinguished carefully: (1) maximizing Sπ(τ)S^\pi(\tau); (2) maximizing Eq∗[∑tlog⁡(1−λt)]E_{q^*}[\sum_t \log(1-\lambda_t)]; maximizing Jα(π)J_\alpha(\pi).

Yes, we will clarify that maximizing Jα(π)J_\alpha(\pi) solves the dual problem of maximizing ESt,At∼qπ[∑tlog⁡(1−λt)]E_{S_t,A_t\sim q^\pi}[\sum_t \log(1-\lambda_t)], a lower bound of log⁡Sπ(τ)\log S^\pi(\tau) with a first-order difference in λ\lambda, within a constrained policy class Πδ={π:ESt,At∼qπ[∑tKL(π(⋅∣St)∥πβ(⋅∣St))]≤δ}\Pi^\delta=\{\pi:E_{S_t,A_t\sim q^\pi}[\sum_t KL(\pi(\cdot\mid S_t)\Vert \pi_\beta(\cdot\mid S_t))]\le \delta\}, where δ=δ(α)\delta=\delta(\alpha) is the KL budget attained by the maximizer πα\pi_\alpha of JαJ_\alpha (Lagrangian sufficiency). We clarify this in the revised manuscript (Figure 2).

W2. Section 3.7 states that uncensored at-risk transitions are used and censored transitions are masked. However, if censoring depends on future health status, simply masking transitions does not recover the full data distribution.

Our claim relies on conditionally independent censoring:

Tg⊥Cg∣St,Atfor all t and regime g,T^{g}\perp C^g\mid S_t,A_t\qquad\text{for all } t \text{ and regime } g,

where TgT^g and CgC^g are counterfactual time-to-event and time-to-censoring. We will clarify this in the revised manuscript.

W3. The IPCW method is introduced only in Appendix D and is not clearly incorporated into Algorithm 1. The IPCW formulation also requires clarification, e.g., does CtC_t denote censoring at time tt, or censoring between tt and t+1t+1? After applying IPCW, does the Bellman target still correspond to Equation (12)?

Yes, Algorithm 1 is for dataset without right-censoring. We will write the complete algorithm with right censoring in Appendix D as Alogorithm 2 in the revised manuscript.

Q1. What is the actual objective optimized by the method: final survival, expected cumulative log-survival, or the KL-regularized ELBO?

The method optimizes the KL-regularized objective Jα(π)J_\alpha(\pi), i.e., the ELBO with tempered optimality variables. Equivalently, it maximizes the expected cumulative log-survival (a lower bound on log⁡Sπ(τ)\log S^\pi(\tau)) within a KL budget δ(α)\delta(\alpha) around the behavior policy.

Q2. In Proposition 1, is the survival evidence defined under the behavior policy πβ\pi_\beta as a fixed evidence term, or under the candidate policy π\pi as a counterfactual survival?

The evidence is defined under the behavior prior πβ\pi_\beta and is a fixed constant, independent of the candidate policy. The candidate policy π\pi is the variational distribution. For every π\pi,

log⁡pπβ(O1:τ=1)=J(π)+DKL(qπ ∥ pπβ(X∣O1:τ=1)),\log p^{\pi_\beta}(O_{1:\tau}=1)=J(\pi)+D_{\mathrm{KL}}\big(q^\pi\,\Vert\,p^{\pi_\beta}(X\mid\mathcal{O}_{1:\tau}=1)\big),

so maximizing J(π)J(\pi) over π\pi minimizes the KL to the posterior, and the maximizer is the posterior policy π⋆\pi^\star.

Q3. What value of γ\gamma is used in the experiments? In proposition 4, it seems like the results only hold when γ=1\gamma = 1. If γ<1\gamma < 1, what causal target does the resulting objective correspond to?

Thank you for catching this. γ\gamma is the discount factor in the critic target (Eq. 23). We used γ=1.0\gamma=1.0 in all experiments. The reviewer is correct that Proposition 4, and the objective of Eqs. (10) and (13), hold only for γ=1\gamma=1: the objective is an undiscounted sum over a finite horizon.

Q4. Why should the learned state StS_t satisfy sequential ignorability? Are there theoretical conditions or diagnostic experiments supporting representation sufficiency?

We need the representation sufficiency for StS_t on top of untestable identifiability condition on the observed data:

  • Hazard sufficiency: Yt⊥Ht∣St,AtY_t\perp H_t\mid S_t,A_t; and
  • State sufficiency: St+1⊥Ht∣St,AtS_{t+1}\perp H_t\mid S_t,A_t; and
  • Balancing: At⊥Ht∣StA_t\perp H_t\mid S_t,

where Ht=[L1:t,A1:t−1,Y1:t−1]H_t = [L_{1:t}, A_{1:t-1}, Y_{1:t-1}] is the history up to time tt. Identification under these assumptions are described in the Appendix F of the revised manuscript.

We tested these assumptions with the CIRCS semi-synthetic data by evaluating the predictive performance gain by adding h(Ht)=Lt,Lt−1,Lt−2h(H_t)=L_t, L_{t-1}, L_{t-2}, At−1,At−2A_{t-1}, A_{t-2} to StS_t or [St,At][S_t, A_t].

TargetMetricGainVerdict
YtY_tlog-loss−0.0016 to +0.0007holds
St+1S_{t+1}nMSE+0.0080 to +0.0094supported
AtA_tlog-loss−0.0016 to +0.0011holds

Diagnostics support hazard sufficiency and balancing. Adding history reduces next-state prediction error by only 4–5%, supporting approximate state sufficiency for the tested 20 leading PCs.

Q5. Is IPCW used in the main experiments? If not, are the censoring-related identification claims limited to conditionally independent censoring?

No, IPCW is not used in any experiment, and yes, the claims are limited to conditionally independent censoring.

Q6. What is the exact direction of the KL divergence, and what propensity clipping procedure is used?

We use the reverse KL, DKL(πθ(⋅∣St)∥πβ(⋅∣St))D_{\mathrm{KL}}(\pi_\theta(\cdot\mid S_t)\Vert\pi_\beta(\cdot\mid S_t)). Training uses neither importance sampling nor IPCW, so no propensity clipping is involved.

Q7. Why does log reward not consistently outperform linear reward in the CIRCS experiments? Is the main contribution intended to be dense reward design rather than a performance advantage of the logarithmic reward?

Yes, the empirical contribution is dense reward design. We added theoretical analysis on the first-order equivalence between the log and linear rewards in Appendix H of the revised manuscript.

Q8. Can the authors report the true simulator value, estimated value, and policy-value estimation error, rather than only the final survival rate?

Yes. Table 1 is the true simulator value. We also perform 4 off-policy evaluations (OPEs): doubly robust (DR; Jiang & Li, 2016; Thomas & Brunskill, 2016), self-normalized doubly robust (SNDR; Kallus & Uehara, 2019), the direct method (DM) via fitted Q iteration (Ernst et al., 2005; Riedmiller, 2005), and weighted importance sampling (WIS; Precup et al., 2000). The following table shows bias / MAE over cells from the true simulator values.

KLCellsDRSNDRDMWIS
< 26+0.002 / 0.044+0.021 / 0.050−0.003 / 0.007+0.028 / 0.059
2 – 45+0.068 / 0.068+0.113 / 0.113+0.036 / 0.036+0.096 / 0.208
≥ 47+0.089 / 0.096+0.168 / 0.168+0.064 / 0.074+0.289 / 0.289
Reviewer_pPUkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates using a less sparse reward function than just point-survival for the task of counterfactual survival maximization using longitudinal data.

Critical strengths

None

Critical weaknesses

The paper is laid out as a 6 page theory + 1 page algorithm construction and 1 page evaluations. 95% of the theory is not novel. In fact, many of the remarks almost verbatim follow the CaI paper (Levine, 2018). Empirical evaluations compare only against generalist deep learning baselines despite the domain being extremely safety-sensitive and therefore --- admittedly: in my opinion --- probably having many many in-domain approaches. Given the lack of novelty and serious comparisons to the domain of application, I cannot recommend acceptance.

Since I am not from this field, I am not sure whether any non-semi-synthetic datasets exist or not, but I strongly suspect they do. Which would imply the empirical setup is truly lacking in depth.

We evaluate the proposed algorithm along two axes: policy performance and positivity adherence. The central empirical question is whether survival-derived dense rewards improve learning from sparse terminal survival outcomes while keeping the learned policy within the support of the observational behavior policy.

This is also telling: The authors do not even claim a proper edge on application domain, and lack novelty otherwise.

Questions for authors

The paper is essentially "X theory may work on Y domain" stage. This is not complete without actual demonstrations.

1 comment

We thank the reviewer for the review and for raising concerns about the contribution and the evaluation. We address each point below.

The paper is laid out as a 6 page theory + 1 page algorithm construction and 1 page evaluations. 95% of the theory is not novel. In fact, many of the remarks almost verbatim follow the CaI paper (Levine, 2018).

Our contribution connects CaI to causal survival policy learning in three ways:

  1. Behavior-anchored objective. Using πβ\pi_\beta as the prior yields a survival objective regularized by DKL(π∥πβ)D_{\mathrm{KL}}(\pi\Vert\pi_\beta).
  2. Positivity and identification. π⋆∝πβexp⁡(Q)\pi^\star\propto\pi_\beta\exp(Q) preserves behavior support, enforcing the action-positivity requirement for g-formula identification (Proposition 2). We connect this policy to incremental propensity interventions (Kennedy, 2019) and positivity-free policy learning (Zhao et al., 2024).
  3. Survival reward identities. We show that sparse terminal and linear intensity rewards, which can be viewed as the CaI reward for the Poisson process, recover the same counterfactual survival target, giving a principled basis for reward choices in clinical RL.

Empirical evaluations compare only against generalist deep learning baselines despite the domain being extremely safety-sensitive and therefore --- admittedly: in my opinion --- probably having many many in-domain approaches.

Yes, learning the optimal treatment policy has a long history as dynamic treatment regime (DTR) in the statistics and biostatistics community. We added four DTR baselines: censored Q-learning (Goldberg & Kosorok, 2012), dWSurv (Simoneau et al., 2020), BOWL (Zhao et al., 2015), and a doubly robust DTR learner (Sakaguchi, 2025). For the CIRCS semi-synthetic dataset, only BOWL achieved KL below 8. BA-SAC (α=0.05\alpha=0.05) achieved higher Monte Carlo survival with less deviation from the behavior policy: 0.515 ± 0.007 at KL 0.38, versus 0.499 ± 0.005 at KL 0.93 for BOWL.

Since I am not from this field, I am not sure whether any non-semi-synthetic datasets exist or not, but I strongly suspect they do. Which would imply the empirical setup is truly lacking in depth.

The fundamental problem of causal inference prevents observing counterfactual outcomes, which is necessary for policy evaluation, in real clinical data. Hence, semi-synthetic benchmarks is the best possible approach.

We evaluate the proposed algorithm along two axes: policy performance and positivity adherence. The central empirical question is whether survival-derived dense rewards improve learning from sparse terminal survival outcomes while keeping the learned policy within the support of the observational behavior policy.

This is also telling: The authors do not even claim a proper edge on application domain, and lack novelty otherwise.

The practical advantage is higher survival with less deviation from observed clinical practice than baselines.

The paper is essentially "X theory may work on Y domain" stage. This is not complete without actual demonstrations.

We derive a survival-grounded, behavior-anchored objective and demonstrate its practical value on the CIRCS semi-synthetic benchmark, achieving higher survival with lower behavior-policy KL than existing methods.

Reviewer_XYsxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a survival policy learning framework based on control as inference for longitudinal observational data. It derives a logarithmic survival reward from survival probabilities and introduces KL regularization to anchor the learned policy to the behavior policy. The method is evaluated on synthetic data and a semi-synthetic dataset based on the CIRCS epidemiological study. Experimental results suggest that survival-derived dense rewards can improve policy learning compared with sparse terminal rewards, although the logarithmic reward does not consistently outperform the linear intensity reward.

Critical strengths

The paper provides a theoretically motivated framework for survival policy learning by deriving dense rewards from survival probabilities, offering a principled alternative to sparse terminal rewards.

Critical weaknesses

The practical advantage of the proposed logarithmic survival reward remains unclear, as it does not consistently outperform the simpler linear intensity reward in the experiments.

Questions for authors

Can the authors provide additional evidence that the learned simulator accurately estimates counterfactual survival under different treatment policies, and assess how simulator misspecification may affect the reported policy comparisons?

1 comment

We thank the reviewer for the careful reading and the two concrete questions.

The practical advantage of the proposed logarithmic survival reward remains unclear, as it does not consistently outperform the simpler linear intensity reward in the experiments.

Yes, this is expected: The linear reward is a first-order approximation of the log reward, hence both objectives end up with similar policies under sufficient KL regularization when the intensity is small with high probability as in our cases. We state this claim in Appendix H of the revised manuscript.

Can the authors provide additional evidence that the learned simulator accurately estimates counterfactual survival under different treatment policies, and assess how simulator misspecification may affect the reported policy comparisons?

We apologize for the confusion in our description. In the semi-synthetic experiments the simulator is not an estimator of the true process: it is the data-generating process (DGP). We clarify this in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.