acceptodds
Under review as a conference paper at ICLR 2027

Hybrid RL for Optimal Dynamic Treatment Regimes: Impact of Unmeasured Confounding

Abstract

Hybrid Reinforcement Learning (RL), where an agent learns from both an offline dataset and online explorations in an unknown environment, has attracted significant recent interest. Existing hybrid RL studies have largely focused on two cases for the offline data-generating process, either imposing no restriction on the degree of unmeasured confounding or assuming that such confounding is entirely absent. Many applications, however, lie between these extremes, with unmeasured confounding present but bounded in magnitude. In this paper, we investigate how the degree of unmeasured confounding in the offline data impacts online learning for optimal dynamic treatment regimes. We show that the offline data never hurt the guarantee of the purely online setting. Moreover, the guarantee improves monotonically as the assumed degree of unmeasured confounding decreases. When the offline data are fully unconfounded, the guarantee is at least as strong as those in the purely online, unrestricted, and bounded unmeasured confounding settings.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Dynamic Treatment Regimes, Unmeasured Confounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jjUYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies offline-to-online reinforcement learning for dynamic treatment regimes (DTRs). It considers a setting in which the agent has access to a potentially confounded offline dataset, which is used to guide initial treatment decisions and facilitate policy refinement through online exploration as new patients arrive. The author(s) considers three scenarios corresponding to different degrees of unmeasured confounding in the offline data: (i) unrestricted confounding, (ii) bounded confounding, and (iii) no unmeasured confounding. They develop algorithms and establish regret guarantees for each scenario. Methodologically, the paper extends existing approaches to offline-to-online learning for DTRs from model-based to model-free methods, and from tabular representations to linear function approximation.

Critical strengths

The paper has several critical strengths:

  • Methodologically, existing offline-to-online approaches to DTRs are model-based and rely on tabular representations. Both can be restrictive in DTR settings, where the Markov property may not hold and the history-action space can grow rapidly.

  • Theoretically, the paper extends existing results by covering a more comprehensive range of scenarios regarding unmeasured confounding, including (i) unrestricted confounding, (ii) bounded confounding, and (iii) no unmeasured confounding. The authors establish regret guarantees for all three settings. The theoretical results appear sound, and I did not spot any major issues.

Critical weaknesses

Despite these strengths, the paper has several limitations. In my view, the main concern is the limited novelty beyond existing work of Zhang and Bareinboim (2019, 2020). Let me elaborate.

First, the paper's empirical evaluation is limited, relying on tabular environments despite its emphasis on linear function approximation. Moreover, the comparison does not include existing approaches. Thus, the paper's contributions are primarily methodological and theoretical. From a methodological perspective, extending the model-based, tabular approaches of Zhang and Bareinboim (2019, 2020) to model-free learning with linear function approximation is meaningful. However, the central idea closely follows these earlier works. Specifically, the proposed algorithm selects treatments by maximizing an optimistic Q-function estimate, constructed by taking the minimum of an online upper confidence bound and an offline causal upper bound. This follows essentially the same principle as Zhang and Bareinboim (2019, 2020), with the differences lying in how the Q-functions are constructed.

Second, from a theoretical perspective, while the paper considers a broader range of confounding scenarios, the structure of the regret guarantees closely follows that of Zhang and Bareinboim (2019, 2020). In particular, the regret bounds consist of an exploration term and the minimum of two quantities: one corresponding to the regret of purely online learning and the other reflecting the information gained from the offline data. This basic structure has already been established in Zhang and Bareinboim (2019, 2020).

Finally, the paper does not establish minimax lower bounds to demonstrate the sharpness of its regret upper bounds.

Questions for authors

Please refer to the weaknesses above.

3 comments

Thank you for your careful reading. We appreciate your time and feedback. Below, we address the main comments provided in the review.

W1 Limited novelty beyond existing work of Zhang and Bareinboim (2019, 2020).

We illustrate the novelty compared with (Zhang & Bareinboim, 2019, 2020) from four angles below.

  1. Problem setting.

    Those prior works on hybrid DTR learning (Zhang & Bareinboim, 2019, 2020) only considered the worst case of the offline data being arbitrarily (i.e., unrestricted) confounded (Γ=∞\Gamma=\infty). They also only considered the impractical setting of unlimited offline data. They did not handle any of the uncertainty that comes from working with finite offline data.

    In many applications, such as clinical trials, the unmeasured confounding is present but bounded in magnitude. In particular, Γ=1\Gamma=1 for randomized controlled trials (the best case). We study three regimes: Γ=1\Gamma=1 for offline data from randomized controlled trials (best case), Γ=∞\Gamma=\infty for (worst case), and the intermediate regime of bounded confounding (Γ∈(1,∞)\Gamma\in(1,\infty)).

    For all three regimes, we consider the data is finite and handle the resulting uncertainty.

  2. Methodology & Technical challenges

    (Zhang & Bareinboim, 2019, 2020) are model-based and specifically for the tabular setting: they estimate all transition kernels and the reward distribution over histories. Our work is model-free and for the more general linear setting. We consider learning the QQ-functions with linear value function approximation. There are two key technical challenges:

    (1). Considering the linear setting for hybrid DTR learning is not direct, even under unrestricted confounding (Γ=∞\Gamma=\infty), despite being standard in RL/MDPs.

    • A DTR is non-Markovian: imposing linearity directly on the transition law and the terminal mean outcome, as is done in MDPs (Jin et al., 2020), in the DTR setting forces dk≥∏i=1k∣Si∣∣Ai∣d_k\ge\prod_{i=1}^{k}\lvert\mathcal S_i\rvert\lvert\mathcal A_i\rvert for every k<Kk < K (Lemma 1).
    • For online DTR learning, (Hu & Kallus, 2022) assume that the optimal QQ-functions are linear and impose a margin condition. When their setting is extended to KK stages with ∣A∣\lvert\mathcal A\rvert treatments at every stage, the margin condition requires O(∣A∣K)\mathcal O(\lvert\mathcal A\rvert^{K}) inequalities to hold which grows exponentially with the horizon KK.

    We adapt the optimistic closure condition of (Wang et al., 2021b) to our DTR setting, imposing it over a specified class of value functions, permitting smaller feature dimensions.

    (2). Under bounded unmeasured confounding (Γ∈(1,∞)\Gamma\in(1,\infty)), which (Zhang & Bareinboim, 2019, 2020) did not consider, we derive new causal bounds (Proposition 6). The main difficulty is the identification uncertainty: the unmeasured variables UU affect the behavior policy πb,k(ak∣sˉk,aˉk−1,u)\pi_{b,k}(a_k\mid\bar s_k,\bar a_{k-1},u) at every stage, so the uncertainty that the confounding adds to the causal bounds multiplies across stages. For unrestricted confounding (Γ=∞\Gamma=\infty), the identification uncertainty has a simpler form. It depends only on probabilities of the offline data distribution, so it can be estimated directly from the offline data and requires no bounds on how the unmeasured variables affect the treatment.

    (3). We consider finite offline data. The causal bounds must account for estimation uncertainties. Our estimated causal bounds depend on the estimated propensities π^b,k(ak∣sˉk,aˉk−1)\hat\pi_{b,k}(a_k\mid\bar s_k,\bar a_{k-1}) of all stages. Therefore, the estimation errors of these propensities compound, amplified by a factor of O(Γk)\mathcal O(\Gamma k) at stage kk, and if handled in a straightforward manner can make the estimated causal bounds vacuous/trivial. We handle this in Proposition 3.

  1. Theoretical Results

    (1). We derive new causal bounds on the transition probabilities Paˉk(sk+1∣sˉk)P_{\bar a_k}(s_{k+1}\mid\bar s_k) and the expected outcome EaˉK[Y∣sˉK]\mathbb E_{\bar a_K}[Y\mid\bar s_K] under bounded confounding (Proposition 6). For the Γ=∞\Gamma=\infty setting, our bounds recover those of (Zhang & Bareinboim, 2019, Theorem 5 and Corollary 1), which are known to be tight. For every Γ<∞\Gamma < \infty, our bounds are strictly tighter than Γ=∞\Gamma=\infty. At Γ=1\Gamma=1, our bounds match the point-identified g-formula (and thus are tight). To our knowledge, Proposition 6 gives the first causal bounds on the stage-wise interventional transition probabilities Paˉk(sk+1∣sˉk)P_{\bar a_k}(s_{k+1}\mid\bar s_k) and mean outcomes EaˉK[Y∣sˉK]\mathbb E_{\bar a_K}[Y\mid\bar s_K] of a (non-Markovian) DTR under bounded confounding (Γ∈(1,∞)\Gamma\in(1,\infty)).

    See Figure 4 in the updated paper for a comparison of the causal bounds of Proposition 6 with those of Zhang & Bareinboim (2019), with Γtrue=1\Gamma_{\mathrm{true}}=1. The table below reports the bounds plotted in Figure 4.

    Γ\GammaTransition Pa1(s2∣s1)P_{a_1}(s_2\mid s_1): BoundWidthOutcome Eaˉ2[Y∣sˉ2]\mathbb E_{\bar a_2}[Y\mid\bar s_2]: BoundWidth
    1[0.7158, 0.7158]0.0000[0.2716, 0.2716]0.0000
    1.25[0.6511, 0.7967]0.1456[0.2039, 0.3661]0.1622
    1.5[0.6079, 0.8443]0.2364[0.1691, 0.4717]0.3026
    2[0.5540, 0.8443]0.2903[0.1420, 0.6191]0.4771
    3[0.5001, 0.8443]0.3442[0.1173, 0.6854]0.5681
    5[0.4569, 0.8443]0.3874[0.0992, 0.7339]0.6347
    10[0.4245, 0.8443]0.4198[0.0866, 0.7676]0.6810
    ∞\infty[0.3922, 0.8443]0.4521[0.0749, 0.7991]0.7242

    (2). Our regret bounds are tighter (See our response to "W2" for details). The regret bounds of (Zhang & Bareinboim, 2019, 2020) depend on the domains of states and actions, growing exponentially with the number of stages. Our regret bounds differ in four respects: (i) ours depend on the feature dimension. (ii) Our results account for the uncertainty of finite offline samples. (iii) We cover bounded unmeasured confounding and the regret bound is never worse than under unrestricted confounding. (iv) We also study the fully unconfounded offline data, and the regret bound is never worse than under unrestricted or bounded confounding.

  2. Empirical results

    (1). Baseline. We include the methods of (Zhang & Bareinboim, 2019) and (Hu & Kallus, 2022) as baselines. For (Hu & Kallus, 2022), we use the authors' replication code. For (Zhang & Bareinboim, 2019), we use a publicly available third party implementation. The implementation of (Zhang & Bareinboim, 2020) is not publicly available at the time of this work.

    (2). We update synthetic experiments with tabular features. See Figure 2 in the updated paper.

    (3). We also construct a DTR environment calibrated to the simulated data set BmiData (Linn et al., 2015), in which Assumption 1 holds with a low-dimensional feature map (d1=8d_1=8, d2=16d_2=16, against 64 and 512 for tabular features). See Appendix H.2 BmiData Experiments with Low-Dimensional Features in the revised PDF for details.

    We report the cumulative regret and standard deviation of all methods (including baselines) at Noff∈5×102,5×103,5×104,5×105N_{\mathrm{off}} \in \\{5 \times 10^2, 5 \times 10^3, 5 \times 10^4, 5 \times 10^5\\} (see Figure 3 in the updated paper). The regret gap between our hybrid methods and Causal UC-DTR (Zhang & Bareinboim, 2019) is substantially larger in the BmiData experiment than in the synthetic experiment. In the BmiData experiment, our methods use linear function approximation with feature maps ψ1(s1,a1)\psi_1(s_1,a_1) and ψ2(sˉ2,aˉ2)\psi_2(\bar{s}_2,\bar{a}_2) of dimensions 8 and 16. Causal UC-DTR (Zhang & Bareinboim, 2019) uses a tabular representation with 64 and 512 state-action pairs at the two stages. The table below reports the bounds plotted in Figure 3 for Noff=500000N_{\mathrm{off}} = 500000.

    MethodT = 2,0004,0006,0008,00010,000
    Purely online18.3 ± 2.621.9 ± 3.024.0 ± 3.226.0 ± 3.627.7 ± 3.9
    Unrestricted conf.14.1 ± 2.817.7 ± 3.420.0 ± 3.821.8 ± 4.123.6 ± 4.6
    Bounded conf.11.6 ± 3.114.9 ± 3.617.3 ± 3.919.0 ± 4.120.6 ± 4.5
    Fully unconf.0.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.0
    Causal UC-DTR (Zhang & Bareinboim, 2019)114.6 ± 4.0191.3 ± 7.1256.5 ± 10.2320.8 ± 9.7379.9 ± 10.6
    DTR bandit (Hu & Kallus, 2022)47.1 ± 9.075.2 ± 15.4101.6 ± 20.9125.8 ± 26.6149.9 ± 31.8
    • Linn, K. A., Laber, E. B., and Stefanski, L. A. iqLearn: Interactive Q-learning in R. Journal of Statistical Software, 2015.

W2 No minimax lower bounds.

For fully unconfounded offline data, the bound in Theorem 3 can be written as O~(TK+dKKTmax⁡dK,γoffT+Noff)\widetilde{\mathcal O}\Big(\sqrt{TK}+\frac{d_K K T\sqrt{\max\\{d_K,\gamma_{\mathrm{off}}\\}}}{\sqrt{T+N_{\mathrm{off}}}}\Big) (see last row in table below). It matches the lower bound Ω(dKTT+Noff)\Omega\Big(\frac{dKT}{\sqrt{T+N_{\mathrm{off}}}}\Big) for hybrid linear MDP (Zhang & Sinclair, 2026) up to a factor max⁡dK,γoff\sqrt{\max\\{d_K,\gamma_{\mathrm{off}}\\}}, and for a single stage (K=1K=1) it matches the lower bound Ω(dTT+Noff)\Omega\Big(\frac{dT}{\sqrt{T+N_{\mathrm{off}}}}\Big) for hybrid linear MAB (Chandak et al., 2026) up to the same factor. The lower bounds of both (Zhang & Sinclair, 2026) and (Chandak et al., 2026) are stated under coverage conditions: they hold only when the offline data cover all feature directions well. Suppose the offline data have good coverage (e.g., γoff=O(d)\gamma_{\mathrm{off}}=\mathcal O(d)) for our fully unconfounded setting, then specializing Theorem 3 to MDPs then gives O~(TK+d3/2KTT+Noff)\widetilde{\mathcal O}\Big(\sqrt{TK}+\frac{d^{3/2}KT}{\sqrt{T+N_{\mathrm{off}}}}\Big), which just have an extra d\sqrt{d} term.

Under unrestricted and bounded confounding, Theorems 1 and 2 matches the dependence on TT of the lower bounds in (Zhang & Sinclair, 2026; Chandak et al., 2026). For the dependence on NoffN_{\mathrm{off}}, the offline widths shrink with NoffN_{\mathrm{off}} only down to the width of the causal bounds, that is, Wkoff=Wk+O~(1/Noff)W_k^{\mathrm{off}} = W_k + \widetilde{\mathcal O}(1/\sqrt{N_{\mathrm{off}}}) and Wk,offΓ=Wk(Γ)+O~(1/Noff)W_{k,\mathrm{off}}^{\Gamma} = W_k(\Gamma) + \widetilde{\mathcal O}(1/\sqrt{N_{\mathrm{off}}}). Here WkW_k and Wk(Γ)W_k(\Gamma) are the widths of the causal bounds on the optimal QQ-function (Q∗Q^*) under unrestricted and bounded confounding, respectively. They do not depend on NoffN_{\mathrm{off}}. This is because the optimal QQ-function is only partially identified, and no amount of offline data removes the width WkW_k or Wk(Γ)W_k(\Gamma) (Γ>1\Gamma > 1) (see Remarks 1 and 2 in the paper). Neither (Zhang & Sinclair, 2026) nor (Chandak et al., 2026) considers unmeasured confounding in the offline data.

Table 1: Regret bounds in related settings. Notation: d,dKd, d_K: feature dimension; KK: horizon; TT: number of online episodes; NoffN_{\mathrm{off}}: number of offline samples (trajectories); γoff\gamma_{\mathrm{off}} offline coverage coefficients; S\mathcal S, A\mathcal A: domains of the state and action. † The lower bound in (Zhang & Bareinboim, 2019, Theorem 3) is for the purely online setting, without offline data. ‡ For the Γ=1\Gamma=1 case (e.g. randomized control trial data), our Algorithm 2's regret bound (Theorem 3) can be rewritten/simplified to a form (see Remark 5 in the updated paper) that enables simpler comparison with lower bounds from the hybrid MAB/MDP setting.

PaperSettingModelRegret boundUnmeasured confounding
Chandak et al. (2026)Hybrid MABLinearΩ(dTT+Noff)\Omega\big(\frac{dT}{\sqrt{T+N_{\mathrm{off}}}}\big)None
Zhang & Sinclair (2026)Hybrid MDPLinear mixtureΩ(dKTT+Noff)\Omega\big(\frac{dKT}{\sqrt{T+N_{\mathrm{off}}}}\big)None
Zhang & Bareinboim (2019)Online DTRTabularΩ(∣S∣,∣A∣,T)\Omega\big(\sqrt{\lvert\mathcal S\rvert\\,\lvert\mathcal A\rvert\\, T}\big)N/A (online) †
Corollary 1Online DTRLinearO~(dK3/2KT)\widetilde{\mathcal O}\big(d_K^{3/2} K\sqrt{T}\big)N/A
Theorem 1Hybrid DTRLinearO~(TK+min⁡dK3/2KT,T∑kWkoff)\widetilde{\mathcal O}\big(\sqrt{TK} + \min\\{d_K^{3/2}K\sqrt{T},\\ T\sum_k W_k^{\mathrm{off}}\\}\big)Unrestricted
Theorem 2Hybrid DTRLinearO~(TK+min⁡dK3/2KT,T∑kWk,offΓ)\widetilde{\mathcal O}\big(\sqrt{TK} + \min\\{d_K^{3/2}K\sqrt{T},\\ T\sum_k W_{k,\mathrm{off}}^{\Gamma}\\}\big)Bounded
Theorem 3Hybrid DTRLinearO~(TK+min⁡dK3/2KT,dKKTγoff/Noff,T∑kWk,offΓ=1)\widetilde{\mathcal O}\big(\sqrt{TK} + \min\\{d_K^{3/2}K\sqrt{T},\\ d_K K T\sqrt{\gamma_{\mathrm{off}}/N_{\mathrm{off}}},\\ T\sum_k W_{k,\mathrm{off}}^{\Gamma=1}\\}\big)None
Theorem 3 ‡Hybrid DTRLinearO~(TK+dKKTmax⁡dK,γoffT+Noff)\widetilde{\mathcal O}\Big(\sqrt{TK} + \frac{d_K K T\sqrt{\max\\{d_K,\gamma_{\mathrm{off}}\\}}}{\sqrt{T+N_{\mathrm{off}}}}\Big)None
  • Zhang, Z. and Sinclair, S. R. Offline-online reinforcement learning for linear mixture MDPs. In Advances in Neural Information Processing Systems, 2026.

  • Chandak, K., Tkachuk, V., Tan, X., and Szepesvári, C. Gaussian-width bounds for offline-to-online linear bandits. arXiv preprint arXiv:2610.12990, 2026.

Reviewer_fhkEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper focuses on learning optimal dynamic treatment regime (DTR) policies, where a treatment path (the policy at each stage) leads to a sequence of states and a terminal outcome. Moreover they focus on a setting where there exists a potentially confounded offline dataset with access to online interaction for T episodes, the latter for which they provide regret guarantees. The key assumption they make is linearity on the QQ function (in particular the Kth stage QQ function, along with Bellman completeness prior to KK). This delivers linearity in the QQ function across all stages and hence a bound that scales at dK32Kd_K^\frac{3}{2} K versus ∏k∣Sk∣∣Ak∣\prod_k |S_k||A_k| where dKd_K is the dimension of the feature map at stage KK. Moreover they show that access to the offline data, no matter how confounded (spanning unrestricted, bounded, unconfounded), never worsens the regret guarantee.

Critical strengths

The paper extends Zhang & Bareinboim (2019) in two ways. First, it handles finite offline data. The prior work treats the causal bounds as exact inputs, which amounts to assuming infinite offline data. Here the estimation uncertainty in those bounds enters the analysis explicitly through the O(1/Noff)O(1/\sqrt{N_{\text{off}}}) term in WkoffW_k^{\text{off}}. Second, it covers the full range of confounding levels, from unrestricted, through bounded, to unconfounded, where prior work considers only the worst case. The Γ\Gamma-sensitivity parameterization is especially useful: practitioners can reason about Γ\Gamma directly and use it to assess how sensitive the bounds, and hence the regret, are to the assumed degree of confounding.

Critical weaknesses

Strength of Assumption 1(ii). Proposition 1 shows that optimistic closure is strictly weaker than the linear DTR assumption, but the witness used (Appendix C) is degenerate: constant features with d1=d2=1d_1 = d_2 = 1, S2S_2 uniform and Y∼Bernoulli(1/2)Y \sim \text{Bernoulli}(1/2) under every intervention, so nothing depends on the history or the treatment. The paper gives no meaningful instance in which Assumption 1(ii) holds with dK≪∏k∣Sk∣∣Ak∣d_K \ll \prod_k |\mathcal S_k||\mathcal A_k|; the only other instance is tabular (Fact 1), where dK=∏k∣Sk∣∣Ak∣d_K = \prod_k |\mathcal S_k||\mathcal A_k|. Natural low-dimensional instances do seem to exist, but they come from a specific kind of structure. Suppose the history enters the dynamics only through a low-cardinality summary zkz_k, so that zk+1z_{k+1} depends only on (zk,ak)(z_k, a_k) and YY only on (zK,aK)(z_K, a_K), and take ψk\psi_k to be one-hot over (zk,ak)(z_k, a_k). Every function in Qk+1enl\mathcal Q_{k+1}^{\text{enl}} is then a function of (zk+1,ak+1)(z_{k+1}, a_{k+1}), its backup is a function of (zk,ak)(z_k, a_k), and closure holds with dk=∣Z∣∣A∣d_k = |\mathcal Z||\mathcal A|. In such cases the improvement over tabular methods comes from assuming the DTR is Markov in a compressed state. That is precisely the structure DTRs are usually defined to avoid; the paper itself notes that DTRs are non-Markovian even without unmeasured confounding. Under unmeasured confounding, a shared UU makes every past state informative about UU, which will generally break such compression. When the features are not one-hot, the polytope term sup⁡ϑ∈P⟨ψk,ϑ⟩\sup_{\vartheta \in \mathfrak P} \langle \psi_k, \vartheta \rangle produces nonlinear functions of ψk\psi_k, and closure over all P∈Pk\mathfrak P \in \mathfrak P_k effectively requires the dynamics to be linear in feature space. Either way, it remains unclear which realistic DTRs satisfy Assumption 1(ii) with dKd_K small enough for the claimed gains to materialize.

Table 1 comparison overclaims. Contribution (ii) says Theorem 1 improves on existing hybrid DTR results in its dependence on the cardinality of the state and action spaces, but this only holds when dK≪∏k∣Sk∣∣Ak∣d_K \ll \prod_k |\mathcal S_k||\mathcal A_k|. The authors acknowledge this in Related Work ("for low-dimensional settings", line 099), but the qualifier is missing from the contributions and from Table 1. In the one setting where both results apply directly, the tabular instantiation of Fact 1, the features are one-hot over histories, so dK=∏k=1K∣Sk∣∣Ak∣d_K = \prod_{k=1}^K |\mathcal S_k||\mathcal A_k|. Theorem 1's online term then becomes (∏k∣Sk∣∣Ak∣)3/2KT(\prod_k |\mathcal S_k||\mathcal A_k|)^{3/2} K \sqrt{T}, which is worse than the T∏k∣Sk∣∣Ak∣\sqrt{T \prod_k |\mathcal S_k||\mathcal A_k|} of Zhang & Bareinboim (2019) by a factor of roughly K∏k∣Sk∣∣Ak∣K \prod_k |\mathcal S_k||\mathcal A_k|. The result trades off a cardinality dependence for a structural assumption, which is not a free win. The comparison also omits Zhang & Bareinboim (2020), whose bound sums T∣Dk∣\sqrt{T |\mathcal D_k|} over the reduced state set S′\mathbf S' rather than taking a product over stages, and which is the stronger tabular baseline.

Offline data helps only in a narrow regime. Because the bound in Theorem 1 is min⁡{dK3/2KT,T∑kWkoff}\min \lbrace d_K^{3/2} K \sqrt{T}, \quad T \sum_k W_k^{\text{off}} \rbrace, the offline branch binds only when ∑kWkoff≲dK3/2K/T\sum_k W_k^{\text{off}} \lesssim d_K^{3/2} K / \sqrt{T}. Under unrestricted confounding, Wkoff=Wk+O~(1/Noff)W_k^{\text{off}} = W_k + \tilde O(1/\sqrt{N_{\text{off}}}), where the width WkW_k of the causal bounds cannot be reduced by collecting more offline data (Remark 1). So unless the causal bounds are tight (Wk=0W_k = 0), there is a TT beyond which the online branch always binds, and offline data improves the guarantee only at finite TT. The same holds for Theorem 2, with Wk(Γ)W_k(\Gamma) vanishing only as Γ→1\Gamma \to 1. This structure is shared with Zhang & Bareinboim (2019), whose offline term is ∥C∥1T\lVert \mathbf C \rVert_1 T, so it is not a flaw unique to this paper. Still, the "offline data never degrades the guarantee" framing is not accompanied by a characterization of when the data actually helps. Moreover, WkoffW_k^{\text{off}} is a worst-case width over all histories, so a single history with loose bounds makes the offline branch useless, even if the bounds are tight on most of the space. A finer analysis, for example one that partitions the space as in Tan et al. (2024), might show benefits that the current bound hides.

Overall assessment. The theory appears sound, and handling finite offline data across a range of confounding levels is a useful extension of Zhang & Bareinboim (2019). However, the main ingredients (optimistic LSVI, optimistic closure, the causal bounds of Zhang & Bareinboim, and taking the minimum of online and offline branches) are largely established, and the paper's key claimed improvement, the reduced dependence on state-action cardinality, holds only when dK≪∏k∣Sk∣∣Ak∣d_K \ll \prod_k |\mathcal S_k||\mathcal A_k|. Low-dimensional instances of Assumption 1(ii) appear to require the DTR to be Markov in a compressed summary of the history, which is a strong restriction in a non-Markovian setting with persistent unmeasured confounding. The benefit of offline data under confounding is also limited to finite TT. Overall I view the contribution as incremental. A convincing non-tabular example satisfying Assumption 1(ii) with small dKd_K would substantially strengthen the paper and could change my assessment.

Questions for authors
  1. Beyond the tabular case (Fact 1) and the degenerate witness for Proposition 1, can the authors give a DTR that satisfies Assumption 1(ii) with dK≪∏k∣Sk∣∣Ak∣d_K \ll \prod_k |\mathcal S_k||\mathcal A_k|, does not reduce to a Markov process in a low-cardinality summary of the history, and allows unmeasured confounding that persists across stages? If no such example exists, how should the claimed improvement in Table 1 be interpreted?

  2. The algorithm only ever uses the single estimated offline polytope C^koff\hat C_k^{\text{off}}. Could optimistic closure be required only for that set (or a small family of such sets), rather than for every P∈Pk\mathfrak P \in \mathfrak P_k? This would make Assumption 1(ii) considerably easier to justify.

  3. In what regime of dKd_K does Theorem 1 improve on both Zhang & Bareinboim (2019) and Zhang & Bareinboim (2020)? Please qualify the claim in contribution (ii) accordingly.

  4. Could the synthetic experiments (Appendix H) report the crossover point in TT beyond which the online branch of the minimum binds, and show which branch is active as Γ\Gamma and NoffN_{\text{off}} vary?

5 comments

Thank you for your careful reading. We appreciate your time and feedback. Below, we address the main comments provided in the review.

W1 Strength of Assumption 1(ii).

Q1 Beyond the tabular case and the degenerate witness for Proposition 1, can the authors give a DTR that satisfies Assumption 1(ii) with dK≪∏k∣Sk∣∣Ak∣d_K \ll \prod_k \lvert\mathcal{S}_k\rvert\lvert\mathcal{A}_k\rvert, does not reduce to a Markov process in a low-cardinality summary of the history, and allows unmeasured confounding that persists across stages?

Yes, the following is one such DTR example. The number of stages K≥3K\ge3,

  • Unmeasured variable U∼Bernoulli(1/2)U\sim\mathrm{Bernoulli}(1/2).

  • Actions. For k∈[K]k\in[K], Ak∈0,1A_k\in\\{0,1\\}.

  • States. For k∈[K]k\in[K], Sk=US_k=U with probability ρ∈(12,1)\rho\in(\frac12,1) and Sk=1−US_k=1-U otherwise.

  • Behavior policy. πb,k(Ak=1∣sˉk,aˉk−1,U)=σ(κ(2U−1))\pi_{b,k}(A_k=1\mid\bar s_k,\bar a_{k-1},U)=\sigma\bigl(\kappa(2U-1)\bigr), where κ>0\kappa > 0.

  • Outcome. Y∼Bernoulli(c,[12+AK−14(2U−1)])Y\sim\mathrm{Bernoulli}\bigl(c\\,\bigl[\frac12+\frac{A_{K-1}}4(2U-1)\bigr]\bigr), where c:=1c:=1 if AK=1A_K=1, and c:=12+12∑i=1K−1(2Si+Ai),4−ic:=\frac12+\frac12\sum_{i=1}^{K-1}(2S_i+A_i)\\,4^{-i} if AK=0A_K=0.

  • Feature mapping. Let nk:=s1+⋯+skn_k:=s_1+\dots+s_k. The feature maps are ψk:=enk⊗eak\psi_k:=e_{n_k}\otimes e_{a_k} for k<Kk < K and ψK:=c,enK⊗eaK−1\psi_K:=c\\,e_{n_K}\otimes e_{a_{K-1}}, where en∈Rk+1e_n\in\mathbb R^{k+1} and ea∈R2e_a\in\mathbb R^2 are standard basis vectors.

For the above construction, we then have (i) Assumption 1 holds with feature dimension dk=2(k+1)d_k=2(k+1) against 4k4^k for the tabular feature mapping; (ii) the DTR does not reduce to a Markov process in a low-cardinality summary of the history; (iii) persistent unmeasured confounding UU.

Real-world motivation for above DTR example. In the ExTENd trial for alcohol dependence (Lei et al., 2012), drinking was monitored during the first eight weeks of naltrexone treatment. Taking each period to be a day,

  • UU is the patient's underlying drinking propensity;

  • ak∈0,1a_k\in\\{0,1\\} indicates whether the current treatment is continued or changed (e.g., switched to a behavioral intervention);

  • sk∈0,1s_k\in\\{0,1\\} indicates whether period kk was a heavy drinking day;

  • nk=s1+⋯+skn_k=s_1+\dots+s_k is the number of heavy drinking days so far;

  • Y∈0,1Y\in\\{0,1\\} indicates no heavy drinking in the final weeks.

Heavy drinking days occur independently across periods given the patient's underlying drinking propensity, so the probability of a heavy drinking day in the next period depends on the history only through the number of heavy drinking days so far (nkn_k). As for the outcome, whether the patient avoids heavy drinking in the final weeks depends on the patient's underlying drinking propensity and treatments. The number of heavy drinking days reveals patient's underlying drinking propensity: a patient with many heavy drinking days is likely to have a high propensity. Therefore, the expected outcome depends on the number of heavy drinking days so far and the treatments. It may also depend on when these days occurred, e.g., early in the treatment or shortly before the final weeks.

  • Lei, H., Nahum-Shani, I., Lynch, K., Oslin, D., and Murphy, S. A. A "SMART" design for building individualized treatment sequences. Annual Review of Clinical Psychology, 8:21–48, 2012.

Q2 The algorithm only ever uses the single estimated offline polytope C^koff\widehat{\mathcal{C}}_k^{\mathrm{off}}. Could optimistic closure be required only for that set (or a small family of such sets), rather than for every P∈Pk\mathfrak{P} \in \mathfrak{P}_k?

No. Requiring optimistic closure only for the estimated offline confidence set C^koff\widehat{\mathcal C}_k^{\mathrm{off}} is indeed weaker. The issue is that C^koff\widehat{\mathcal C}_k^{\mathrm{off}} is random, since it is built from the offline sample. An assumption on the DTR and its feature maps should not depend on the offline sample. If Assumption 1 referred to a set built from the sample, the same DTR would satisfy it for some offline samples and violate it for others.

W2 Table 1 comparison overclaims.

Q3 In what regime of dKd_K does Theorem 1 improve on both Zhang & Bareinboim (2019) and Zhang & Bareinboim (2020)? Please qualify the claim in contribution (ii) accordingly.

For unrestricted confounding, up to logarithmic factors, Theorem 1 is smaller than that of (Zhang & Bareinboim, 2019, Theorem 7) when dK3≲∏k=1K∣Sk∣,∣Ak∣d_K^3\lesssim\prod_{k=1}^{K}\lvert\mathcal S_k\rvert\\,\lvert\mathcal A_k\rvert, and smaller than that of (Zhang & Bareinboim, 2020, Theorem 6) when K2dK3≲(∑Sk∈S′∣Dk∣)2K^2 d_K^3\lesssim\big(\sum_{S_k\in\boldsymbol{S}'}\sqrt{\lvert D_k\rvert}\big)^2. See the table below.

PaperRegretTheorem 1 is smaller if
Zhang & Bareinboim (2019)O~(KT+min⁡K∏k=1K∣Sk∣,∣Ak∣,T,∥C∥1T)\widetilde{\mathcal O}\big(K\sqrt{T}+\min\\{K\sqrt{\prod_{k=1}^{K}\lvert\mathcal S_k\rvert\\,\lvert\mathcal A_k\rvert\\,T},\\ \lVert C\rVert_1 T\\}\big)dK3≲∏k=1K∣Sk∣,∣Ak∣d_K^3\lesssim\prod_{k=1}^{K}\lvert\mathcal S_k\rvert\\,\lvert\mathcal A_k\rvert
Zhang & Bareinboim (2020)O~(∣S′∣T+∑Sk∈S′min⁡∣Dk∣T,∥Ck∥1T)\widetilde{\mathcal O}\big(\lvert\boldsymbol{S}'\rvert\sqrt{T}+\sum_{S_k\in\boldsymbol{S}'}\min\\{\sqrt{\lvert D_k\rvert T},\\ \lVert C_k\rVert_1 T\\}\big)K2dK3≲(∑Sk∈S′∣Dk∣)2K^2 d_K^3\lesssim\big(\sum_{S_k\in\boldsymbol{S}'}\sqrt{\lvert D_k\rvert}\big)^2
Theorem 1O~(TK+min⁡dK3/2KT,T∑kWkoff)\widetilde{\mathcal O}\big(\sqrt{TK}+\min\\{d_K^{3/2}K\sqrt{T},\\ T\sum_k W_k^{\mathrm{off}}\\}\big)N/A

W3 Offline data helps only in a narrow regime.

Q4 Could the synthetic experiments (Appendix H) report the crossover point in TT beyond which the online branch of the minimum binds, and show which branch is active as Γ\Gamma and NoffN_{\mathrm{off}} vary?

For the synthetic experiments, we report the crossover horizon T∗T^* for different numbers of offline trajectories Noff∈500,5,000,50,000,500,000N_{\mathrm{off}}\in\\{500,\\ 5{,}000,\\ 50{,}000,\\ 500{,}000\\}, under unrestricted confounding (Γ=∞\Gamma=\infty), bounded confounding, and fully unconfounded offline data. T∗T^* is the first number of online episodes at which the online branch of the regret bound is smaller than every branch that uses the offline data. See Figure 9, 10, 11, 12 in the updated paper. The table below reports the crossover horizon T∗T^* in the synthetic experiments for different numbers of offline trajectories NoffN_{\mathrm{off}}, under unrestricted confounding, bounded confounding and fully unconfounded offline data.

MethodNoffN_{\mathrm{off}} = 500500050000500000
Unrestricted confounding (Theorem 1)4,0964,1034,2784,638
Bounded confounding, Γ=1\Gamma=1 (Theorem 2)4,0964,1135,47915,863
Fully unconfounded (Theorem 3)4,0964,11310,132101,321

The results show that under unrestricted confounding, T∗T^* barely changes with NoffN_{\mathrm{off}}, because the widths of the causal bounds do not vanish. Under bounded confounding, T∗T^* grows as the widths shrink, and for fully unconfounded offline data it grows linearly in NoffN_{\mathrm{off}} when dKKγoff/Noff≤∑kWk,offΓ=1d_K K\sqrt{\gamma_{\mathrm{off}}/N_{\mathrm{off}}}\le\sum_k W^{\Gamma=1}_{k,\mathrm{off}}.

W4 Overall I view the contribution as incremental. A convincing non-tabular example satisfying Assumption 1(ii) with small dKd_K would substantially strengthen the paper and could change my assessment.

Please see our response to "W1" and "W2" of Reviewer jjUY for contribution, where we argue from (i) problem setting; (ii) methodology & technical challenges; (iii) theoretical results; and (iv) empirical results.

In particular, we construct a DTR environment calibrated to the simulated data set BmiData (Linn et al., 2015), in which Assumption 1 holds with a low-dimensional feature map (d1=8d_1=8, d2=16d_2=16, against 64 and 512 for tabular features). See Appendix H.2 BmiData Experiments with Low-Dimensional Features in the revised PDF for details.

We report the cumulative regret and standard deviation of all methods (including baselines) at Noff∈5×102,5×103,5×104,5×105N_{\mathrm{off}} \in \\{5 \times 10^2, 5 \times 10^3, 5 \times 10^4, 5 \times 10^5\\} (see Figure 3 in the updated paper). The regret gap between our hybrid methods and Causal UC-DTR (Zhang & Bareinboim, 2019) is substantially larger in the BmiData experiment than in the synthetic experiment. In the BmiData experiment, our methods use linear function approximation with feature maps ψ1(s1,a1)\psi_1(s_1,a_1) and ψ2(sˉ2,aˉ2)\psi_2(\bar{s}_2,\bar{a}_2) of dimensions 8 and 16. Causal UC-DTR (Zhang & Bareinboim, 2019) uses a tabular representation with 64 and 512 state-action pairs at the two stages. See the table in Common Response: Numerical Experiments above, which reports the cumulative regret plotted in Figure 3 for Noff=500000N_{\mathrm{off}} = 500000.

  • Linn, K. A., Laber, E. B., and Stefanski, L. A. iqLearn: Interactive Q-learning in R. Journal of Statistical Software, 2015.

Thank you for the detailed response and the additional experiments. Some of my concerns are resolved:

  • The corrected bounds for Zhang & Bareinboim (2019, 2020) make the comparison fair in Table 1. The explicit conditions in the response, dK3≲∏k∣Sk∣∣Ak∣d_K^3 \lesssim \prod_k |\mathcal S_k||\mathcal A_k| and K2dK3≲(∑Sk∈S′∣Dk∣)2K^2 d_K^3 \lesssim (\sum_{S_k \in \mathbf S'} \sqrt{|\mathcal D_k|})^2, are helpful. I suggest stating them in the caption of Table 1 and in contribution (ii), since the table alone still suggests an unconditional improvement.
  • Crossover analysis: The reported crossover horizons help clarify when the offline branch binds, and the behavior under unrestricted confounding matches the theory.

My remaining concerns:

  1. The new example. I don’t think it shows quite shows what Q1 asked for. The dependence of the outcome on the full history enters only through cc when AK=0A_K = 0. Since c<1c < 1 in that branch while c=1c = 1 when AK=1A_K = 1, and every function in QKenl\mathcal Q_K^{\text{enl}} is positively homogeneous in cc before clipping, the AK=1A_K = 1 branch always attains the maximum. Hence VKV_K depends only on (nK,aK−1)(n_K, a_{K-1}), and the full-history dependence never enters any backup. Moreover, the states SkS_k do not depend on the treatments, so A1,…,AK−2A_1, \dots, A_{K-2} affect the outcome only through the dominated branch and are irrelevant to the optimal policy. For every quantity the algorithm uses, the DTR is therefore Markov in the count nkn_k, which what Q1 was asking about excluding. I agree that a persistent latent trait observed through exchangeable measurements is a natural clinical structure, but the example supports the view that low-dimensional instances of Assumption 1(ii) arise from Markov-type compressibility of the value-relevant dynamics. Finally, with dK=2(K+1)d_K = 2(K+1) and a tabular product of 4K4^K, the condition dK3≲∏k∣Sk∣∣Ak∣d_K^3 \lesssim \prod_k |\mathcal S_k||\mathcal A_k| holds only for K≥6K \geq 6, whereas the paper notes K is typically small, say K=2K = 2 or 33 (first footnote). The motivating ExTENd trial reaches this regime only by treating each day as a decision stage.

  2. The BmiData experiment. With dK=16d_K = 16 against 512512 tabular state-action pairs, dK3=4096>512d_K^3 = 4096 > 512, so this experiment lies outside the regime in which Theorem 1 improves on Zhang & Bareinboim (2019) by the authors' own criterion. The empirical gains are encouraging, but not explained by the theory.

  3. Bounded confounding. The bounded-confounding row in the crossover table uses Γ=1\Gamma = 1, i.e. unconfounded data processed without pooling. Crossover horizons for intermediate confounding, e.g. Γ∈{1.5,2,4}\Gamma \in \lbrace 1.5, 2, 4 \rbrace, would show how the bounded-confounding guarantee behaves in the regime that motivates Theorem 2.

Overall I think the response improves the presentation. However, my main concern is not resolved: the improvement claimed over tabular methods requires a degree of compressibility that existing examples obtain through Markov-type structure, and neither new example falls in the improvement regime at the small KK typical of DTRs. As such, I maintain my score.

Reviewer_KLY9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies online learning of dynamic treatment regimes with access to potentially confounded offline trajectories. It combines optimistic least-squares value iteration with offline causal confidence bounds under a linear optimistic-closure assumption. The analysis covers unrestricted confounding, bounded confounding through a sensitivity parameter, and fully unconfounded data, for which an additional pooled regression estimator is introduced. Synthetic experiments illustrate the resulting regret behavior.

Critical strengths
  1. A useful treatment of observational uncertainty. The offline confidence construction accounts for both causal non-identification and finite-sample uncertainty. Separating these sources of uncertainty is consequential: additional observational data cannot generally eliminate uncertainty caused by unrestricted confounding.

  2. A coherent connection between confounding regimes. The sensitivity model connects unrestricted confounding to sequential ignorability, and the proposed regret bounds expose how tighter causal bounds can help online learning. The distinction between causal-bound clipping and pooled estimation at the unconfounded endpoint is conceptually useful.

Critical weaknesses
  1. Theorem 3's pooled-data concentration argument does not account for reuse of the offline sample. Algorithm 2 retains the offline confidence set from Proposition 3 while also inserting the offline trajectories into the Bellman regression in Equation (16). In Lemma 9, however, the covering-number argument treats the clipping function derived from that confidence set as fixed conditional on the offline sample, with no additional complexity cost (p. 49). This conditioning is legitimate for an analysis using only subsequent online observations, but it does not establish concentration over those same offline observations. Lemma 7 proves the martingale property for a fixed value function; its Equations (235)–(237) explicitly use a deterministic parameter and independence of the offline trajectories. Conditioning on the entire offline sample to fix the clipping function does not preserve this argument for the offline residuals. Lemma 15 likewise assumes a fixed clipping function. A uniform argument covering the possible offline-derived clipping functions, or an independent sample for their construction, appears necessary. This affects the confidence and optimism claims used in Theorem 3; taking the minimum with a valid online upper bound does not protect optimism if another branch can underestimate. I do not regard the current proof as establishing the claimed pooled-data guarantee.

  2. The evidence for the main extension beyond tabular, unconfounded learning remains limited. Appendix H explicitly sets the behavior-policy confounding coefficient to zero, so all experiments have true sensitivity parameter equal to one; even the bounded-confounding curve uses this unconfounded endpoint. The features are one-hot history indicators with dimensions 4 and 16. Thus, the experiments demonstrate neither learning under actual unmeasured confounding nor generalization with features smaller than the history space. These omissions matter because both are central motivations relative to the cited tabular DTR literature. The computational claim also needs qualification: Equations (4)–(7) construct bounds and constraints over all histories. Calling the final optimization a linear program does not establish scalability in feature dimension when the number of constraints can grow exponentially with the horizon. The regret result may still be statistically valuable, but the paper does not yet establish how broadly its advertised advantage is usable.

Questions for authors
  1. Clarification: Are the offline trajectories used to construct the confidence sets independent of those used in Equation (16)? If they are the same, please provide the missing uniform-concentration argument for the random clipping function in Lemma 9. If they are separate, please revise Algorithm 2 and state the sample sizes and resulting guarantee explicitly.

  2. Evidence for the confounded setting: Can you report the existing synthetic experiment with nonzero confounding and several valid sensitivity bounds greater than one? Reporting causal-bound widths alongside regret would clarify whether the proposed mechanism is informative in the setting motivating the paper.

  3. Scope of function approximation: Can you give a nontrivial treatment-selection example satisfying Assumption 1 with substantially fewer features than histories, and state the time and memory required to construct and query its offline confidence sets? This would clarify the practical significance of the dimension-dependent regret bound.

2 comments

Thank you for your careful reading. We appreciate your time and feedback. Below, we address the main comments provided in the review.

W1 Theorem 3's pooled-data concentration argument does not account for reuse of the offline sample.

Q1 Are the offline trajectories used to construct the confidence sets independent of those used in Equation (16)?

We agree that the analysis of Alg. 2 did not account for reuse of the offline sample. We will revise Alg. 2 to randomly split the offline trajectories into two disjoint sets D1off\mathcal D_1^{\mathrm{off}} and D2off\mathcal D_2^{\mathrm{off}}. The offline confidence set C^k,offΓ=1\widehat{\mathcal C}^{\Gamma=1}_{k,\mathrm{off}} is now only built from D1off\mathcal D_1^{\mathrm{off}} (Proposition 3), and only D2off\mathcal D_2^{\mathrm{off}} enters the regression in Eq. (16). Theorem 3 holds with NoffN_{\mathrm{off}} replaced by Noff/2N_{\mathrm{off}}/2. The regret bound dependence on TT, dKd_K, KK and NoffN_{\mathrm{off}} remains unchanged.

Q2 Can you report the existing synthetic experiment with nonzero confounding and several valid sensitivity bounds greater than one?

Yes. For the synthetic experiments, we are adding new results with Γtrue=1.5\Gamma_{\mathrm{true}}=1.5, i.e., the unmeasured confounding shifts the probability of treatment by at most about 0.10. It lies within the range studied by (Bruns-Smith & Zhou, 2023) in their MIMIC-III sepsis case study (Γ∈[1,2.5]\Gamma\in[1,2.5]), and is close to Γ=1.42\Gamma=1.42, which they report as the 90% quantile of the odds ratios obtained by dropping each observed covariate feature.

We run the bounded-confounding algorithm with the valid parameters Γ∈1.5,2,5,∞\Gamma\in\\{1.5,2,5,\infty\\} (Γ=∞\Gamma=\infty is the unrestricted-confounding algorithm) for Noff∈500,5000,50000,500000N_{\mathrm{off}}\in\\{500,\\ 5000,\\ 50000,\\ 500000\\}. We report the cumulative regret and the causal-bound width ∑kWk,offΓ\sum_k W^{\Gamma}_{k,\mathrm{off}}. The results show that (i) the causal bound width ∑kWk,offΓ\sum_k W^{\Gamma}_{k,\mathrm{off}} decreases as NoffN_{\mathrm{off}} grows, and is smaller for smaller Γ\Gamma; (ii) offline data help (i.e., smaller regret) under unmeasured confounding (Γtrue=1.5\Gamma_{\mathrm{true}}=1.5) once NoffN_{\mathrm{off}} is large; (iii) the gain is largest when Γ\Gamma equals Γtrue(1.5)\Gamma_{\mathrm{true}}(1.5) and shrinks as Γ\Gamma grows. See Figure 5, 6, 7, 8 in the updated paper. The table below reports the results of Figure 8 for Noff=500000N_{\mathrm{off}}=500000.

Method∑kWk,offΓ\sum_k W^{\Gamma}_{k,\mathrm{off}}Regret at TT = 10,000
Purely online–246.97 ± 16.49
Bounded conf. (Γ=1.5\Gamma=1.5)1.475147.87 ± 28.67
Bounded conf. (Γ=2\Gamma=2)1.638196.89 ± 18.85
Bounded conf. (Γ=5\Gamma=5)1.811220.28 ± 17.38
Unrestricted conf. (Γ=∞\Gamma=\infty)1.879225.69 ± 15.38
  • Bruns-Smith, D. and Zhou, A. Robust fitted-Q-evaluation and iteration under sequentially exogenous unobserved confounders. NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World (ReALML), 2023.

W2 The evidence for the main extension beyond tabular, unconfounded learning remains limited.

Q3 Can you give a nontrivial treatment-selection example satisfying Assumption 1 with substantially fewer features than histories, and state the time and memory required to construct and query its offline confidence sets?

Yes, the following is one such DTR example. The number of stages K≥3K\ge3,

  • Unmeasured variable U∼Bernoulli(1/2)U\sim\mathrm{Bernoulli}(1/2).

  • Actions. For k∈[K]k\in[K], Ak∈0,1A_k\in\\{0,1\\}.

  • States. For k∈[K]k\in[K], Sk=US_k=U with probability ρ∈(12,1)\rho\in(\frac12,1) and Sk=1−US_k=1-U otherwise.

  • Behavior Policy. πk(Ak=1∣sˉk,aˉk−1,U)=σ(κ(2U−1))\pi_{k}(A_k=1\mid\bar s_k,\bar a_{k-1},U)=\sigma\bigl(\kappa(2U-1)\bigr), κ>0\kappa > 0.

  • Outcome. Y∼Bernoulli(c,[12+AK−14(2U−1)])Y\sim\mathrm{Bernoulli}\bigl(c\\,\bigl[\frac12+\frac{A_{K-1}}4(2U-1)\bigr]\bigr), where c:=1c:=1 if AK=1A_K=1, and c:=12+12∑i=1K−1(2Si+Ai),4−ic:=\frac12+\frac12\sum_{i=1}^{K-1}(2S_i+A_i)\\,4^{-i} if AK=0A_K=0.

  • Feature mapping. Let nk:=s1+⋯+skn_k:=s_1+\dots+s_k. The feature maps are ψk:=enk⊗eak\psi_k:=e_{n_k}\otimes e_{a_k} for k<Kk < K and ψK:=c,enK⊗eaK−1\psi_K:=c\\,e_{n_K}\otimes e_{a_{K-1}}, where en∈Rk+1e_n\in\mathbb R^{k+1} and ea∈R2e_a\in\mathbb R^2 are standard basis vectors.

Assumption 1 holds with feature dimension dk=2(k+1)d_k=2(k+1) against 4k4^k for the tabular feature mapping. The optimal treatment at stage K−1K-1 is 11 if nK−1>(K−1)/2n_{K-1} > (K-1)/2 and 00 if nK−1<(K−1)/2n_{K-1} < (K-1)/2, so it depends on all past states.

Time and memory. For the above construction,

  • Bounds. For observed pair (sˉk,aˉk)(\bar s_k,\bar a_k), the bounds LkΓ(sˉk,aˉk),UkΓ(sˉk,aˉk)L_k^\Gamma(\bar s_k,\bar a_k),\\ U_k^\Gamma(\bar s_k,\bar a_k) (Proposition 3) are computed in O(KNoff)O(KN_{\mathrm{off}}) time and O(∑kmin⁡Noff,4k)O(\sum_k\min\\{N_{\mathrm{off}},4^k\\}) memory.

  • Offline confidence sets. C^k,offΓ\widehat{\mathcal C}^{\Gamma}_{k,\mathrm{off}} is computed in O(∑kmin⁡Noff,4k)O(\sum_k\min\\{N_{\mathrm{off}},4^k\\}) time and stored in O(∑kdk)O(\sum_k d_k) memory.

  • Queries. A query of the offline confidence set, max⁡θ∈C^k,offΓ⟨ψk,θ⟩\max_{\theta\in\widehat{\mathcal C}^{\Gamma}_{k,\mathrm{off}}}\langle\psi_k,\theta\rangle, takes constant (O(1)O(1)) time.

Real-world motivation for above DTR example. In the ExTENd trial for alcohol dependence (Lei et al., 2012), drinking was monitored during the first eight weeks of naltrexone treatment. Taking each period to be a day,

  • UU is the patient's underlying drinking propensity;

  • ak∈0,1a_k\in\\{0,1\\} indicates whether the current treatment is continued or changed (e.g., switched to a behavioral intervention);

  • sk∈0,1s_k\in\\{0,1\\} indicates whether period kk was a heavy drinking day;

  • nk=s1+⋯+skn_k=s_1+\dots+s_k is the number of heavy drinking days so far;

  • Y∈0,1Y\in\\{0,1\\} indicates no heavy drinking in the final weeks.

Heavy drinking days occur independently across periods given the patient's underlying drinking propensity, so the probability of a heavy drinking day in the next period depends on the history only through the number of heavy drinking days so far (nkn_k). As for the outcome, whether the patient avoids heavy drinking in the final weeks depends on the patient's underlying drinking propensity and treatments. The number of heavy drinking days reveals patient's underlying drinking propensity: a patient with many heavy drinking days is likely to have a high propensity. Therefore, the expected outcome depends on the number of heavy drinking days so far and the treatments. It may also depend on when these days occurred, e.g., early in the treatment or shortly before the final weeks.

We also construct a DTR environment calibrated to the simulated data set BmiData (Linn et al., 2015), in which Assumption 1 holds with a low-dimensional feature map (d1=8d_1=8, d2=16d_2=16, against 64 and 512 for tabular features). See Common Response: Numerical Experiments above.

  • Lei, H., Nahum-Shani, I., Lynch, K., Oslin, D., and Murphy, S. A. A "SMART" design for building individualized treatment sequences. Annual Review of Clinical Psychology, 8:21–48, 2012.

  • Linn, K. A., Laber, E. B., and Stefanski, L. A. iqLearn: Interactive Q-learning in R. Journal of Statistical Software, 2015.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response: Numerical Experiments

1. Synthetic Experiments with Tabular Features

  • We add the baselines Causal UC-DTR (Zhang & Bareinboim, 2019) and the DTR bandit (Hu & Kallus, 2022) to the synthetic experiments. See Figure 2 in the revised paper. The table below reports the bounds plotted in Figure~2 for Noff=500000N_{\mathrm{off}}=500000. | Method | T = 2,000 | 4,000 | 6,000 | 8,000 | 10,000 | |---|---|---|---|---|---| | Purely online | 108.8 ± 25.5 | 161.4 ± 33.0 | 194.8 ± 40.1 | 223.2 ± 44.6 | 247.0 ± 50.0 | | Unrestricted conf. (Γ=∞\Gamma=\infty) | 85.6 ± 28.5 | 137.4 ± 30.0 | 171.3 ± 35.7 | 203.1 ± 44.7 | 225.2 ± 50.3 | | Bounded conf. (Γ=1\Gamma=1) | 7.2 ± 13.5 | 12.0 ± 16.6 | 16.5 ± 16.5 | 21.4 ± 19.6 | 28.8 ± 30.0 | | Fully unconf. | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | | Causal UC-DTR (Zhang & Bareinboim, 2019) | 87.8 ± 28.3 | 136.0 ± 37.5 | 168.6 ± 40.0 | 193.2 ± 44.7 | 214.0 ± 50.9 | | DTR bandit (Hu & Kallus, 2022) | 65.3 ± 40.2 | 94.0 ± 69.7 | 121.2 ± 100.1 | 142.5 ± 130.1 | 163.5 ± 161.4 |

  • We report crossover horizon T∗T^* in the synthetic experiments for Noff=500,5000,50000,500000N_{\mathrm{off}}=500,\\ 5000,\\ 50000,\\ 500000. We plot the branches (online, offline and, pooled (for Theorem 3)) of the regret bounds in Theorem 1 (unrestricted confounding), Theorem 2 (bounded confounding) and Theorem 3 (fully unconfounded). See Figures 9, 10, 11, 12 in the revised paper.

  • We add an experiment with unmeasured confounding level Γtrue=1.5\Gamma_{\mathrm{true}}=1.5. We run bounded confounding with Γ∈1.5,2,5\Gamma\in\\{1.5,2,5\\}, unrestricted confounding (Γ=∞\Gamma=\infty) and purely online, for Noff=500,5000,50000,500000N_{\mathrm{off}}=500,\\ 5000,\\ 50000,\\ 500000. We report the cumulative regret and the offline width ∑kWk,offΓ\sum_k W^{\Gamma}_{k,\mathrm{off}}. See Figures 5, 6, 7, 8 in the revised paper.

2. BmiData Experiments with Low-Dimensional Features

We construct a DTR environment calibrated to the simulated data set BmiData (Linn et al., 2015), in which Assumption 1 holds with a low-dimensional feature map. BmiData is a simulated data set of 210 adolescent patients. It mimics the clinical trial of (Berkowitz et al., 2011), which compares meal replacement with a conventional diet for adolescent obesity. See Appendix H.2 BmiData Experiments with Low-Dimensional Features in the revised PDF for details.

We report the cumulative regret and standard deviation of all methods (including baselines) at Noff∈5×102,5×103,5×104,5×105N_{\mathrm{off}} \in \\{5 \times 10^2, 5 \times 10^3, 5 \times 10^4, 5 \times 10^5\\} (see Figure 3 in the updated paper). The regret gap between our hybrid methods and Causal UC-DTR (Zhang & Bareinboim, 2019) is substantially larger in the BmiData experiment than in the synthetic experiment. In the BmiData experiment, our methods use linear function approximation with feature maps ψ1(s1,a1)\psi_1(s_1,a_1) and ψ2(sˉ2,aˉ2)\psi_2(\bar{s}_2,\bar{a}_2) of dimensions 8 and 16. Causal UC-DTR (Zhang & Bareinboim, 2019) uses a tabular representation with 64 and 512 state-action pairs at the two stages. The table below reports the bounds plotted in Figure 3 for Noff=500000N_{\mathrm{off}} = 500000.

MethodT = 2,0004,0006,0008,00010,000
Purely online18.3 ± 2.621.9 ± 3.024.0 ± 3.226.0 ± 3.627.7 ± 3.9
Unrestricted conf.14.1 ± 2.817.7 ± 3.420.0 ± 3.821.8 ± 4.123.6 ± 4.6
Bounded conf.11.6 ± 3.114.9 ± 3.617.3 ± 3.919.0 ± 4.120.6 ± 4.5
Fully unconf.0.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.0
Causal UC-DTR (Zhang & Bareinboim, 2019)114.6 ± 4.0191.3 ± 7.1256.5 ± 10.2320.8 ± 9.7379.9 ± 10.6
DTR bandit (Hu & Kallus, 2022)47.1 ± 9.075.2 ± 15.4101.6 ± 20.9125.8 ± 26.6149.9 ± 31.8
  • Linn, K. A., Laber, E. B., and Stefanski, L. A. iqLearn: Interactive Q-learning in R. Journal of Statistical Software, 2015.

  • Berkowitz, R. I., Wadden, T. A., Gehrman, C. A., Bishop-Gilyard, C. T., Moore, R. H., Womble, L. G., Cronquist, J. L., Trumpikas, N. L., Levitt-Katz, L. E., and Xanthopoulos, M. S. Meal replacements in the treatment of adolescent obesity: A randomized controlled trial. Obesity, 19(6):1193–1199, 2011.

Dear Reviewer jjUY, we thank for your reply.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.