- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies offline-to-online reinforcement learning for dynamic treatment regimes (DTRs). It considers a setting in which the agent has access to a potentially confounded offline dataset, which is used to guide initial treatment decisions and facilitate policy refinement through online exploration as new patients arrive. The author(s) considers three scenarios corresponding to different degrees of unmeasured confounding in the offline data: (i) unrestricted confounding, (ii) bounded confounding, and (iii) no unmeasured confounding. They develop algorithms and establish regret guarantees for each scenario. Methodologically, the paper extends existing approaches to offline-to-online learning for DTRs from model-based to model-free methods, and from tabular representations to linear function approximation.
Critical strengths
The paper has several critical strengths:
-
Methodologically, existing offline-to-online approaches to DTRs are model-based and rely on tabular representations. Both can be restrictive in DTR settings, where the Markov property may not hold and the history-action space can grow rapidly.
-
Theoretically, the paper extends existing results by covering a more comprehensive range of scenarios regarding unmeasured confounding, including (i) unrestricted confounding, (ii) bounded confounding, and (iii) no unmeasured confounding. The authors establish regret guarantees for all three settings. The theoretical results appear sound, and I did not spot any major issues.
Critical weaknesses
Despite these strengths, the paper has several limitations. In my view, the main concern is the limited novelty beyond existing work of Zhang and Bareinboim (2019, 2020). Let me elaborate.
First, the paper's empirical evaluation is limited, relying on tabular environments despite its emphasis on linear function approximation. Moreover, the comparison does not include existing approaches. Thus, the paper's contributions are primarily methodological and theoretical. From a methodological perspective, extending the model-based, tabular approaches of Zhang and Bareinboim (2019, 2020) to model-free learning with linear function approximation is meaningful. However, the central idea closely follows these earlier works. Specifically, the proposed algorithm selects treatments by maximizing an optimistic Q-function estimate, constructed by taking the minimum of an online upper confidence bound and an offline causal upper bound. This follows essentially the same principle as Zhang and Bareinboim (2019, 2020), with the differences lying in how the Q-functions are constructed.
Second, from a theoretical perspective, while the paper considers a broader range of confounding scenarios, the structure of the regret guarantees closely follows that of Zhang and Bareinboim (2019, 2020). In particular, the regret bounds consist of an exploration term and the minimum of two quantities: one corresponding to the regret of purely online learning and the other reflecting the information gained from the offline data. This basic structure has already been established in Zhang and Bareinboim (2019, 2020).
Finally, the paper does not establish minimax lower bounds to demonstrate the sharpness of its regret upper bounds.
Questions for authors
Please refer to the weaknesses above.
3 comments
-
Theoretical Results
(1). We derive new causal bounds on the transition probabilities and the expected outcome under bounded confounding (Proposition 6). For the setting, our bounds recover those of (Zhang & Bareinboim, 2019, Theorem 5 and Corollary 1), which are known to be tight. For every , our bounds are strictly tighter than . At , our bounds match the point-identified g-formula (and thus are tight). To our knowledge, Proposition 6 gives the first causal bounds on the stage-wise interventional transition probabilities and mean outcomes of a (non-Markovian) DTR under bounded confounding ().
See Figure 4 in the updated paper for a comparison of the causal bounds of Proposition 6 with those of Zhang & Bareinboim (2019), with . The table below reports the bounds plotted in Figure 4.
Transition : Bound Width Outcome : Bound Width 1 [0.7158, 0.7158] 0.0000 [0.2716, 0.2716] 0.0000 1.25 [0.6511, 0.7967] 0.1456 [0.2039, 0.3661] 0.1622 1.5 [0.6079, 0.8443] 0.2364 [0.1691, 0.4717] 0.3026 2 [0.5540, 0.8443] 0.2903 [0.1420, 0.6191] 0.4771 3 [0.5001, 0.8443] 0.3442 [0.1173, 0.6854] 0.5681 5 [0.4569, 0.8443] 0.3874 [0.0992, 0.7339] 0.6347 10 [0.4245, 0.8443] 0.4198 [0.0866, 0.7676] 0.6810 [0.3922, 0.8443] 0.4521 [0.0749, 0.7991] 0.7242 (2). Our regret bounds are tighter (See our response to "W2" for details). The regret bounds of (Zhang & Bareinboim, 2019, 2020) depend on the domains of states and actions, growing exponentially with the number of stages. Our regret bounds differ in four respects: (i) ours depend on the feature dimension. (ii) Our results account for the uncertainty of finite offline samples. (iii) We cover bounded unmeasured confounding and the regret bound is never worse than under unrestricted confounding. (iv) We also study the fully unconfounded offline data, and the regret bound is never worse than under unrestricted or bounded confounding.
-
Empirical results
(1). Baseline. We include the methods of (Zhang & Bareinboim, 2019) and (Hu & Kallus, 2022) as baselines. For (Hu & Kallus, 2022), we use the authors' replication code. For (Zhang & Bareinboim, 2019), we use a publicly available third party implementation. The implementation of (Zhang & Bareinboim, 2020) is not publicly available at the time of this work.
(2). We update synthetic experiments with tabular features. See Figure 2 in the updated paper.
(3). We also construct a DTR environment calibrated to the simulated data set
BmiData(Linn et al., 2015), in which Assumption 1 holds with a low-dimensional feature map (, , against 64 and 512 for tabular features). See Appendix H.2BmiDataExperiments with Low-Dimensional Features in the revised PDF for details.We report the cumulative regret and standard deviation of all methods (including baselines) at (see Figure 3 in the updated paper). The regret gap between our hybrid methods and Causal UC-DTR (Zhang & Bareinboim, 2019) is substantially larger in the
BmiDataexperiment than in the synthetic experiment. In theBmiDataexperiment, our methods use linear function approximation with feature maps and of dimensions 8 and 16. Causal UC-DTR (Zhang & Bareinboim, 2019) uses a tabular representation with 64 and 512 state-action pairs at the two stages. The table below reports the bounds plotted in Figure 3 for .Method T = 2,000 4,000 6,000 8,000 10,000 Purely online 18.3 ± 2.6 21.9 ± 3.0 24.0 ± 3.2 26.0 ± 3.6 27.7 ± 3.9 Unrestricted conf. 14.1 ± 2.8 17.7 ± 3.4 20.0 ± 3.8 21.8 ± 4.1 23.6 ± 4.6 Bounded conf. 11.6 ± 3.1 14.9 ± 3.6 17.3 ± 3.9 19.0 ± 4.1 20.6 ± 4.5 Fully unconf. 0.0 ± 0.0 0.0 ± 0.0 0.0 ± 0.0 0.0 ± 0.0 0.0 ± 0.0 Causal UC-DTR (Zhang & Bareinboim, 2019) 114.6 ± 4.0 191.3 ± 7.1 256.5 ± 10.2 320.8 ± 9.7 379.9 ± 10.6 DTR bandit (Hu & Kallus, 2022) 47.1 ± 9.0 75.2 ± 15.4 101.6 ± 20.9 125.8 ± 26.6 149.9 ± 31.8 - Linn, K. A., Laber, E. B., and Stefanski, L. A. iqLearn: Interactive Q-learning in R. Journal of Statistical Software, 2015.
W2 No minimax lower bounds.
For fully unconfounded offline data, the bound in Theorem 3 can be written as (see last row in table below). It matches the lower bound for hybrid linear MDP (Zhang & Sinclair, 2026) up to a factor , and for a single stage () it matches the lower bound for hybrid linear MAB (Chandak et al., 2026) up to the same factor. The lower bounds of both (Zhang & Sinclair, 2026) and (Chandak et al., 2026) are stated under coverage conditions: they hold only when the offline data cover all feature directions well. Suppose the offline data have good coverage (e.g., ) for our fully unconfounded setting, then specializing Theorem 3 to MDPs then gives , which just have an extra term.
Under unrestricted and bounded confounding, Theorems 1 and 2 matches the dependence on of the lower bounds in (Zhang & Sinclair, 2026; Chandak et al., 2026). For the dependence on , the offline widths shrink with only down to the width of the causal bounds, that is, and . Here and are the widths of the causal bounds on the optimal -function () under unrestricted and bounded confounding, respectively. They do not depend on . This is because the optimal -function is only partially identified, and no amount of offline data removes the width or () (see Remarks 1 and 2 in the paper). Neither (Zhang & Sinclair, 2026) nor (Chandak et al., 2026) considers unmeasured confounding in the offline data.
Table 1: Regret bounds in related settings. Notation: : feature dimension; : horizon; : number of online episodes; : number of offline samples (trajectories); offline coverage coefficients; , : domains of the state and action. † The lower bound in (Zhang & Bareinboim, 2019, Theorem 3) is for the purely online setting, without offline data. ‡ For the case (e.g. randomized control trial data), our Algorithm 2's regret bound (Theorem 3) can be rewritten/simplified to a form (see Remark 5 in the updated paper) that enables simpler comparison with lower bounds from the hybrid MAB/MDP setting.
| Paper | Setting | Model | Regret bound | Unmeasured confounding |
|---|---|---|---|---|
| Chandak et al. (2026) | Hybrid MAB | Linear | None | |
| Zhang & Sinclair (2026) | Hybrid MDP | Linear mixture | None | |
| Zhang & Bareinboim (2019) | Online DTR | Tabular | N/A (online) † | |
| Corollary 1 | Online DTR | Linear | N/A | |
| Theorem 1 | Hybrid DTR | Linear | Unrestricted | |
| Theorem 2 | Hybrid DTR | Linear | Bounded | |
| Theorem 3 | Hybrid DTR | Linear | None | |
| Theorem 3 ‡ | Hybrid DTR | Linear | None |
-
Zhang, Z. and Sinclair, S. R. Offline-online reinforcement learning for linear mixture MDPs. In Advances in Neural Information Processing Systems, 2026.
-
Chandak, K., Tkachuk, V., Tan, X., and Szepesvári, C. Gaussian-width bounds for offline-to-online linear bandits. arXiv preprint arXiv:2610.12990, 2026.
Thank you for your careful reading. We appreciate your time and feedback. Below, we address the main comments provided in the review.
We illustrate the novelty compared with (Zhang & Bareinboim, 2019, 2020) from four angles below.
Problem setting.
Those prior works on hybrid DTR learning (Zhang & Bareinboim, 2019, 2020) only considered the worst case of the offline data being arbitrarily (i.e., unrestricted) confounded (Γ=∞). They also only considered the impractical setting of unlimited offline data. They did not handle any of the uncertainty that comes from working with finite offline data.
In many applications, such as clinical trials, the unmeasured confounding is present but bounded in magnitude. In particular, Γ=1 for randomized controlled trials (the best case). We study three regimes: Γ=1 for offline data from randomized controlled trials (best case), Γ=∞ for (worst case), and the intermediate regime of bounded confounding (Γ∈(1,∞)).
For all three regimes, we consider the data is finite and handle the resulting uncertainty.
Methodology & Technical challenges
(Zhang & Bareinboim, 2019, 2020) are model-based and specifically for the tabular setting: they estimate all transition kernels and the reward distribution over histories. Our work is model-free and for the more general linear setting. We consider learning the Q-functions with linear value function approximation. There are two key technical challenges:
(1). Considering the linear setting for hybrid DTR learning is not direct, even under unrestricted confounding (Γ=∞), despite being standard in RL/MDPs.
We adapt the optimistic closure condition of (Wang et al., 2021b) to our DTR setting, imposing it over a specified class of value functions, permitting smaller feature dimensions.
(2). Under bounded unmeasured confounding (Γ∈(1,∞)), which (Zhang & Bareinboim, 2019, 2020) did not consider, we derive new causal bounds (Proposition 6). The main difficulty is the identification uncertainty: the unmeasured variables U affect the behavior policy πb,k(ak∣sˉk,aˉk−1,u) at every stage, so the uncertainty that the confounding adds to the causal bounds multiplies across stages. For unrestricted confounding (Γ=∞), the identification uncertainty has a simpler form. It depends only on probabilities of the offline data distribution, so it can be estimated directly from the offline data and requires no bounds on how the unmeasured variables affect the treatment.
(3). We consider finite offline data. The causal bounds must account for estimation uncertainties. Our estimated causal bounds depend on the estimated propensities π^b,k(ak∣sˉk,aˉk−1) of all stages. Therefore, the estimation errors of these propensities compound, amplified by a factor of O(Γk) at stage k, and if handled in a straightforward manner can make the estimated causal bounds vacuous/trivial. We handle this in Proposition 3.