acceptodds
Under review as a conference paper at ICLR 2027

Hybrid Reinforcement Learning under Dynamics Shift: Policy Optimization, Certification, and Adaptation

Abstract

Hybrid reinforcement learning combines offline data with online interaction. When these data come from environments with different transition dynamics, the learner must decide how to use the source and when to rely on target observations. We study policy optimization, target certification, and adaptation under such dynamics shift. For general function classes satisfying realizability and Bellman completeness in both environments, we analyze pessimistic policy optimization with source-consistent critics. Its guarantees depend on target Bellman residuals along the learned policies and a target-optimal comparator, quantified through policy-dependent transfer moduli. A block-based algorithm uses its own target observations to constrain the deployed-policy residual; a separate comparator term and the cost of failed tests remain. An incompatibility test supports a handoff to online learning, while a return-based master adapts between source-informed copies and an online baseline, with coefficient-dependent overheads. Structured upper bounds separate source uncertainty from dynamics shift in tabular, linear, and block MDPs. Lower bounds show that the resulting square-root modulus dependence is necessary on hard families when target observations cannot distinguish the competing models. In finite-class experiments, an optimal source policy can trigger handoff while a better unvisited route remains undetected, and return-based selection partly recovers from a misleading source at a substantial cost relative to target-only controls.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Hybrid RL, Transfer Modulus, Dynamics Shift, Instance-dependent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BkSKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of hybrid RL. The learner is given an offline dataset generated from the source environment and is allowed to interact with the target environment, while the source environment differs from the target in terms of the transition dynamics. The authors propose several hybrid RL algorithms incorporating pessimism for offline learning, block-wise adaptation, "handoff" mechanism, and regret balancing. They also present upper bounds on their suboptimality and regret as well as a lower bound. They finally reported the results of numerical experiments.

Critical strengths
  • Tackling relevant problem, i.e., theoretical understanding of dynamics shift in hybrid RL.
Critical weaknesses
  • Weak positioning: lacking comparison with robust MDP literature in the main text.
  • Weak presentation: The paper is cluttered with poorly defined terms (e.g., "balancing master" (L74), "that set" (L161), "the pairs" (L201), "block" (L257)), while it skips many theoretically important details in the main text; For example, Theorem 1 is hard to interpret without the definition and intuition of ϵoff\epsilon_{off} as well as what the values of δh∙(ϵoff)\delta_h^{\bullet}(\epsilon_{off}) would be, and most claims in the subsequent "Reading the bound" section do not follow from the displayed bound (Eq. 7) alone. This paper contains five more "Reading the bound" sections in the main text with similar flaws.
Questions for authors

Please correct my assessment if it contains factual error.

3 comments
AuthorsOpenReview ↗

Response to Reviewer BkSK

Although clarity is partly subjective, several points in this assessment overlook definitions and discussion already present in the submission.

Definitions and terminology

Section 2.1 defines and interprets the transfer modulus before Theorem 1. It explains the separate effects of source coverage and dynamics shift, and gives the bound δh(ϵ)≤Cϵ\delta_h(\epsilon)\le C\epsilon when the environments coincide and the relevant occupancy is covered. The quantity ϵoff\epsilon_{\mathrm{off}} in Theorem 1 is the source Bellman-error tolerance associated with the statistical validity event (6). It accounts for finite source data; the optimization error is the separate final term in (7). Thus the quantities needed to interpret the theorem are defined before or within its statement.

The terminology examples also have identifiable meanings in context. “Balancing master” appears in the introduction's synopsis of contributions, with the algorithmic definition given later. “That set” refers to the immediately preceding set of source-consistent critics, which contains the source value function under realizability. “The pairs that π⋆\pi^\star visits” are state-action pairs under that policy's occupancy. A block is the group of episodes collected under a fixed policy in one iteration of Algorithm 1. We have made these references more explicit. The updated PDF includes the new results, with changes shown in blue.

Robust MDPs

The comparison with robust MDPs was already included in Appendix A.2 and the related-work table. We have brought that comparison into the main text. Robust MDPs optimize worst-case value over a specified transition uncertainty set. Our analysis concerns performance in a fixed target environment, and relates source consistency to target Bellman error through policy-dependent transfer quantities. These are different objectives.

Interpretation and contributions

The discussion following Theorem 1 compressed the displayed bound together with refinements established in the appendix. We have separated these implications and made their supporting results explicit. We also state precisely what the lower bound establishes: necessity of the δh\sqrt{\delta_h} dependence on the stated hard families and target budgets.

The contribution is a framework for policy optimization, certification, and adaptation under dynamics shift. Its results distinguish when source data provide useful guarantees, what observations under the deployed policy can reveal, and how online learning can recover from harmful transfer. The revision makes the comparisons with source-only and target-only learning explicit. With matched dynamics and sufficiently abundant source data covering a target-optimal policy, the tabular learner achieves O~(H2T)\widetilde O(H^2\sqrt T) regret with no polynomial state-action dependence; the source sample carries that statistical cost. The return test gives a detection-time bound under sustained overprediction, while the regret-balancing master compares source-informed and online guarantees with explicit adaptation costs.

Summary

Given the rebuttal above, we disagree with the evaluation and consider the ratings of 1 for presentation, informativeness, and soundness unjustified. The presentation assessment overlooks definitions and explanations already present in the submission. On informativeness, we note that the review does not engage with the paper's results on transfer, certification, adaptation, and their limitations. On soundness, we note that the review identifies no incorrect theorem, invalid proof step, or counterexample. We request you to reconsider all three ratings in light of these responses and the revised paper.

We have addressed all the concerns raised in your review and encourage you to read our responses carefully and review the changes in the updated PDF. The changes are shown in blue color. We would be happy to answer any further questions or address any remaining concerns.

I appreciate the authors’ response and the revisions to the manuscript. However, my main concerns have not yet been sufficiently addressed.

  • I still expect clearer positioning of the contributions relative to previous work for a paper to be accepted at ICLR. To phrase my previous concern differently, the main text needs a coherent account of which limitations of existing work motivate this paper and how its results address them. I believe robust MDP research, or more specifically robust hybrid RL/MDP research, is one of the most closely related fields and deserves more than a couple of sentences in the main text to contextualize the contribution.

  • The presentation needs further improvement, though I appreciate the revisions addressing the terminology and the "Reading the bound" issues. The theoretical results remain difficult to follow as a whole. The manuscript should explain more clearly what question each result answers, how the results relate to one another, and how they support the central contributions.

Therefore, I maintain my Presentation and Informativeness scores. As for the Soundness score, my previous score reflects the difficulty of evaluating the paper due to its presentation issues. I will reevaluate it later based on the revised manuscript.

AuthorsOpenReview ↗

We vehemently disagree

We thank the reviewer for the follow-up. We disagree with the assessment that the paper does not adequately position its contributions or explain its theoretical results, and we are disappointed that the response does not engage with the substantial material already devoted to these purposes.

Positioning and discussion of prior work

The introduction, spanning pp. 1–3, motivates the problem, discusses the most closely related work, explains the limitations that motivate our study, and explicitly lists the paper’s contributions. Appendix A extends this discussion with detailed comparisons. It includes a table summarizing our formal guarantees and two tables comparing theoretical settings and guarantee forms with related work. The extended literature discussion occupies more than four pages, including discussion of robust MDPs, and the bibliography spans pp. 11–18.

We recognize that the remaining request concerns the emphasis given to robust MDPs in the main text. However, the paper studies hybrid RL under dynamics shift, which connects several bodies of work, including offline RL, online RL, transfer learning, and robust MDPs. Its contributions should be assessed in that broader context. The discussion cannot cover every paper within every related area.

The absence of an exhaustive treatment of a particular subfield does not, by itself, establish a basis for rejection. A substantive concern about positioning would identify an omitted result that changes the novelty assessment, a claimed distinction that is incorrect, or a comparison that misrepresents prior work. Neither the review nor the follow-up identifies such a result or comparison.

Explanation of the theoretical results

The main theoretical results are accompanied by explanations of their terms, implications, and limitations. Corollaries and special cases make the guarantees concrete, while further remarks and discussions appear in the appendix. The experiments also illustrate distinctions that are central to the analysis, including when source information helps, when it misleads the learner, and what observations under a deployed policy can reveal.

The revised manuscript makes these explanations more accessible in several places. The changes extend beyond the terminology and the single passage mentioned in the follow-up. The updated PDF includes the new results, with changes shown in blue.

A reader may prefer a different organization or further explanation, but the follow-up does not specify which relationship between results remains unclear or which explanation is inadequate after the revisions. We therefore do not believe it provides a substantive justification for maintaining the lowest Presentation and Informativeness ratings. Your clarification that the Soundness score reflected difficulty evaluating the presentation also leaves no identified error in a theorem or proof for us to address.

We summarize the contributions again in the accompanying comment. We ask that you consider these explanations together with the revised manuscript and reconsider your evaluation. We would be happy to answer any further questions or address any remaining concerns.

Reviewer_Hk3tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies hybrid reinforcement learning under transition dynamics shift, where the learner receives nn offline episodes collected under a behavior policy μ\mu in a source MDP and interacts with a target MDP for TT episodes over horizon HH. Assuming a critic class F\mathcal{F} satisfies realizability and Bellman completeness in both environments, the authors analyze pessimistic policy optimization over source consistent critics using per state multiplicative weights updates. To quantify how source consistency controls target errors, the paper defines a policy dependent weak transfer modulus as the supremum of the target Bellman error over source consistent critics, distinguishing the backup policy from the evaluation policy. Using this weak modulus, the authors establish a target suboptimality bound for the offline mixture policy (Theorem 1) and an online regret bound for source constrained updates (Theorem 21). To incorporate target observations during deployment, the paper introduces a target conditioned strong transfer modulus and a block based algorithm over blocks of length ww (Algorithm 1), proving certified regret bounds (Theorem 2 and Corollary 3) and a certified handoff to an online learner upon critic incompatibility (Theorem 4), alongside a return based balancing master (Theorem 5). Finally, the authors provide explicit modulus bounds and matching lower bounds for structured MDPs.

Critical strengths

The authors provide both upper and lower bounds for the hybrid reinforcement learning analysis under dynamics shift, pairing weak and strong transfer modulus upper bounds with matching lower bounds across general Bellman complete classes (Theorem 7) and structured tabular and linear MDPs (Theorem 6 and Appendix G).

Critical weaknesses
  • The online phase relies purely on pessimism, which gives up online exploration. The value of online target interaction lies in its ability to gather information that cannot be obtained from source data, specifically along trajectories where the source data are uninformative or erroneous. Because Algorithms 1 and 3 only select a pessimistic critic from the source constrained confidence set and test residuals along the deployed policy, source data and target interactions are never truly integrated for exploration. When target observations reject the source critics in Theorem 4, the system simply hands off the remaining episodes to a purely online learner started fresh from scratch, either relying on the source constrained critics or abandoning the source data completely. This raises doubts about the hybrid modeling in this paper.
  • The paper starts from the premise that "Offline data can reduce the interaction needed to learn a good policy," but the regret bounds with target certification are too loose. Because Algorithm 1 holds the deployed policy fixed over blocks of length ww to certify target residuals, Theorem 2 and Corollary 3 incur unconditional regret remainders of order T5/6T^{5/6} and T3/4T^{3/4} even when no fallback occurs. Both rates are far above T\sqrt{T}, the rate a purely online learner achieves on the target MDP alone without any offline data, such as UCBVI in tabular MDPs.
Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer Hk3t (Weakness 1)

We thank the reviewer for their comments.

W1. Pessimism, exploration, and handoff

Your characterization of how our framework combines source data with exploration in the target environment is incomplete. It describes the pessimistic base learners and one handoff rule, but does not account for how target data change policy improvement or how the adaptive framework provides exploration.

Algorithms 1 and 3 do not explicitly direct exploration toward uncertain trajectories. However, target observations serve a substantive role beyond deciding whether to abandon the source. In Algorithm 1, the critic must satisfy both the source constraints and the Bellman-error constraints estimated from target episodes. These observations can rule out source-consistent critics and change the subsequent policy update, while retaining the source constraints. Theorem 2 quantifies this benefit. On a certified block, the deployed-policy contribution is bounded by min⁡{δhπk(ϵoff),γˉk},\sqrt{\min\{\delta_h^{\pi_k}(\epsilon_{\mathrm{off}}),\bar\gamma_k\}}, rather than the source-only term δhπk(ϵoff)\sqrt{\delta_h^{\pi_k}(\epsilon_{\mathrm{off}})}. The comparator term is also evaluated over critics satisfying both constraints. Thus, the two datasets are jointly used for policy improvement, and the analysis captures information about the target dynamics that the source data cannot supply.

The framework incorporates exploration through the regret-balancing master (Theorem 5 in the original submission, Theorem 7 in the revised PDF). It runs source-informed learners alongside an exploratory online learner and allocates episodes using their observed returns. The online learner operates without waiting for the critic test to reject, which matters when the source predicts the deployed policy accurately but undervalues a better unvisited route. The guarantee compares the source-informed and online bounds with explicit adaptation costs. This is a modular construction. The base learners combine source and target constraints, while the online learner supplies exploration.

The description of Theorem 4's fresh restart is correct. That theorem analyzes a particular first-rejection handoff rule, not the only adaptation mechanism in the paper. The master keeps source-informed learners available alongside the online learner and therefore does not require an irreversible decision to discard the source.

The revision makes the benefits more explicit (changes are shown in blue color). The new return monitor permits policy updates after every episode, and Corollary 6 bounds the time needed to detect sustained overprediction. Corollary 9 shows that, under matched dynamics, coverage of a target-optimal policy, and sufficient source data, monitoring preserves the source learner's policy sequence and yields O~(H2T)\widetilde O(H^2\sqrt T) regret. The dependence on the numbers of states and actions is paid through the source sample requirement, with only logarithmic action dependence remaining in this regret bound. A second specialization allows arbitrary dynamics changes away from a covered target-optimal route and includes the online continuation cost. These results establish explicit regimes in which source data reduce the cost of learning in the target environment, alongside guarantees for recovery when predictions from the source become unreliable.

AuthorsOpenReview ↗

Response to Reviewer Hk3t (Weakness 2)

W2. Regret rates and the benefit of source data

The T5/6T^{5/6} and T3/4T^{3/4} expressions are particular specializations for controlling Bellman error through target certification. They are not the best regret guarantees available in the submission. The full Theorem 2 already permits O~(T)\widetilde O(\sqrt T) regret with an informative source, and the regret-balancing master retains the online O~(T)\widetilde O(\sqrt T) dependence under arbitrary transfer (Theorem 5 in the original submission, Theorem 7 in the revised PDF). We have made these comparisons explicit in the revision.

Theorem 2 allows any block length 1≤w≤T1\le w\le T and retains min⁡{δhπ(k)(ϵoff),γˉk}\sqrt{\min\{\delta_h^{\pi^{(k)}}(\epsilon_{\rm off}),\bar\gamma_k\}} for the deployed-policy error. Its comparator term is also no larger than the corresponding weak modulus. The displayed T5/6T^{5/6} and T3/4T^{3/4} specializations use the target-radius bound and choose long blocks to balance certification accuracy against policy updates. When source data already control the relevant errors, those block lengths are unnecessary.

For example, suppose the dynamics match and the source distribution covers the comparator and deployed-policy occupancies with coefficient CC. On the confidence event, the true policy value satisfies both constraints, so no fallback occurs. Taking w=1w=1 in the existing Theorem 2 gives

\le 2HT\sqrt{C\epsilon_{\rm off}} +H^2\sqrt{T\log A}+H.$$ Consequently, a source sample large enough that $C\epsilon_{\rm off}\le 1/T$ gives $\widetilde O(H^2\sqrt T)$ regret. **This follows directly from Theorem 2 and requires no change to Algorithm 1.** The source-constrained online learner also has a regret bound of $T\Gamma_T+H^2\sqrt{T\log A/2}$. **When the source is unreliable, the regret-balancing master provides a comparison with learning online from scratch (Theorem 5 in the original submission, Theorem 7 in the revised PDF).** With an online learner satisfying $R_{\rm on}(t)=\widetilde O(c_{\rm on}\sqrt t)$, the master retains the $\sqrt T$ dependence, with multiplier $M+\sqrt M\,c_{\rm on}/H^2$ and the additional terms stated in the theorem. Thus, the framework does not require paying the large-block rates under arbitrary transfer. The revision develops these favorable cases more explicitly. Corollary 9 gives a tabular specialization requiring only coverage of a target-optimal policy. With the stated source budget, regret is $\widetilde O(H^2\sqrt T)$, with **no polynomial dependence on the numbers of states and actions**; that statistical cost is carried by the source sample. This is the benefit relative to the leading $\sqrt{SAT}$ dependence of tabular online bounds at fixed horizon. The return monitor preserves this guarantee under matched dynamics and extends the analysis to sources accurate only along a covered target-optimal route, while allowing arbitrary changes elsewhere and accounting for online continuation. These refinements make the source-sample requirements and the benefits under dynamics shift explicit; the availability of $\sqrt T$ regret was already part of the submitted framework. The updated PDF includes the new results, with changes shown in blue. We have addressed all the concerns raised in your review and encourage you to read our responses carefully and review the changes in the updated PDF. We ask you to reconsider your evaluation in light of this response and the revised paper. We would be happy to answer any further questions or address any remaining concerns.
Reviewer_PqTxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds a theoretical framework to conceptually characterize a special setting of hybrid RL: instead of assuming the same dynamics for the offline and online RL, it considers a more realistic setting where the dynamics of source and target are different. Theoretical analysis regarding offline policy optimization, target certification, and inference time adaptation under dynamics shift are established, providing several novel insights under the off-dynamics setting. A toy-example experiment illustrates some conceptual behavior.

Critical strengths

This is a clear and interesting work establishing a theoretical framework to conceptualize hybrid RL with dynamics shift, one step towards a more realistic setting. It adopts techniques from many previous works, I appreciate the effort. It is also very clear about what it does and what doesn't, the discussion of related works is thorough. I also appreciate the experiments done to illustrate the idea, like failure mode. In general I think this works offer something interesting and novel to the field, worth my recommendation.

Critical weaknesses

Not sure how the conceptual framework could be used to guide practice. The experiment is also a weak part, it is a toy example and the proposed Algorithm 1 is not implemented to my understanding. The paper is written tight to contain more information, making a outsider hard to follow the idea.

Questions for authors
  1. I am not familiar with this field and the terminologies - what is the "backup policy" π and "evaluation policy" ν?

  2. The presentation is tight, expanding some places could help understanding, such as "Policy improvement in the target instead involves target occupancies of both the current policy and a comparator, while retaining the current policy in the backup. Transferring between these errors therefore involves both a change of distribution and a change of Bellman operator."

  3. Line 166-168: "Two effects can make this implication weak. Limited source coverage leaves critics unconstrained on target-relevant state-action pairs, and a change of dynamics can leave a nonzero target residual even for the exact source value function. " I have the same question, in that case the quantity is vacuous and so is the Theorem 1, and we don't know that in advance.

  4. From the perspective of implementation, what would practitioners do under the setting considered in the paper? Probably use the stale policy learned from the offline source dataset temporarily, gather enough target data and learn a new policy, compare the two policies to determine switching or not. I don't think this is what this paper do. Can you summarize the protocol in this paper and justify it (intuitively why the stale source data should be used to constraint the value function, how practical is it)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer PqTx (part 1 of 2)

We thank the reviewer for their positive assessment and helpful comments.

Q1, the backup policy π\pi and the evaluation policy ν\nu. The Bellman error in (2) has two policy arguments. The backup policy π\pi fixes which Bellman equation is tested. A critic ff for π\pi should satisfy fh=Tπfh+1f_h=\mathcal T^\pi f_{h+1}, where the action at the next state is drawn from π\pi. The evaluation policy ν\nu fixes where the squared residual is averaged, namely over the state-action distribution dhνd^\nu_h. In Theorem 1 the backup policy is always the current iterate π(t)\pi^{(t)}, because the critic evaluates the policy being improved. The proof needs its residual in two places, along the iterate's own target distribution (ν=π(t)\nu=\pi^{(t)}) and along that of the target-optimal policy (ν=πtar⋆\nu=\pi^\star_{\mathrm{tar}}), which is the comparator. Source data control the residual on the behavior distribution (ν=μ\nu=\mu) under the source dynamics. Section 2 now explains these roles immediately after (2).

Q2, clarity and accessibility. Fitting the material into nine pages made some parts of the exposition dense and affected accessibility. With the additional page, we have expanded the explanations in several places and clarified the definitions and interpretation of the results. These changes are shown in blue in the revised PDF.

Q3, a vacuous modulus. You are right that the modulus can be large, and then Theorem 1 says little. The rest of the paper addresses this situation in three ways.

(a) A large modulus can reflect a hard problem. The lower bound (Theorem 7 in the original submission, Theorem 10 in the revised PDF) shows that the δh\sqrt{\delta_h} dependence on the comparator modulus cannot be improved in general. On its hard families, every algorithm that uses nn source and TT target episodes loses order ∑hδh\sum_h\sqrt{\delta_h}.

(b) The online procedures do not need the modulus in advance. In certified blocks, Theorem 2 bounds the deployed policy's Bellman error using data collected online in the target environment, with a radius that shrinks as the block grows. Theorem 5 (new in the revised PDF) compares predicted and observed returns. Corollary 6 (new in the revised PDF) gives an explicit detection period under sustained overprediction, followed by the online learner's guarantee. The first bound for the master (Theorem 5 in the original submission, Theorem 7 in the revised PDF) keeps it within stated overheads of an online learner at any transfer level.

(c) The modulus can be bounded explicitly. The tabular result (Theorem 6 in the original submission, Theorem 8 in the revised PDF) bounds it through coverage and total-variation shift on the pairs that the evaluated policy visits. The new specialization (Corollary 9 and Section 5.2 in the revised PDF) gives explicit source budgets and T\sqrt T guarantees. In the matched case, regret in the target environment has no state-count dependence and only logarithmic action-count dependence; source data carry the statistical cost. Agreement only along a covered target-optimal route also suffices for a T\sqrt T rate with a t\sqrt t online continuation, even under arbitrary shift elsewhere. The plug-in estimation result (Proposition 24 in the original submission, Proposition 27 in the revised PDF) estimates the modulus of a fixed policy from target data, given a maximization oracle.

AuthorsOpenReview ↗

Response to Reviewer PqTx (part 2 of 2)

Q4, what a practitioner would do. Section 4.2 (new in the revised PDF) explains the return test using offline source transitions, known rewards, and returns observed online in the target environment. The learner selects a source-consistent critic with the smallest initial prediction and uses it to update the policy. After each episode, it adds the difference between the predicted and observed return to a running total. It starts a target-only learner when this total crosses the threshold.

The protocol you describe is close to this, and the paper supplies the parts that the protocol leaves open. Running one fixed policy online in the target environment for several episodes is one block of Algorithm 1, and retraining on that block is the policy-finetuning template of Xie et al. (2021b), as Remark 4 notes. Comparing and switching is what the return test and the master do, with an explicit threshold and a stated cost. Two cautions apply, and the new experiments show both. Data collected along one route do not reveal a better route elsewhere. In the hidden-route condition the return test never fires and the regret stays at 4,654, so finding the better route needs a learner that explores, which can be registered with the master. A switch can also be unnecessary. In the useful-source condition, the return test stops a near-optimal policy whose value the source overestimates, and the regret rises from 239 to 1,570.

Why the source constrains the critics. Pooling source transitions with target transitions leaves a bias, governed by the shift, that more source data do not remove. As a constraint, the source excludes only the value functions that contradict it. Section 3 now explains what pessimism adds. On the source-validity event, the current policy's source value is feasible, so the pessimistic prediction never exceeds it. With matched dynamics the learner's own overestimation then vanishes, and only comparator coverage is required (Corollary 9 in the revised PDF).

How practical it is. For tabular critic classes, Appendix I (new in the revised PDF) gives explicit radii, pessimistic selection is a second-order cone program, and the source-only case admits a verified closed form on these instances (Appendix J.2.2 in the revised PDF). For general classes, one must solve the critic optimization, as in Bellman-consistent pessimism for offline RL (Xie et al., 2021a). We have not run a deep implementation. The experiments with existing deep methods (Appendix H.3 in the original submission, Appendix J.5 in the revised PDF) examine friction shift and do not test our procedure.

W2, implementation and experiments. We implement Algorithm 1 in the revised experiments and provide the full code and configurations needed to reproduce them. The implementation uses the algorithm's exponential-weights update over complete tabular critic classes (Section 6.3 and Appendix J.2.1 in the revised PDF). The seven-state MDP allows exact evaluation; we average over eight equivalent MDP labelings for each of ten seeds. With the handoff of Theorem 4, regret in the misleading-source condition falls from 10,500 to 1,286, against 1,024 for the target-only control. The return test with calibrated radii reduces it from 13,886 to 1,499 at T=20,000T=20{,}000 (Table 11 and Appendix J.2.2 in the revised PDF). At that budget both handoff variants trail the target-only control in all four conditions. At T=2,000T=2{,}000 the return-test learner is ahead under the useful source (195 against 525) and under poor coverage (278 against 292). We report both outcomes.

W3, the writing. The introduction names and explains the algorithms. Section 6.1 and the experimental appendix (Appendix H.1.1 in the original submission, Appendix J.1.1 in the revised PDF) now describe the experimental MDP stage by stage, with explicit conditions and a diagram in the appendix.

The updated PDF includes the new results, with changes shown in blue. We have addressed all the concerns raised in your review and encourage you to read our responses carefully and review the changes in the updated PDF. We would be happy to answer any further questions or address any remaining concerns.

Thanks for the response. I acknowledge I have read it.

Comments on the paper

3 comments

As the discussion period is approaching its end, please take a moment to review the authors’ responses and share any remaining thoughts or updates to your assessment. Thanks for your thoughtful reviews and discussion!

AuthorsOpenReview ↗

Raising concern regarding reviewer's BkSK's follow-up comments

Dear Area Chair,

We are very disappointed by Reviewer BkSK’s follow-up and strongly disagree with its assessment of the paper’s positioning, exposition, and informativeness. We ask that these claims be assessed against the specific material in the manuscript and our responses.

Positioning and literature discussion

The introduction spans pp. 1–3 and is devoted to motivating the problem, discussing closely related work, explaining the limitations that motivate our study, and identifying the contributions. Appendix A provides extended comparisons and three tables covering related settings, our formal guarantees, and the forms of competing guarantees. The related-work discussion occupies more than four pages and includes robust MDP research. The bibliography spans pp. 11–18.

The reviewer requests greater main-text emphasis on robust MDPs. That request does not establish that the paper lacks positioning or a substantial literature discussion. Nor does the follow-up identify a missing result that invalidates a novelty claim or an incorrect comparison with prior work. Our paper addresses hybrid RL under dynamics shift and should be assessed against its stated problem and contributions across the relevant literature.

Explanation of the results and the ratings

The main theoretical results are accompanied by explanations of their meaning, implications, and limitations, with corollaries, special cases, and further discussion in the appendix. The experiments illustrate the same theoretical distinctions. We have also revised the exposition in several places, with changes shown in blue.

Despite these changes and our detailed response, the follow-up does not identify a particular theorem whose interpretation remains unsupported or explain which connection between results remains unclear. It maintains the lowest Presentation and Informativeness ratings without engaging with the specific contributions summarized in the paper and our response.

The reviewer has now explicitly clarified that the original Soundness rating of 1 reflected difficulty evaluating the presentation. They have indicated that they will reevaluate it, which we welcome. No specific error in a theorem or proof has been identified in either the review or the follow-up.

Even taking the remaining presentation requests at face value, the comments do not explain the exceptionally negative evaluation across all three dimensions. Broad requests for clearer positioning and exposition, without identifying what remains deficient after the revisions, give us little concrete opportunity to resolve the disagreement. This falls short of the substantive engagement that the discussion period is intended to support.

We respectfully ask you to examine the introduction, Appendix A, the explanations accompanying the main results, and our responses when weighing this review. In particular, we ask that difficulty assessing the presentation be distinguished from an identified mathematical flaw, and that the Presentation and Informativeness ratings also receive independent scrutiny against the paper’s actual content.

The updated PDF includes the new results, with changes shown in blue. We would be happy to address any specific remaining concern.

AuthorsOpenReview ↗

Clarifying the Paper’s Contributions, Positioning, and Main Results

We disagree with Reviewer BkSK’s assessment of the paper’s positioning and exposition, and encourage the other reviewers to read our response and engage with BkSK on these points. We summarize the contributions and the relationships between the results below.

The central question

The paper asks when offline data from one environment can reduce the cost of learning in another environment with different transition dynamics, and how the learner should respond when the source information is inadequate.

Robust MDP research is relevant, but its usual objective is worst-case performance over a collection of models. Our objective is performance against the optimal policy of the particular target environment in which the learner interacts. These objectives can favor different policies, so guarantees for one do not directly resolve the other. We discuss robust MDP research and explain these distinctions as part of an extensive literature survey, with the most directly related work covered in the main text, more than four pages of additional discussion in Appendix A, and a bibliography spanning eight pages. The paper’s positioning and contributions are explicitly set out in the PDF, and we summarize them again below for the reviewers’ convenience.

How the results answer the central question above

First, we characterize what source fitting can guarantee in the target. Theorem 1 separates the target errors associated with the comparator policy from those associated with the learner’s deployed policies. This identifies the two difficulties that the subsequent results address.

Second, we show what target observations can establish. Theorem 2 uses observations under the deployed policy to control its Bellman error, and Theorem 4 provides a guarantee for switching to an online learner when the test fails. The new Theorem 5 in the revised PDF instead monitors predicted and observed episode returns, allowing policy updates every episode.

Third, we address failures that monitoring alone cannot reveal. Accurate predictions about the current policy do not rule out a better policy elsewhere. The regret-balancing master therefore combines source-informed learners with an online learner that explores, comparing their guarantees with explicit adaptation costs (Theorem 5 in the original submission, Theorem 7 in the revised PDF).

Finally, we make the benefits and limitations concrete. The structured bounds separate coverage from dynamics shift. The new Corollary 9 shows that, with matched dynamics and sufficient source data covering a target-optimal policy, regret is O~(H2T)\widetilde O(H^2\sqrt T) without polynomial dependence on the numbers of states and actions. The required source sample carries that dependence. The lower bounds establish necessary dependence on transfer error for the stated hard families and target budgets.

These results explain the benefit of useful source information, the role and limitations of target monitoring, and the need for exploration when the deployed policies cannot reveal a better alternative. The experiments illustrate these same distinctions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.