- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the problem of hybrid RL. The learner is given an offline dataset generated from the source environment and is allowed to interact with the target environment, while the source environment differs from the target in terms of the transition dynamics. The authors propose several hybrid RL algorithms incorporating pessimism for offline learning, block-wise adaptation, "handoff" mechanism, and regret balancing. They also present upper bounds on their suboptimality and regret as well as a lower bound. They finally reported the results of numerical experiments.
Critical strengths
- Tackling relevant problem, i.e., theoretical understanding of dynamics shift in hybrid RL.
Critical weaknesses
- Weak positioning: lacking comparison with robust MDP literature in the main text.
- Weak presentation: The paper is cluttered with poorly defined terms (e.g., "balancing master" (L74), "that set" (L161), "the pairs" (L201), "block" (L257)), while it skips many theoretically important details in the main text; For example, Theorem 1 is hard to interpret without the definition and intuition of as well as what the values of would be, and most claims in the subsequent "Reading the bound" section do not follow from the displayed bound (Eq. 7) alone. This paper contains five more "Reading the bound" sections in the main text with similar flaws.
Questions for authors
Please correct my assessment if it contains factual error.
3 comments
I appreciate the authors’ response and the revisions to the manuscript. However, my main concerns have not yet been sufficiently addressed.
-
I still expect clearer positioning of the contributions relative to previous work for a paper to be accepted at ICLR. To phrase my previous concern differently, the main text needs a coherent account of which limitations of existing work motivate this paper and how its results address them. I believe robust MDP research, or more specifically robust hybrid RL/MDP research, is one of the most closely related fields and deserves more than a couple of sentences in the main text to contextualize the contribution.
-
The presentation needs further improvement, though I appreciate the revisions addressing the terminology and the "Reading the bound" issues. The theoretical results remain difficult to follow as a whole. The manuscript should explain more clearly what question each result answers, how the results relate to one another, and how they support the central contributions.
Therefore, I maintain my Presentation and Informativeness scores. As for the Soundness score, my previous score reflects the difficulty of evaluating the paper due to its presentation issues. I will reevaluate it later based on the revised manuscript.
We vehemently disagree
We thank the reviewer for the follow-up. We disagree with the assessment that the paper does not adequately position its contributions or explain its theoretical results, and we are disappointed that the response does not engage with the substantial material already devoted to these purposes.
Positioning and discussion of prior work
The introduction, spanning pp. 1–3, motivates the problem, discusses the most closely related work, explains the limitations that motivate our study, and explicitly lists the paper’s contributions. Appendix A extends this discussion with detailed comparisons. It includes a table summarizing our formal guarantees and two tables comparing theoretical settings and guarantee forms with related work. The extended literature discussion occupies more than four pages, including discussion of robust MDPs, and the bibliography spans pp. 11–18.
We recognize that the remaining request concerns the emphasis given to robust MDPs in the main text. However, the paper studies hybrid RL under dynamics shift, which connects several bodies of work, including offline RL, online RL, transfer learning, and robust MDPs. Its contributions should be assessed in that broader context. The discussion cannot cover every paper within every related area.
The absence of an exhaustive treatment of a particular subfield does not, by itself, establish a basis for rejection. A substantive concern about positioning would identify an omitted result that changes the novelty assessment, a claimed distinction that is incorrect, or a comparison that misrepresents prior work. Neither the review nor the follow-up identifies such a result or comparison.
Explanation of the theoretical results
The main theoretical results are accompanied by explanations of their terms, implications, and limitations. Corollaries and special cases make the guarantees concrete, while further remarks and discussions appear in the appendix. The experiments also illustrate distinctions that are central to the analysis, including when source information helps, when it misleads the learner, and what observations under a deployed policy can reveal.
The revised manuscript makes these explanations more accessible in several places. The changes extend beyond the terminology and the single passage mentioned in the follow-up. The updated PDF includes the new results, with changes shown in blue.
A reader may prefer a different organization or further explanation, but the follow-up does not specify which relationship between results remains unclear or which explanation is inadequate after the revisions. We therefore do not believe it provides a substantive justification for maintaining the lowest Presentation and Informativeness ratings. Your clarification that the Soundness score reflected difficulty evaluating the presentation also leaves no identified error in a theorem or proof for us to address.
We summarize the contributions again in the accompanying comment. We ask that you consider these explanations together with the revised manuscript and reconsider your evaluation. We would be happy to answer any further questions or address any remaining concerns.
Response to Reviewer BkSK
Although clarity is partly subjective, several points in this assessment overlook definitions and discussion already present in the submission.
Definitions and terminology
Section 2.1 defines and interprets the transfer modulus before Theorem 1. It explains the separate effects of source coverage and dynamics shift, and gives the bound δh(ϵ)≤Cϵ when the environments coincide and the relevant occupancy is covered. The quantity ϵoff in Theorem 1 is the source Bellman-error tolerance associated with the statistical validity event (6). It accounts for finite source data; the optimization error is the separate final term in (7). Thus the quantities needed to interpret the theorem are defined before or within its statement.
The terminology examples also have identifiable meanings in context. “Balancing master” appears in the introduction's synopsis of contributions, with the algorithmic definition given later. “That set” refers to the immediately preceding set of source-consistent critics, which contains the source value function under realizability. “The pairs that π⋆ visits” are state-action pairs under that policy's occupancy. A block is the group of episodes collected under a fixed policy in one iteration of Algorithm 1. We have made these references more explicit. The updated PDF includes the new results, with changes shown in blue.
Robust MDPs
The comparison with robust MDPs was already included in Appendix A.2 and the related-work table. We have brought that comparison into the main text. Robust MDPs optimize worst-case value over a specified transition uncertainty set. Our analysis concerns performance in a fixed target environment, and relates source consistency to target Bellman error through policy-dependent transfer quantities. These are different objectives.
Interpretation and contributions
The discussion following Theorem 1 compressed the displayed bound together with refinements established in the appendix. We have separated these implications and made their supporting results explicit. We also state precisely what the lower bound establishes: necessity of the δh dependence on the stated hard families and target budgets.
The contribution is a framework for policy optimization, certification, and adaptation under dynamics shift. Its results distinguish when source data provide useful guarantees, what observations under the deployed policy can reveal, and how online learning can recover from harmful transfer. The revision makes the comparisons with source-only and target-only learning explicit. With matched dynamics and sufficiently abundant source data covering a target-optimal policy, the tabular learner achieves O(H2T) regret with no polynomial state-action dependence; the source sample carries that statistical cost. The return test gives a detection-time bound under sustained overprediction, while the regret-balancing master compares source-informed and online guarantees with explicit adaptation costs.
Summary
Given the rebuttal above, we disagree with the evaluation and consider the ratings of 1 for presentation, informativeness, and soundness unjustified. The presentation assessment overlooks definitions and explanations already present in the submission. On informativeness, we note that the review does not engage with the paper's results on transfer, certification, adaptation, and their limitations. On soundness, we note that the review identifies no incorrect theorem, invalid proof step, or counterexample. We request you to reconsider all three ratings in light of these responses and the revised paper.
We have addressed all the concerns raised in your review and encourage you to read our responses carefully and review the changes in the updated PDF. The changes are shown in blue color. We would be happy to answer any further questions or address any remaining concerns.