- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Offline RL algorithms are often based on conservative policy while this slows the online fine-tuning speed. The authors point out the gap that this relation has not been properly measured and studied. To resolve this gap, authors propose an adaptive switching between conservative and explorative policy based on the fitted Q-evaluation on the offline dataset. If the estimated Q-evaluation decreases, switch to conservative and vice-versa. The authors argue an improved degradation and fine-tuning performance by this approach.
Critical strengths
-
The authors highlight an underexplored phenomenon in fine-tuning literature of offline-to-online RL : the policy degradation problem while fine-tuning. Although, this has been known, re-visiting the baselines and reporting their result on Mean-FP versus VaR-MPD metric seems to be useful to understand this phenomenon metrically. Moreover, even though the switching schedule seems heuristic, the choices are well supported by their ablation studies.
-
Overall, the experiments section seems to be the strength of the paper. In particular, the finding in Section 4 is interesting. In average in d4rl dataset, InAC is betttr but there are cases where SAC can be better. This finding motivates combining conservative updates with more exploratory online learning.
Critical weaknesses
My major concern is that i) their theoretical results does not give proper explanation on their methods ii) No proper comparison and credit to theoretical works are given.; iii) the theoretical statements are not clearly stated. Below are the details
-
In the introduction, the authors state that "Our results support this last issue, showing that out-of-distribution actions and states are a key culprit". But none of the theoretical support this claim. If this is the main claim of experimental results, the theory should support this claim.
-
The authors theoretically explain how zero degradation and policy improvement is not both achievable by Thm 6.1. Nonetheless the theorems should clarify "whether this is specific to their algorithm' or "whether this is a result from standard results". For example, it seems to be a direct application from the bandit literature, and whether it applies to "offline-to-online learning algorithm" or "any learning algorithm". The authors should only use minimal assumptions or scopes to state a theorem so that only necessary components are presented.
-
The paper requires improvement on the clarity of the writing:
i) "backstepping approach"-> the term backstepping is only used in the abstract. If the term is only used in the abstract, for consistency of the paper, it can be replaced with other wordings.
ii) "A typical way to reflect this is to use an upper percentile over samples, such as the 90th percentile, wherein higher values indicate more severe performance degradation." -> Please provide citation if such metrics were used in the literature.
iii) The terms "batch" and "offline" are used interchangeably. Sticking either to "batch" or "offline" would be clearer.
iv) Figure 2 is not used in main text.
Questions for authors
- My other concern is that the paper does not fully explain why SAC is better than InAC in specific environments (HalfCheetah) in Section 4. Why particularly HalfCheetah?
4 comments
Thanks for the detailed response. I feel there are still unclear parts on Section 6 how it explains the advantage and motivation of COE. The result of Theorem 1 and 2 seems to be not COE specific and similar theoretical results on these seems to be presented by JSRL (backward approach) or safe policy improvement (Theorem 2 in Laroche). Therefore, I do not think Section 6 properly addresses the advantage of proposed COE.
Moreover, I have some comments about Proposition 1.
i) The term "rollback mechanism" appears which is only used in Section 6 and not else in the other sections.
ii) With , , and large , the probability bound can be vacous. Can the authors clarify this?
iii) Overall, Proposition 1 seems to be too overly informal. Such as the terms "pre-expansion history" and "post-expansion episode"--not only those-- are used without definition.
Thanks for the detailed response. I feel there are still unclear parts on Section 6 how it explains the advantage and motivation of COE. The result of Theorem 1 and 2 seems to be not COE specific and similar theoretical results on these seems to be presented by JSRL (backward approach) or safe policy improvement (Theorem 2 in Laroche). Therefore, I do not think Section 6 properly addresses the advantage of proposed COE.
Theorem 6.1 and 6.2 are not meant to be specific to COE, they were intended to motivate some of its design choices rather than establishing its advantage over other algorithms.
JSRL (Uchendu et al., 2023) establishes sample complexity benefits of backward exploration under guide-policy coverage assumptions. Our Theorem 6.2 gives a complementary result. We show that placing exploratory actions at the end of an episode uniquely minimizes worst-case performance degradation for a fixed number of exploratory steps, without requiring these coverage assumptions. Note also that JSRL was motivated by improving exploration (using a guide policy) rather than performance degradation. SPIBB (Laroche et al., 2019) gives an approximate high-probability baseline-improvement guarantee. Our Theorem 6.1 gives an impossibility result of universal almost-sure zero degradation with nontrivial improvement under finite online interaction. We have added references to these works in Section 6 and clarify how our theoretical results are different from theirs.
These results motivate some of COE’s design choices. Its comparative advantages are well supported empirically: across 22 environments, COE achieves a strong balance between performance degradation and final performance compared to existing offline-to-online methods. Our ablations show the importance of COE’s components, particularly adaptive switching and return-based rollback, in reducing degradation and maintaining strong final performance.
Moreover, I have some comments about Proposition 1. i) The term "rollback mechanism" appears which is only used in Section 6 and not else in the other sections.
We agree that we should introduce this terminology earlier. In Section 5, we have now explicitly defined “expansion” as decreasing the switch horizon , and “rollback” as increasing following a detected performance drop. This mechanism is already implemented in Algorithm 1 and explained in Section 5.
ii) With m=5, gamma=0.99, and large H, the probability bound can be vacous. Can the authors clarify this?
The bound is not vacuous, but we do agree that the bound can be loose, particularly when the degradation gap is small relative to the return range. Proposition 1 provides a distribution-free concentration bound using the worst-case return range. For , the bound is , which will approach 1 when is small. So, the proposition does not guarantee reliable detection within 5 episodes in every environment. Instead, it shows how the probability of delayed rollback decreases with the magnitude of persistent degradation. We appreciate you pointing out this point, and have added a remark after the proposition highlighting this point.
iii) Overall, Proposition 1 seems to be too overly informal. Such as the terms "pre-expansion history" and "post-expansion episode"--not only those-- are used without definition.
We have now made the terminology more precise. We have clarified that pre-expansion history contains all information available before a decrease in , and post-expansion episodes are the subsequent episodes.
In Appendix A, we had already formalized this with a filtration , post-expansion returns , and first rollback time . So, the assumption is until rollback, where is the rollback threshold fixed by the pre-expansion history. In the proof, we use a stopped-martingale concentration argument, and do not assume independent episode returns. We have now incorporated these formal definitions directly into Section 6 and revised the statement of Proposition 1 accordingly.
Update: To provide additional evidence for the advantage and motivation of COE, we ran an additional experiment where the exploratory policy acts from the beginning of the episode, and the switch horizon h expands from there. This is the opposite scheme of what we are proposing with COE (exploratory policy at the end of episode). We train this variant on 4 adroit and 2 locomotion datasets while keeping every other component of the COE the same as before. The results are shown in Figure 8 of the main text. We observe that not only the performance degradation is significantly higher in this case, but the final policy is not able to improve as much as if the exploration was done from the end of the episodes.
Thank you for the detailed response. I am not yet convinced how the theories support motivation of COE given the establisehd works. The authors emphasize different condition and results but it is not clear enough whether they are stronger or weaker--or whether it is truly different. Moreover, the loose bound in the proposition requires further discussion whether it is meaningful. Nonetheless, given the strong empirical results, I adjusted my score from 1 to 3.
We thank the reviewer for their constructive comments. We outline how we have addressed some concerns below and hope to further discuss your primary concerns around the theory.
This statement is supported empirically, and have modified the paper to emphasize empirical (rather than theoretical) support and to point the reader toward the relevant section in the appendix. However, this is not the main focus of this paper, which is why we relegate this additional support in appendix F.1.
This result is an impossibility claim, and is not specific to our algorithm. The construction is a simple bandit problem, but a bandit is an instance of an MDP. Most impossibility results construct an instance, and that instance need not be complex to prove impossibility for the broader class.
Note also that this is not a standard result from bandits, our result has not been shown before. We believe we provided minimal assumptions, allowing for any learner with a finite offline dataset that then learns further from online samples. We are more than happy to make this section clearer, and can add more preamble as we have outlined here or try to reduce assumptions if there is a particular set you feel is too complex. We will wait for further response from you, to see if this clarifies and if further clarification would be useful.
We have updated the abstract to use the terminology consistent with the paper: “backwards expansion”.
We have updated this paragraph in the paper with relevant citations:
[1] Rockafellar, R. T., & Uryasev, S. (2000). Optimization of conditional value-at-risk. Journal of risk, 2, 21-42.
[2] Chan, S. C., Fishman, S., Canny, J., Korattikara, A., & Guadarrama, S. (2019). Measuring the reliability of reinforcement learning algorithms. arXiv preprint arXiv:1912.05663.
This is a fair point. We wanted to emphasize the fact that these algorithms learn from a batch (or buffer) of data, and they need not only be used offline. Such batch algorithms (like InAC and PARS) can be used both offline and online. When we say an offline algorithm, it is only being applied to offline data. We have now gone through and made the wording consistent, calling them batch algorithms and otherwise using the modifier offline only when its strictly learning offline.
We have updated section 3 to reference Figure 2.
We want to first note that in section 4, HalfCheetah is used for illustrative purposes, and is not the only environment where SAC is better than InAC. But to nonetheless answer this question:
We have updated the main text with additional analysis comparing SAC and InAC in section 4 and Figure 5, which shows empirically that SAC’s final policy almost always learns to sample actions outside the dataset’s support, while InAC stays within the support. This allows SAC to explore more and learn a better policy in some cases
Note also that we have added additional 6 environments from NeoRL2, and here the final performance of the policy learned by SAC is higher than InAC across all environments (see Figures 6, 13, 14 and Table 1).