acceptodds
Under review as a conference paper at ICLR 2027

COE: Mitigating Degradation while Improving Fine-tuning in Offline-to-Online RL

Abstract

In offline-to-online RL, agents often experience performance degradation in the early stages of fine-tuning. There is some evidence that offline (batch) RL algorithms can mitigate this degradation, but can also learn too slowly during fine-tuning. The majority of progress in offline-to-online RL has been on achieving faster fine-tuning, but with insufficiently controlled performance degradation. We propose a new algorithm, called Cautious Online Expansion (COE), that combines a conservative batch algorithm and a faster-learning online algorithm, using off-policy estimation to gradually hand off control. We show the resulting algorithm mitigates degradation while allowing for fast learning online, compared to several batch algorithms and offline-to-online algorithms. We provide theoretical support for the backstepping approach in the algorithm and how the policies are evaluated to effectively decide on the hand-off. We additionally ablate several design decisions in COE, providing empirical support for each component.

Primary area:
reinforcement learning
Keywords:
offline-to-online, performance degradation, continual fine-tuning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_w6cXOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Offline RL algorithms are often based on conservative policy while this slows the online fine-tuning speed. The authors point out the gap that this relation has not been properly measured and studied. To resolve this gap, authors propose an adaptive switching between conservative and explorative policy based on the fitted Q-evaluation on the offline dataset. If the estimated Q-evaluation decreases, switch to conservative and vice-versa. The authors argue an improved degradation and fine-tuning performance by this approach.

Critical strengths
  1. The authors highlight an underexplored phenomenon in fine-tuning literature of offline-to-online RL : the policy degradation problem while fine-tuning. Although, this has been known, re-visiting the baselines and reporting their result on Mean-FP versus VaR-MPD metric seems to be useful to understand this phenomenon metrically. Moreover, even though the switching schedule seems heuristic, the choices are well supported by their ablation studies.

  2. Overall, the experiments section seems to be the strength of the paper. In particular, the finding in Section 4 is interesting. In average in d4rl dataset, InAC is betttr but there are cases where SAC can be better. This finding motivates combining conservative updates with more exploratory online learning.

Critical weaknesses

My major concern is that i) their theoretical results does not give proper explanation on their methods ii) No proper comparison and credit to theoretical works are given.; iii) the theoretical statements are not clearly stated. Below are the details

  1. In the introduction, the authors state that "Our results support this last issue, showing that out-of-distribution actions and states are a key culprit". But none of the theoretical support this claim. If this is the main claim of experimental results, the theory should support this claim.

  2. The authors theoretically explain how zero degradation and policy improvement is not both achievable by Thm 6.1. Nonetheless the theorems should clarify "whether this is specific to their algorithm' or "whether this is a result from standard results". For example, it seems to be a direct application from the bandit literature, and whether it applies to "offline-to-online learning algorithm" or "any learning algorithm". The authors should only use minimal assumptions or scopes to state a theorem so that only necessary components are presented.

  3. The paper requires improvement on the clarity of the writing:

i) "backstepping approach"-> the term backstepping is only used in the abstract. If the term is only used in the abstract, for consistency of the paper, it can be replaced with other wordings.

ii) "A typical way to reflect this is to use an upper percentile over samples, such as the 90th percentile, wherein higher values indicate more severe performance degradation." -> Please provide citation if such metrics were used in the literature.

iii) The terms "batch" and "offline" are used interchangeably. Sticking either to "batch" or "offline" would be clearer.

iv) Figure 2 is not used in main text.

Questions for authors
  1. My other concern is that the paper does not fully explain why SAC is better than InAC in specific environments (HalfCheetah) in Section 4. Why particularly HalfCheetah?
4 comments

We thank the reviewer for their constructive comments. We outline how we have addressed some concerns below and hope to further discuss your primary concerns around the theory.

In the introduction, the authors state that "Our results support this last issue, showing that out-of-distribution actions and states are a key culprit". But none of the theoretical support this claim. If this is the main claim of experimental results, the theory should support this claim.

This statement is supported empirically, and have modified the paper to emphasize empirical (rather than theoretical) support and to point the reader toward the relevant section in the appendix. However, this is not the main focus of this paper, which is why we relegate this additional support in appendix F.1.

The authors theoretically explain how zero degradation and policy improvement is not both achievable by Thm 6.1. Nonetheless the theorems should clarify "whether this is specific to their algorithm' or "whether this is a result from standard results".

This result is an impossibility claim, and is not specific to our algorithm. The construction is a simple bandit problem, but a bandit is an instance of an MDP. Most impossibility results construct an instance, and that instance need not be complex to prove impossibility for the broader class.

Note also that this is not a standard result from bandits, our result has not been shown before. We believe we provided minimal assumptions, allowing for any learner with a finite offline dataset that then learns further from online samples. We are more than happy to make this section clearer, and can add more preamble as we have outlined here or try to reduce assumptions if there is a particular set you feel is too complex. We will wait for further response from you, to see if this clarifies and if further clarification would be useful.

The paper requires improvement on the clarity of the writing: i) "backstepping approach"-> the term backstepping is only used in the abstract.

We have updated the abstract to use the terminology consistent with the paper: “backwards expansion”.

ii) "A typical way to reflect this is to use an upper percentile over samples, such as the 90th percentile, wherein higher values indicate more severe performance degradation." -> Please provide citation if such metrics were used in the litearature.

We have updated this paragraph in the paper with relevant citations:

[1] Rockafellar, R. T., & Uryasev, S. (2000). Optimization of conditional value-at-risk. Journal of risk, 2, 21-42.

[2] Chan, S. C., Fishman, S., Canny, J., Korattikara, A., & Guadarrama, S. (2019). Measuring the reliability of reinforcement learning algorithms. arXiv preprint arXiv:1912.05663.

iii) The terms "batch" and "offline" are used interchangeably. Sticking either to "batch" or "offline" would be clearer.

This is a fair point. We wanted to emphasize the fact that these algorithms learn from a batch (or buffer) of data, and they need not only be used offline. Such batch algorithms (like InAC and PARS) can be used both offline and online. When we say an offline algorithm, it is only being applied to offline data. We have now gone through and made the wording consistent, calling them batch algorithms and otherwise using the modifier offline only when its strictly learning offline.

Figure 2 is not used in main text.

We have updated section 3 to reference Figure 2.

My other concern is that the paper does not fully explain why SAC is better than InAC in specific environments (HalfCheetah) in Section 4. Why particularly HalfCheetah?

We want to first note that in section 4, HalfCheetah is used for illustrative purposes, and is not the only environment where SAC is better than InAC. But to nonetheless answer this question:

  1. The high return behavior in HalfCheetah lies outside the dataset’s action support, and learning this behavior requires leaving that support.
  2. InAC’s in-sample update keeps the learning policy near the dataset’s action, so it remains close to its offline performance.
  3. HalfCheetah is a task with dense rewards and no early termination, so even though SAC degrades at the beginning and leaves the support, it is still able to receive full length informative trajectories and recover quickly.

We have updated the main text with additional analysis comparing SAC and InAC in section 4 and Figure 5, which shows empirically that SAC’s final policy almost always learns to sample actions outside the dataset’s support, while InAC stays within the support. This allows SAC to explore more and learn a better policy in some cases

Note also that we have added additional 6 environments from NeoRL2, and here the final performance of the policy learned by SAC is higher than InAC across all environments (see Figures 6, 13, 14 and Table 1).

Thanks for the detailed response. I feel there are still unclear parts on Section 6 how it explains the advantage and motivation of COE. The result of Theorem 1 and 2 seems to be not COE specific and similar theoretical results on these seems to be presented by JSRL (backward approach) or safe policy improvement (Theorem 2 in Laroche). Therefore, I do not think Section 6 properly addresses the advantage of proposed COE.

Moreover, I have some comments about Proposition 1.
i) The term "rollback mechanism" appears which is only used in Section 6 and not else in the other sections.

ii) With m=5m=5, γ=0.99\gamma=0.99, and large HH, the probability bound can be vacous. Can the authors clarify this?

iii) Overall, Proposition 1 seems to be too overly informal. Such as the terms "pre-expansion history" and "post-expansion episode"--not only those-- are used without definition.

Thanks for the detailed response. I feel there are still unclear parts on Section 6 how it explains the advantage and motivation of COE. The result of Theorem 1 and 2 seems to be not COE specific and similar theoretical results on these seems to be presented by JSRL (backward approach) or safe policy improvement (Theorem 2 in Laroche). Therefore, I do not think Section 6 properly addresses the advantage of proposed COE.

Theorem 6.1 and 6.2 are not meant to be specific to COE, they were intended to motivate some of its design choices rather than establishing its advantage over other algorithms.

JSRL (Uchendu et al., 2023) establishes sample complexity benefits of backward exploration under guide-policy coverage assumptions. Our Theorem 6.2 gives a complementary result. We show that placing exploratory actions at the end of an episode uniquely minimizes worst-case performance degradation for a fixed number of exploratory steps, without requiring these coverage assumptions. Note also that JSRL was motivated by improving exploration (using a guide policy) rather than performance degradation. SPIBB (Laroche et al., 2019) gives an approximate high-probability baseline-improvement guarantee. Our Theorem 6.1 gives an impossibility result of universal almost-sure zero degradation with nontrivial improvement under finite online interaction. We have added references to these works in Section 6 and clarify how our theoretical results are different from theirs.

These results motivate some of COE’s design choices. Its comparative advantages are well supported empirically: across 22 environments, COE achieves a strong balance between performance degradation and final performance compared to existing offline-to-online methods. Our ablations show the importance of COE’s components, particularly adaptive switching and return-based rollback, in reducing degradation and maintaining strong final performance.

Moreover, I have some comments about Proposition 1. i) The term "rollback mechanism" appears which is only used in Section 6 and not else in the other sections.

We agree that we should introduce this terminology earlier. In Section 5, we have now explicitly defined “expansion” as decreasing the switch horizon hh, and “rollback” as increasing hh following a detected performance drop. This mechanism is already implemented in Algorithm 1 and explained in Section 5.

ii) With m=5, gamma=0.99, and large H, the probability bound can be vacous. Can the authors clarify this?

The bound is not vacuous, but we do agree that the bound can be loose, particularly when the degradation gap is small relative to the return range. Proposition 1 provides a distribution-free concentration bound using the worst-case return range. For m=5m=5, the bound is exp−10ε2/B2exp^{-10\varepsilon^2/B^2}, which will approach 1 when ε/B\varepsilon/B is small. So, the proposition does not guarantee reliable detection within 5 episodes in every environment. Instead, it shows how the probability of delayed rollback decreases with the magnitude of persistent degradation. We appreciate you pointing out this point, and have added a remark after the proposition highlighting this point.

iii) Overall, Proposition 1 seems to be too overly informal. Such as the terms "pre-expansion history" and "post-expansion episode"--not only those-- are used without definition.

We have now made the terminology more precise. We have clarified that pre-expansion history contains all information available before a decrease in hh, and post-expansion episodes are the subsequent episodes.

In Appendix A, we had already formalized this with a filtration Fi\mathcal F_i, post-expansion returns ZiZ_i, and first rollback time τ\tau. So, the assumption is E[Zi∣Fi−1]≤c−ε\mathbb E[Z_i\mid\mathcal F_{i-1}]\le c-\varepsilon until rollback, where cc is the rollback threshold fixed by the pre-expansion history. In the proof, we use a stopped-martingale concentration argument, and do not assume independent episode returns. We have now incorporated these formal definitions directly into Section 6 and revised the statement of Proposition 1 accordingly.


Update: To provide additional evidence for the advantage and motivation of COE, we ran an additional experiment where the exploratory policy acts from the beginning of the episode, and the switch horizon h expands from there. This is the opposite scheme of what we are proposing with COE (exploratory policy at the end of episode). We train this variant on 4 adroit and 2 locomotion datasets while keeping every other component of the COE the same as before. The results are shown in Figure 8 of the main text. We observe that not only the performance degradation is significantly higher in this case, but the final policy is not able to improve as much as if the exploration was done from the end of the episodes.

Thank you for the detailed response. I am not yet convinced how the theories support motivation of COE given the establisehd works. The authors emphasize different condition and results but it is not clear enough whether they are stronger or weaker--or whether it is truly different. Moreover, the loose bound in the proposition requires further discussion whether it is meaningful. Nonetheless, given the strong empirical results, I adjusted my score from 1 to 3.

Reviewer_ZkkUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Authors study the problem of performance degradation during offline-to-online RL and the trade-offs with fast online improvement. They introduce COE, which is a method that switches between a conservative offline-trained agent and a more aggressive exploratory agent during training within episodes. They show that their method slightly outperforms the closest baseline with respect to final performance.

Critical strengths
  • Authors perform thorough experimentation comparing multiple baselines and reasonable ablations.
  • The paper is well written and easy to follow. It flows very well. Figures generally are also quite informative.
  • Results show overall minor final performance improvements, while retaining low degradation from their base method.
  • The performance plots of Var-MPD vs mean-FP and overall focus on worse performance degradation over trajectories is by itself a valuable contribution (even though the analysis of what actually contributes to the performance characteristics of different methods is mostly lacking).
Critical weaknesses
  • The choice of switching agents at some point in the middle of trajectories, always starting with the conservative agent, is a heuristic that is not well-motivated enough and not analyzed enough in the paper. Theorem 6.2 shows that this backwards expansion minimizes worst-case degradation, which makes sense because degradation at later parts matters less than earlier parts. Nonetheless, there may be many tasks and scenarios in which this heuristic would break down. For example, in robotics tasks in which the strategy taken at the beginning of episodes might determine future success (say, early grasping strategy might determine how likely an object placement will be to succeed), backwards expansion might not be exploratory enough.

  • The core idea of switching between a more conservative agent and a faster learning online agent is also a poorly-motivated heuristic. This does not address the underlying reasons why methods specifically designed for offline-to-online learning both suffer from degradation and might exhibit slower learning, and rather bypasses that by combining both methods. Appendix F mentions that a possible culprit is overestimation of unseen states (as opposed to unseen actions for seen states, which previous methods directly address). If authors believe that this is causing degradation, why not specifically combat that, instead of switching to an SAC policy, which does not actually address this issue?

  • There is nice analysis about comparing different methods in a VaR-MPD vs mean-FP plot, but I would have liked more analysis about what specifically is making some of these methods significantly better or more robust than others. E.g. InAC by itself already shows the best lack of degradation out of all considered methods (equivalent to COE itself). What about it makes it more robust than other offline-to-online methods?

Overall, the paper is technically solid, enjoyable to read, and provides interesting analysis. I do not think it addresses the underlying bottlenecks of offline-to-online RL, and the proposed method is based on heuristics and unrelated changes on top of its base method, InAC. On the balance, I think the strengths outweigh the weaknesses.

Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Author response (Part 1/2)

Thank you for your thoughtful comments. We address some of your points below and will also incorporate this to improve the paper.

The choice of switching agents at some point in the middle of trajectories, always starting with the conservative agent, is a heuristic that is not well-motivated enough…

This is an astute observation. We agree that this backward expansion approach will not be as effective for preventing degradation for all environments. For example, consider a setting where an agent decides on turning left or right at the beginning of an episode and it leads to very different parts of the MDP. If the conservative policy never turns right (and its update does not cause it to consider turning right), then the agent cannot learn about this until stepping all the way back to the start. At this point, it needs to essentially learn the policy for the right side from scratch.

Even in this scenario, however, it has learned a much better policy for going left and overall the policy needs to change less than it would if it started learning about both the left and right sides from the beginning of the fine-tuning phase. Furthermore, allowing a more exploratory policy (like SAC) to run from the first state once it expands back to it means the agent will actually try the right action, whereas the conservative policy may forever stick with the left. In general, slowly expanding from the end contains the degradation, since degradation at latter parts affects performance less (as you also say). But the agent can still get the benefits of a less conservative policy for generating new data and improving the policy overall. We will add a Remark in the paper highlighting that as yet there is more theoretical work to understand exactly which MDPs this benefits and which it does not. Our current theory largely provides conceptual motivation for the idea of backwards expansion, without specific guarantees on performance degradation and improvement.

We would like to point out that we tested COE on Adroit tasks, which are similar to your example of a manipulation task where the early grasping strategy can decide the outcome. There, the methods that explore from the start perform very poorly. For example: SAC from the InAC checkpoint ends at around -1.9 on pen and 36 on door (see Figures 14 and 15), whereas COE ends at 145 and 105. On these tasks, the early decision is the part of the episode that the offline policy already gets right (though not yet optimally), and giving it to an algorithm that explores uncontrollably first is what ruins the entire trajectory.

Note also that the switch horizon h does eventually reach the start of the episode in cases where the early decision needs to change. The backward direction of expansion does not put a cap on the exploration, it just limits the exploration on the beginning of the episode until the horizon eventually reaches it. Furthermore, the conservative policy is also able to do some limited exploration during the beginning of the episodes, as is evident from the COE component evaluations in Figure 11.

The core idea of switching between a more conservative agent and a faster learning online agent is also a poorly-motivated heuristic. This does not address the underlying reasons why...

We show in the paper that just learning with the conservative batch agents can actually already largely resolve the issues of avoiding performance degradation and improving online, particularly with either PARS or InAC. However, we find that these methods can get stuck at a suboptimal solution, potentially because they limit exploration (by nature of being conservative). Previous work, as we discuss in the introduction, has similarly found these methods can be slow to learn in fine-tuning. Once trying to promote faster online learning, however, the proposed methods then start suffering again from degradation. Some of these methods involve strategies like using SAC online. Switching to SAC is not an approach to mitigate degradation but rather to learn faster online.

We do find that overestimation for unseen states is an issue. Inherently learning for new states may result in some degradation, but the impact on the action-values is exacerbated by allowing the policy and state distribution to change too quickly like it does under SAC. Backwards expansion limits these changes. The conservative agent learns more slowly with minimal degradation and then backwards expansion with a more exploratory agent (e.g., SAC) allows COE to reduce the likelihood of getting stuck at a suboptimal solution. In summary, the design of COE is intended to slowly start visiting new (unseen) states, without destabilizing learning.

We can acknowledge that COE is as yet a heuristic, without sufficient theoretical evidence under which conditions it guarantees improvement and mitigates degradation. However, we do believe it is conceptually motivated and empirically has strong evidence to support it.

AuthorsOpenReview ↗

Author response (Part 2/2)

Continued from above:

There is nice analysis about comparing different methods in a VaR-MPD vs mean-FP plot, but I would have liked more analysis about what specifically is making some of these methods significantly better or more robust than others. E.g. InAC by itself already shows the best lack of degradation out of all considered methods (equivalent to COE itself). What about it makes it more robust than other offline-to-online methods?

For InAC's lack of degradation, we can analyze it using the SAC baseline. SAC starts from the same offline policy and critic as the InAC, and differs only in the online update. Removing the InAC’s in-sample constraint raises the VaR-MPD from 8.1+-1.3 to 54+-1.4. So, the robustness in this case likely comes from keeping the constraint during fine-tuning. We have updated the main text with an additional analysis (Figure 5), which shows that for the dataset’s states, the mean actions from SAC’s final policy almost always leaves the dataset’s support, whereas InAC’s final policy remains within this support.

Note that we expanded the empirical evaluation to 6 more environments from the NeoRL2 bench(see Figures 6, 14, 15 and Table 1 for updated results). We see similar conclusions on these new environments: COE achieves high overall final performance and low VaR-MPD.

Overall, the paper is technically solid, enjoyable to read, and provides interesting analysis. I do not think it addresses the underlying bottlenecks of offline-to-online RL, and the proposed method is based on heuristics and unrelated changes on top of its base method, InAC. On the balance, I think the strengths outweigh the weaknesses.

Thank you for the positive comment. Regarding the “unrelated changes on top of InAC”: The three normalizations that we applied to the conservative (InAC) policy (layer-norm, observation and reward scaling) are applied to every baseline method in the evaluation. The only other change to the conservative policy is the added behavior cloning term, which is added to make the conservative policy more conservative and is ablated in Figure 7.

AuthorsOpenReview ↗

Author response (update)

The choice of switching agents at some point in the middle of trajectories, always starting with the conservative agent, is a heuristic that is not well-motivated enough and not analyzed enough in the paper. Theorem 6.2 shows that this backwards expansion minimizes worst-case degradation, which makes sense because degradation at later parts matters less than earlier parts. Nonetheless, there may be many tasks and scenarios in which this heuristic would break down. For example, in robotics tasks in which the strategy taken at the beginning of episodes might determine future success (say, early grasping strategy might determine how likely an object placement will be to succeed), backwards expansion might not be exploratory enough.

To provide additional evidence to support our previous response to this question, we ran an additional experiment where the exploratory policy acts from the beginning of the episode, and the switch horizon h expands from there. This is the opposite scheme of what we are proposing with COE (exploratory policy at the end of episode). We train this variant on 4 adroit and 2 locomotion datasets while keeping every other component of the COE the same as before. The results are shown in Figure 8 of the main text. We observe that not only the performance degradation is significantly higher in this case, but the final policy is not able to improve as much as if the exploration was done from the end of the episodes.

Reviewer_hPzPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Cautious Online Expansion (COE) algorithm for offline-to-online RL. The idea is to avoid performance degradation during the initial part of the online RL phase as the agent rapidly shifts from using the offline teacher distribution prior to exploring on its own. The main idea of COE is to maintain two agents, a conservative agent updated periodically via an offline RL algorithm (such as InAC, a modification of which the authors use), and an exploratory agent updated online (such as SAC), gradually expanding the ability of the exploratory agent to take actions while resorting to the conservative agent in most turns. The authors provide both theoretical and small-scale experimental support for the algorithm especially highlighting the lack of initial performance degradation.

Critical strengths

The algorithm makes intuitive sense (at least why it would work), probably due to good presentation by the authors. The theory substantially motivates the algorithm: within the setting of the paper, Theorem 6.1 says that the best we can hope for is a gradual transition from offline-trained agents to online-trained agents, and that is the spirit of the COE algorithm. Other theory motivates different parts of the algorithm. The empirical verification is quite thorough, testing over 16 environments, many ablations on each component of the COE algorithm, etc. It is nice that the authors present and justify the success metrics ahead of the empirical results. And the algorithm seems both highest-performing and lowest-degradation, which is a strong empirical finding.

Critical weaknesses

A lot of hyperparameters seem empirically chosen via thorough sweeps. It would be nice to have proofs of optimality of these schemes, understanding of how certain parameters (hh in particular) should change with others, etc.

Questions for authors

See weaknesses.

1 comment

Thank you for the positive summary. Note that we have updated the paper to expand the experimental support with 6 additional environments from a different benchmark suite for all algorithms (see Figures 6, 14, 15 and Table 1). These results further demonstrate the effectiveness of our method.

A lot of hyperparameters seem empirically chosen via thorough sweeps. It would be nice to have proofs of optimality of these schemes, understanding of how certain parameters (h in particular) should change with others, etc

We have clarified how hyperparameters are selected in the paper, specifically we have added “We aimed for minimal tuning, setting to reasonable defaults for most hyperparameters based on performance in the original work and only tuning one or two hyperparameters that were particularly sensitive. This tuning was done across environments (using only a random subset of 5 environments) to find the best set per algorithm, and then fixed for all datasets. This tuning, the resulting hyperparameters and other experimental details are described in more detail in Appendix E2.” We would like to point out that in the new environments (from NeoRL2), we reused these same hyperparameters without any additional hyperparameter sweeping and observed that the performance is consistent, providing some evidence that they do generalize.

We do not have theoretical proofs of optimality for the hyperparameters but we do empirically study the parameter sensitivity (for the newly introduced hyperparameters) in Appendix E.2. As we see from Figure 10, changing the values of these hyperparameters does not cause a very significant change in performance or degradation across the suite. The exception is the n_min (minimum steps before the switch horizon h can reach 0): since most of the degradation happens in the early phase of online training, setting n_min such that the value is not too low generally performs equivalently well. Additionally, Figure 12 shows how the horizon h changes over time across all the environments that we tested.


Update: We ran an additional experiment where the exploratory policy acts from the beginning of the episode, and the switch horizon h expands from there. This is the opposite scheme of what we are proposing with COE (exploratory policy at the end of episode). We train this variant on 4 adroit and 2 locomotion datasets while keeping every other component of the COE the same as before. The results are shown in Figure 8 of the main text. We observe that when exploration is done from the beginning of the episode, the performance degradation is significantly higher and final policy performance is significantly lower than the proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.