acceptodds
Under review as a conference paper at ICLR 2027

The Three Regimes of Offline-to-Online Reinforcement Learning

Abstract

Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online fine-tuning that work well in one setting can fail completely in another. Inspired by the Stability–Plasticity principle, we propose a framework that can explain this inconsistency: We argue that efficient fine-tuning must preserve the utility of the stronger offline prior, whether that is the pretrained policy or the offline dataset, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 52 out of 73 cases, with only 3 opposite mismatches. This work provides a framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, offline-to-online RL, online fine-tuning, stability and plasticity

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_5nH8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Offline-to-online RL has a problem of different algorithms performing better or worse on different types of tasks, which makes it hard for practitioners to know which method to use. Authors introduce a framework that attempts to explain why certain classes of methods will perform better for certain classes of environment-dataset tuples, through the lens of stability-plasticity trade-off.

Critical strengths
  • Authors perform experiments on a wide range of environments and base algorithms.
Critical weaknesses
  • Authors do not mention an explicit policy of how the order of performances was decided (Table 9), and there are some inconsistencies. E.g. For hopper-random-v2, ReBRAC was given ≈\approx with a difference of 0.07 with the dataset performance, but given > on walker2d-random-v2 with a difference of 0.058 in performance. Likewise, on the antmaze domains, on ultra-diverse CalQL and ReBRAC were give ≈\approx with differences of 0.065 and 0.074 respectively. And BC was give ≈\approx for antmaze-ultra-diverse while < for antmaze-large-play, while both have 0. Since these choices of ordering alter the main results of the paper, can the authors share the policy that they used across these comparisons to obtain all the orderings from Tables 9 and 11? Currently, orderings appear hand-picked whenever ambiguous.

  • The method classifications into dataset-centric or π0\pi_0-centric are broken. For example, RLPD is categorized as a method that uses stability relative to the offline dataset. While RLPD does use the offline dataset during training, it is inaccurate to say that it “promotes stability around the offline dataset”. It uses transitions from the offline dataset for TD updates, but it does not use any regularization that pulls either the policy toward the behavior distribution from the dataset, or the value function towards assigning seen states or actions higher values.

Lines 240-247 argue that whenever offline data is used with regularization, they additionally categorize it as promoting stability towards both π0\pi_0 and D. They also mention that this regularization is applied to data that comes from a sequence of policies that includes π0\pi_0, this implicitly anchors learning around the region visited by π0\pi_0, even if π0\pi_0 is not stored during fine-tuning. These generalizations make the category meaningless.

Lines 255-256 argue that RLPD can be interpreted as using a parameter reset to increase stability. This categorization does not seem reasonable, since RLPD just does not use pretraining.

Overall, the categories used in the paper seem unnatural, and are very poorly justified. In fact, there is no explicit list or table for which methods belong to which category, even though this categorization plays a key role in their main results.

  • The categorization of a method relies on seeing its performance compared with the performance of the training data, after having performed training. What is the practical value of the framework, if it only offers recommendations after having performed the expensive training?

  • Lines 317-319 argue that when pretrained policy achieves substantially higher performance than offline dataset, e.g. when dealing with suboptimal datasets, the primary concern becomes preserving stability relative to π0\pi_0. This is not necessarily true. The mechanism that most offline RL algorithms use to learn from suboptimal datasets to begin with is TD learning. And during online fine-tuning, that same mechanism can be used to keep using the suboptimal offline dataset, irrespective of π0\pi_0.

Questions for authors
  • What was the algorithm used to create tables 9 and 11?
  • Can authors provide a table with algorithm classifications that they used for their main results?
4 comments

We thank you for your careful review and valuable feedback. We address your comments and concerns below.

W1&Q1: ...Since these choices of ordering alter the main results of the paper, can the authors share the policy that they used across these comparisons to obtain all the orderings from Tables 9 and 11? Currently, orderings appear hand-picked whenever ambiguous.

The regime assignments in Tables 9 and 11 follow the same fixed statistical procedure described in Appendix B.1 (Eqs. 6–7). Specifically, we conduct two one-sided Welch t-tests with a performance margin δ=0.05\delta=0.05 and significance level α=0.05\alpha=0.05.

Let Δ=μ0−μD\Delta=\mu_0-\mu_{\mathcal D} denote the difference between the mean returns of the pretrained policy and offline dataset. We test

\quad \text{vs.} \quad H_{1,+}: \Delta > \delta,$$ $$H_{0,-}: \Delta \geq -\delta \quad \text{vs.} \quad H_{1,-}: \Delta < -\delta.$$ A setting is classified as Superior if $H_{0,+}$ is rejected, Inferior if $H_{0,-}$ is rejected, and Comparable otherwise. Thus, regime assignment depends on both the mean return difference and its uncertainty, rather than the mean difference alone. The corresponding one-sided p-values are reported below. Here, $p_+$ corresponds to the test of $\Delta>\delta$, and $p_-$ corresponds to the test of $\Delta<-\delta$. Values are rounded for readability. | Pretrained policy | Environment | $p_+$ | $p_-$ | Assigned regime | |---|---|---:|---:|---| | ReBRAC | `hopper-random-v2` | 0.0568 | $\approx 1$ | Comparable | | ReBRAC | `walker2d-random-v2` | $3.81\times10^{-10}$ | $\approx 1$ | Superior | | CalQL | `antmaze-ultra-diverse-v2` | 0.2709 | 0.9997 | Comparable | | ReBRAC | `antmaze-ultra-diverse-v2` | 0.2920 | 0.9921 | Comparable | | BC | `antmaze-ultra-diverse-v2` | $\approx 1$ | 0.3335 | Comparable | | BC | `antmaze-large-play-v2` | $\approx 1$ | $8.88\times10^{-9}$ | Inferior | These p-values reproduce all six original assignments under the same decision rule and explain the examples raised in the review: - **ReBRAC on Hopper-random versus Walker2d-random:** Hopper-random does not meet the significance threshold for Superior ($p_+=0.0568$), whereas Walker2d-random does. Despite Hopper’s larger mean gap, its policy scores have substantially larger standard deviation (0.036 versus 0.001), illustrating why the mean gap alone does not determine the assignment. - **CalQL and ReBRAC on AntMaze-ultra-diverse:** Although their mean gaps exceed 0.05, neither superiority test is significant ($p_+=0.2709$ and $0.2920$, respectively). Neither inferiority test is significant either, so both settings are classified as Comparable. - **BC on AntMaze-ultra-diverse versus AntMaze-large-play:** BC scores zero in both settings, but the dataset means differ (0.053 versus 0.105). The inferiority test is not significant on Ultra-diverse ($p_-=0.3335$), but is significant on Large-play ($p_-=8.88\times10^{-9}$), yielding Comparable and Inferior assignments, respectively. We have added a reference in the main text of the revised manuscript to the detailed regime assignment procedure in the appendix.

W2-1: The method classifications into dataset-centric or π0\pi_0-centric are broken. For example, RLPD is categorized as a method that uses stability relative to the offline dataset. While RLPD does use the offline dataset during training, it is inaccurate to say that it “promotes stability around the offline dataset”. It uses transitions from the offline dataset for TD updates, but it does not use any regularization that pulls either the policy toward the behavior distribution from the dataset, or the value function towards assigning seen states or actions higher values.

We agree that RLPD does not use any regularization that pulls either the policy toward the behavior distribution from the dataset, but the offline data used during the online training lead to still learn from the knowledge in the offline data.

Section 5.4 (Figure 5) provides empirical support for this interpretation: offline replay reduces offline TD errors and avoids the Q-value divergence. These results support a role for replay in stabilizing critic learning on the offline data.

W2-2: Lines 240-247 argue that whenever offline data is used with regularization, they additionally categorize it as promoting stability towards both π0\pi_0 and D\mathcal D. They also mention that this regularization is applied to data that comes from a sequence of policies that includes π0\pi_0, this implicitly anchors learning around the region visited by π0\pi_0, even if π0\pi_0 is not stored during fine-tuning. These generalizations make the category meaningless.

We classify continued regularization as π0\pi_0-centric because it uses online data initially generated by π0\pi_0 as the reference for regularized learning. ReBRAC penalizes differences between policy actions and recorded actions, so early samples carry pretrained behavior without requiring a frozen copy of π0\pi_0. For CalQL, the connection is through conservative critic regularization relative to the training data, rather than an explicit penalty on policy actions. The online reference evolves as updated policies collect new experience.

In the mixed variant, both online and offline samples contribute to regularization. This distinguishes the mixed variant from offline replay alone, which uses these data without the additional offline RL regularization. Aggregate stability, plasticity, and improvement measurements across all 73 settings, including the mixed variant, are provided in Common Response C1, together with three illustrative examples.

W2-3: Lines 255-256 argue that RLPD can be interpreted as using a parameter reset to increase stability. This categorization does not seem reasonable, since RLPD just does not use pretraining.

Section 4.3 discusses parameter reset as a way to increase plasticity, rather than stability. RLPD trains from random initialization using both offline and online data. Our baseline fine-tunes a pretrained agent using standard online RL with only online data and no additional components. Relative to this baseline, our + offline data + reset variant captures RLPD’s initialization and replay choices in a controlled comparison.

We do not mean that RLPD literally resets a pretrained agent. Representing random initialization as a full reset makes it easier to compare training from scratch with fine-tuning a pretrained agent under the same training protocol.

Q2: Can authors provide a table with algorithm classifications that they used for their main results?

The main experiments evaluate six variants built from the design choices in Section 4. Their initialization, added components, and classifications are summarized below. The categories describe which source of prior knowledge each added component emphasizes.

Fine-tuning variantInitializationAdded componentCategory
BaselinePretrained agentNone; standard online RL using online dataBaseline
+ warmupPretrained agentCollect 5,000 online transitions using π0\pi_0 before updatesπ0\pi_0-centric
+ offline RLPretrained agentContinue pretraining regularization using online dataπ0\pi_0-centric
+ offline dataPretrained agentReplay offline data with α=0.5\alpha=0.5D\mathcal D-centric
+ offline data + resetRandom initializationReplay offline data with α=0.5\alpha=0.5D\mathcal D-centric
+ offline RL + offline dataPretrained agentCombine regularization and offline data replayMixed π0+D\pi_0+\mathcal D

W3: The categorization of a method relies on seeing its performance compared with the performance of the training data, after having performed training. What is the practical value of the framework, if it only offers recommendations after having performed the expensive training?

We would like to clarify that our framework distinguishes between method classification and regime assignment, neither of which requires observing online fine-tuning results.

  • First, fine-tuning methods are classified based on their design choices, rather than their performance. For example, warmup is classified as π0\pi_0-centric because it emphasizes the pretrained policy, while offline data replay is classified as D\mathcal D-centric because it continues to use the offline dataset. These classifications are independent of the observed fine-tuning outcomes.
  • Second, regime assignment depends on the relative performance of the pretrained policy and offline dataset. In the offline-to-online RL setting studied in our paper, offline pretraining has already been completed before selecting a fine-tuning strategy. Thus, J(π0)J(\pi_0) and J(πD)J(\pi_{\mathcal D}) can be estimated before online fine-tuning, without training any candidate fine-tuning methods.

In our empirical study, we compare the strongest observed variants within each method family to evaluate the regime-based predictions. This retrospective comparison is separate from the regime assignment itself.

The practical goal of our framework is to guide the choice of fine-tuning strategies before expensive online training, by indicating whether to emphasize the pretrained policy or the offline dataset. It aims to narrow the choice of methods rather than requiring practitioners to train and compare all candidate methods.

W4. Lines 317-319 argue that when pretrained policy achieves substantially higher performance than offline dataset, e.g. when dealing with suboptimal datasets, the primary concern becomes preserving stability relative to π0\pi_0. This is not necessarily true. The mechanism that most offline RL algorithms use to learn from suboptimal datasets to begin with is TD learning. And during online fine-tuning, that same mechanism can be used to keep using the suboptimal offline dataset, irrespective of π0\pi_0.

We agree that suboptimal offline data can still support further improvement through TD learning. The relative returns do not, by themselves, determine how much additional value the dataset can provide. Our hypothesis is that they nevertheless provide useful guidance on which prior to emphasize during fine-tuning.

Our experiments support this hypothesis: in the Superior regime, the strongest tested π0\pi_0-centric variant outperforms the strongest tested D\mathcal D-centric variant in 24 of 32 settings, with six showing no statistically significant difference and two favoring D\mathcal D-centric methods.

Figure 5 further illustrates the distinction between learning from offline data and improving fine-tuning performance. Offline replay shows lower offline TD loss in both illustrated settings, but warmup achieves higher return in the Superior setting, whereas replay achieves higher return in the Inferior setting. Thus, the benefit of replay for offline TD learning does not translate into the same performance advantage across regimes.

We revised the statement in Section 5.1 to distinguish this empirical recommendation from a general claim.

Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed rebuttal addressing your comments and concerns. We would greatly appreciate it if you could take a look at our responses and let us know whether they adequately address your concerns.

Thank you for your time and consideration.

Reviewer_bZQ7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper conducts an empirical study to analyze how the offline pre-trained policies and the offline dataset predicts fine-tuning performances of different algorithms. These algorithms are grouped by how much it relies on the pre-trained policy (π\pi-centric) vs. offline dataset (D\mathcal{D}-centric). The authors uncover a trend where π0\pi_0-centric fine-tuning algorithms tend to work well when pre-trained policy has better performance compared to the the behavior policy, and vice versa.

Critical strengths
  • The idea of organizing algorithms based on stability anchors is nice. The experiments are comprehensive.
Critical weaknesses
  • The stability/plasticity decomposition is not being measured, and it is unclear how they connect to the regimes in practice.
Questions for authors

N/A

3 comments

We thank you for your careful review and valuable feedback. We address your comments and concerns below.

W: The stability/plasticity decomposition is not being measured, and it is unclear how they connect to the regimes in practice.

We provide aggregate stability, plasticity, and improvement changes relative to baseline across all 73 settings, covering all six fine-tuning variants, in Common Response C1 (“Stability and plasticity measurements”). The aggregate results are equally weighted means across settings with pointwise 95% confidence intervals from a hierarchical bootstrap over environment–dataset clusters and training seeds. We also retain three illustrative examples reporting absolute measurements as mean ± standard deviation across ten seeds.

The aggregate analysis shows how the trade-off differs across regimes: offline RL regularization increases stability but reduces plasticity and improvement in the Superior regime, while replay with or without reset increases plasticity and improvement in the Inferior regime. We have added these measurements and their interpretation to the appendix, with a pointer in Section 5.

Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed rebuttal addressing your comments and concerns. We would greatly appreciate it if you could take a look at our responses and let us know whether they adequately address your concerns.

Thank you for your time and consideration.

Thanks for the additional experiments on measuring stability/plasticity/improvement under different regimes. Overall, the results roughly support the main arguments of the paper (that algorithms under different regimes exhibit different stability/plasticity profiles). The caveat is that these results do seem to be a bit noisy.

I remain positive about the paper because I believe the analysis framework of stability (anchored to either π0\pi_0 or D\mathcal{D}) is valuable.

Reviewer_34NeOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates why different offline-to-online RL strategies succeed or fail across settings. It proposes a three-regime taxonomy based on the relative returns of the pretrained policy and the offline dataset: Superior, Comparable, and Inferior. Through a stability–plasticity perspective, the authors hypothesize that fine-tuning should primarily preserve the stronger offline prior. They evaluate this hypothesis using warmup, offline regularization, data replay, and parameter reset across 73 experimental settings, reporting agreement with the predicted outcomes in 52 cases.

Critical strengths
  1. The research question is relevant and practically meaningful. Understanding when to rely on pretrained parameters versus continued access to offline data is important for designing effective fine-tuning procedures. The proposed distinction provides an intuitive way to organize seemingly inconsistent empirical findings.

  2. The empirical evaluation is substantial and systematic. The study includes multiple D4RL domains, several pretraining algorithms, and additional Robomimic experiments with flow-based policies.

Critical weaknesses
  1. The proposed taxonomy relies solely on the average return of the offline dataset, overlooking the heterogeneity of data quality. Even when the pretrained policy outperforms the dataset on average, the dataset may still contain high-return trajectories that provide valuable information for further policy improvement[1]. Therefore, the inferior performance of DD-centric methods in the Superior regime may result from uniformly replaying low-quality transitions rather than the limited utility of offline data itself. The paper only evaluates uniform offline data replay with a fixed ratio of 0.5, without investigating whether selectively replaying high-return trajectories would change the observed conclusions.

  2. Dependence on online policy evaluation limits the practical applicability of the proposed framework. The regime classification requires evaluating the return of the pretrained policy before online fine-tuning. However, directly evaluating the pretrained policy may be infeasible in real-world applications where environment interactions are constrained by safety requirements or deployment costs. Conversely, if online evaluation is feasible, the collected interaction data could also be used for online data warmup, which is not incorporated into the D-centric methods evaluated in the paper, potentially introducing an inconsistency in the comparison. One possible alternative is to estimate the pretrained policy's return using off-policy evaluation (OPE). However, this raises an important question that remains unexplored: whether estimation errors would affect regime classification and whether the proposed framework could maintain reliable predictive performance when policy returns are estimated solely from offline data.

  3. The proposed performance decomposition provides limited guidance for fine-tuning strategy selection across the three regimes. The stability–plasticity decomposition in Section 3.2 is essentially an algebraic identity derived directly from the definitions of the corresponding metrics and holds for any online fine-tuning method. However, it only provides a post-hoc characterization of performance without explaining how different fine-tuning strategies affect stability and plasticity, or how these effects relate to the relative returns of the two offline priors. Consequently, even given the relative performance of the pretrained policy and the offline dataset, the decomposition cannot determine which category of fine-tuning methods would achieve better performance. Therefore, the proposed decomposition does not provide substantive explanatory or practical guidance for selecting fine-tuning strategies across the three regimes.

  4. Although the paper aims to provide general guidance for offline-to-online RL fine-tuning, its empirical analysis mainly focuses on a limited set of basic design choices, including warmup, offline regularization, replay, and parameter reset. Comparisons with other representative approaches, such as OPT [2], are missing, while WSRL [3] is represented primarily through its warmup strategy. These methods suggest that retaining the entire offline dataset may not always be necessary, and that selectively utilizing high-quality offline data or improving critic estimation may be more important for effective online fine-tuning. In particular, OPT highlights the importance of critic adaptation during the offline-to-online transition, suggesting that fine-tuning performance may depend on value estimation quality rather than solely on the relative returns of the pretrained policy and offline dataset.

[1] Yu, Zishun, and Xinhua Zhang. "Actor-critic alignment for offline-to-online reinforcement learning." International Conference on Machine Learning. PMLR, 2023. [2] Shin, Yongjae, et al. "Online pre-training for offline-to-online reinforcement learning." arXiv preprint arXiv:2507.08387 (2025). [3] Zhou, Zhiyuan, et al. "Efficient online reinforcement learning fine-tuning need not retain offline data." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors

See Critical Weaknesses

7 comments

We thank you for your careful review and valuable feedback. We address your comments and concerns below.

W1: The proposed taxonomy relies solely on the average return of the offline dataset, overlooking the heterogeneity of data quality. Even when the pretrained policy outperforms the dataset on average, the dataset may still contain high-return trajectories that provide valuable information for further policy improvement. Therefore, the inferior performance of DD-centric methods in the Superior regime may result from uniformly replaying low-quality transitions rather than the limited utility of offline data itself. The paper only evaluates uniform offline data replay with a fixed ratio of 0.5, without investigating whether selectively replaying high-return trajectories would change the observed conclusions.

To address this concern, we conducted additional experiments in four Superior AntMaze settings, restricting offline replay to successful trajectories. We provide the replay configurations, complete results, and discussion in Common Response C2 (“Selective replay and the value of offline data”).

W2-1: Dependence on online policy evaluation limits the practical applicability of the proposed framework. The regime classification requires evaluating the return of the pretrained policy before online fine-tuning. However, directly evaluating the pretrained policy may be infeasible in real-world applications where environment interactions are constrained by safety requirements or deployment costs.

We agree that evaluating the pretrained policy can be costly or infeasible in some real-world applications. However, in settings where online fine-tuning is feasible, estimating the performance of the pretrained policy π0\pi_0 typically requires relatively few interactions compared with the entire fine-tuning process.

W2-2:Conversely, if online evaluation is feasible, the collected interaction data could also be used for online data warmup, which is not incorporated into the D-centric methods evaluated in the paper, potentially introducing an inconsistency in the comparison.

We would like to clarify that policy evaluation and online data warmup serve different purposes. Evaluation measures the pretrained policy's performance without exploration noise, while warmup collects exploratory data. We do not use evaluation trajectories for training in any of the fine-tuning methods. Therefore, all methods follow the same evaluation protocol, and the additional exploratory data collected during warmup are a design choice being evaluated, rather than an inconsistency in the comparison.

W2-3:One possible alternative is to estimate the pretrained policy's return using off-policy evaluation (OPE). However, this raises an important question that remains unexplored: whether estimation errors would affect regime classification and whether the proposed framework could maintain reliable predictive performance when policy returns are estimated solely from offline data.

Following this suggestion, we used fitted Q evaluation (FQE), an off-policy evaluation (OPE) method, to estimate the pretrained policy’s return and replace its online evaluation scores in regime assignment. For each frozen policy π0\pi_0, we fit a separate value network on offline transitions, using actions from π0\pi_0 in the Bellman targets. We then average the predicted values over initial states to estimate policy performance, using each task’s evaluation horizon and return or success metric. Initial states come from known benchmark reset distributions for D4RL and offline demonstration starts for Robomimic. This requires no new policy rollouts.

We substitute these estimates for the online evaluation scores in the same regime-assignment procedure (δ=0.05\delta=0.05, α=0.05\alpha=0.05), keeping the dataset scores and historical fine-tuning outcomes unchanged. Across 73 settings, agreement with these outcomes is 38/73 (52.1%), compared with 52/73 (71.2%) using online evaluation.

Fine-tuning outcome / Pretraining regime (FQE)SuperiorComparableInferior
π0\pi_0-centric >> D\mathcal{D}-centric15111
π0\pi_0-centric ≈\approx D\mathcal{D}-centric490
π0\pi_0-centric << D\mathcal{D}-centric61314

The confusion matrix shows 28 adjacent mismatches and 7 opposite mismatches. Appendix H provides the FQE evaluation procedure and results.

These results show that OPE estimates can replace online evaluation as inputs to the framework, but the tested FQE implementation does not preserve the agreement obtained with online evaluation.

W3: The proposed performance decomposition provides limited guidance for fine-tuning strategy selection across the three regimes. The stability–plasticity decomposition in Section 3.2 is essentially an algebraic identity derived directly from the definitions of the corresponding metrics and holds for any online fine-tuning method. However, it only provides a post-hoc characterization of performance without explaining how different fine-tuning strategies affect stability and plasticity, or how these effects relate to the relative returns of the two offline priors. Consequently, even given the relative performance of the pretrained policy and the offline dataset, the decomposition cannot determine which category of fine-tuning methods would achieve better performance. Therefore, the proposed decomposition does not provide substantive explanatory or practical guidance for selecting fine-tuning strategies across the three regimes.

We agree that the performance decomposition is an algebraic identity. Its purpose is to provide a common perspective for understanding fine-tuning behavior in terms of stability and plasticity, rather than directly determining the best fine-tuning method.

Our regime-based predictions are a separate empirical hypothesis: fine-tuning should emphasize preserving information from the stronger offline prior while maintaining sufficient plasticity. The relative performance of the pretrained policy and offline dataset provides a simple metric for identifying which prior to emphasize.

To quantify these trade-offs, we computed stability, plasticity, and improvement using Eqs. 4–5 for all six fine-tuning variants across all 73 settings. Common Response C1 (“Stability and plasticity measurements”) reports aggregate changes relative to baseline with pointwise 95% confidence intervals from a hierarchical bootstrap, alongside three illustrative examples with absolute measurements.

The aggregate results show that, in the Superior regime, offline RL regularization increases stability but reduces plasticity and improvement relative to baseline. In the Inferior regime, replay with or without reset increases plasticity and improvement. These measurements quantify the observed stability–plasticity trade-offs across settings.

The regime classification provides guidance on which source of prior knowledge to emphasize before online fine-tuning, while the decomposition helps interpret the observed behavior of different methods. We have added the aggregate measurements, bootstrap procedure, and three illustrative examples to the appendix, with a pointer in Section 5.

W4:Although the paper aims to provide general guidance for offline-to-online RL fine-tuning, its empirical analysis mainly focuses on a limited set of basic design choices, including warmup, offline regularization, replay, and parameter reset. Comparisons with other representative approaches, such as OPT, are missing, while WSRL is represented primarily through its warmup strategy. These methods suggest that retaining the entire offline dataset may not always be necessary, and that selectively utilizing high-quality offline data or improving critic estimation may be more important for effective online fine-tuning. In particular, OPT highlights the importance of critic adaptation during the offline-to-online transition, suggesting that fine-tuning performance may depend on value estimation quality rather than solely on the relative returns of the pretrained policy and offline dataset.

Our study examines how common fine-tuning design choices interact with the relative performance of the pretrained policy and offline dataset. We compare six variants under a shared training protocol across 73 settings, allowing us to isolate the common fine-tuning design choices. This controlled comparison is central to our research question.

These components capture training strategies used by representative methods. Baseline + warmup captures WSRL-style fine-tuning, which initializes online training with data collected by the pretrained policy and discards offline data. Baseline + reset + offline data replay captures RLPD-style training from random initialization using both offline and online data. Baseline + offline data replay + offline RL regularization represents the common practice of continuing regularized learning with offline and online data during fine-tuning, as in AWAC [4], CalQL [5], and ReBRAC [6]. Comparing these combinations under a shared protocol allows us to examine how their effectiveness changes across regimes.

OPT highlights the importance of critic adaptation. We agree that critic quality can influence fine-tuning outcomes, and Section 5.4 examines critic learning through TD errors and Q-values. Our hypothesis is that relative prior performance provides useful guidance for the design choices we study; it does not require prior returns to explain every factor affecting performance. Our experiments do not establish whether the same relationship holds for OPT’s additional critic-adaptation mechanism. We have added OPT to the related work in the revised manuscript, describing its additional critic-training phase and its use of two critics during fine-tuning.

We also examined the selective use of offline data through additional experiments in four Superior AntMaze settings, replaying only successful offline trajectories. Warmup achieves the highest mean in three settings, while replay remains competitive in the fourth. The configurations and complete results are provided in Common Response C2 (“Selective replay and the value of offline data”).

[4] Nair, Ashvin, et al. "AWAC: Accelerating Online Reinforcement Learning with Offline Datasets." arXiv preprint arXiv:2006.09359 (2020).

[5] Nakamoto, Mitsuhiko, et al. "Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning." Advances in Neural Information Processing Systems 36 (2023).

[6] Tarasov, Denis, et al. "Revisiting the Minimalist Approach to Offline Reinforcement Learning." Advances in Neural Information Processing Systems 36 (2023).

Reviewer_R71UOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a three-regime taxonomy for offline-to-online RL based on the relative return of the pretrained policy J(π0)J(\pi_0) and the offline dataset J(πD)J(\pi_D): Superior when J(π0)>J(πD)J(\pi_0)>J(\pi_D), Comparable when J(π0)≈J(πD)J(\pi_0)\approx J(\pi_D), and Inferior when J(π0)<J(πD)J(\pi_0)<J(\pi_D). The main idea is that online fine-tuning should anchor stability to whichever offline prior is stronger. That means π0\pi_0-centric designs such as online data warmup and offline RL regularization in the Superior regime, and DD-centric designs such as offline data replay, with or without a parameter reset, in the Inferior regime. The paper motivates this with a decomposition of the best fine-tuning return into an offline prior term, a stability term, and a plasticity term, defined in Eq. 2–5. The study covers 21 D4RL task–dataset pairs with CalQL, ReBRAC, and BC pretraining, plus 10 Robomimic settings with SAC Flow. Table 1 reports that the regime label predicts the better method class in 52 of 73 settings, with 3 opposite mismatches. Section 5.4 further analyzes offline TD loss and Q-values during fine-tuning, and Section 5.5 compares against alternative taxonomies.

Critical strengths
  • A large and transparent study. The study runs 6 fine-tuning variants with 10 seeds each across 73 settings and three pretraining algorithms, and the code is released. The confusion matrix in Table 1 makes the central claim easy to check.
  • A simple diagnostic for a real conflict in the literature. Figure 1 shows a useful reversal between WSRL and RLPD. In Table 2, the raw-return label is right 71% of the time, compared with 49% for a Q-value taxonomy and 36% for a BC-performance taxonomy. Section 5.4 also gives a plausible mechanism: Figure 5 shows that without offline replay in the Inferior setting, offline TD loss exceeds 10310^3 and the Q-values diverge.
Critical weaknesses
  1. The stability–plasticity framework does not explain the results beyond the regime label.

    • The decomposition after Eq. 5 is an algebraic identity. With Stability =min⁡nJ(πn)−Joff∗=\min_n J(\pi_n)-J^*_{\text{off}} and Plasticity =max⁡nJ(πn)−min⁡nJ(πn)=\max_n J(\pi_n)-\min_n J(\pi_n), the three terms sum to max⁡nJ(πn)\max_n J(\pi_n) for any training run, so the decomposition alone predicts nothing. Also, Section 5 never measures the quantities defined in Eq. 2–5.
    • Part of the evidence contradicts the framing. Section 5.1 reports that, in the Superior regime, the π0\pi_0-centric method with the strongest stability, offline RL regularization, loses to warmup in 27 of 32 settings.
    • To me, the Superior finding reduces to "do not replay low-quality data when π0\pi_0 is already better", which is the message of WSRL [1]. The Inferior finding reduces to "replay near-expert data, possibly with fresh networks", which is the message of RLPD [2]. Figure 1 is exactly a comparison between these two methods.

    As a result, the paper reads more as a post-hoc taxonomy of known findings than as a predictive principle. The authors should report the stability and plasticity values from Eq. 4–5 for each method class and regime, and show that these quantities predict something the regime label alone does not.

  2. The regime assignment and the statistics are fragile.

    • Accuracy depends on the margin δ\delta. Table 4 shows that with δ=0.1\delta=0.1 accuracy drops to 41 of 73 settings, or 56%. Table 3 shows that with δ=0\delta=0 accuracy stays at 52 of 73, but no Comparable setting is predicted correctly.
    • Section 5.3 reports that at δ=0.05\delta=0.05 only 4 of 13 Comparable settings are statistically indistinguishable. The paper then reinterprets them with a mean-gap below 0.1, a criterion not used anywhere else. The evidence therefore supports a two-regime rule at best.
    • Appendix B.2 pools the last 10 evaluations of each seed for the t-tests. These evaluations are strongly correlated, so treating them as independent samples inflates significance.
    • Each setting compares only the strongest π0\pi_0-centric method against the strongest DD-centric method. The mixed π0+D\pi_0+D variant is left out, and the warmup length K=5000K=5000 and the offline ratio α=0.5\alpha=0.5 are fixed in all settings.
  3. The benchmarks make the prediction easy.

    • Table 9 shows that the Inferior regime is dominated by datasets with J(πD)=1.0J(\pi_D)=1.0, namely pen-, door-, and relocate-binary and kitchen-complete, and by BC policies with zero return on AntMaze. The Superior regime is dominated by MuJoCo random and medium-replay data. In both extremes, it is obvious which offline prior is stronger without any framework.
    • Table 10 shows that Robomimic adds no Superior case.
    • The informative case is medium-quality data where both π0\pi_0 and DD are useful. That is exactly the Comparable regime, where the framework is weakest.
    • Appendix D.1 notes that the AntMaze dense proxy ignores the maze walls. This makes the sparse-reward comparison in Table 2, 76% for raw return against 59% for the dense proxy, less informative.

    It will be great if the authors could evaluate on OGBench [3], for example its navigate, stitch, and explore datasets, where dataquality varies without being random or expert.

[1] Zhou et al. Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data. ICLR 2025.

[2] Ball et al. Efficient Online Reinforcement Learning with Offline Data. ICML 2023.

[3] Park et al. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR 2025.

Questions for authors

See weaknesses

6 comments

We thank you for your careful review and valuable feedback. We address your comments and concerns below.

W1: The stability–plasticity framework does not explain the results beyond the regime label.

W1-1: The decomposition after Eq. 5 is an algebraic identity. The authors should report the stability and plasticity values from Eq. 4–5 for each method class and regime, and show that these quantities predict something the regime label alone does not.

We agree that the decomposition is an identity. Its purpose is to help understand the trade-off between retaining useful pretrained knowledge and allowing further improvement during online fine-tuning. We provide aggregate stability, plasticity, and improvement changes relative to baseline across all 73 settings, covering all six variants, in Common Response C1 (“Stability and plasticity measurements”). These results include pointwise 95% confidence intervals from a hierarchical bootstrap, alongside three illustrative examples with absolute measurements. The quantities are computed from completed fine-tuning results and do not establish additional predictive power beyond the regime labels. Their role is to help assess whether the observed behavior of different fine-tuning methods is consistent with the stability–plasticity trade-off.

W1-2: Part of the evidence contradicts the framing. Section 5.1 reports that, in the Superior regime, the (π0)(\pi_0)-centric method with the strongest stability, offline RL regularization, loses to warmup in 27 of 32 settings.

The 27/32 comparison is consistent with the stability–plasticity trade-off. Our framework emphasizes preserving useful pretrained knowledge while retaining sufficient plasticity for improvement. Stronger stability alone need not lead to better fine-tuning performance.

The aggregate measurements in Common Response C1 (“Stability and plasticity measurements”) show this across all 32 Superior settings. Relative to baseline, offline RL regularization increases stability by 0.199 on average, but reduces plasticity by 0.311 and improvement by 0.112. Warmup increases improvement by 0.038 on average. The corresponding 95% confidence intervals and three illustrative examples are provided in Common Response C1.

W1-3: To me, the Superior finding reduces to "do not replay low-quality data when (π0)(\pi_0) is already better", which is the message of WSRL. The Inferior finding reduces to "replay near-expert data, possibly with fresh networks", which is the message of RLPD. Figure 1 is exactly a comparison between these two methods.

Our contribution is a framework for identifying which source of offline knowledge to emphasize during fine-tuning and for interpreting the stability–plasticity trade-offs of common design choices. The central criterion is the performance of the pretrained policy relative to the offline dataset, which can guide different choices even when the dataset is unchanged.

For example, on kitchen-partial-v0, the dataset return is 0.586, while CalQL and ReBRAC achieve pretrained returns of 0.764 and 0.133, respectively (Table 9). These initializations therefore fall into the Superior and Inferior regimes, and Figures 6–7 show the corresponding reversal in the preferred method family. Dataset quality alone does not distinguish these two settings.

Figure 1 motivates this question through the WSRL–RLPD comparison. We evaluate the proposed criterion through controlled comparisons of six variants across 73 settings, complemented by the critic-learning analysis in Section 5.4 and comparisons with alternative taxonomies in Section 5.5. Together, these analyses examine when emphasizing each offline prior is effective and how the associated fine-tuning behavior differs.

W2. The regime assignment and statistics are fragile.

W2-1. Accuracy depends on the margin δ\delta.

We agree that prediction accuracy depends on the margin. We did not optimize δ\delta for prediction accuracy. We report results for δ=0\delta=0, 0.050.05, and 0.10.1 to examine how the framework’s predictions change as we require a larger performance gap between the two offline priors.

We summarize the results from Tables 1, 3, and 4 below:

Margin δ\deltaSuperior + Inferior settingsAccuracy in Superior and InferiorOverall accuracyAdjacent mismatchOpposite mismatch
071/7352/71 (73.2%)52/73 (71.2%)15/73 (20.5%)6/73 (8.2%)
0.0560/7348/60 (80.0%)52/73 (71.2%)18/73 (24.7%)3/73 (4.1%)
0.142/7334/42 (81.0%)41/73 (56.2%)30/73 (41.1%)2/73 (2.7%)

The results show a clear trend. As the margin increases, fewer settings are classified as Superior or Inferior: 71, 60, and 42, respectively. Within these settings, prediction accuracy increases from 73.2% to 80.0% to 81.0%. Opposite mismatches decrease, while adjacent mismatches increase as more settings move into Comparable.

Overall agreement remains 52/73 at both δ=0\delta=0 and δ=0.05\delta=0.05, showing that this result is not unique to δ=0.05\delta=0.05. At δ=0.1\delta=0.1, agreement decreases to 41/73. The larger margin requires stronger evidence of a substantial advantage for either offline prior and therefore places more settings into Comparable. This broader group includes more settings where the two method families have different fine-tuning outcomes, reducing the overall agreement.

W2-2: Only 4 of 13 Comparable settings have statistically indistinguishable outcomes. The paper then reinterprets them with a mean-gap below 0.1, a criterion not used anywhere else. The evidence therefore supports a two-regime rule at best.

We would like to clarify that the observation that 8/13 settings have a mean gap below 0.1 describes the size of the performance differences; it does not introduce another criterion for correct predictions. The reported count remains 4/13.

When the offline priors have similar performance, other factors may have a greater influence and lead to different fine-tuning outcomes that our current return-based criterion does not fully capture. This can make outcomes in the Comparable regime harder to predict. We revised Section 5.3 to state the prediction more carefully and explain these limitations.

W2-3: Appendix B.2 pools the last 10 evaluations of each seed for the t-tests. These evaluations are strongly correlated, so treating them as independent samples inflates significance.

For each seed, we first average performance over the final 10 evaluations to reduce sensitivity to performance fluctuations. The t-test therefore uses 10 per-seed averages per method, and does not treat the 100 individual evaluations as independent samples. We have clarified this procedure in Appendix B.2.

W2-4(1):Each setting compares only the strongest π0\pi_0-centric method against the strongest D\mathcal{D}-centric method. The mixed π0+D\pi_0+\mathcal{D} variant is left out,

The mixed variant is included in our experiments, while the main confusion matrix compares the two method families that emphasize different offline priors. Common Response C1 (“Stability and plasticity measurements”) includes aggregate measurements for the mixed variant across all 73 settings, together with three illustrative examples.

Relative to baseline, the mixed variant increases stability in all three regimes, but this does not consistently translate into greater improvement. In the Superior regime, its mean stability change is +0.275, while plasticity decreases by 0.436 and improvement decreases by 0.160. In the Comparable and Inferior regimes, its improvement confidence intervals include zero. These measurements illustrate why combining regularization with replay does not necessarily improve fine-tuning performance.

W2-4(2):...and the warmup length KK and offline ratio α=0.5\alpha=0.5 are fixed in all settings.

We fixed KK and α\alpha to compare design choices under a common training protocol, following the default values reported in the original papers.

For the warmup length, we follow WSRL’s default of K=5,000K=5{,}000. Its warmup ablation (Appendix D, Figure 17) compares K∈{1,000,5,000,20,000}K\in\{1{,}000,5{,}000,20{,}000\}, showing that insufficient warmup can hurt stability, while excessive warmup can slow online improvement. We keep it fixed across settings to support consistent comparisons.

We provide the additional selective-replay configurations and full results in Common Response C2 (“Selective replay and the value of offline data”). These experiments compare selective append, selective replay with a fixed offline ratio of 0.1, and selective replay with a fixed offline ratio of 0.5 across four Superior AntMaze settings.

W3. The benchmarks make the prediction easy.

W3-1: Table 9 shows that the Inferior regime is dominated by datasets with J(πD)=1.0J(\pi_D)=1.0,..., and by BC policies with zero return on AntMaze. The Superior regime is dominated by MuJoCo random and medium-replay data. In both extremes, it is obvious which offline prior is stronger without any framework.

To test whether the observed pattern is driven by the extreme cases identified in the review, we provide the confusion matrix after excluding:

  • MuJoCo random and medium-replay datasets.
  • D4RL datasets with J(πD)=1.0J(\pi_{\mathcal D})=1.0.
  • AntMaze settings with zero-return BC initializations.

This leaves 40 settings: 30 from D4RL and 10 from Robomimic.

Fine-tuning outcome / Pretraining regimeSuperiorComparableInferior
π0\pi_0-centric >> D\mathcal{D}-centric1311
π0\pi_0-centric ≈\approx D\mathcal{D}-centric332
π0\pi_0-centric << D\mathcal{D}-centric2411

The predictions align with the observed outcomes in 27/40 settings (67.5%), compared with 52/73 (71.2%) before the exclusions. On the remaining D4RL settings alone, the accuracy is 20/30 (66.7%). These results show that the observed relationship extends beyond the extreme cases identified in the review.

W3-2: Table 10 shows that Robomimic adds no Superior case.

Regime assignments depend on the relative performance of the pretrained policy and the offline dataset, so different benchmarks need not cover all three regimes. Our Robomimic settings fall into the Comparable and Inferior regimes, providing an additional evaluation of the framework in these two regimes.

W3-3: The informative case is medium-quality data where both π0\pi_0 and DD are useful. That is exactly the Comparable regime, where the framework is weakest.

We want to clarify that medium-quality data do not necessarily correspond to the Comparable regime. The regime depends on the relative performance of the pretrained policy and the dataset, rather than the dataset’s quality alone. After excluding the extreme cases identified in the review, 32 of the 40 remaining settings still belong to the Superior or Inferior regime.

W3-4:Appendix D.1 notes that the AntMaze dense proxy ignores the maze walls. This makes the sparse-reward comparison in Table 2, 76% for raw return against 59% for the dense proxy, less informative.

We designed the dense proxy to provide a simple measure of progress that can be constructed with little task knowledge. The comparison in Table 2 therefore shows that raw return achieves higher prediction accuracy than the particular coarse proxies we tested, rather than establishing an advantage over all dense progress measures.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common response (1)

We thank all four reviewers for their careful reading and constructive feedback. Below, we provide the shared stability–plasticity measurements and selective-replay results. Other questions are addressed in the individual reviewer responses. We have also revised the manuscript, with changes highlighted in blue.

Common Response C1. Stability and plasticity measurements

Addresses: R71U W1-1, W1-2, and W2-4(1); 34Ne W3; bZQ7 W; 5nH8 W2-2.

We computed stability, plasticity, and improvement using Eqs. 4–5 for all six fine-tuning variants across all 73 settings. Below, we report aggregate changes relative to baseline, followed by three examples showing the absolute values. We have added these measurements and the bootstrap procedure to the appendix (“Empirical Measurements of Stability and Plasticity”), with a pointer in Section 5.

Aggregate changes relative to baseline

For each setting, we average each metric across ten training seeds and subtract the corresponding baseline average. Each setting receives equal weight within its regime. Values are mean changes [pointwise 95% CI], with ΔI=ΔS+ΔP\Delta I=\Delta S+\Delta P before rounding. Positive ΔS\Delta S means less performance degradation than baseline.

Superior regime — 32 settings, 15 environment–dataset clusters

Fine-tuning methodΔ StabilityΔ PlasticityΔ Improvement
+ warmup0.005 [−0.014, 0.024]0.033 [−0.004, 0.079]0.038 [0.009, 0.079]
+ offline RL0.199 [0.095, 0.296]−0.311 [−0.399, −0.216]−0.112 [−0.183, −0.042]
+ offline data0.023 [−0.029, 0.085]−0.036 [−0.116, 0.041]−0.013 [−0.054, 0.035]
+ offline data + reset−0.216 [−0.314, −0.129]0.176 [0.104, 0.263]−0.040 [−0.084, 0.011]
+ offline RL + offline data0.275 [0.148, 0.390]−0.436 [−0.532, −0.327]−0.160 [−0.250, −0.075]

Comparable regime — 13 settings, 8 environment–dataset clusters

Fine-tuning methodΔ StabilityΔ PlasticityΔ Improvement
+ warmup0.002 [−0.0005, 0.004]−0.006 [−0.043, 0.023]−0.005 [−0.042, 0.024]
+ offline RL0.057 [−0.001, 0.177]−0.053 [−0.176, 0.033]0.005 [−0.052, 0.044]
+ offline data0.002 [−0.0002, 0.006]0.167 [0.010, 0.295]0.169 [0.012, 0.298]
+ offline data + reset−0.039 [−0.128, −0.001]0.225 [0.032, 0.446]0.187 [−0.022, 0.410]
+ offline RL + offline data0.166 [0.008, 0.450]−0.038 [−0.399, 0.231]0.128 [−0.031, 0.276]

Inferior regime — 28 settings, 14 environment–dataset clusters

Fine-tuning methodΔ StabilityΔ PlasticityΔ Improvement
+ warmup0.016 [−0.006, 0.050]−0.004 [−0.052, 0.038]0.011 [−0.023, 0.041]
+ offline RL0.044 [0.008, 0.094]−0.208 [−0.295, −0.115]−0.164 [−0.259, −0.067]
+ offline data0.014 [−0.002, 0.042]0.257 [0.078, 0.459]0.272 [0.103, 0.464]
+ offline data + reset−0.047 [−0.110, −0.002]0.364 [0.203, 0.543]0.316 [0.143, 0.509]
+ offline RL + offline data0.141 [0.050, 0.265]−0.097 [−0.216, 0.021]0.044 [−0.034, 0.128]

Bootstrap procedure. We use 20,000 hierarchical bootstrap replicates, resampling environment–dataset clusters within each regime and then ten training seeds with replacement. Associated initialization settings are retained together, and the same sampled seed indices are used across methods and initializations within each cluster. Each replicate averages the seed-averaged differences from baseline with equal weight per sampled setting. The 2.5th and 97.5th percentiles give the pointwise 95% confidence intervals. Regime labels remain fixed, and repeated evaluations within each training run are kept together. The intervals describe uncertainty in the average change, rather than variation across individual settings.

These aggregate results show that:

  • Superior: regularization increases stability but reduces plasticity and overall improvement, whereas warmup provides a positive average improvement.
  • Inferior: replay, with or without reset, increases stability, plasticity and overall improvement.
AuthorsOpenReview ↗

Common response (2)

Three illustrative examples

The following examples show absolute stability, plasticity, and improvement values as mean ± standard deviation across ten seeds, with I=S+PI=S+P.

For the Superior regime, we computed the three terms from ReBRAC runs on walker2d-medium-v2:

Fine-tuning methodStabilityPlasticityImprovement = Plasticity + Stability
baseline-0.582 ± 0.2111.014 ± 0.2120.432 ± 0.023
+ warmup (π0\pi_0-centric)-0.457 ± 0.2100.903 ± 0.1830.445 ± 0.040
+ offline RL (π0\pi_0-centric)-0.094 ± 0.0670.271 ± 0.0650.177 ± 0.030
+ offline data (D\mathcal{D}-centric)-0.547 ± 0.2370.948 ± 0.2310.402 ± 0.030
+ offline data + reset (D\mathcal{D}-centric)-0.846 ± 0.0091.184 ± 0.0410.338 ± 0.043
+ offline RL + offline data (mixed π0+D\pi_0+\mathcal{D})-0.048 ± 0.0530.168 ± 0.0510.120 ± 0.016

For the Inferior regime, we computed the three terms using BC pretraining on hopper-medium-expert-v2.

Fine-tuning methodStabilityPlasticityImprovement = Plasticity + Stability
baseline-0.591 ± 0.0631.028 ± 0.0600.437 ± 0.014
+ warmup (π0\pi_0-centric)-0.537 ± 0.0800.974 ± 0.0830.437 ± 0.016
+ offline RL (π0\pi_0-centric)-0.381 ± 0.1290.815 ± 0.1270.434 ± 0.012
+ offline data (D\mathcal{D}-centric)-0.571 ± 0.0521.050 ± 0.0530.479 ± 0.004
+ offline data + reset (D\mathcal{D}-centric)-0.635 ± 0.0071.110 ± 0.0080.475 ± 0.006
+ offline RL + offline data (mixed π0+D\pi_0+\mathcal{D})-0.164 ± 0.1200.633 ± 0.1210.469 ± 0.004

For the Comparable regime, we computed the same terms from ReBRAC runs on hopper-random-v2:

Fine-tuning methodStabilityPlasticityImprovement = Plasticity + Stability
baseline-0.019 ± 0.0230.860 ± 0.3050.841 ± 0.299
+ warmup (π0\pi_0-centric)-0.015 ± 0.0260.868 ± 0.3070.853 ± 0.303
+ offline RL (π0\pi_0-centric)-0.018 ± 0.0280.871 ± 0.3090.853 ± 0.304
+ offline data (D\mathcal{D}-centric)-0.012 ± 0.0170.766 ± 0.3730.753 ± 0.367
+ offline data + reset (D\mathcal{D}-centric)-0.070 ± 0.0370.939 ± 0.2540.869 ± 0.257
+ offline RL + offline data (mixed π0+D\pi_0+\mathcal{D})-0.011 ± 0.0130.790 ± 0.3450.778 ± 0.341

These measurements illustrate how different fine-tuning choices affect stability and plasticity across the three regimes:

  • Superior: Offline RL regularization achieves stronger stability than warmup, but its lower plasticity limits further improvement. Combining regularization with offline replay further increases stability while reducing improvement. This shows why preserving the stronger pretrained policy must be balanced with sufficient plasticity.
  • Inferior: Offline data replay achieves the highest mean improvement (0.479), while replay with reset achieves the highest plasticity (1.110) but also experiences the largest performance drop (−0.635). The mixed method provides the strongest stability (−0.164) but lower plasticity (0.633). These results illustrate how different stability–plasticity balances affect improvement when the pretrained policy is weaker than the dataset.
  • Comparable: Different methods exhibit different stability and plasticity values, while several achieve similar improvements in the examined setting. This illustrates that methods with different stability–plasticity balances can achieve similar performance.

Together, the aggregate results and individual examples quantify the observed stability–plasticity trade-offs. These measurements are computed after fine-tuning and do not establish additional predictive power beyond the regime labels.

AuthorsOpenReview ↗

Common response (3)

Common Response C2. Selective replay and the value of offline data

Addresses: R71U W2-4(2); 34Ne W1 and W4.

To investigate whether the lower performance of offline replay in the Superior regime is caused by uniformly sampling all transitions, we conducted additional experiments in four Superior AntMaze settings, using CalQL and ReBRAC pretrainings on antmaze-large-diverse and antmaze-large-play. During fine-tuning, we restricted replay from the offline dataset to successful trajectories. The selected subsets contain 106 of 999 trajectories (10.6%) in antmaze-large-diverse and 105 of 999 trajectories (10.5%) in antmaze-large-play.

We evaluated three variants of replay: selective append, selective replay with a fixed offline data ratio of 0.1, and selective replay with a fixed offline data ratio of 0.5. In selective append, all successful trajectories from the offline dataset initialize the replay buffer. Newly collected online transitions are then added during fine-tuning, so the offline data ratio changes as online data accumulate.

The table below reports success rates on a 0–1 scale. We first average performance over the final 10 evaluations for each seed, then report the mean ± standard deviation across 10 seeds. Bold values indicate the highest displayed mean in each setting.

PretrainingAntMaze environmentAll-data α=0.5\alpha=0.5 (offline data replay)Selective α=0.5\alpha=0.5Selective appendSelective α=0.1\alpha=0.1Warmup
CalQLLarge Diverse0.72 ± 0.290.00 ± 0.000.01 ± 0.010.88 ± 0.230.97 ± 0.01
CalQLLarge Play0.75 ± 0.080.03 ± 0.080.05 ± 0.090.78 ± 0.050.97 ± 0.02
ReBRACLarge Diverse0.94 ± 0.040.69 ± 0.050.86 ± 0.040.94 ± 0.030.92 ± 0.07
ReBRACLarge Play0.70 ± 0.100.50 ± 0.080.67 ± 0.070.70 ± 0.090.94 ± 0.03

Warmup achieves the highest mean success rate in three of the four settings. In the remaining setting, it achieves 0.92, compared with the highest reported mean of 0.94. These results show that warmup often retains its advantage even when offline data replay is restricted to successful trajectories, consistent with our Superior-regime recommendation. Furthermore, selective replay at α=0.5\alpha=0.5 underperforms uniform replay at the same ratio in all four settings, showing that selecting successful trajectories alone does not necessarily improve fine-tuning performance. We have added the experimental setup and selective-replay results to the appendix, with a summary and pointer in Section 5.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.