- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Offline-to-online RL has a problem of different algorithms performing better or worse on different types of tasks, which makes it hard for practitioners to know which method to use. Authors introduce a framework that attempts to explain why certain classes of methods will perform better for certain classes of environment-dataset tuples, through the lens of stability-plasticity trade-off.
Critical strengths
- Authors perform experiments on a wide range of environments and base algorithms.
Critical weaknesses
-
Authors do not mention an explicit policy of how the order of performances was decided (Table 9), and there are some inconsistencies. E.g. For hopper-random-v2, ReBRAC was given with a difference of 0.07 with the dataset performance, but given > on walker2d-random-v2 with a difference of 0.058 in performance. Likewise, on the antmaze domains, on ultra-diverse CalQL and ReBRAC were give with differences of 0.065 and 0.074 respectively. And BC was give for antmaze-ultra-diverse while < for antmaze-large-play, while both have 0. Since these choices of ordering alter the main results of the paper, can the authors share the policy that they used across these comparisons to obtain all the orderings from Tables 9 and 11? Currently, orderings appear hand-picked whenever ambiguous.
-
The method classifications into dataset-centric or -centric are broken. For example, RLPD is categorized as a method that uses stability relative to the offline dataset. While RLPD does use the offline dataset during training, it is inaccurate to say that it “promotes stability around the offline dataset”. It uses transitions from the offline dataset for TD updates, but it does not use any regularization that pulls either the policy toward the behavior distribution from the dataset, or the value function towards assigning seen states or actions higher values.
Lines 240-247 argue that whenever offline data is used with regularization, they additionally categorize it as promoting stability towards both and D. They also mention that this regularization is applied to data that comes from a sequence of policies that includes , this implicitly anchors learning around the region visited by , even if is not stored during fine-tuning. These generalizations make the category meaningless.
Lines 255-256 argue that RLPD can be interpreted as using a parameter reset to increase stability. This categorization does not seem reasonable, since RLPD just does not use pretraining.
Overall, the categories used in the paper seem unnatural, and are very poorly justified. In fact, there is no explicit list or table for which methods belong to which category, even though this categorization plays a key role in their main results.
-
The categorization of a method relies on seeing its performance compared with the performance of the training data, after having performed training. What is the practical value of the framework, if it only offers recommendations after having performed the expensive training?
-
Lines 317-319 argue that when pretrained policy achieves substantially higher performance than offline dataset, e.g. when dealing with suboptimal datasets, the primary concern becomes preserving stability relative to . This is not necessarily true. The mechanism that most offline RL algorithms use to learn from suboptimal datasets to begin with is TD learning. And during online fine-tuning, that same mechanism can be used to keep using the suboptimal offline dataset, irrespective of .
Questions for authors
- What was the algorithm used to create tables 9 and 11?
- Can authors provide a table with algorithm classifications that they used for their main results?
4 comments
W2-1: The method classifications into dataset-centric or -centric are broken. For example, RLPD is categorized as a method that uses stability relative to the offline dataset. While RLPD does use the offline dataset during training, it is inaccurate to say that it “promotes stability around the offline dataset”. It uses transitions from the offline dataset for TD updates, but it does not use any regularization that pulls either the policy toward the behavior distribution from the dataset, or the value function towards assigning seen states or actions higher values.
We agree that RLPD does not use any regularization that pulls either the policy toward the behavior distribution from the dataset, but the offline data used during the online training lead to still learn from the knowledge in the offline data.
Section 5.4 (Figure 5) provides empirical support for this interpretation: offline replay reduces offline TD errors and avoids the Q-value divergence. These results support a role for replay in stabilizing critic learning on the offline data.
W2-2: Lines 240-247 argue that whenever offline data is used with regularization, they additionally categorize it as promoting stability towards both and . They also mention that this regularization is applied to data that comes from a sequence of policies that includes , this implicitly anchors learning around the region visited by , even if is not stored during fine-tuning. These generalizations make the category meaningless.
We classify continued regularization as -centric because it uses online data initially generated by as the reference for regularized learning. ReBRAC penalizes differences between policy actions and recorded actions, so early samples carry pretrained behavior without requiring a frozen copy of . For CalQL, the connection is through conservative critic regularization relative to the training data, rather than an explicit penalty on policy actions. The online reference evolves as updated policies collect new experience.
In the mixed variant, both online and offline samples contribute to regularization. This distinguishes the mixed variant from offline replay alone, which uses these data without the additional offline RL regularization. Aggregate stability, plasticity, and improvement measurements across all 73 settings, including the mixed variant, are provided in Common Response C1, together with three illustrative examples.
W2-3: Lines 255-256 argue that RLPD can be interpreted as using a parameter reset to increase stability. This categorization does not seem reasonable, since RLPD just does not use pretraining.
Section 4.3 discusses parameter reset as a way to increase plasticity, rather than stability. RLPD trains from random initialization using both offline and online data. Our baseline fine-tunes a pretrained agent using standard online RL with only online data and no additional components. Relative to this baseline, our + offline data + reset variant captures RLPD’s initialization and replay choices in a controlled comparison.
We do not mean that RLPD literally resets a pretrained agent. Representing random initialization as a full reset makes it easier to compare training from scratch with fine-tuning a pretrained agent under the same training protocol.
Q2: Can authors provide a table with algorithm classifications that they used for their main results?
The main experiments evaluate six variants built from the design choices in Section 4. Their initialization, added components, and classifications are summarized below. The categories describe which source of prior knowledge each added component emphasizes.
| Fine-tuning variant | Initialization | Added component | Category |
|---|---|---|---|
| Baseline | Pretrained agent | None; standard online RL using online data | Baseline |
| + warmup | Pretrained agent | Collect 5,000 online transitions using before updates | -centric |
| + offline RL | Pretrained agent | Continue pretraining regularization using online data | -centric |
| + offline data | Pretrained agent | Replay offline data with | -centric |
| + offline data + reset | Random initialization | Replay offline data with | -centric |
| + offline RL + offline data | Pretrained agent | Combine regularization and offline data replay | Mixed |
W3: The categorization of a method relies on seeing its performance compared with the performance of the training data, after having performed training. What is the practical value of the framework, if it only offers recommendations after having performed the expensive training?
We would like to clarify that our framework distinguishes between method classification and regime assignment, neither of which requires observing online fine-tuning results.
- First, fine-tuning methods are classified based on their design choices, rather than their performance. For example, warmup is classified as -centric because it emphasizes the pretrained policy, while offline data replay is classified as -centric because it continues to use the offline dataset. These classifications are independent of the observed fine-tuning outcomes.
- Second, regime assignment depends on the relative performance of the pretrained policy and offline dataset. In the offline-to-online RL setting studied in our paper, offline pretraining has already been completed before selecting a fine-tuning strategy. Thus, and can be estimated before online fine-tuning, without training any candidate fine-tuning methods.
In our empirical study, we compare the strongest observed variants within each method family to evaluate the regime-based predictions. This retrospective comparison is separate from the regime assignment itself.
The practical goal of our framework is to guide the choice of fine-tuning strategies before expensive online training, by indicating whether to emphasize the pretrained policy or the offline dataset. It aims to narrow the choice of methods rather than requiring practitioners to train and compare all candidate methods.
W4. Lines 317-319 argue that when pretrained policy achieves substantially higher performance than offline dataset, e.g. when dealing with suboptimal datasets, the primary concern becomes preserving stability relative to . This is not necessarily true. The mechanism that most offline RL algorithms use to learn from suboptimal datasets to begin with is TD learning. And during online fine-tuning, that same mechanism can be used to keep using the suboptimal offline dataset, irrespective of .
We agree that suboptimal offline data can still support further improvement through TD learning. The relative returns do not, by themselves, determine how much additional value the dataset can provide. Our hypothesis is that they nevertheless provide useful guidance on which prior to emphasize during fine-tuning.
Our experiments support this hypothesis: in the Superior regime, the strongest tested -centric variant outperforms the strongest tested -centric variant in 24 of 32 settings, with six showing no statistically significant difference and two favoring -centric methods.
Figure 5 further illustrates the distinction between learning from offline data and improving fine-tuning performance. Offline replay shows lower offline TD loss in both illustrated settings, but warmup achieves higher return in the Superior setting, whereas replay achieves higher return in the Inferior setting. Thus, the benefit of replay for offline TD learning does not translate into the same performance advantage across regimes.
We revised the statement in Section 5.1 to distinguish this empirical recommendation from a general claim.
Dear Reviewer,
Thank you again for your valuable feedback. We have provided a detailed rebuttal addressing your comments and concerns. We would greatly appreciate it if you could take a look at our responses and let us know whether they adequately address your concerns.
Thank you for your time and consideration.
We thank you for your careful review and valuable feedback. We address your comments and concerns below.
The regime assignments in Tables 9 and 11 follow the same fixed statistical procedure described in Appendix B.1 (Eqs. 6–7). Specifically, we conduct two one-sided Welch t-tests with a performance margin δ=0.05 and significance level α=0.05.
Let Δ=μ0−μD denote the difference between the mean returns of the pretrained policy and offline dataset. We test
\quad \text{vs.} \quad H_{1,+}: \Delta > \delta,$$ $$H_{0,-}: \Delta \geq -\delta \quad \text{vs.} \quad H_{1,-}: \Delta < -\delta.$$ A setting is classified as Superior if $H_{0,+}$ is rejected, Inferior if $H_{0,-}$ is rejected, and Comparable otherwise. Thus, regime assignment depends on both the mean return difference and its uncertainty, rather than the mean difference alone. The corresponding one-sided p-values are reported below. Here, $p_+$ corresponds to the test of $\Delta>\delta$, and $p_-$ corresponds to the test of $\Delta<-\delta$. Values are rounded for readability. | Pretrained policy | Environment | $p_+$ | $p_-$ | Assigned regime | |---|---|---:|---:|---| | ReBRAC | `hopper-random-v2` | 0.0568 | $\approx 1$ | Comparable | | ReBRAC | `walker2d-random-v2` | $3.81\times10^{-10}$ | $\approx 1$ | Superior | | CalQL | `antmaze-ultra-diverse-v2` | 0.2709 | 0.9997 | Comparable | | ReBRAC | `antmaze-ultra-diverse-v2` | 0.2920 | 0.9921 | Comparable | | BC | `antmaze-ultra-diverse-v2` | $\approx 1$ | 0.3335 | Comparable | | BC | `antmaze-large-play-v2` | $\approx 1$ | $8.88\times10^{-9}$ | Inferior | These p-values reproduce all six original assignments under the same decision rule and explain the examples raised in the review: - **ReBRAC on Hopper-random versus Walker2d-random:** Hopper-random does not meet the significance threshold for Superior ($p_+=0.0568$), whereas Walker2d-random does. Despite Hopper’s larger mean gap, its policy scores have substantially larger standard deviation (0.036 versus 0.001), illustrating why the mean gap alone does not determine the assignment. - **CalQL and ReBRAC on AntMaze-ultra-diverse:** Although their mean gaps exceed 0.05, neither superiority test is significant ($p_+=0.2709$ and $0.2920$, respectively). Neither inferiority test is significant either, so both settings are classified as Comparable. - **BC on AntMaze-ultra-diverse versus AntMaze-large-play:** BC scores zero in both settings, but the dataset means differ (0.053 versus 0.105). The inferiority test is not significant on Ultra-diverse ($p_-=0.3335$), but is significant on Large-play ($p_-=8.88\times10^{-9}$), yielding Comparable and Inferior assignments, respectively. We have added a reference in the main text of the revised manuscript to the detailed regime assignment procedure in the appendix.