- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focuses on O2O RL without retaining offline data during online fine-tuning. The authors claim that many existing methods assume the offline dataset remains available throughout fine-tuning. However, this setting has already been studied for years. Moreover, the paper only considers a specific class of methods that employ expressive behavioral priors, which limits the generality of its claims. Overall, the paper reads more like an empirical case study than a methodological contribution or any insight. The writing also requires further proofreading.
Critical strengths
None.
Critical weaknesses
-
The paper introduces a general O2O RL setting without retaining offline data, but the actual investigation is restricted to a specific class of O2O RL methods that rely on pretrained behavioral priors. This limits the generality of the paper's claims. More importantly, I am concerned about the motivation for continuing BC in the no-retention setting. If the offline dataset is no longer available, why should the pretrained behavioral prior continue to be trained using online data? Such updates naturally introduce a distribution shift relative to the original offline training distribution, potentially causing out-of-distribution adaptation and degrading the pretrained behavioral prior, as demonstrated by the experimental results. Given that this failure mode appears largely predictable from the training setup, it is unclear what additional insight this case study provides beyond confirming the expected consequences of continuing BC on a shifted data distribution.
-
While the comparison between continuing and stopping BC is reasonable for this case study, the experiments do not fully support some of the conclusions. First, the authors attribute the failure of Gaussian policies to their limited expressiveness, but replacing the flow prior with a Gaussian actor also introduces differences in pretrained policy quality and action coverage. Thus, the experiments do not sufficiently isolate the role of an expressive behavioral prior. Furthermore, the authors argue that continued BC causes actor collapse, which subsequently leads to critic degeneration. However, the evidence is largely correlational, and it remains unclear whether critic degeneration is directly caused by reduced behavioral diversity or by other factors, such as distribution shift and bootstrapping errors during online learning. More controlled experiments are needed to support these claims.
Questions for authors
See in weaknesses.
5 comments
W1. The paper introduces a general O2O RL setting .... it is unclear what additional insight this case study provides beyond confirming the expected consequences of continuing BC on a shifted data distribution.
“is restricted to a specific class of O2O RL methods that rely on pretrained behavioral priors…limits the papers generality claims”
We agree that our claims concern a specific, and increasingly common, class of methods, and we have revised the paper to state this scope explicitly. Our claim is that when an expressive behavioral prior is (i) trained by imitation on the offline data to capture multimodal behavior, (ii) preserved during no-retention fine-tuning rather than refit to online data, and (iii) exploited by a decision rule that learns online, effective fine-tuning is possible without the offline data.
This class includes many recent offline-to-online methods: flow-based priors (IFQL, FQL [2]), action-chunked priors (QC [3]), and, as we now show, diffusion-based priors (IDQL [4]), where freezing the prior also enables effective fine-tuning (Fig. 3: https://tinyurl.com/bdd584n3 in the updated paper).
“motivation for continuing BC in the no-retention setting … expected consequences of continuing BC on a shifted data distribution.”
Continuing BC is usually the default: offline-to-online algorithms typically retain the offline data and keep training the policy toward the replay buffer [2, 3, 8, 9, 10, 11] . Applying their reference implementations naively in the no-retention setting is therefore exactly BC-ON, and we show that this yields poor results. We believe it is a useful contribution to demonstrate that these methods remain usable even when offline data cannot be retained. Continuing BC is also not unreasonable a priori: as the agent improves, its online data reflects better behavior, and Luo et al. [15] find that fitting a BC regularizer to online data alone can accelerate fine-tuning of TD3-BC. Our results show that for expressive priors trained exclusively through BC, this refitting instead erases the prior.
While the failure may appear intuitive in hindsight, several of our findings are less directly predictable from the training setup alone. Seeding the replay buffer with up to 100k transitions from the pretrained policy, the remedy proposed by WSRL [1], reduces the degradation of continued BC but, in most settings, does not close the gap to stopping BC. Conversely, stopping BC at intermediate points of fine-tuning still recovers a substantial fraction of this gap. Finally, freezing the prior does not appear to be sufficient on its own: the benefit relies on the prior being expressive, since a Gaussian prior in IFQL fails even with BC off, and on the decision rule retaining an active pathway to the prior, since FQL without distillation collapses.
W2. While the comparison between....More controlled experiments are needed to support these claims.
First, the authors attribute the failure of Gaussian policies to their limited expressiveness…sufficiently isolate the role of an expressive behavioral prior.
Prior work has shown that unimodal policies trained by maximum likelihood on multimodal data average between behavioral modes, placing probability mass where the data has no actions [13, 4]. A Gaussian prior therefore proposes actions away from the valid behavior in the data, which alters its action coverage and, in turn, the quality of the resulting policy. The differences in policy quality and action coverage the reviewer mentions are thus consequences of the policy class, and we view them as the mechanism through which expressiveness matters rather than as independent confounds. We verify each step of this chain below. To show that the Gaussian prior averages between behaviorally represented modes in the data, we do the following experiment using held-out validation datasets for the domains below. For each state in the dataset, we sample K = 32 proposals from each prior type and compute the mean Euclidean distance of each proposal to the nearest dataset action. If a prior captures the behavioral modes, its proposals lie close to valid demonstrations; if it averages between modes, its proposals land between demonstrator clusters and this distance grows.
| Domain | Flow prior | Gaussian prior |
|---|---|---|
| Cube Double Play | 0.20 ± 0.01 | 0.99 ± 0.01 |
| Ant Soccer Arena | 0.75 ± 0.02 | 1.20 ± 0.02 |
As shown in the table above, the Gaussian prior’s proposals are substantially farther from valid demonstrator actions, consistent with mode averaging.
To rule out state coverage as the cause, we plot PCA projections of online rollouts at 1.2M steps. For consistency, all panels are colored using a single critic: the critic from the Gaussian-prior run at the end of offline pretraining (1M steps), before any online updates. In IFQL, the critic is pretrained via in-sample value learning on the dataset's actions, independently of the prior, so the offline critics from both (Gaussian and Flow) runs are trained identically. With BC off, the Gaussian prior explores the state space comparably to the flow prior, yet it achieves 0% success on both antsoccer-arena and cube-double, whereas the flow prior achieves 43% and 30%, respectively. Moreover, the flow prior’s rollouts progress into regions the critic assigns high value, whereas the Gaussian prior’s rollouts predominantly remain in low-value regions despite their broad coverage. Therefore, we believe the loss of expressiveness is the reason behind the differences in action coverage and the resulting poor policy.
The plot can be seen here (App. E.3, Fig. 20): https://tinyurl.com/2p8x7dpr
Finally, to demonstrate generalization beyond flow priors, we extend the evaluation by replacing the flow prior with a diffusion prior (IDQL [4]), which also enables policy improvement with BC off (Fig. 3 in the updated paper: https://tinyurl.com/bdd584n3). This shows that the benefit comes from expressiveness rather than from flow matching specifically. Note that for Cube Double IDQL, we used 3M offline steps and saw dramatic improvements in our approach within the first 1M steps, despite running for an additional 2M. We also allocated the same offline training budget to the Gaussian version and saw no performance improvements.
continued below
We thank the reviewer for their feedback. We agree that fine-tuning without retaining offline data has established precedents, including WSRL [1]. Our work focuses on a specific question that arises in recent offline-to-online methods such as FQL [2] and Q-chunking [3]. In these methods, a separately trained expressive prior proposes actions, and online learning improves how these proposals are used. By default, the prior continues to be trained with BC on a replay buffer that includes the offline data. Once the offline data are discarded, this same BC update fits the prior only to the agent’s own online experience. We study whether this update should continue.
Prior work has studied related, but different, choices. IDQL [4] compares freezing its diffusion behavior model with fine-tuning it, but retains offline data during fine-tuning. PEX [5] freezes a pretrained policy during fine-tuning, but this policy is not an expressive prior. EXPO [6] fine-tunes without retained offline data after collecting warm-up rollouts from the pretrained policy, but it continues to update its expressive prior with imitation during fine-tuning. DICE-RL [7] proposed a new RL method that keeps a diffusion or flow-based prior frozen; however, it is primarily an online RL method: only the prior is pretrained, and the critic is learned from scratch during the online phase. In contrast, we do not propose a new method. We study the role of BC in existing offline-to-online methods and show that, once the offline data are unavailable, simply turning off their BC update, with no other change to the algorithms or their hyperparameters, enables effective fine-tuning. We further explain why continued BC fails without the offline data, showing that its effect propagates to the critic through the data the agent collects (see our response to the critic-degeneration concern below). We have clarified this scope in the revision and addressed the specific concerns below.