acceptodds
Under review as a conference paper at ICLR 2027

Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

Abstract

Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on five OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.

Primary area:
reinforcement learning
Keywords:
deep rl, offline to online rl, bc

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6TbyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on O2O RL without retaining offline data during online fine-tuning. The authors claim that many existing methods assume the offline dataset remains available throughout fine-tuning. However, this setting has already been studied for years. Moreover, the paper only considers a specific class of methods that employ expressive behavioral priors, which limits the generality of its claims. Overall, the paper reads more like an empirical case study than a methodological contribution or any insight. The writing also requires further proofreading.

Critical strengths

None.

Critical weaknesses
  1. The paper introduces a general O2O RL setting without retaining offline data, but the actual investigation is restricted to a specific class of O2O RL methods that rely on pretrained behavioral priors. This limits the generality of the paper's claims. More importantly, I am concerned about the motivation for continuing BC in the no-retention setting. If the offline dataset is no longer available, why should the pretrained behavioral prior continue to be trained using online data? Such updates naturally introduce a distribution shift relative to the original offline training distribution, potentially causing out-of-distribution adaptation and degrading the pretrained behavioral prior, as demonstrated by the experimental results. Given that this failure mode appears largely predictable from the training setup, it is unclear what additional insight this case study provides beyond confirming the expected consequences of continuing BC on a shifted data distribution.

  2. While the comparison between continuing and stopping BC is reasonable for this case study, the experiments do not fully support some of the conclusions. First, the authors attribute the failure of Gaussian policies to their limited expressiveness, but replacing the flow prior with a Gaussian actor also introduces differences in pretrained policy quality and action coverage. Thus, the experiments do not sufficiently isolate the role of an expressive behavioral prior. Furthermore, the authors argue that continued BC causes actor collapse, which subsequently leads to critic degeneration. However, the evidence is largely correlational, and it remains unclear whether critic degeneration is directly caused by reduced behavioral diversity or by other factors, such as distribution shift and bootstrapping errors during online learning. More controlled experiments are needed to support these claims.

Questions for authors

See in weaknesses.

5 comments

We thank the reviewer for their feedback. We agree that fine-tuning without retaining offline data has established precedents, including WSRL [1]. Our work focuses on a specific question that arises in recent offline-to-online methods such as FQL [2] and Q-chunking [3]. In these methods, a separately trained expressive prior proposes actions, and online learning improves how these proposals are used. By default, the prior continues to be trained with BC on a replay buffer that includes the offline data. Once the offline data are discarded, this same BC update fits the prior only to the agent’s own online experience. We study whether this update should continue.

Prior work has studied related, but different, choices. IDQL [4] compares freezing its diffusion behavior model with fine-tuning it, but retains offline data during fine-tuning. PEX [5] freezes a pretrained policy during fine-tuning, but this policy is not an expressive prior. EXPO [6] fine-tunes without retained offline data after collecting warm-up rollouts from the pretrained policy, but it continues to update its expressive prior with imitation during fine-tuning. DICE-RL [7] proposed a new RL method that keeps a diffusion or flow-based prior frozen; however, it is primarily an online RL method: only the prior is pretrained, and the critic is learned from scratch during the online phase. In contrast, we do not propose a new method. We study the role of BC in existing offline-to-online methods and show that, once the offline data are unavailable, simply turning off their BC update, with no other change to the algorithms or their hyperparameters, enables effective fine-tuning. We further explain why continued BC fails without the offline data, showing that its effect propagates to the critic through the data the agent collects (see our response to the critic-degeneration concern below). We have clarified this scope in the revision and addressed the specific concerns below.

W1. The paper introduces a general O2O RL setting .... it is unclear what additional insight this case study provides beyond confirming the expected consequences of continuing BC on a shifted data distribution.

“is restricted to a specific class of O2O RL methods that rely on pretrained behavioral priors…limits the papers generality claims”

We agree that our claims concern a specific, and increasingly common, class of methods, and we have revised the paper to state this scope explicitly. Our claim is that when an expressive behavioral prior is (i) trained by imitation on the offline data to capture multimodal behavior, (ii) preserved during no-retention fine-tuning rather than refit to online data, and (iii) exploited by a decision rule that learns online, effective fine-tuning is possible without the offline data.

This class includes many recent offline-to-online methods: flow-based priors (IFQL, FQL [2]), action-chunked priors (QC [3]), and, as we now show, diffusion-based priors (IDQL [4]), where freezing the prior also enables effective fine-tuning (Fig. 3: https://tinyurl.com/bdd584n3 in the updated paper).

“motivation for continuing BC in the no-retention setting … expected consequences of continuing BC on a shifted data distribution.”

Continuing BC is usually the default: offline-to-online algorithms typically retain the offline data and keep training the policy toward the replay buffer [2, 3, 8, 9, 10, 11] . Applying their reference implementations naively in the no-retention setting is therefore exactly BC-ON, and we show that this yields poor results. We believe it is a useful contribution to demonstrate that these methods remain usable even when offline data cannot be retained. Continuing BC is also not unreasonable a priori: as the agent improves, its online data reflects better behavior, and Luo et al. [15] find that fitting a BC regularizer to online data alone can accelerate fine-tuning of TD3-BC. Our results show that for expressive priors trained exclusively through BC, this refitting instead erases the prior.

While the failure may appear intuitive in hindsight, several of our findings are less directly predictable from the training setup alone. Seeding the replay buffer with up to 100k transitions from the pretrained policy, the remedy proposed by WSRL [1], reduces the degradation of continued BC but, in most settings, does not close the gap to stopping BC. Conversely, stopping BC at intermediate points of fine-tuning still recovers a substantial fraction of this gap. Finally, freezing the prior does not appear to be sufficient on its own: the benefit relies on the prior being expressive, since a Gaussian prior in IFQL fails even with BC off, and on the decision rule retaining an active pathway to the prior, since FQL without distillation collapses.

W2. While the comparison between....More controlled experiments are needed to support these claims.

First, the authors attribute the failure of Gaussian policies to their limited expressiveness…sufficiently isolate the role of an expressive behavioral prior.

Prior work has shown that unimodal policies trained by maximum likelihood on multimodal data average between behavioral modes, placing probability mass where the data has no actions [13, 4]. A Gaussian prior therefore proposes actions away from the valid behavior in the data, which alters its action coverage and, in turn, the quality of the resulting policy. The differences in policy quality and action coverage the reviewer mentions are thus consequences of the policy class, and we view them as the mechanism through which expressiveness matters rather than as independent confounds. We verify each step of this chain below. To show that the Gaussian prior averages between behaviorally represented modes in the data, we do the following experiment using held-out validation datasets for the domains below. For each state in the dataset, we sample K = 32 proposals from each prior type and compute the mean Euclidean distance of each proposal to the nearest dataset action. If a prior captures the behavioral modes, its proposals lie close to valid demonstrations; if it averages between modes, its proposals land between demonstrator clusters and this distance grows.

DomainFlow priorGaussian prior
Cube Double Play0.20 ± 0.010.99 ± 0.01
Ant Soccer Arena0.75 ± 0.021.20 ± 0.02

As shown in the table above, the Gaussian prior’s proposals are substantially farther from valid demonstrator actions, consistent with mode averaging.

To rule out state coverage as the cause, we plot PCA projections of online rollouts at 1.2M steps. For consistency, all panels are colored using a single critic: the critic from the Gaussian-prior run at the end of offline pretraining (1M steps), before any online updates. In IFQL, the critic is pretrained via in-sample value learning on the dataset's actions, independently of the prior, so the offline critics from both (Gaussian and Flow) runs are trained identically. With BC off, the Gaussian prior explores the state space comparably to the flow prior, yet it achieves 0% success on both antsoccer-arena and cube-double, whereas the flow prior achieves 43% and 30%, respectively. Moreover, the flow prior’s rollouts progress into regions the critic assigns high value, whereas the Gaussian prior’s rollouts predominantly remain in low-value regions despite their broad coverage. Therefore, we believe the loss of expressiveness is the reason behind the differences in action coverage and the resulting poor policy.

The plot can be seen here (App. E.3, Fig. 20): https://tinyurl.com/2p8x7dpr

Finally, to demonstrate generalization beyond flow priors, we extend the evaluation by replacing the flow prior with a diffusion prior (IDQL [4]), which also enables policy improvement with BC off (Fig. 3 in the updated paper: https://tinyurl.com/bdd584n3). This shows that the benefit comes from expressiveness rather than from flow matching specifically. Note that for Cube Double IDQL, we used 3M offline steps and saw dramatic improvements in our approach within the first 1M steps, despite running for an additional 2M. We also allocated the same offline training budget to the Gaussian version and saw no performance improvements.

continued below

Reviewer_EEGAOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to effectively perform offline-to-online reinforcement learning with flow policies when the offline dataset is not available during online fine-tuning. The empirical analyses show that continuing behavioral cloning (BC) the prior using only online data can cause catastrophic forgetting of the pretrained behavioral prior, leading to reduced action diversity and degraded critic learning. Instead, freezing the pretrained prior while continuing to update the critic and other policy improvement components would enable more effective online fine-tuning.

Critical strengths

S1 (Relevant problem setting). The paper investigates offline-to-online RL in a setting where the offline dataset is no longer available during online fine-tuning, leaving only the pretrained models and newly collected online data.

S2 (Reasonable empirical findings). The observation that continued online BC can degrade the pretrained behavioral prior is reasonable, as replacing the offline dataset with limited online rollouts changes the training distribution and may lead to overfitting. The analyses in Figures 6 and 7 provide empirical evidence of behavioral forgetting, reduced action diversity, and degraded critic representations.

Critical weaknesses

W1 (Limited practical applicability and generalizability). The paper focuses on behavioral priors pretrained on task-specific offline datasets, making it unclear how broadly the observations extend to more general pretrained models. For example, large-scale pretrained policies, such as VLA models, are typically trained on diverse tasks and datasets and are often kept frozen during downstream adaptation (e.g., using residual or refinement action heads). Therefore, it is unclear in which practical settings the identified failure mode and proposed freezing strategy would provide additional benefits. Providing a compelling practical use case or concrete example where the proposed strategy offers clear advantages would help establish the broader relevance and applicability of the observations.

W2 (Limited methodological novelty). Furthermore, freezing the pretrained prior may fundamentally conflict with the goal of offline-to-online RL, which is to discover better actions and improve beyond the pretrained policy. In particular, for IFQL and QC, preserving the pretrained behavioral distribution comes at the cost of limiting exploration and the potential for policy improvement beyond behaviors represented in the offline dataset.

W3 (Limited policy improvement and insufficient alternative strategies). Freezing the pretrained behavioral prior (the actor for IFQL and QC) may restrict online policy improvement, particularly when the offline dataset is suboptimal or has limited coverage. For IFQL and QC, which select actions or action chunks from the frozen prior, this may prevent the discovery of high-value behaviors that are poorly represented in the offline data. Although the paper evaluates low-quality offline datasets, these experiments do not fully address scenarios with severely limited action coverage. The warmup experiments are also limited to 25K online interactions, leaving it unclear whether a larger warmup buffer could mitigate the degradation caused by continued BC. Additional comparisons would help clarify whether complete freezing is necessary and better characterize the trade-off between preserving pretrained behavior and discovering improved actions.

W4 (Insufficient justification for expressive priors). The paper argues that expressive flow-based priors are important for successful no-retention fine-tuning (Column 2 in Figure 4), supported by the Gaussian-policy and ReBRAC variants. However, the failure of ReBRAC without BC regularization term may simply reflect the importance of behavioral regularization, which is already well established in offline RL, rather than demonstrating the necessity of preserving behavior priors.

W5 (Limited experimental coverage and analysis). The OGBench experiments evaluate only 3 of the 5 available tasks per task category, with three random seeds per task. Since OGBench performance can exhibit substantial variability across random seeds, reporting results on the commonly used five tasks per category with eight random seeds would provide more reliable and comprehensive empirical evidence. Furthermore, although QC is included in the performance comparisons, the detailed analyses of behavioral forgetting, action diversity, and critic degradation primarily focus on IFQL and FQL. Since action-chunked policies can model temporally extended behaviors, additional analyses for QC would help establish whether the identified failure mechanism extends to this different policy structure.

Questions for authors

Could the authors provide additional explanations or experimental results addressing the concerns above?

6 comments

We thank the reviewer for the thoughtful review. We appreciate the recognition that the paper studies a “relevant problem setting” and that the “analyses in Figures 6 and 7 provide empirical evidence of behavioral forgetting, reduced action diversity, and degraded critic representations”. We respond to your concerns below.

W1. Limited practical applicability and generalizability

We agree that vision-language-action (VLA) models are an important setting for extending this work, as we highlighted in the Opportunities section of the original submission. In response to the reviewer’s suggestion, we have added a dedicated discussion in Appendix F that clarifies how our setting relates to VLA adaptation with a frozen pretrained policy.

As the reviewer notes, VLA pipelines that adapt a residual or refinement head often keep the base policy frozen. These VLAs are typically pretrained by behavioral cloning (BC) alone, without a critic or a policy-improvement objective [1, 2]. We study a different regime: offline-to-online RL, where the expressive prior is pretrained together with a critic used for policy improvement and, by default, continues to be trained with BC during fine-tuning, typically with much smaller models and datasets. In this regime, our results show that without the original data, continuing BC on the agent’s own rollouts erases the pretrained prior and, through the data it collects, degrades the critic. Then we find that freezing is effective only when the prior is expressive, and the decision rule retains a pathway to it. Whether these findings carry over to large, multitask VLAs remains an open question; our experiments provide evidence for task-specific expressive priors, as is often studied in the offline-to-online RL setting.

A concrete use case is a robot fine-tuned via offline-to-online RL from a checkpoint pretrained on human teleoperation demonstrations, in which the original demonstrations remain unavailable. Our Robomimic Multi-Human experiments [3] model this setting in simulation on Can and Lift, using demonstrations from six operators of varying proficiency. After excluding the demonstrations from online replay, continued behavioral cloning (BC) degraded performance, while freezing the prior enabled effective online fine-tuning (Fig. 4). This gives a practical example of how a task-specific policy can continue learning from interaction after losing access to its training demonstrations, using the offline-to-online methods we study.


W2. Limited methodological novelty

Thanks for raising this important question. This response addresses both W2 and the first point for W3. Freezing the prior does bound the local action support: at each state, candidate actions come from the offline action distribution. However, the decision rule is not frozen. The online critic chooses which in-support action to take at each step, and chaining these per-step choices yields trajectories that the dataset never contains. Improvement beyond the dataset therefore happens at the trajectory level: the critic composes in-support fragments into new, task-solving behavior. This also explains our low-quality-data results (Fig. 5), where even near-random data provides usable fragments that the critic assembles.

For each domain from OGBench represented in the plot below, we compare the fraction of trajectories in the offline dataset that succeed to the success rate of BC-OFF at 1M online steps. Offline trajectory success is very low across all settings (1.7–7.0%), whereas BC-OFF achieves much higher success. The frozen prior, therefore, does not cap improvement at the level of behavior present in the dataset.

The plot can be seen here (App. E.1, Fig. 13): https://tinyurl.com/swjnmzww

As a visual aid to the result above, we plot trajectories on antmaze-large-explore with IFQL. We chose this dataset because it consists of short exploratory trajectories, from which goal-directed behavior would not be expected to arise without policy improvement. The left panel shows 20 randomly sampled offline trajectories: disjoint, localized walks that wander within confined corridors and never connect start to goal. The shading shows the offline data support across all transitions, which covers nearly all corridors. The middle panel shows 5 evaluation rollouts of BC-OFF at 1M steps. Successful rollouts follow goal-directed routes from start to goal, while every step remains within the offline data support. The rightmost panel shows BC-ON rollouts whose trajectories are mostly undirected and often fail to reach the goal.

The plot can be seen here (App. E.1, Fig. 14): https://tinyurl.com/3daebahw

W3. Limited policy improvement and insufficient alternative strategies

“Freezing the pretrained behavioral prior (the actor for IFQL and QC) may restrict online policy improvement”

The response to W2 above addresses this.

“The warmup experiments are also limited to 25K online interactions…..”

We have extended the warm-up experiments to 5k and 100k transitions. We initially used 5k, following the recommendation of WSRL [4], and added 25k because 5k alone did not prevent collapse. Larger warmups reduce BC-ON degradation; however, BC-OFF mostly performs better or matches performance.

The plot can be seen here (Fig. 5, Cols. 4–5): https://tinyurl.com/2anfm7mu

W4. Insufficient justification for expressive priors

We would like to clarify that ReBRAC is trained with its BC term on during the offline stage; BC is removed only during online fine-tuning. Moreover, keeping BC on during online fine-tuning does not help: without retained offline data, ReBRAC collapses with BC on as well (Fig. 5, Col. 2), which suggests that its failure is not solely due to the removal of behavioral regularization. The role of expressiveness is most directly isolated by the Gaussian IFQL ablation (Fig. 5, Col. 1). In IFQL, the actor is trained only by BC, so replacing the flow prior with a Gaussian changes only the policy class, while the data, critic, and training procedure remain unchanged. With BC off, the flow prior enables continued improvement while the Gaussian prior flatlines. We provide further analysis of this comparison, including evidence of mode averaging in the Gaussian prior.

Prior work has shown that unimodal policies trained by maximum likelihood on multimodal data average between behavioral modes, placing probability mass where the data has no actions [7, 8]. A Gaussian prior therefore proposes actions away from the valid behavior in the data, which alters its action coverage and, in turn, the quality of the resulting policy. The differences in policy quality and action coverage the reviewer mentions are thus consequences of the policy class, and we view them as the mechanism through which expressiveness matters rather than as independent confounds. We verify each step of this chain below.

To show that the Gaussian prior averages between behaviorally represented modes in the data, we do the following experiment using held-out validation datasets for the domains below. For each state in the dataset, we sample K = 32 proposals from each prior type and compute the mean Euclidean distance of each proposal to the nearest dataset action. If a prior captures the behavioral modes, its proposals lie close to valid demonstrations; if it averages between modes, its proposals land between demonstrator clusters and this distance grows.

DomainFlow priorGaussian prior
Cube Double Play0.20 ± 0.010.99 ± 0.01
Ant Soccer Arena0.75 ± 0.021.20 ± 0.02

As shown in the table above, the Gaussian prior’s proposals are substantially farther from valid demonstrator actions, consistent with mode averaging.

To rule out state coverage as the cause, we plot PCA projections of online rollouts at 1.2M steps. For consistency, all panels are colored using a single critic: the critic from the Gaussian-prior run at the end of offline pretraining (1M steps), before any online updates. In IFQL, the critic is pretrained via in-sample value learning on the dataset’s actions, independently of the prior, so the offline critics from both (Gaussian and Flow) runs are trained identically. With BC off, the Gaussian prior explores the state space comparably to the flow prior, yet it achieves 0% success on both antsoccer-arena and cube-double, whereas the flow prior achieves 43% and 30%, respectively. Moreover, the flow prior’s rollouts progress into regions the critic assigns high value, whereas the Gaussian prior’s rollouts predominantly remain in low-value regions despite their broad coverage. Therefore, we believe the loss of expressiveness is the reason behind the differences in action coverage and the resulting poor policy.

The plot can be seen here (App. E.3, Fig. 20): https://tinyurl.com/2p8x7dpr

Finally, to demonstrate generalization beyond flow priors, we extend the evaluation by replacing the flow prior with a diffusion prior (IDQL [8]), which also enables policy improvement with BC off (Fig. 3 in the updated paper). This shows that the benefit comes from expressiveness rather than from flow matching specifically. Note that for Cube Double IDQL, we used 3M offline steps and saw dramatic improvements in our approach within the first 1M steps, despite running for an additional 2M. We also allocated the same offline training budget to the Gaussian version and saw no performance improvements.

The plot can be seen here (Fig.3): https://tinyurl.com/bdd584n3

Reviewer_edPsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors find that in offline-to-online RL, when the offline dataset is not retained during online finetuning, one should not use the BC loss. They demonstrate that for three different RL agents on five OGBench domains that not using the BC constraint prevents catastrophic forgetting and improves online performance. They also provide analysis experiments that show why not using the BC constraint prevents critic collapse.

Critical strengths
  1. The presentation of the paper is very clear and easy to follow, and addresses an interesting setting (no offline data retention) in offline-to-online RL, and is very relevant to the community.
  2. The authors offer a lot of different ablation and informative analysis experiments that show why removing the BC constraint online is a good idea.
Critical weaknesses
  1. Novelty concerns: removing the BC constraint during the online phase is not a novel contribution, and done in several prior works, including the cited WSRL paper [1], as well as [2,3]. Therefore, this paper should not present itself as the work that introduced this technique.

  2. Experimental evaluation is lacking: the paper only uses 3 seeds per experiment, which is not sufficient to get significant results. The paper also does not compare to relevant baseline methods in offline-to-online RL.

[1] Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data. [2] Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions. [3] Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble

Questions for authors
  1. Why do you only consider 3 tasks per OGBench domain? Each OGBench domain comes with 5 tasks. How are the three tasks selected? It would be good to show results on all tasks for all the OGBench domains.
  2. Presentation could be improved: the paper offers a lot of ablations and experimental insights that are captured through the paragraph headings. While informative, they appear a bit disjoint from each other. There is an opportunity to improve the writing of the paper to unify all the ablations under some central insight.
4 comments

Final Author Response to the Area Chair and Reviewers

We sincerely thank the reviewers for their careful evaluation and constructive feedback. We appreciate that the additional experiments have addressed several empirical concerns, while recognizing that the main remaining question concerns the novelty and significance of our contribution. We agree that offline-to-online RL without retaining offline data, and removing BC during fine-tuning, have established precedents. Our contribution is not the introduction of BC removal, but a systematic investigation of how continually refitting an expressive behavioral prior to online-only data can undermine learning, why this happens, and under what conditions the pretrained prior can instead be preserved while policy improvement continues.

We believe that identifying and explaining a consequential failure mode is a meaningful scientific contribution, even when the resulting intervention is simple. Our findings go beyond the general intuition that distribution shift may cause forgetting. They provide evidence that degradation of the behavioral prior can impair subsequent critic learning through the data collected by the agent, that freezing alone is insufficient without an expressive prior and an effective policy-improvement mechanism, and that frozen priors can support successful trajectories absent from the offline dataset. Together, these findings offer an actionable design principle for future offline-to-online RL methods, namely preserving expressive pretrained behavioral knowledge while continuing to improve how that knowledge is used.

In response to the reviews, we substantially strengthened the experimental evidence, clarified the motivation and scope of our study, and revised the manuscript accordingly. All revised text is highlighted in blue.

  • Motivation for continued BC (6Tby). Continuing BC is the default in FQL, QC, and IDQL, so running them unchanged after the offline data is discarded is exactly BC-ON, which fails. These methods already apply BC to online samples, mixed with offline data; without retention, the prior is fit to online data alone. Simply turning BC off, with no other change, makes them effective without any other hyperparameter changes.
  • Expanded evaluation (edPs, EEGA) — Fig. 1. We extended the evaluation to 35 OGBench tasks across seven domains, with five seeds per task, totaling 175 runs per method and condition.
  • Generalization across policy architectures (EEGA, 6Tby) — Fig. 3, App. E.2. We added diffusion-based IDQL experiments and extended the behavioral and critic analyses to action-chunked QC.
  • Role of prior expressiveness (EEGA, 6Tby) — Fig. 5, Col. 1, App. E.3, Fig. 20. We added analyses of Gaussian-prior mode averaging, action validity, and state-space coverage to clarify why expressiveness matters.
  • Improvement beyond offline trajectories (EEGA) — App. E.1, Figs. 13–14. We added quantitative and trajectory-level evidence showing how frozen priors can support successful new trajectories by composing behaviors represented in the offline data.
  • Controlled analysis of critic degradation (6Tby) — Sec. 4, App. E.5, Fig. 18. We introduced a controlled replay-buffer experiment showing that data collected by BC-ON policies can degrade initially healthy critics under otherwise identical training procedures.
  • Additional baselines and practical evaluation (edPs, EEGA, 6Tby) — Figs. 4–5, Apps. E.4 and F. We added Robomimic human-demonstration experiments, extended replay warmups to 100k transitions, evaluated a WSRL-style baseline in Fig. 19, and clarified the relationship to VLA adaptation.
  • Improved presentation (edPs, 6Tby) — Secs. 3–4. We reorganized the analyses around the central experimental questions, clarified the relationship to prior work, and made the scope and limitations of our conclusions more explicit.

We acknowledge that our conclusions concern a specific class of offline-to-online RL methods with expressive behavioral priors, rather than all offline-to-online algorithms. We believe this clearly defined scope, together with the expanded evaluation and controlled analyses, strengthens the contribution. We respectfully ask the Area Chair to consider the work for the new understanding it establishes and the guidance it provides for developing more reliable offline-to-online RL methods, rather than evaluating its significance solely by the simplicity of the intervention.

We additionally evaluate a WSRL-style baseline, applying WSRL’s fine-tuning recipe (warmup, no behavioral constraint, unconstrained actor updates) to pretrained ReBRAC checkpoints, the strongest unimodal offline RL method we tested, on the domains where ReBRAC performs best offline (cube-double-play, puzzle-4x4-play, and scene-play). Across all three domains, the WSRL-style baseline underperforms compared to freezing the expressive prior with IFQL, FQL, and QC (details in App. E.4). We do not include [5] separately, as it follows WSRL’s recipe without a pretrained critic, which differs from ours.

The plot can be seen here (App. E.4, Fig. 19): https://tinyurl.com/2692z8s6

We have added a discussion of these new experiments to Appendix E.4 and E.6 of the revised manuscript.


[Q-1]. Why do you only consider 3 tasks per OGBench domain? Each OGBench domain comes ...

Please refer to the answer above [CW-2].


[Q-2]. Presentation could be improved: the paper offers a lot of ablations and experimental insights that ...

We thank the reviewer for this suggestion. We reorganized Sections 3 and 4. Section 3 now groups the ablations by the questions they answer. We present the Gaussian-prior and FQL distillation ablations together to clarify what is necessary for a frozen prior to be useful for policy improvement. We then group the BC-weight, replay-warmup, and delayed-stopping experiments to show how much performance can be recovered when BC continues. Section 4 traces what changes when the prior is refit to online-only replay, from forgetting offline behavior and proposing less diverse actions to collecting narrower rollouts and degrading critic learning. Each section ends with a short “Key observations” box that explains how its findings fit together. We also improved the paper's content by adding more experimental analysis in the appendix, along with corresponding plots.


[1] Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data. (Zhou, 2025, ICLR)

[2] Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions. (Luo, 2023)

[3] Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble. (Lee, 2022, CoRL)

[4] Deep Reinforcement Learning at the Edge of the Statistical Precipice. (Agarwal, 2021, NeurIPS)

[5] Efficient Online RL Fine-Tuning with Offline Pre-trained Policy Only. (Xiao, 2025, arXiv)

[6] Flow Q-learning. (Park, 2025, ICML)

Dear Reviewer,

As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. If any questions remain, we would be happy to provide further clarification. Otherwise, we would greatly appreciate it if you could consider revisiting your assessment and updating your score accordingly. Thanks for reviewing our work.

I thank the authors for their detailed response! The new experimental results strengthens the paper, and makes the empirical results more compelling. However, my concern about novelty (CW-1) is not entirely resolved. The authors have acknowledge that the central contribution of the paper is not the introduction of the "removal of BC" during the online phase, but the analysis of why this is needed. I think this is great to clarify in the paper. However, while this is an interesting insight, I'm not sure it is significant enough for publication at this conference, and so I am not entirely convinced to raise my current score. I thank the authors again for their response and new experiments.

Comments on the paper

1 comment

Final Author Response to the Area Chair and Reviewers

We sincerely thank the reviewers for their careful evaluation and constructive feedback. We appreciate that the additional experiments have addressed several empirical concerns, while recognizing that the main remaining question concerns the novelty and significance of our contribution. We agree that offline-to-online RL without retaining offline data, and removing BC during fine-tuning, have established precedents. Our contribution is not the introduction of BC removal, but a systematic investigation of how continually refitting an expressive behavioral prior to online-only data can undermine learning, why this happens, and under what conditions the pretrained prior can instead be preserved while policy improvement continues.

We believe that identifying and explaining a consequential failure mode is a meaningful scientific contribution, even when the resulting intervention is simple. Our findings go beyond the general intuition that distribution shift may cause forgetting. They provide evidence that degradation of the behavioral prior can impair subsequent critic learning through the data collected by the agent, that freezing alone is insufficient without an expressive prior and an effective policy-improvement mechanism, and that frozen priors can support successful trajectories absent from the offline dataset. Together, these findings offer an actionable design principle for future offline-to-online RL methods, namely preserving expressive pretrained behavioral knowledge while continuing to improve how that knowledge is used.

In response to the reviews, we substantially strengthened the experimental evidence, clarified the motivation and scope of our study, and revised the manuscript accordingly. All revised text is highlighted in blue.

  • Motivation for continued BC (6Tby). Continuing BC is the default in FQL, QC, and IDQL, so running them unchanged after the offline data is discarded is exactly BC-ON, which fails. These methods already apply BC to online samples, mixed with offline data; without retention, the prior is fit to online data alone. Simply turning BC off, with no other change, makes them effective without any other hyperparameter changes.
  • Expanded evaluation (edPs, EEGA) — Fig. 1. We extended the evaluation to 35 OGBench tasks across seven domains, with five seeds per task, totaling 175 runs per method and condition.
  • Generalization across policy architectures (EEGA, 6Tby) — Fig. 3, App. E.2. We added diffusion-based IDQL experiments and extended the behavioral and critic analyses to action-chunked QC.
  • Role of prior expressiveness (EEGA, 6Tby) — Fig. 5, Col. 1, App. E.3, Fig. 20. We added analyses of Gaussian-prior mode averaging, action validity, and state-space coverage to clarify why expressiveness matters.
  • Improvement beyond offline trajectories (EEGA) — App. E.1, Figs. 13–14. We added quantitative and trajectory-level evidence showing how frozen priors can support successful new trajectories by composing behaviors represented in the offline data.
  • Controlled analysis of critic degradation (6Tby) — Sec. 4, App. E.5, Fig. 18. We introduced a controlled replay-buffer experiment showing that data collected by BC-ON policies can degrade initially healthy critics under otherwise identical training procedures.
  • Additional baselines and practical evaluation (edPs, EEGA, 6Tby) — Figs. 4–5, Apps. E.4 and F. We added Robomimic human-demonstration experiments, extended replay warmups to 100k transitions, evaluated a WSRL-style baseline in Fig. 19, and clarified the relationship to VLA adaptation.
  • Improved presentation (edPs, 6Tby) — Secs. 3–4. We reorganized the analyses around the central experimental questions, clarified the relationship to prior work, and made the scope and limitations of our conclusions more explicit.

We acknowledge that our conclusions concern a specific class of offline-to-online RL methods with expressive behavioral priors, rather than all offline-to-online algorithms. We believe this clearly defined scope, together with the expanded evaluation and controlled analyses, strengthens the contribution. We respectfully ask the Area Chair to consider the work for the new understanding it establishes and the guidance it provides for developing more reliable offline-to-online RL methods, rather than evaluating its significance solely by the simplicity of the intervention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.