acceptodds
Under review as a conference paper at ICLR 2027

Asymmetric Actor-Critic Pretraining for Efficient Flow-Based Online RL Fine-tuning

Abstract

Offline-to-online (O2O) reinforcement learning employs synchronized, symmetric offline training of the actor and critic for downstream online fine-tuning. Our analysis reveals that this symmetric training process impairs performance during online fine-tuning: while the policy network acquires a sampling prior through offline training to ensure efficient online fine-tuning, the value network may lose plasticity due to prolonged offline training, thereby failing to rapidly adapt to new environments when faced with distribution shifts between the offline and online settings. Based on this distinction, we identify a trade-off mechanism regarding critic offline training and investigate the impact of offline data coverage on the actor's online performance. Our research reveals an asymmetry in offline training: the critic is highly sensitive to the extent of offline training, whereas the actor relies more heavily on the coverage of effective regions within the offline data. Leveraging this asymmetry, we propose Critic-Free Pretraining—a simple, asymmetric training framework widely applicable to various algorithms. CFP involves pretraining the actor without a critic, subsequently initializing a fresh critic with a joint offline warm-up before entering the online interaction phase. Across 8 challenging manipulation domains, spanning OGBench and Robomimic, CFP achieves performance comparable to or superior to naive O2O methods.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, offline-to-online RL, flow-matching policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_roR4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper studies offline-to-online (O2O) fine-tuning of flow-based actor-critic methods. It argues that prolonged offline critic training impairs online fine-tuning through loss of plasticity and value mismatch, whereas the actor mainly benefits from offline data coverage. It proposes Critic-Free Pretraining (CFP): pretrain the actor with behavior cloning only, then initialize a fresh critic, run a short offline warm-up, and fine-tune online. A theorem bounds downstream performance by the behavior data’s coverage and the imitation error. Experiments with four FQL/QC-family algorithms on OGBench and Robomimic show CFP comparable or superior to standard O2O, with lower offline compute.

Critical strengths

It is a simple and efficient method. CFP is a workflow-level change that applies to several flow-based algorithms without modifying their objectives, and it reduces offline training time.

A clear empirical phenomenon on hard tasks, with useful ablations. The gains are large on some difficult Cube tasks.

Critical weaknesses

Novelty is limited, and the claimed mechanism is not isolated.

  • PORL (Xiao et al., 2025) already proposes policy-only pretraining with a freshly initialized critic. The differences are Gaussian vs. flow policies and CFP’s short offline warm-up. PORL is discussed only in the appendix related work, and the main text cites it only as a pessimism reference.
  • The novelty therefore mainly rests on the analysis, which does not establish its causal claim. Relative to O2O, CFP changes both the actor’s offline objective (BC-only instead of Q-guided) and the critic (fresh with a short warm-up instead of long-trained), so the gains cannot be attributed to the critic alone. Resetting the critic of a standard O2O agent before the same warm-up would isolate this effect.
  • Theorem 1 contains no critic and holds for any BC-pretrained actor, so it does not explain why critic pretraining hurts online fine-tuning.

The experiments are narrow and the comparisons incomplete.

  • The functionally closest baseline is missing: PORL (flow version). All four base methods are from the FQL/QC family.
  • All 28 tasks are sparse-reward, low-dimensional manipulation, and the main text shows only one task per domain.
  • The gains concentrate on a few Cube tasks and algorithms. Robomimic is roughly a tie, and CFP does not help all four algorithms consistently.
  • CFP adds a per-task hyperparameter (warm-up steps searched over 6 values) that O2O does not have. So the efficiency claim should account for tuning cost.
Questions for authors

See the weaknesses

10 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the thorough, rigorous, and penetrating evaluation of our work. We summarize your comment as follows: whether the gains can be attributed to the critic alone, how CFP genuinely differs from PORL, the precise scope of Theorem 1, and considerations regarding the generalizability and practical utility of the CFP experiments.

Our detailed responses are provided below:


1. Mechanism Isolation

Reviewer's concern: Relative to O2O, CFP changes both the actor’s offline objective (BC-only instead of Q-guided) and the critic (fresh with a short warm-up instead of long-trained), so the gains cannot be attributed to the critic alone. Resetting the critic of a standard O2O agent before the same warm-up would isolate this effect.

We are profoundly grateful to the reviewer for suggesting this crucial experiment.

Experiment: Following your exact recommendation, we took the actor checkpoint from a standard 1M-step O2O run, completely reset its critic, and applied the identical 10k offline warm-up before online fine-tuning. We first evaluated this workflow on Cube Triple across 5 seeds with QCFQL as the backbone:

Table: QCFQL Success Rates on Cube Triple after 1M Online Interactions

WorkflowTraining before online interactionTask 1Task 2Task 3Task 4Task 5Avg.
O2O (standard)1M joint updates0.996±0.0110.836±0.1440.656±0.1460.256±0.1270.000±0.0000.549±0.072
O2O + fresh critic1M joint updates, reset QQ, 10k warm-up1.000±0.0000.984±0.0110.960±0.0300.988±0.0220.000±0.0000.786±0.006

The above table shows that resetting the critic of a standard O2O model immediately boosts its performance from 0.549 to 0.786. This rigorously isolates the mechanism and directly validates your hypothesis and our core claim: the primary performance bottleneck in standard O2O is indeed the critic's loss of plasticity from prolonged offline training. The experiments above confirm the general applicability of the CFP concept and its underlying mechanism; by eliminating the time-consuming and costly long-time offline training of a critic, we are actually able to achieve superior results during online fine-tuning.

To verify the generality of this mechanism, we used different algorithm backbones on Cube Double and Cube Triple:

Table: O2O w/ wo a fresh critic after 1M online interactions with different backbones

BackboneWorkflowCube Double Task 4Cube Triple Task 4
QCFQL-nstepO2O0.920 ± 0.0390.688 ± 0.109
QCFQL-nstepO2O + fresh critic0.996 ± 0.0110.984 ± 0.021
QCO2O0.868 ± 0.1650.340 ± 0.140
QCO2O + fresh critic0.968 ± 0.0330.500 ± 0.152
FQLO2O0.016 ± 0.0270.000 ± 0.000
FQLO2O + fresh critic0.864 ± 0.1070.000 ± 0.000

The Practical Value of CFP: Given that "O2O + fresh critic" matches CFP, why is CFP still preferable?

  • Computation: O2O spends massive compute optimizing the critic and backpropagating Q-gradients during the 1M-step pretraining, only to discard that critic. CFP eliminates critic computation entirely during actor pretraining. As we mentioned in our original manuscript under "Q: How efficient is CFP for offline training:"

    In particular, QC (Best-of-N) evaluates 32 or even more action chunks with the critic ensemble. Removing these critic updates during actor pretraining reduces the measured training time from 83 to 32 minutes in the reported setting, corresponding to a 2.59× speedup.

  • Multi-task Reusability: In domain benchmarks like OGBench, tasks share the same offline state-action dataset but have different rewards. Standard O2O requires a separate, expensive 1M-step joint training run for every single task. In contrast, CFP’s actor pretraining is reward-free, meaning a single pretrained actor can be reused to initialize all tasks in that domain, requiring only the brief 10k warm-up to adapt to specific tasks.

We have incorporated these new experiments directly into Section 5 of the main text under the dedicated Q&A: "Q: How does CFP compare with other offline-to-online workflows?" and summarized it in the newly added Table 1. We also listed the full results of different backbones in Appendix E.3 under "Q: How does CFP compare to other workflows?".

AuthorsOpenReview ↗

Response 2

2. Comparison with PORL.

Reviewer's concern: PORL (Xiao et al., 2025) already proposes policy-only pretraining with a freshly initialized critic. The differences are Gaussian vs. flow policies and CFP’s short offline warm-up. PORL is discussed only in the appendix related work, and the main text cites it only as a pessimism reference.

We agree that a direct empirical comparison with a flow-based implementation of PORL was a missing piece necessary to contextualize our contribution.

Original Submission vs. Revisions: In our original submission (Appendix C.1, Lines 864–870), we conceptually contrasted CFP with PORL:

"Xiao et al. (2025)... discover the phenomenon that the pretrained critic could potentially hinder online fine-tuning. Their work focuses primarily on the specific case of the Gaussian algorithm; PORL studies online fine-tuning from a pretrained policy without retaining a pretrained critic and emphasizes persistent value underestimation... This conclusion differs from the finding of reduced plasticity associated with CFP."

However, we fully agree with the reviewer that a conceptual mention was insufficient and that an explicit, side-by-side comparison of the workflows is essential. To make this distinction fully transparent, we have added explicit comparisons in both the main text and Appendix:

  1. In the Main Text (Section 5, Table 1): We added a flow-based PORL baseline (using the exact same QCFQL backbone and BC-pretrained flow actor, with 20k online warm-up). On Cube Triple, PORL records 0.000 final success across all five tasks, whereas CFP reaches 1.000, 0.940, 0.980, and 0.968 on Tasks 1–4.

  2. In Appendix C.1: We added formal side-by-side pseudocode (Algorithm 3 vs. Algorithm 4) detailing the algorithmic mechanics. In summary, both CFP and PORL share the core intuition of avoiding a fully pretrained offline critic to prevent negative transfer, choosing instead to preserve the pretrained actor while initializing a fresh critic. The fundamental distinction lies in the data source for critic calibration: PORL discards the offline dataset to warm up the critic exclusively on online interactions collected by the frozen actor, whereas CFP conducts a brief warm-up directly on the offline dataset and retains these offline transitions in the replay buffer.

  3. In Appendix E.3: We expanded the evaluation to all 15 tasks across Cube Double, Triple, and Quadruple:

    Task SuiteTask IDQCFQL-CFP Online SRQCFQL-PORL Online SR
    Cube-Double10.996 ± 0.0090.992 ± 0.011
    (1M dataset)2 to 50.996 ± 0.009 (all)0.000 ± 0.000 (all)
    Cube-Triple11.000 ± 0.0000.000 ± 0.000
    (1M dataset)20.940 ± 0.0790.000 ± 0.000
    30.980 ± 0.0140.000 ± 0.000
    40.968 ± 0.0230.000 ± 0.000
    50.000 ± 0.0000.000 ± 0.000
    Cube-Quadruple1 to 30.996 ± 0.009 (all)0.000 ± 0.000 (all)
    (100M dataset)41.000 ± 0.0000.000 ± 0.000
    50.000 ± 0.0000.000 ± 0.000

With the same BC-pretrained actor, our flow-based PORL implementation achieves nonzero final success on one of the 15 Cube tasks. CFP achieves nonzero success on 13 of the 15 tasks. One possible explanation is that the frozen actor collects few successful trajectories during online warm-up on the harder tasks, whereas CFP can train its critic on the broader set of transitions already present in the offline dataset. The comparison captures the combined effect of offline warm-up and continued access to offline replay.

To better showcase this comparative analysis and present the full training progression, we have incorporated clear, multi-domain learning curve figures alongside an in-depth discussion in Appendix E.3, under the subsection "Q: How does CFP compare to PORL?".

AuthorsOpenReview ↗

Response 3

3. What Theorem 1 Establishes.

Reviewer's concern: Theorem 1 contains no critic and holds for any BC-pretrained actor, so it does not explain why critic pretraining hurts online fine-tuning.

The reviewer is completely correct: Theorem 1 is strictly an actor-side guarantee and is not intended to explain the critic's failure mode.

In our original submission (Lines 270–274 and Lines 316–320), we introduced this theorem to address a specific skepticism regarding critic-free actor pretraining:

"CFP uses supervised actor pretraining to acquire a behavioral prior before value-guided improvement. We analyze when this prior is sufficient to support downstream learning... Thus, the theorem provides a conditional sufficiency statement for critic-free actor pretraining."

  • The exact role of Theorem 1: When we propose removing the critic during actor pretraining, a natural concern arises: *does omitting value guidance offline cripple the actor's downstream capabilities? Theorem 1 characterizes how offline data coverage, the pretrained actor’s imitation error, and downstream local regret enter the performance bound. Its role is to identify conditions under which a BC-pretrained actor can support subsequent policy improvement. We study the effect of critic pretraining separately through empirical comparisons.
  • The critic's failure: The negative impact of prolonged critic pretraining is a representational phenomenon (loss of plasticity and critic mismatch), which we investigate empirically via gradient norms, parameter shifts (Figure 1), Q-mean tracking (Figure 2), and the newly added "O2O + fresh critic" ablation.

We fully agree with reviewer's concerns regarding Theorem 1. We have further clarified the role of Theorem 1 in the main text Section 4 to address any potential doubts:

Theorem 1 explains how offline data coverage and the actor’s imitation error enter the downstream performance bound.

Reviewer_u4w1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Authors introduce CFP, a framework for RL online fine-tuning that moves critic pre-training to happen after having done BC pre-training for the policy. Authors find that after selecting the hyperparameter of warm-up steps that performed the best, this style of training outperforms training the critic and policy concurrently with the same number of steps.

Critical strengths

The paper is generally easy to read, and authors test their method on top of multiple base algorithms.

Critical weaknesses
  • Lines 152-155: “Figure 1 shows that, following online training, the parameter shift in the CFP critic is significantly greater than that in the O2O critic. This indicates that online training has a more pronounced impact on the CFP critic, suggesting a superior ability to adapt to new environments.”

This is not necessarily true. Authors make the observation that parameters change more during online fine-tuning if the critic is not pretrained (or pre-trained to a lesser, undisclosed, extent), and infer that this is due to superior adaptability. However, it stands to reason that a critic that has been pretrained (at all, or for longer) simply needs smaller adjustments to fit new data from the same environment when compared to a critic from scratch, which has to fit all the basic features too.

The same argument applies to the gradient norm (Fig 1 c). During the beginning of training grad norms are higher for CFP critics - which is entirely expected, since it has to fit entirely new data. After ~400k training steps, differences might not be significant anymore. Generally, neither parameter shifts nor gradient norms show loss of plasticity by themselves. E.g. the pre-trained critic may in fact have been better at fitting the data from the start, which would lead to lower gradient norms, but clearly this is not a sign of loss of plasticity.

  • Line 268: “CFP introduces a radical change for the actor: it is pretrained entirely without critic guidance”.

In vanilla FQL, the flow policy is also pretrained entirely without critic guidance. Only the one-step policy gets critic guidance.

  • Figure 4 goes against the narrative of the paper, and the results have two big unexplained gaps.

First, compare the performances of O2O and CFP in the offline phase. Across seed numbers CFP is markedly better than O2O. The paper’s narrative is that critic pretraining introduces loss of plasticity when switching to the online phase. This does not explain why there would be such a big gap in performance during the offline phase. The difference between O2O and CFP during the offline phase is not clearly communicated here, but if I understand correctly, O2O trains the critic and one-step distilled policy alongside the BC flow policy, whereas CFP first trains the BC flow policy, and then keeps it frozen and trains the critic and the one-step distilled policy. Since loss of plasticity does not apply during the offline phase, this gap in performance is unexplained, and needs to be clarified.

Second, note the sudden jump in performance when moving to the online phase for O2O. This goes against the argument of loss of plasticity during pretraining.

Third, note the sudden drop in performance when moving to the online phase for CFP, and further note that at convergence CFP generally does not surpass the offline level of performance, and for some numbers of reset seeds it does not reach it either. The motivation of the paper mentions the severe performance drops and inefficient fine-tuning when transitioning to the online phase. This experiment seems to show that CFP does not address either.

  • Lines 1060-1062: “Regarding the hyperparameter of warm-up steps, we conduct an exhaustive search within the range of [5k, 10k, 30k, 50k, 0.1M, 0.2M] and select the value that achieves the best balance between final performance and computational efficiency.” Table 4 shows warmup steps specifically picked across different methods for the same task, as well as across tasks and across specific initial conditions for a given task. There are two big concerns with this: 1. This tuning was not performed for the O2O baselines. These baselines used a common hyperparam of 1M offline training steps. 2. As far as it is mentioned in the paper, authors did not perform an independent evaluation after choosing the winner hyperparameters, and rather kept the results after the argmax step. All else being equal, this practice will give unfairly high results.

Overall, I find that the explanations authors give are unsubstantiated, and that there is no substantial takeaway a reader could get from this paper. If my understanding is correct, the CFP framework can be summarized as: rather than pre-training the policy alongside the critic, first pre-train the policy, and then pretrain the critic. Any performance or efficiency claims on top of this framework seem to come from tuning the amount of critic pre-training rather than on solving an underlying offline-to-online issue. CFP moves the sensitivity of critic pre-training to the amount of training steps used to a “warmup” phase, but does not actually address the problem of how to make this critic pretraining more stable or give a systematic way of picking warm-up steps without performing online fine-tuning first.

Readability concerns:

  • Section 3.1 already shows experiments with CFP critics, before they are introduced. It is not clear which algorithm is used to train these critics, or whether they are trained entirely from scratch.
  • The theory in Section 4 is poorly motivated in the writing, and underexplained.
Questions for authors
  • Lines 79-80: "We conduct extensive experiments across 28 manipulation tasks spanning 8 domains.” Results only show experiments across 8 tasks. What are the mentioned 28 manipulation tasks?
  • Lines 372-373: “For OGBench, we select Task 4 from each domain as the representative task”. What exactly does “Task 4” represent? Is it an individual starting condition from the task? If so, why is this done, rather than showing randomized initial conditions, as is standard practice for RL in robotic manipulation?
16 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the careful, critical, and constructive evaluation of our manuscript. We summarize your first major concern as follows: whether the larger parameter shifts and early gradient norms of the CFP critic in Figure 1 genuinely reflect higher plasticity rather than simply starting from scratch (or with less pretraining) compared to an already well-fitted O2O critic on the same environment.

Following your insightful critique, we clarify the exact diagnostic intent of Figure 2 in our original submission, contextualize our findings with prior work on distribution shift and plasticity, and report a new cross-environment transfer experiment (Puzzle 3×3 →\rightarrow Cube Quadruple) that directly tests the "same-environment" hypothesis. We have also updated Section 3.1 and the Appendix to make the experimental setup and claims fully transparent. Our detailed response is provided below:


1. Parameter Shifts, Gradient Norms, and Loss of Plasticity

Reviewer's concern: Authors make the observation that parameters change more during online fine-tuning if the critic is not pretrained (or pre-trained to a lesser, undisclosed, extent), and infer that this is due to superior adaptability. However, it stands to reason that a critic that has been pretrained (at all, or for longer) simply needs smaller adjustments to fit new data from the same environment when compared to a critic from scratch, which has to fit all the basic features too.

We completely agree with the reviewer that parameter shifts and gradient norms alone cannot establish a loss of plasticity, as a well-fitted critic may naturally require smaller updates.

Original Submission Context & Intent: In our original submission, we explicitly noted that plasticity metrics alone are insufficient if the critic has already converged (Section 3.1 Line 159-161):

"Merely demonstrating a decline in the plasticity of the critic network is insufficient to prove that offline training is detrimental to online fine-tuning. After all, if the critic has already converged to the optimal point during offline training, there is no need for online training."

Precisely to address the possibility that the O2O critic moves less simply because it is already well-fitted, we paired Figure 1 with Figure 2, where we wrote (Section 3.1 Line 188-193):

"Figure 2 further illustrates the mean Q-values output by the CFP critic and the O2O critic, along with the corresponding agent success rate (SR). It is evident that the outputs of the two critics are very close at the onset of online training; however, as fine-tuning progresses, the Q-mean of the CFP critic rises rapidly, whereas that of the O2O critic increases slowly. This further confirms that the low fine-tuning efficiency of O2O flow RL is not due to critic pessimism, but is instead a consequence of prolonged offline training under distribution shifts."

Our purpose in presenting Figure 2 alongside Figure 1 was to show that both critics begin online fine-tuning at comparable Q-value levels—rather than the O2O critic already sitting at the optimal online values—while diverging sharply in their subsequent online adaptation.

AuthorsOpenReview ↗

Response 1.1 & 1.2 & 1.3

Our Detailed Response:

1. Figure 2 gives no indication that the pretrained critic had already converged to the values reached during online fine-tuning. We agree that a well-fitted critic may require smaller updates. Section 3.1 explicitly raises this possibility, which is why we report mean Q-values and policy success rates alongside the parameter diagnostics. In our Cube Triple Task 4 run, the fresh critic has already received a 10k-step offline warm-up before online interaction begins. The two critics’ earliest logged mean predictions are close: −228.77 for CFP and −231.38 for O2O, a difference of only 2.61. Relative to the initial evaluation rollouts, their approximate mean-Q gaps are 41.5 and 38.9, respectively. Both predictions are appreciably above this empirical return reference, and the pretrained critic shows little initial advantage by this aggregate measure.

Figure 2 also lets us examine each critic against its own subsequent predictions. Over 1M online interactions, the CFP critic’s mean Q-value rises from −228.77 to −163.81 (+64.96), while the O2O critic’s rises from −231.38 to −217.00 (+14.38). Thus, even the pretrained critic subsequently revises its estimates upward as its policy improves. The difference is particularly clear early in fine-tuning: by 400k interactions, CFP’s mean Q-value has increased by 23.77 and its policy reaches 80% success. Over the same interval, O2O’s mean Q-value has decreased by 1.81 and its policy reaches 8% success. These observations weaken the explanation that O2O exhibits smaller parameter shifts simply because offline pretraining had already supplied the values needed downstream.

The subsequent rise in Q-values shows why continued value adaptation matters as the policy and its visitation distribution change. To test initial estimation accuracy directly against a known online Q-function, we turn to the controlled experiment in Appendix F. There, the pretrained critic can start farther from the true online values and still adapt more slowly than the fresh critic.

2. The toy experiment directly tests whether longer pretraining gives the critic a better initial fit. In our Appendix F toy MDP, the exact ground-truth online Q-function (QonQ^{\text{on}}) is known, allowing us to measure estimation error directly (Figure 22 in original version, right panel). When the offline dataset contains few successful trajectories, the pretrained O2O critic can enter online fine-tuning with larger initial error relative to QonQ^{\text{on}} than the fresh CFP critic, yet still adapts much more slowly. This provides a counterexample to the reviewer’s proposed explanation: a critic can fit the available offline data extensively and still require substantial correction on the states and actions relevant to policy improvement.

3. Prior work identifies both critic mismatch and reduced plasticity. In particular, the reviewer’s observation that online data come from the same environment still allows the state–action distribution to shift substantially as the policy improves:

  • Lee et al. [1] show that this distribution shift can expose errors in offline Q-estimates and amplify them through bootstrapping.
  • Cal-QL [2] demonstrates that the scale of an offline critic’s predictions matters critically for subsequent online fine-tuning.
  • WSRL [3] further identifies a mismatch between offline data and online rollouts as a cause of value-function divergence at the O2O transition.

These results explain why an offline critic may need to change substantially even when the environment is fixed. Complementary research on primacy bias [4] and plasticity [5] addresses how readily a network can make that change after extensive prior optimization.


References:

[1] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic Q-ensemble. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2022.

[2] Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.

[3] Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data. In International Conference on Learning Representations (ICLR), 2025.

[4] Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning (ICML), 2022.

[5] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning (ICML), 2023.

AuthorsOpenReview ↗

Response 1.4

4. A new cross-domain experiment tests whether prior fitting on the same task explains the smaller critic updates. The reviewer suggests that a pretrained critic may require only small adjustments because it has already learned useful features from the same environment. To test this explanation, we pretrained a QCFQL critic on Puzzle 3×3 and transferred it to Cube Quadruple Task 4. The two domains have matching state and action dimensions, allowing direct weight transfer, while the Puzzle-pretrained critic has received no offline updates on the target task. We compared it with a fresh critic during online fine-tuning on Cube Quadruple and tracked two quantities: the change in each output-layer parameter from its own online initialization, and the critic-loss gradient norm, ∥∇ϕLQ∥2\|\nabla_\phi \mathcal{L}_Q\|_2, throughout training.

Both measurements are consistent with our hypothesis. Over 1M online interactions, the fresh critic exhibits more pronounced output-layer shifts. It also exhibits larger gradient-norm peaks early in fine-tuning, when the two critics first adapt to Cube Quadruple. The gradient traces overlap more frequently later in training. This timing is informative: the strongest difference in gradient response occurs during the initial adaptation period, while the output-layer comparison records the cumulative displacement over the full run. The same qualitative pattern appears in the original Cube Triple comparison and persists when the pretrained critic comes from a different domain.

Prior work provides a basis for examining gradient magnitude in this setting. Abbas et al. [2] observed diminishing gradients in value-based agents under nonstationarity and linked them to increasingly sparse network activations. Lyle et al. [1] [3] studied plasticity through a network’s ability to fit changing targets and examined how its optimization dynamics affect that ability. We therefore interpret the gradient norm as a diagnostic of the critic’s response to online updates. The corresponding changes in predictions and task performance in Figure 2, and the direct comparison against the true online Q-function in Appendix F, provide the evidence about how effectively those updates translate into adaptation.

Summary and revision. We understand the reviewer’s concern: a critic that already fits the relevant values well can have both small gradients and small parameter updates. To address this possibility more directly, we have expanded Figure 1 in Section 3.1 of the revised main text. The left two panels show output-layer shifts and critic gradient norms on Cube Triple Task 4; the new right two panels report the same measurements for the Puzzle-to-Cube transfer. We have also revised the accompanying discussion to connect these diagnostics with the other evidence. Figure 2 shows that the two critics begin with similar mean predictions and subsequently adapt at different rates. Appendix F shows that, where the true online Q-function is known, the pretrained critic can start with greater error and still adapt more slowly. The new Figure 1 shows that smaller critic updates persist even after removing prior training on the target domain. Together, these findings support our interpretation that prolonged offline training can reduce a critic’s responsiveness during online fine-tuning.

​


References:

[1] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning (ICML), 2023.

[2] Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. In Proceedings of the 2nd Conference on Lifelong Learning Agents (CoLLAs), 2023.

[3] Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. In Proceedings of the 3rd Conference on Lifelong Learning Agents (CoLLAs), 2025.

Reviewer_J8JMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies offline-to-online reinforcement learning with flow-based policies. The authors argue that the standard practice of jointly pretraining the actor and critic can hurt subsequent online fine-tuning. I.e., extensive offline training may reduce the critic's plasticity while the actor benefits from offline pretraining by learning a useful behavioral prior. Based on this observation, the authors propose Critic-Free Pretraining (CFP), which first pretrains the actor using behavioral cloning, then initializes a fresh critic and jointly trains both networks for a short offline warm-up before online fine-tuning. Experiments showcase the efficacy of the proposed method. The paper also provides a theoretical bound on downstream performance to the state coverage of the pretrained actor.

Critical strengths

The observed asymmetry between actor and critic offline training is intuitive and informative. Actor training on learning distributions is useful for downstream exploration while critic may not favor overlong training at this stage.

The proposed method is straightforward and aligns well with the observation. It can be applied to several existing flow-based RL algorithms, which makes the method potentially useful in practice. The improvement in offline training efficiency is also a nice benefit.

Minor: The QA format in the experimental section is quite interesting and straight to the point.

Critical weaknesses

The paper mentions that the duration of offline critic warm-up is quite important for the final performance of CFP, and Appendix E.1 denotes that this is lrgely based on hyperparameter search that changes with tasks. If this hypaprameter search need to be done for every task without a principled and efficient method to choose it, it will hinder the practical value of the proposed method.

Questions for authors

The authors cited [1] but did not compare with it, since both methods are against fully pretrained critic and support retaining the actor while training a new critic, it may worth a comparison to discuss pros and cons of each.

[1] Xiao et al; Efficient online rl fine tuning with offline pre-trained policy only.

3 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the positive evaluation of our work, particularly for recognizing our method as "intuitive," "straightforward," and "potentially useful in practice," as well as appreciating the QA format in our experimental section.

We deeply appreciate your constructive feedback regarding the hyperparameter tuning cost and the suggestion to compare with PORL [1]. In response, we have added two dedicated Q&A ablation subsections with new experimental curves at the end of Appendix E.3 in our revised manuscript to thoroughly address these points. We have carefully addressed your concerns below:

1. Warm-up duration and tuning cost.

Reviewer's concern: If this hypaprameter search need to be done for every task without a principled and efficient method to choose it, it will hinder the practical value of the proposed method.

First and foremost, we sincerely apologize for a typo in our Appendix regarding the hyperparameters, which understandably caused your concern about tuning difficulty. We mistakenly wrote that QCFQL-nstep used 0.1M (100k) warm-up steps on cube-triple-task5. After carefully re-verifying our code and logs, we confirm that we used exactly 10k warm-up steps across ALL algorithms on ALL 5 cube-triple tasks, as well as ALL 5 cube-double tasks. We will correct this typo in Table 4 of the final version.

To clarify our statement regarding the "sensitivity" of the warm-up duration: this primarily refers to extreme changes across orders of magnitude in the offline training budget (e.g., 10k vs. 10M updates). Within a reasonable local range, the online fine-tuning performance is actually highly robust and insensitive to minor changes.

New Ablation in Appendix E.3: To eliminate your concerns regarding local sensitivity, we have added a dedicated Q&A subsection at the end of Appendix E.3 ("Q: How sensitive is CFP to the number of warm-up steps?") along with new ablation curves, presenting a local parameter sweep on Cube Triple Task 4:

Table: QCFQL-CFP warm-up sweep on Cube Triple Task 4 (Mean ± sample SD over 5 seeds)

Offline warm-up updatesEnd of warm-up SRAfter 1M online steps SR
5k0.000 ± 0.0000.968 ± 0.023
10k0.000 ± 0.0000.968 ± 0.023
200k0.000 ± 0.0000.936 ± 0.052

As shown, performance remains nearly identical between 5k and 10k updates, proving that exhaustive fine-grained search is unnecessary around the chosen budget. Furthermore, the slight performance degradation at 200k perfectly corroborates our core claim: excessive offline critic updates begin to induce a loss of plasticity, hurting downstream online fine-tuning.

To better illustrate the dynamics across different warm-up budgets, we have included a clear training curve plot along with a more comprehensive and detailed discussion in the ablation section of our revised manuscript, specifically under the subsection "Q: How sensitive is CFP to the number of warm-up steps?" in Appendix E.3.


AuthorsOpenReview ↗

Response 2

2. Comparison with PORL [1].

Reviewer's question: The authors cited [1] but did not compare with it, since both methods are against fully pretrained critic and support retaining the actor while training a new critic, it may worth a comparison to discuss pros and cons of each.

We thank the reviewer for raising this insightful suggestion. We would like to gently note that in our original submission (Appendix C.1, Lines 864–870), we had briefly discussed the relationship with PORL [1], noting that while PORL emphasizes value underestimation in Gaussian policies and initializes the critic via online interaction, CFP targets the loss of plasticity in flow-based policies and advocates a brief offline warm-up:

Xiao et al. (2025) also discover the phenomenon that the pretrained critic could potentially hinder online fine-tuning. Their work focuses primarily on the specific case of the Gaussian algorithm; PORL studies online fine-tuning from a pretrained policy without retaining a pretrained critic and emphasizes persistent value underestimation associated with offline training. This conclusion differs from the finding of reduced plasticity associated with CFP.

However, we fully agree with the reviewer that a conceptual mention was insufficient and that an explicit, side-by-side comparison of the workflows and empirical pros and cons is essential.

Revisions in Both Main Text and Appendix:

  1. In the Main Text (Section 5): We have added a dedicated Q&A subsection ("Q: How does CFP compare with other offline-to-online workflows?") accompanied by a new comparison table directly comparing CFP, PORL, and various O2O workflow variants on all five Cube Triple tasks.
  2. In Appendix C.1: We have substantially expanded our discussion with a new explanatory paragraph and side-by-side pseudocode (Algorithm 3 vs. Algorithm 4).

In summary, both CFP and PORL share the core intuition of avoiding a fully pretrained offline critic to prevent negative transfer, choosing instead to preserve the pretrained actor while initializing a fresh critic. The fundamental distinction lies in the data source for critic calibration: PORL discards the offline dataset to warm up the critic exclusively on online interactions collected by the frozen actor, whereas CFP conducts a brief warm-up directly on the offline dataset and retains these offline transitions in the replay buffer.


[1] Xiao et al; Efficient online rl fine tuning with offline pre-trained policy only.

AuthorsOpenReview ↗

Response 2 (continued from above)

Empirical Comparison: Furthermore, we added a dedicated Q&A subsection at the end of Appendix E.3 ("Q: How does CFP compare to PORL?") with new multi-task learning curves. We conducted extensive evaluations comparing CFP and the PORL workflow (both with a QCFQL backbone) across all 15 manipulation tasks in Cube Double, Cube Triple, and Cube Quadruple. To ensure rigorous fairness, PORL’s 20k online warm-up interactions are counted toward the same 1M-step interaction budget.

Task SuiteTask IDQCFQL-CFP (Ours) Online SRQCFQL-PORL Online SR
Cube-Double10.996 ± 0.0090.992 ± 0.011
(1M dataset)2 to 50.996 ± 0.009 (all)0.000 ± 0.000 (all)
Cube-Triple11.000 ± 0.0000.000 ± 0.000
(1M dataset)20.940 ± 0.0790.000 ± 0.000
30.980 ± 0.0140.000 ± 0.000
40.968 ± 0.0230.000 ± 0.000
50.000 ± 0.0000.000 ± 0.000
Cube-Quadruple1 to 30.996 ± 0.009 (all)0.000 ± 0.000 (all)
(100M dataset)41.000 ± 0.0000.000 ± 0.000
50.000 ± 0.0000.000 ± 0.000

(Note: Tasks 2–5 on Double and 1–3 on Quadruple achieved identical scores across all seeds and are aggregated for concise presentation. Task 5s are unsolvable for all current baselines due to extreme exploration difficulty).

As shown in the table above, the empirical results reveal a stark contrast across all 15 tasks: QCFQL-PORL achieves positive final success on only a single task (Cube-Double Task 1), while collapsing to complete failure across all other 14 tasks. In contrast, QCFQL-CFP consistently achieves near-perfect success rates on all 12 solvable tasks across the Cube-Double, Cube-Triple, and Cube-Quadruple suites (with Task 5 being unsolvable for all methods due to extreme exploration difficulty).

Consequently, while PORL's online-only workflow can be effective in dense-reward settings, CFP’s retention of offline data ensures the critic receives vital reward signals from static demonstrations, making it substantially more robust and effective in challenging sparse-reward environments.

To better showcase this comparative analysis and present the full training progression, we have incorporated clear, multi-domain learning curve figures alongside an in-depth discussion in Appendix E.3, under the subsection "Q: How does CFP compare to PORL?".


We believe these additions (specifically the newly added ablation plots and expanded discussions in Appendix C.1 & E.3) have substantially strengthened the empirical foundation and practical value of our work. Thank you again for your constructive review, which has helped us make our paper much more comprehensive. We remain at your disposal should you have any further questions!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their thoughtful, thorough, and constructive evaluations of our paper. We have carefully addressed each reviewer’s comments individually and revised both the main text and the Appendix accordingly. Below, we summarize the major updates made to the manuscript in order of appearance:

(1) Clarifying the CFP Workflow Implementation and the Relationship Between QCFQL and FQL (Sections 3–3.2 & Appendix D):

(2) Cross-Environment Transfer Experiment on Critic Plasticity (Section 3.1, Figure 1):

(3) Clearer Motivation for Theorem 1 and Refined Framing of Figure 4 (Section 4):

(4) Expanded Baseline Analysis Across Algorithmic Paradigms (Section 5):

(5) Controlled Workflow Comparisons: O2O Budget Sweep, O2O with Fresh Critic, and PORL (Section 5.1, Table 1; Appendix C.1 & E.3, Algorithms 3–4, Figure 23, Table 6):

(6) Generalization to QAM and Locomotion Tasks (Appendix D & E.3, Table 7 & Figure 25):

(7) Warm-Up Selection Protocol, Unification, and Local Robustness (Appendix E.1 & E.3, Table 5 & Figure 22):


In summary, we are deeply grateful to all three reviewers for their probing, constructive critiques. Their questions prompted us to investigate the offline-to-online critic mechanism far more rigorously, resulting in a much stronger chain of empirical evidence and comprehensive workflow comparisons. These additions not only verify the consistency of our core claims, but also underscore CFP’s advantages in both downstream performance and training efficiency—making our paper clearer, more rigorous, and substantially more solid. We hope that our revisions and responses have satisfactorily addressed your concerns. We look forward to answering any further questions you may have.

The revised paper has been uploaded, with all changes highlighted in blue. Our detailed responses to specific questions are provided below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.