- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies offline-to-online (O2O) fine-tuning of flow-based actor-critic methods. It argues that prolonged offline critic training impairs online fine-tuning through loss of plasticity and value mismatch, whereas the actor mainly benefits from offline data coverage. It proposes Critic-Free Pretraining (CFP): pretrain the actor with behavior cloning only, then initialize a fresh critic, run a short offline warm-up, and fine-tune online. A theorem bounds downstream performance by the behavior data’s coverage and the imitation error. Experiments with four FQL/QC-family algorithms on OGBench and Robomimic show CFP comparable or superior to standard O2O, with lower offline compute.
Critical strengths
It is a simple and efficient method. CFP is a workflow-level change that applies to several flow-based algorithms without modifying their objectives, and it reduces offline training time.
A clear empirical phenomenon on hard tasks, with useful ablations. The gains are large on some difficult Cube tasks.
Critical weaknesses
Novelty is limited, and the claimed mechanism is not isolated.
- PORL (Xiao et al., 2025) already proposes policy-only pretraining with a freshly initialized critic. The differences are Gaussian vs. flow policies and CFP’s short offline warm-up. PORL is discussed only in the appendix related work, and the main text cites it only as a pessimism reference.
- The novelty therefore mainly rests on the analysis, which does not establish its causal claim. Relative to O2O, CFP changes both the actor’s offline objective (BC-only instead of Q-guided) and the critic (fresh with a short warm-up instead of long-trained), so the gains cannot be attributed to the critic alone. Resetting the critic of a standard O2O agent before the same warm-up would isolate this effect.
- Theorem 1 contains no critic and holds for any BC-pretrained actor, so it does not explain why critic pretraining hurts online fine-tuning.
The experiments are narrow and the comparisons incomplete.
- The functionally closest baseline is missing: PORL (flow version). All four base methods are from the FQL/QC family.
- All 28 tasks are sparse-reward, low-dimensional manipulation, and the main text shows only one task per domain.
- The gains concentrate on a few Cube tasks and algorithms. Robomimic is roughly a tie, and CFP does not help all four algorithms consistently.
- CFP adds a per-task hyperparameter (warm-up steps searched over 6 values) that O2O does not have. So the efficiency claim should account for tuning cost.
Questions for authors
See the weaknesses
10 comments
Response 2
2. Comparison with PORL.
Reviewer's concern: PORL (Xiao et al., 2025) already proposes policy-only pretraining with a freshly initialized critic. The differences are Gaussian vs. flow policies and CFP’s short offline warm-up. PORL is discussed only in the appendix related work, and the main text cites it only as a pessimism reference.
We agree that a direct empirical comparison with a flow-based implementation of PORL was a missing piece necessary to contextualize our contribution.
Original Submission vs. Revisions: In our original submission (Appendix C.1, Lines 864–870), we conceptually contrasted CFP with PORL:
"Xiao et al. (2025)... discover the phenomenon that the pretrained critic could potentially hinder online fine-tuning. Their work focuses primarily on the specific case of the Gaussian algorithm; PORL studies online fine-tuning from a pretrained policy without retaining a pretrained critic and emphasizes persistent value underestimation... This conclusion differs from the finding of reduced plasticity associated with CFP."
However, we fully agree with the reviewer that a conceptual mention was insufficient and that an explicit, side-by-side comparison of the workflows is essential. To make this distinction fully transparent, we have added explicit comparisons in both the main text and Appendix:
-
In the Main Text (Section 5, Table 1): We added a flow-based PORL baseline (using the exact same QCFQL backbone and BC-pretrained flow actor, with 20k online warm-up). On Cube Triple, PORL records 0.000 final success across all five tasks, whereas CFP reaches 1.000, 0.940, 0.980, and 0.968 on Tasks 1–4.
-
In Appendix C.1: We added formal side-by-side pseudocode (Algorithm 3 vs. Algorithm 4) detailing the algorithmic mechanics. In summary, both CFP and PORL share the core intuition of avoiding a fully pretrained offline critic to prevent negative transfer, choosing instead to preserve the pretrained actor while initializing a fresh critic. The fundamental distinction lies in the data source for critic calibration: PORL discards the offline dataset to warm up the critic exclusively on online interactions collected by the frozen actor, whereas CFP conducts a brief warm-up directly on the offline dataset and retains these offline transitions in the replay buffer.
-
In Appendix E.3: We expanded the evaluation to all 15 tasks across Cube Double, Triple, and Quadruple:
Task Suite Task ID QCFQL-CFP Online SR QCFQL-PORL Online SR Cube-Double 1 0.996 ± 0.009 0.992 ± 0.011 (1M dataset) 2 to 5 0.996 ± 0.009 (all) 0.000 ± 0.000 (all) Cube-Triple 1 1.000 ± 0.000 0.000 ± 0.000 (1M dataset) 2 0.940 ± 0.079 0.000 ± 0.000 3 0.980 ± 0.014 0.000 ± 0.000 4 0.968 ± 0.023 0.000 ± 0.000 5 0.000 ± 0.000 0.000 ± 0.000 Cube-Quadruple 1 to 3 0.996 ± 0.009 (all) 0.000 ± 0.000 (all) (100M dataset) 4 1.000 ± 0.000 0.000 ± 0.000 5 0.000 ± 0.000 0.000 ± 0.000
With the same BC-pretrained actor, our flow-based PORL implementation achieves nonzero final success on one of the 15 Cube tasks. CFP achieves nonzero success on 13 of the 15 tasks. One possible explanation is that the frozen actor collects few successful trajectories during online warm-up on the harder tasks, whereas CFP can train its critic on the broader set of transitions already present in the offline dataset. The comparison captures the combined effect of offline warm-up and continued access to offline replay.
To better showcase this comparative analysis and present the full training progression, we have incorporated clear, multi-domain learning curve figures alongside an in-depth discussion in Appendix E.3, under the subsection "Q: How does CFP compare to PORL?".
Response 3
3. What Theorem 1 Establishes.
Reviewer's concern: Theorem 1 contains no critic and holds for any BC-pretrained actor, so it does not explain why critic pretraining hurts online fine-tuning.
The reviewer is completely correct: Theorem 1 is strictly an actor-side guarantee and is not intended to explain the critic's failure mode.
In our original submission (Lines 270–274 and Lines 316–320), we introduced this theorem to address a specific skepticism regarding critic-free actor pretraining:
"CFP uses supervised actor pretraining to acquire a behavioral prior before value-guided improvement. We analyze when this prior is sufficient to support downstream learning... Thus, the theorem provides a conditional sufficiency statement for critic-free actor pretraining."
- The exact role of Theorem 1: When we propose removing the critic during actor pretraining, a natural concern arises: *does omitting value guidance offline cripple the actor's downstream capabilities? Theorem 1 characterizes how offline data coverage, the pretrained actor’s imitation error, and downstream local regret enter the performance bound. Its role is to identify conditions under which a BC-pretrained actor can support subsequent policy improvement. We study the effect of critic pretraining separately through empirical comparisons.
- The critic's failure: The negative impact of prolonged critic pretraining is a representational phenomenon (loss of plasticity and critic mismatch), which we investigate empirically via gradient norms, parameter shifts (Figure 1), Q-mean tracking (Figure 2), and the newly added "O2O + fresh critic" ablation.
We fully agree with reviewer's concerns regarding Theorem 1. We have further clarified the role of Theorem 1 in the main text Section 4 to address any potential doubts:
Theorem 1 explains how offline data coverage and the actor’s imitation error enter the downstream performance bound.
Response 1
We sincerely thank the reviewer for the thorough, rigorous, and penetrating evaluation of our work. We summarize your comment as follows: whether the gains can be attributed to the critic alone, how CFP genuinely differs from PORL, the precise scope of Theorem 1, and considerations regarding the generalizability and practical utility of the CFP experiments.
Our detailed responses are provided below:
1. Mechanism Isolation
We are profoundly grateful to the reviewer for suggesting this crucial experiment.
Experiment: Following your exact recommendation, we took the actor checkpoint from a standard 1M-step O2O run, completely reset its critic, and applied the identical 10k offline warm-up before online fine-tuning. We first evaluated this workflow on Cube Triple across 5 seeds with QCFQL as the backbone:
Table: QCFQL Success Rates on Cube Triple after 1M Online Interactions
The above table shows that resetting the critic of a standard O2O model immediately boosts its performance from 0.549 to 0.786. This rigorously isolates the mechanism and directly validates your hypothesis and our core claim: the primary performance bottleneck in standard O2O is indeed the critic's loss of plasticity from prolonged offline training. The experiments above confirm the general applicability of the CFP concept and its underlying mechanism; by eliminating the time-consuming and costly long-time offline training of a critic, we are actually able to achieve superior results during online fine-tuning.
To verify the generality of this mechanism, we used different algorithm backbones on Cube Double and Cube Triple:
Table: O2O w/ wo a fresh critic after 1M online interactions with different backbones
The Practical Value of CFP: Given that "O2O + fresh critic" matches CFP, why is CFP still preferable?
Computation: O2O spends massive compute optimizing the critic and backpropagating Q-gradients during the 1M-step pretraining, only to discard that critic. CFP eliminates critic computation entirely during actor pretraining. As we mentioned in our original manuscript under "Q: How efficient is CFP for offline training:"
Multi-task Reusability: In domain benchmarks like OGBench, tasks share the same offline state-action dataset but have different rewards. Standard O2O requires a separate, expensive 1M-step joint training run for every single task. In contrast, CFP’s actor pretraining is reward-free, meaning a single pretrained actor can be reused to initialize all tasks in that domain, requiring only the brief 10k warm-up to adapt to specific tasks.
We have incorporated these new experiments directly into Section 5 of the main text under the dedicated Q&A: "Q: How does CFP compare with other offline-to-online workflows?" and summarized it in the newly added Table 1. We also listed the full results of different backbones in Appendix E.3 under "Q: How does CFP compare to other workflows?".