- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether in-sample offline reinforcement learning can separate value learning from policy extraction through the Freeze-then-Extract protocol, which pretrains and freezes critics before training the policy. Experiments with IQL, XQL, and SQL on D4RL and OGBench show that extraction from a single frozen critic underperforms alternating updates on average. The authors identify temporal fluctuations in advantage estimates and concentrated extraction weights as potential contributors, hypothesizing that alternating updates provide implicit temporal ensembling. Motivated by this analysis, they investigate frozen ensembles constructed from multiple training checkpoints or independent initializations, selecting one member per actor update to provide extraction weights. These variants recover much of the performance loss. Additional experiments demonstrate improved actor-update efficiency for vision-language-action policies and the reuse of pretrained goal-conditioned critics to extract policies for held-out goals without further value learning.
Critical strengths
- The motivation for the research is clear. In-sample learning methods can be efficient when policy extraction is relatively expensive, and their pretrainability is also of great benefit.
- Observations and analyses are very well studied. Moreover, the proposed method is reasonably and naturally derived based on the observations.
- The proposed method is simple yet can be applied to various in-sample value learning frameworks, e.g., IQL, XQL, SQL.
Critical weaknesses
- The scope of the study is small, focusing solely on in-sample learning methods among various offline RL methods.
- It remains unclear why we should use in-sample learning methods in terms of task performance. In-sample learning methods are known to perform worse than offline RL methods with policy extraction for value learning, e.g., FQL.
- It is unclear how minimal the added ensemble computation is when compared to the case when we sample actions for value training. The proposed method adds more computation for better task performance, and this can also be achieved through non-in-sample learning methods.
- Writing can be improved by introducing IQL, XQL, and SQL within the Preliminaries section. Further minor details include: (1) At first glance, it was hard to understand what Jaccard overlap is without explaining what A and B are. (2) The title should be modified since the answer to the current title will always be yes when not considering task performance.
Questions for authors
- Q.1. I am curious how minimal the added computation is, considering the improvement in task performance. The task performance of all three methods, IQL, XQL, and SQL, using the proposed method is still far behind the reported results of Flow Q-Learning (FQL). However, FQL maintains a very efficient value learning mechanism through one-step action sampling. What are the training runtimes and floating-point operations (FLOPs) of IQL, XQL, and SQL with the proposed method? Please compare with FQL and the three original in-sampling learning approaches without the proposed method.
- Q.2. Prior work already uses ensembles per critic network by default. The difference is that the authors form ensembles either using critics from different training steps (Checkpoint-based variant) or critics trained with different random initialization (Independent-run variant). What if we just increase the ensemble size, e.g., from 2 to 32, of the networks used in prior work? Please provide quantitative results on whether this still leads to freeze-then-extract failure. If so, further compare results with the proposed two variants on OGBench humanoidmaze medium.
- Q.3. What other possible directions do the authors see in solving the observed problems of Freeze-then-Extract in-sample learning?
5 comments
Thank you for the response.
I value this work highly for its problem formulation, analysis, and a simple yet effective solution.
However, I still lean towards a weak rejection since the task performance gap with state-of-the-art methods remains unclear.
In my opinion, efficiency matters only when the final task performance is similar or negligibly lower. I think incorporating the proposed method into recent well-performing in-sample value learning algorithms, and checking whether it can effectively solve challenging online RL tasks with much higher training efficiency (due to critic pretraining), would strengthen the paper.
Therefore, I think evaluations on (1) more challenging tasks such as OGBench cube-quadruple, and (2) combining it with more recent well-performing algorithms such as Decoupled Q-Chunking (DQC) are needed.
We thank the reviewer for the follow-up and for valuing our problem formulation and analysis.
We agree that final task performance is an important consideration when assessing efficiency, alongside computational cost. Our goal, however, is not to establish state-of-the-art scores on OGBench, but to understand and mitigate frozen-critic degradation and investigate the potential of critic-first training for improving VLA training efficiency. Our VLA experiments demonstrate improved actor-update efficiency while achieving comparable or better final performance than alternating training across four task settings.
Additional VLA experiments and end-to-end efficiency.
We have expanded our VLA evaluation to a different model and simulation environment, in MolmoSpaces, with two additional language-conditioned tasks: Household Picking and Kitchenware Picking (Section 6.2, Fig. 8(e–h)). Together with GR00T N1.5 in RoboCasa, our evaluation now covers two VLA models, two simulation environments, and four task settings.
On both new tasks, Multi-head Freeze-then-Extract exceeds the final mean performance of alternating training after only 5k actor updates, compared with 30k and 20k for Alternating, respectively. Importantly, these gains translate into substantial end-to-end efficiency improvements, reducing the time to reach the final alternating performance by 74.4% and 60.7%, respectively. Unlike the projected GR00T runtimes, these times are reconstructed from completed training-job records and include feature-cache construction, critic pretraining, advantage caching, and actor training. Moreover, Multi-head achieves higher final mean success rates: 70.17 ± 4.32% vs. 60.67 ± 1.86% on Household Picking and 50.89 ± 2.73% vs. 41.11 ± 1.74% on Kitchenware Picking (mean ± SEM over three seeds). We have incorporated these results into Section 6.2, Fig. 8(e–h), and Appendices D.1–D.2 and G.4.2.
Relation to DQC and recent in-sample methods.
DQC [4] also employs in-sample value learning. However, its reported implementation performs policy extraction through test-time Best-of-N sampling rather than training a critic-guided actor. This requires multiple policy samples and critic evaluations at inference time, potentially introducing substantial computational overhead when deploying large VLAs on physical robots. Moreover, DQC does not report experiments with VLA policies. Our focus is instead on efficiently training a deployable VLA policy without requiring critic-guided action selection at inference time.
We have already evaluated our approach with DEAS [5], a recent, competitive in-sample offline RL algorithm with demonstrated effectiveness on VLAs. An independent comparison in Chunk-Guided Q-Learning (CGQ) [6] reports higher average OGBench manipulation performance for DEAS than DQC (71% vs. 46%) under its evaluated setting. Thus, our experiments already establish the applicability of critic-first training to a competitive algorithm with demonstrated VLA performance.
We acknowledge that evaluating Freeze-then-Extract on cube-quadruple would broaden the empirical scope. However, our central question concerns the effect of separating critic learning and actor training, rather than achieving the highest benchmark score through different policy-extraction mechanisms. Our contribution is to identify and mitigate frozen-critic degradation and demonstrate practical efficiency gains with a competitive, VLA-validated offline RL method.
[4] Li et al., "Decoupled Q-Chunking," ICLR 2026.
[5] Kim et al., "DEAS: DEtached Value Learning with Action Sequence for Scalable Offline RL," ICLR 2026.
[6] Song et al., "Chunk-Guided Q-Learning," ICML 2026.
Author Response
We thank the reviewer for recognizing the motivation, detailed analysis, and applicability of our approach.
W1 & W2. Scope of In-sample Offline RL and Comparison with FQL
Our focus on in-sample offline RL is intentional. Rather than proposing a higher-performing offline RL algorithm, we investigate whether policy-independent critic learning in IQL, XQL, and SQL enables critic-first training. This separation is particularly valuable for expensive policies such as VLAs, allowing critics to be pretrained and reused without policy queries.
Contrary to our expectations, naive Freeze-then-Extract degrades performance. Identifying and analyzing this non-obvious failure is a central contribution.
We acknowledge FQL's strong benchmark performance. However, FQL samples actions from its learned policy during value learning, unlike in-sample methods. Thus, it does not directly address our research question. Moreover, FQL's authors describe applying FQL to pretrained VLA flow BC models as future work [2]. A key challenge of applying FQL to large VLAs is computational cost. Its critic learning relies on a one-step policy distilled from a multi-step BC flow model, requiring expensive multi-step inference. In contrast, in-sample offline RL does not require such action sampling, which makes it suitable for VLA models. If the reviewer knows of studies demonstrating successful FQL-based offline RL with pretrained VLA policies, we would appreciate references.
[2] Park et al., Flow Q-Learning, ICML 2025, OpenReview author discussion: https://openreview.net/forum?id=KVf2SFL1pi¬eId=lgmvMGy83D
W3 & Q1. Ensemble Computation and Efficiency
We acknowledge the additional critic-training cost of Independent Ensemble. However, Checkpoint Ensemble requires no additional critic training, while our VLA experiments use a lightweight shared-backbone Multi-head Ensemble. Increasing from one to five heads adds only 1.37% to total parameters and 17.65% to trainable parameters (Appendix D.4). Including critic pretraining and caching, projected end-to-end training time decreases from 20.10 to 17.49 hours on Coffee and from 77.77 to 66.75 hours on Stove (Appendix D.2). Critic reuse provides further benefits: across five AWR temperatures, projected total training time decreases by approximately 57% (Appendix D.3). We have not directly compared FLOPs or runtime against FQL and do not claim superior computational efficiency over it. Our focus is the feasibility of critic-first training within in-sample offline RL.
Q2. Increasing the Number of Q Networks
We first clarify the distinction between conventional Clipped Double Q-learning and our Independent Ensemble. In IQL, two Q-functions are combined using their minimum to mitigate overestimation. Simply extending this approach to 32 Q-functions can lead to excessively pessimistic value estimates [3]. In contrast, our Independent Ensemble uses multiple independently trained (Q1, Q2, V) critics and exposes policy extraction to diverse advantage signals, rather than taking the minimum across all critics.
To address Q2, we additionally evaluated 32 Q-functions with minimum aggregation on OGBench HumanoidMaze Medium. Neither training protocol learned meaningful policies: Alternating achieved 0.20 ± 0.20 and Freeze-then-Extract achieved 0.80 ± 0.37.
We do not claim algorithmic novelty in ensembling itself. Checkpoint Ensemble directly tests our temporal-ensembling hypothesis, while Independent Ensemble provides a practical alternative without checkpoint-schedule tuning. Our main contributions are identifying and analyzing frozen-critic degradation and investigating ways to mitigate it.
[3] An et al., "Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble," NeurIPS 2021.
Q3. Future Research Directions
As discussed in Section 7, we do not consider critic ensembling a fundamental solution. Our analysis shows that advantage estimates and action rankings can fluctuate even late in critic training, making extraction from a single checkpoint fragile. A more fundamental direction is to stabilize critic learning itself, for example by developing value-learning objectives that encourage consistent advantage estimates or action rankings across training checkpoints. Investigating hyperparameters specifically suited to frozen-critic extraction is another promising direction.
W4. Presentation and Clarity
We thank the reviewer for these suggestions. We have expanded the Preliminaries to introduce the value-learning objectives of IQL, XQL, and SQL, and clarified the Jaccard overlap by explicitly defining the sets of high-weight samples. Regarding the title, our question concerns whether separating critic learning and policy extraction can preserve effective policy learning, rather than whether the separation is merely technically possible.