acceptodds
Under review as a conference paper at ICLR 2027

Can In-Sample Offline RL Learn Critics First and Extract Policies Later?

Abstract

Recent progress in robotics and foundation models is driving offline reinforcement learning toward increasingly expressive but computationally expensive policy classes, including diffusion-based, flow-based, and large VLA policies. In-sample value-learning methods such as IQL and XQL are appealing in this setting because they estimate value functions without sampling actions from the current policy. This suggests a simple two-stage strategy: first perform value learning, then freeze the learned critic and extract a policy afterward, which we refer to as the *Freeze-then-Extract* protocol. However, existing in-sample value-learning methods typically still rely on alternating policy and critic updates in practice. Motivated by this gap, we systematically evaluate Freeze-then-Extract across offline RL methods and benchmarks and find that, despite its appeal, it often underperforms standard alternating updates. To understand this degradation, we investigate how the learned critic evolves throughout training. We find that advantage estimates vary substantially across training steps, even toward the end of critic training. When policy extraction relies on a single frozen critic, these fluctuations can lead to transient high-advantage peaks and a reduction in the effective number of training samples. We therefore investigate whether aggregating signals from multiple frozen critics, obtained from different checkpoints or independent initializations, can stabilize policy extraction while preserving policy-sampling-free value estimation. We find that such ensembles improve average extraction performance. In VLA experiments, critic-first training reaches strong performance with fewer expensive policy updates. Our results identify temporal variability in advantage estimates as a key obstacle to reusable frozen critics in scalable offline RL.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, vision-language-action models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u4PnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether in-sample offline reinforcement learning can separate value learning from policy extraction through the Freeze-then-Extract protocol, which pretrains and freezes critics before training the policy. Experiments with IQL, XQL, and SQL on D4RL and OGBench show that extraction from a single frozen critic underperforms alternating updates on average. The authors identify temporal fluctuations in advantage estimates and concentrated extraction weights as potential contributors, hypothesizing that alternating updates provide implicit temporal ensembling. Motivated by this analysis, they investigate frozen ensembles constructed from multiple training checkpoints or independent initializations, selecting one member per actor update to provide extraction weights. These variants recover much of the performance loss. Additional experiments demonstrate improved actor-update efficiency for vision-language-action policies and the reuse of pretrained goal-conditioned critics to extract policies for held-out goals without further value learning.

Critical strengths
  • The motivation for the research is clear. In-sample learning methods can be efficient when policy extraction is relatively expensive, and their pretrainability is also of great benefit.
  • Observations and analyses are very well studied. Moreover, the proposed method is reasonably and naturally derived based on the observations.
  • The proposed method is simple yet can be applied to various in-sample value learning frameworks, e.g., IQL, XQL, SQL.
Critical weaknesses
  • The scope of the study is small, focusing solely on in-sample learning methods among various offline RL methods.
  • It remains unclear why we should use in-sample learning methods in terms of task performance. In-sample learning methods are known to perform worse than offline RL methods with policy extraction for value learning, e.g., FQL.
  • It is unclear how minimal the added ensemble computation is when compared to the case when we sample actions for value training. The proposed method adds more computation for better task performance, and this can also be achieved through non-in-sample learning methods.
  • Writing can be improved by introducing IQL, XQL, and SQL within the Preliminaries section. Further minor details include: (1) At first glance, it was hard to understand what Jaccard overlap is without explaining what A and B are. (2) The title should be modified since the answer to the current title will always be yes when not considering task performance.
Questions for authors
  • Q.1. I am curious how minimal the added computation is, considering the improvement in task performance. The task performance of all three methods, IQL, XQL, and SQL, using the proposed method is still far behind the reported results of Flow Q-Learning (FQL). However, FQL maintains a very efficient value learning mechanism through one-step action sampling. What are the training runtimes and floating-point operations (FLOPs) of IQL, XQL, and SQL with the proposed method? Please compare with FQL and the three original in-sampling learning approaches without the proposed method.
  • Q.2. Prior work already uses ensembles per critic network by default. The difference is that the authors form ensembles either using critics from different training steps (Checkpoint-based variant) or critics trained with different random initialization (Independent-run variant). What if we just increase the ensemble size, e.g., from 2 to 32, of the networks used in prior work? Please provide quantitative results on whether this still leads to freeze-then-extract failure. If so, further compare results with the proposed two variants on OGBench humanoidmaze medium.
  • Q.3. What other possible directions do the authors see in solving the observed problems of Freeze-then-Extract in-sample learning?
5 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for recognizing the motivation, detailed analysis, and applicability of our approach.

W1 & W2. Scope of In-sample Offline RL and Comparison with FQL

Our focus on in-sample offline RL is intentional. Rather than proposing a higher-performing offline RL algorithm, we investigate whether policy-independent critic learning in IQL, XQL, and SQL enables critic-first training. This separation is particularly valuable for expensive policies such as VLAs, allowing critics to be pretrained and reused without policy queries.

Contrary to our expectations, naive Freeze-then-Extract degrades performance. Identifying and analyzing this non-obvious failure is a central contribution.

We acknowledge FQL's strong benchmark performance. However, FQL samples actions from its learned policy during value learning, unlike in-sample methods. Thus, it does not directly address our research question. Moreover, FQL's authors describe applying FQL to pretrained VLA flow BC models as future work [2]. A key challenge of applying FQL to large VLAs is computational cost. Its critic learning relies on a one-step policy distilled from a multi-step BC flow model, requiring expensive multi-step inference. In contrast, in-sample offline RL does not require such action sampling, which makes it suitable for VLA models. If the reviewer knows of studies demonstrating successful FQL-based offline RL with pretrained VLA policies, we would appreciate references.

[2] Park et al., Flow Q-Learning, ICML 2025, OpenReview author discussion: https://openreview.net/forum?id=KVf2SFL1pi&noteId=lgmvMGy83D

W3 & Q1. Ensemble Computation and Efficiency

We acknowledge the additional critic-training cost of Independent Ensemble. However, Checkpoint Ensemble requires no additional critic training, while our VLA experiments use a lightweight shared-backbone Multi-head Ensemble. Increasing from one to five heads adds only 1.37% to total parameters and 17.65% to trainable parameters (Appendix D.4). Including critic pretraining and caching, projected end-to-end training time decreases from 20.10 to 17.49 hours on Coffee and from 77.77 to 66.75 hours on Stove (Appendix D.2). Critic reuse provides further benefits: across five AWR temperatures, projected total training time decreases by approximately 57% (Appendix D.3). We have not directly compared FLOPs or runtime against FQL and do not claim superior computational efficiency over it. Our focus is the feasibility of critic-first training within in-sample offline RL.

Q2. Increasing the Number of Q Networks

We first clarify the distinction between conventional Clipped Double Q-learning and our Independent Ensemble. In IQL, two Q-functions are combined using their minimum to mitigate overestimation. Simply extending this approach to 32 Q-functions can lead to excessively pessimistic value estimates [3]. In contrast, our Independent Ensemble uses multiple independently trained (Q1, Q2, V) critics and exposes policy extraction to diverse advantage signals, rather than taking the minimum across all critics.

To address Q2, we additionally evaluated 32 Q-functions with minimum aggregation on OGBench HumanoidMaze Medium. Neither training protocol learned meaningful policies: Alternating achieved 0.20 ± 0.20 and Freeze-then-Extract achieved 0.80 ± 0.37.

We do not claim algorithmic novelty in ensembling itself. Checkpoint Ensemble directly tests our temporal-ensembling hypothesis, while Independent Ensemble provides a practical alternative without checkpoint-schedule tuning. Our main contributions are identifying and analyzing frozen-critic degradation and investigating ways to mitigate it.

[3] An et al., "Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble," NeurIPS 2021.

Q3. Future Research Directions

As discussed in Section 7, we do not consider critic ensembling a fundamental solution. Our analysis shows that advantage estimates and action rankings can fluctuate even late in critic training, making extraction from a single checkpoint fragile. A more fundamental direction is to stabilize critic learning itself, for example by developing value-learning objectives that encourage consistent advantage estimates or action rankings across training checkpoints. Investigating hyperparameters specifically suited to frozen-critic extraction is another promising direction.

W4. Presentation and Clarity

We thank the reviewer for these suggestions. We have expanded the Preliminaries to introduce the value-learning objectives of IQL, XQL, and SQL, and clarified the Jaccard overlap by explicitly defining the sets of high-weight samples. Regarding the title, our question concerns whether separating critic learning and policy extraction can preserve effective policy learning, rather than whether the separation is merely technically possible.

Thank you for the response.
I value this work highly for its problem formulation, analysis, and a simple yet effective solution.

However, I still lean towards a weak rejection since the task performance gap with state-of-the-art methods remains unclear.
In my opinion, efficiency matters only when the final task performance is similar or negligibly lower. I think incorporating the proposed method into recent well-performing in-sample value learning algorithms, and checking whether it can effectively solve challenging online RL tasks with much higher training efficiency (due to critic pretraining), would strengthen the paper.

Therefore, I think evaluations on (1) more challenging tasks such as OGBench cube-quadruple, and (2) combining it with more recent well-performing algorithms such as Decoupled Q-Chunking (DQC) are needed.

We thank the reviewer for the follow-up and for valuing our problem formulation and analysis.

We agree that final task performance is an important consideration when assessing efficiency, alongside computational cost. Our goal, however, is not to establish state-of-the-art scores on OGBench, but to understand and mitigate frozen-critic degradation and investigate the potential of critic-first training for improving VLA training efficiency. Our VLA experiments demonstrate improved actor-update efficiency while achieving comparable or better final performance than alternating training across four task settings.

Additional VLA experiments and end-to-end efficiency.

We have expanded our VLA evaluation to a different model and simulation environment, π0.5\pi_{0.5} in MolmoSpaces, with two additional language-conditioned tasks: Household Picking and Kitchenware Picking (Section 6.2, Fig. 8(e–h)). Together with GR00T N1.5 in RoboCasa, our evaluation now covers two VLA models, two simulation environments, and four task settings.

On both new tasks, Multi-head Freeze-then-Extract exceeds the final mean performance of alternating training after only 5k actor updates, compared with 30k and 20k for Alternating, respectively. Importantly, these gains translate into substantial end-to-end efficiency improvements, reducing the time to reach the final alternating performance by 74.4% and 60.7%, respectively. Unlike the projected GR00T runtimes, these times are reconstructed from completed training-job records and include feature-cache construction, critic pretraining, advantage caching, and actor training. Moreover, Multi-head achieves higher final mean success rates: 70.17 ± 4.32% vs. 60.67 ± 1.86% on Household Picking and 50.89 ± 2.73% vs. 41.11 ± 1.74% on Kitchenware Picking (mean ± SEM over three seeds). We have incorporated these results into Section 6.2, Fig. 8(e–h), and Appendices D.1–D.2 and G.4.2.

Relation to DQC and recent in-sample methods.

DQC [4] also employs in-sample value learning. However, its reported implementation performs policy extraction through test-time Best-of-N sampling rather than training a critic-guided actor. This requires multiple policy samples and critic evaluations at inference time, potentially introducing substantial computational overhead when deploying large VLAs on physical robots. Moreover, DQC does not report experiments with VLA policies. Our focus is instead on efficiently training a deployable VLA policy without requiring critic-guided action selection at inference time.

We have already evaluated our approach with DEAS [5], a recent, competitive in-sample offline RL algorithm with demonstrated effectiveness on VLAs. An independent comparison in Chunk-Guided Q-Learning (CGQ) [6] reports higher average OGBench manipulation performance for DEAS than DQC (71% vs. 46%) under its evaluated setting. Thus, our experiments already establish the applicability of critic-first training to a competitive algorithm with demonstrated VLA performance.

We acknowledge that evaluating Freeze-then-Extract on cube-quadruple would broaden the empirical scope. However, our central question concerns the effect of separating critic learning and actor training, rather than achieving the highest benchmark score through different policy-extraction mechanisms. Our contribution is to identify and mitigate frozen-critic degradation and demonstrate practical efficiency gains with a competitive, VLA-validated offline RL method.

[4] Li et al., "Decoupled Q-Chunking," ICLR 2026.

[5] Kim et al., "DEAS: DEtached Value Learning with Action Sequence for Scalable Offline RL," ICLR 2026.

[6] Song et al., "Chunk-Guided Q-Learning," ICML 2026.

Reviewer_EnKhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether offline RL critics can be trained first and then frozen for policy extraction. Single-critic extraction often falls short of alternating updates, while critic ensembles help close the gap. The question is practical and the experiments are useful, but checkpoint selection, training-cost savings, and final performance remain open concerns.

Critical strengths
  • Systematic comparisons across three algorithms on D4RL and OGBench provide useful evidence about the limitations of frozen-critic policy extraction.

  • Simple ensemble approaches mitigate these limitations, with promising extensions to VLA training and critic reuse.

Critical weaknesses
  • The practical value of checkpoint tuning remains unclear. Appendix C shows that performance depends on the checkpoint schedule, while Checkpoint (Tuned) selects the best schedule for each algorithm–task pair using downstream scores. Although the paper also reports a fixed schedule and independent ensembles that avoid temporal checkpoint selection, the additional gains from tuning lack a practical selection procedure that does not require repeated policy training and environment evaluation.

  • Evidence for general VLA training savings remains limited. The experiments cover only two task settings. Actor-update savings are substantial on TurnOffStove, but on CoffeeSetupMug the ensemble reaches the final alternating baseline’s mean performance only one evaluation interval earlier (25k versus 30k updates). The runtime analysis appropriately includes critic pretraining and cache construction, but extrapolates from short timing benchmarks rather than measuring complete runs. Broader task coverage and comparisons across multiple performance thresholds would strengthen the efficiency claim.

  • Performance at convergence is not established. On CoffeeSetupMug, alternating training continues to improve near the budget limit and ends with a marginally higher mean than the ensemble, although the difference is small relative to the reported uncertainty. These results demonstrate an early-learning advantage but do not establish whether frozen ensembles preserve the performance achievable with longer alternating training. Extended runs with suitable learning-rate schedules would clarify this distinction.

Questions for authors
  • Can checkpoint schedules be selected without downstream environment evaluation, and what additional training and evaluation cost does selection incur?

  • Do the VLA efficiency gains persist across more tasks and performance thresholds when measured over complete training runs?

  • Do longer runs with suitable learning-rate schedules yield comparable converged performance, particularly on CoffeeSetupMug?

2 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the careful assessment and for recognizing the value of our systematic experiments and VLA applications.

W1 & Q1. Practicality of Checkpoint Selection

We agree that Checkpoint (Tuned) requires additional policy training and environment evaluation, although no additional critic training is needed. Such downstream performance-based hyperparameter tuning is common in RL (e.g., XQL-T [1]). Nevertheless, we do not present Checkpoint (Tuned) as an evaluation-free or computationally optimal selection method. Importantly, Checkpoint Ensemble primarily serves to investigate our hypothesis that alternating training benefits from implicit temporal ensembling. It allows us to examine whether combining critic signals from different training stages can mitigate frozen-extraction degradation. For practical use, we favor Independent Ensemble, which avoids checkpoint-schedule tuning. As shown in Appendix F.1, performance generally improves with ensemble size in the evaluated setting, suggesting that fine-grained tuning is less critical. Accordingly, our VLA experiments adopt a lightweight shared-backbone Multi-head Ensemble rather than Checkpoint Ensemble. Selecting useful critic checkpoints without downstream evaluation remains an important open problem, but we also demonstrate a practical alternative that does not require such selection.

[1] Garg et al., "Extreme Q-Learning: MaxEnt RL without Entropy," ICLR 2023.

W2 & Q2. VLA Efficiency and Runtime Evaluation

We agree that our VLA evaluation is limited to two tasks. However, the efficiency gains on CoffeeSetupMug are not limited to reaching the final Alternating performance one evaluation interval earlier. Comparing multiple success-rate thresholds using Appendix D.1 (Tables 18 and 19) gives:

Task / Success ThresholdAlternatingMulti-head
Coffee / 40%25k10k
Coffee / 47%30k25k
Stove / 35%50k15k
Stove / 40%Not reached35k

These results show improved actor-update efficiency at multiple performance levels on both tasks, rather than only at the final evaluation checkpoint. Furthermore, accounting for critic pretraining and advantage caching, projected end-to-end training time decreases from 20.10 to 17.49 hours on Coffee (13.0%) and from 77.77 to 66.75 hours on Stove (14.2%). Beyond a single training run, critic reuse provides substantial additional benefits for hyperparameter tuning. Reusing pretrained critics and cached advantages across five AWR temperature settings reduces projected total training time by 57.1% on Coffee and 57.3% on Stove (Appendix D.3). Regarding runtime estimation, we measured 2,000 training updates on the same NVIDIA GB200 GPU and extrapolated from the average step time after warm-up (Appendix D.2). Measuring complete runs under identical conditions is challenging on shared clusters due to resource contention and varying workloads. Short, controlled benchmarks provide a consistent basis for comparison, although we acknowledge that these are projections rather than direct full-run measurements. We agree that broader VLA evaluation would strengthen the evidence. However, constructing suitable offline RL datasets for VLA training is computationally expensive, and we independently collected 1,500 trajectories for Coffee and 9,000 for Stove.

W3 & Q3. Performance under Longer Training

We agree that our current experiments do not establish asymptotic performance under substantially longer training budgets. On CoffeeSetupMug, Alternating and Multi-head achieve comparable performance at 30k updates (47.12 ± 1.37% vs. 46.88 ± 1.31%). However, Multi-head already reaches 48.16 ± 2.24% at 25k updates. Our focus is on reaching a given level of policy performance with fewer expensive actor updates, rather than demonstrating superior asymptotic performance. This distinction is particularly important for large VLA policies, where each actor update is computationally costly. We acknowledge that longer training with appropriately adjusted learning-rate schedules could change the final performance comparison. Such experiments would be valuable, but the current results do not establish which protocol performs better at convergence. Our findings instead demonstrate the potential of critic-first training to improve efficiency under limited actor-update budgets.

We thank the reviewer for the constructive feedback. We have expanded our VLA evaluation to π0.5\pi_{0.5} in MolmoSpaces, with two additional language-conditioned tasks, Household Picking and Kitchenware Picking (Section 6.2, Fig. 8(e–h)). Together with GR00T N1.5 in RoboCasa, our evaluation now covers two VLA models, two simulation environments, and four task settings. We have updated the manuscript to include these results.

W2/Q2. Additional VLA experiments and end-to-end efficiency.

Unlike the projected runtimes in our GR00T experiments, the new π0.5\pi_{0.5} end-to-end times are reconstructed from completed training-job records, including feature-cache construction, data staging, training-job preparation, critic pretraining, advantage caching, and actor training. Data collection, scheduling delays, and evaluation are excluded.

Using the final mean success rate of Alternating as the reference, Multi-head Freeze-then-Extract exceeds this performance at the first evaluated checkpoint of only 5k actor updates on both new tasks:

TaskAlternating: updates / timeMulti-head: updates / timeTime-to-reference reduction
Household Picking30k / 34.41 h5k / 8.80 h74.4%
Kitchenware Picking20k / 25.14 h5k / 9.89 h60.7%

These reductions include the upfront critic-training and caching costs. Both protocols use the same precomputed visual-language features from a frozen encoder. Since the remaining trainable critic networks are substantially smaller than the VLA policy, critic preparation, training, and advantage caching take only about 13 minutes even with five critic heads, after feature-cache construction. Thus, the actor-update savings translate into end-to-end improvements.

W3/Q3. Final performance within the evaluated training budgets.

Importantly, the additional experiments also show higher final mean success rates than Alternating at the same actor-update budget:

TaskActor updatesAlternatingMulti-head
Household Picking30k60.67 ± 1.86%70.17 ± 4.32%
Kitchenware Picking20k41.11 ± 1.74%50.89 ± 2.73%

Results are mean ± SEM across three training seeds. These findings extend the evidence beyond early actor-update efficiency: on both additional tasks, Multi-head reaches stronger performance with substantially less end-to-end training time and achieves higher final mean success within the evaluated budgets.

We have incorporated the additional learning curves, runtime measurements, and experimental details into Section 6.2, Fig. 8(e–h), Appendix D.1–D.2 (Tables 20, 21, and 23), and Appendix G.4.2.

Reviewer_JiTKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether in-sample offline RL methods such as IQL, XQL, and SQL can train critics first and extract policies later from a frozen critic, which the authors call Freeze-then-Extract. Across D4RL and OGBench, this protocol underperforms standard alternating updates. The authors attribute this degradation to late-stage variability of advantage estimates and concentration of AWR weights, and hypothesize that alternating updates act as implicit temporal ensembling. Motivated by this, they extract policies from multiple frozen critics, obtained from different checkpoints or independent initializations, which recovers most of the degradation. They further show applications to VLA policies and goal-conditioned critic reuse.

Critical strengths

S1. Identifying a non-obvious challenge.

The paper raises a practical question that has been largely overlooked: whether in-sample offline RL can train critics first and extract policies later. It shows the counterintuitive result that this protocol degrades performance across three algorithms and two benchmarks, even though the critic is learned independently of the policy.

S2. Analysis and a simple remedy.

The authors analyze the cause of this degradation, attributing it to late-stage variability of advantage estimates and concentration of AWR weights, and propose a simple fix using multiple frozen critics. This remedy recovers most of the degradation and is supported by extensive controls, including DDPG+BC and flow-based extraction, inverse-temperature sweeps, and target-critic baselines.

Critical weaknesses

W1. The motivating problem is not validated.

The introduction argues that alternating updates are wasteful because early actor updates use immature value estimates and the policy is expensive to query. However, since in-sample critics never query the policy, each actor update costs the same under both protocols, so the second point is not specific to alternating training. More importantly, the paper never tests whether early updates with immature critics actually harm the final policy. If this were a real problem, Freeze-then-Extract, which performs all actor updates with a fully trained critic, should improve performance. The results show otherwise, as discussed below.

W2. Freeze-then-Extract underperforms alternating training.

Freeze-then-Extract underperforms alternating training across all three algorithms and both benchmarks (Fig. 2). Even with multiple frozen critics, it at best matches alternating training (e.g., IQL: 59.9 vs. 60.1; SQL: 48.9 vs. 52.3, Table 3), and applying the same ensemble to alternating training yields higher performance (66.0 vs. 58.3, App. F.4). This suggests that observing evolving critics is beneficial rather than wasteful, as the paper itself argues in Sec. 4.3, which contradicts the motivation in the introduction.

W3. The efficiency benefits are unclear.

The remaining benefits, fewer actor updates and reduced end-to-end training time, are supported by only two VLA tasks, and the gains are clear on only one of them. On CoffeeSetupMug, the multi-head ensemble reaches the final alternating performance at 25k versus 30k updates, the final success rates are comparable (46.9 vs. 47.1), and the end-to-end curves in Fig. 8(c) largely overlap, partly because no usable policy is available during the roughly 10-hour critic stage. The larger gain on TurnOffStove is measured against an alternating run whose performance drops below the base policy early and plateaus low, which makes the comparison sensitive. Moreover, all runtimes are extrapolated from short benchmarks (App. D.2). Experiments on more tasks, particularly where the efficiency gap is more pronounced, would better support these claims.

Questions for authors

Q1. Early actor updates with immature critics may be suboptimal, but their effect could be recovered by later updates as the critic improves. Could the authors show whether these early updates actually affect the final policy?

Q2. Could the authors provide a setting where Freeze-then-Extract, with or without ensembles, achieves higher performance than alternating training under the same ensemble configuration? Without such a case, it is difficult to see a performance benefit of the protocol.

Q3 Could the authors evaluate the efficiency benefits on more tasks with expensive policies, particularly ones where the alternating baseline trains stably? Showing a clear reduction in actor updates or end-to-end training time across these tasks would better support the claimed efficiency.

5 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for recognizing the importance of our research question and analysis. First, we do not claim that alternating training is generally inefficient or that Freeze-then-Extract is always superior. Critic-first training was motivated by potential gains in actor-update efficiency, yet naive freezing unexpectedly harms final performance. Identifying and analyzing this non-obvious degradation is a central contribution of our work.

W1 & Q1. Motivation and Early Actor Updates

We revised the Introduction from "can become wasteful" to "may not be effective" to frame early-update inefficiency as a hypothesis rather than a fact. We also changed "expensive to query" to "expensive to optimize" to clarify that we mean actor optimization, not policy sampling during critic learning. Regarding Q1, we do not claim that early updates irreversibly harm final performance; later updates may overcome their effects. However, recoverability does not imply efficiency under a limited actor-update budget. We measure actor updates needed to reach a target score. To investigate this, we added learning curves on all six D4RL AntMaze tasks and four locomotion tasks, limiting both critic and actor training to 250k updates each (Appendix B.2, Fig. 13). While naive frozen extraction exhibits unstable or degraded learning on several tasks, Independent Ensemble mitigates this issue and reaches higher performance with fewer actor updates on multiple tasks, particularly AntMaze. Together with the VLA results, this supports the potential of critic-first training to improve actor-update efficiency.

W2 & Q2. Final Performance Compared with Alternating Training

As mentioned above, we would like to clarify that our goal is to investigate why a naive implementation of Freeze-then-Extract often fails to achieve performance comparable to alternating training. Regarding W2, the degradation under Freeze-then-Extract is not a contradiction of our motivation, but the phenomenon we seek to explain. As discussed in Section 4.3, advantage estimates continue to fluctuate even late in critic training. We hypothesize that alternating training benefits from exposure to these evolving critic signals through implicit temporal ensembling. Regarding Q2, our goal is not higher final performance than alternating training, but actor-update efficiency and critic reuse. With a larger budget such as 1M updates, alternating training can perform many actor updates after the critic becomes informative, potentially obscuring differences in early actor-update efficiency. Our additional learning curves (Appendix B.2) show that adding Independent Ensemble to alternating training does not provide a comparable or consistent acceleration in early policy learning, whereas Frozen + Independent Ensemble often reaches higher performance with fewer actor updates. This suggests that the early gains are not simply due to ensembling, but also associated with critic pretraining.

W3 & Q3. Efficiency Benefits with VLA Policies

Regarding W3 and Q3, the unstable early performance of alternating training on TurnOffStove does not make the comparison less meaningful. When learning an informative critic takes longer, actor updates based on immature value estimates may be less effective. Such settings are especially relevant for evaluating critic-first training. On Stove, alternating training initially falls below the pretrained Base Policy, whereas frozen extraction avoids this drop. The observed instability is consistent with our motivation and represents an important test case. Moreover, gains are observed on CoffeeSetupMug, where alternating training improves relatively steadily: at 15k actor updates, Multi-head achieves 44.00% success versus 36.16% for Alternating. On Stove, Multi-head reaches 35.36% at 15k, while Alternating needs 50k to reach comparable performance. Both tasks show improved actor-update efficiency, with larger gains on Stove.

During critic pretraining, only the pretrained BC policy is available, and its performance does not improve during this stage. However, when considering the full training pipeline, projected end-to-end times, including critic pretraining and caching, decrease from 20.10 to 17.49 hours on Coffee and from 77.77 to 66.75 hours on Stove (Appendix D.2). Beyond the benefits of a single training run, critic reuse provides a substantial additional advantage. Reusing pretrained critics and cached advantages across five AWR temperatures reduces projected total training time by 57.1% on Coffee and 57.3% on Stove (Appendix D.3). We agree that broader VLA evaluation would strengthen the evidence. However, constructing suitable offline RL datasets for VLA training is costly, and we independently collected 1,500 trajectories for Coffee and 9,000 for Stove. These results provide initial evidence of actor-update efficiency and critic reuse benefits.

Thank you for the response and the revisions.

W1/W2. The clarified framing improves consistency: the degradation is presented as the phenomenon under study, and the paper no longer claims that early updates harm the final policy. However, this also narrows the contribution to actor-update efficiency and critic reuse, while the protocol still at best matches alternating training. I also note that Fig. 13 excludes critic pretraining from the x-axis, so the faster early improvement of the frozen variants is expected.

W3. This remains my main concern. The efficiency evidence is still limited to two VLA tasks with extrapolated runtimes, and the projected end-to-end reduction is modest.

Since the remaining claims rest on efficiency benefits that are not yet well established, I am keeping my score.

We thank the reviewer for clarifying the remaining concerns. We have expanded our VLA evaluation to a different model and simulation environment: π0.5\pi_{0.5} in MolmoSpaces, with two additional language-conditioned tasks, Household Picking and Kitchenware Picking (Section 6.2, Fig. 8(e–h)). Together with the existing GR00T N1.5 experiments in RoboCasa, the revised evaluation now covers two VLA models, two simulation environments, and four task settings.

W1/W2. Actor-update efficiency and final performance.

Fig. 13 measures actor-update efficiency. Our experiments show that critic-first extraction with Independent Ensemble often achieves higher performance with fewer actor updates. This is consistent with the hypothesis that early actor updates guided by immature critics may be less effective under limited actor-update budgets. The practical question is whether these gains persist after accounting for critic pretraining and other upfront computation. Our newly added VLA experiments demonstrate that these actor-update gains also translate into substantial end-to-end training-time reductions.

W3. Additional VLA results and end-to-end efficiency gains.

For the new π0.5\pi_{0.5} experiments, the end-to-end times are reconstructed from completed training-job records, rather than extrapolated from short timing benchmarks. The reported pipeline includes feature-cache construction, data staging, training-job preparation, critic pretraining, advantage caching, and actor training. Data collection, scheduling delays, and evaluation are excluded.

Using the final mean success rate of Alternating as the reference, our Multi-head Freeze-then-Extract approach exceeds this performance at the first evaluated checkpoint of only 5k actor updates on both new tasks:

TaskAlternating: updates / timeMulti-head: updates / timeEnd-to-end time reduction
Household Picking30k / 34.41 h5k / 8.80 h74.4%
Kitchenware Picking20k / 25.14 h5k / 9.89 h60.7%

These reductions include the upfront critic-training and caching costs. Both methods use the same cached features from a frozen copy of the pretrained visual-language encoder. Since the remaining trainable critic networks are substantially smaller than the VLA policy, critic training is inexpensive: even with five critic heads, critic preparation, training, and advantage caching take only about 13 minutes after feature-cache construction. Thus, the upfront cost of critic pretraining is small relative to policy optimization, allowing the actor-update gains to translate into end-to-end improvements.

The efficiency gains also accompany higher final mean task performance within the evaluated training budgets. At 30k actor updates on Household Picking, Multi-head achieves 70.17 ± 4.32%, compared with 60.67 ± 1.86% for Alternating. At 20k updates on Kitchenware Picking, it achieves 50.89 ± 2.73%, compared with 41.11 ± 1.74%. Results are mean ± SEM across three training seeds.

We have incorporated these additional learning curves, runtime results, and experimental details into Section 6.2, Fig. 8(e–h), Appendix D.1–D.2 (Tables 20, 21, and 23), and Appendix G.4.2. These additions strengthen the evidence for practical critic-first VLA training by demonstrating shorter end-to-end time to strong performance, together with higher final mean success, on a different VLA model and simulation environment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response to the Area Chair and Reviewers

We sincerely thank the Area Chair and all reviewers for their constructive feedback and continued discussion. We appreciate that all three reviewers recognized key strengths of our work, including the non-obvious degradation of frozen-critic extraction, its systematic analysis, and ensemble-based mitigation.

Much of the remaining discussion focused on the practical implications of these findings, particularly actor-update efficiency, end-to-end training time, final task performance, and computational costs. We have revised the manuscript and conducted additional experiments to address these concerns. We summarize the key revisions below.

We clarified that our goal is not to establish that Freeze-then-Extract always outperforms alternating training, but to understand why naive frozen-critic extraction often degrades performance, investigate how to mitigate this degradation, and explore the potential of critic-first training to improve actor-update efficiency for expensive policies such as VLAs.

1. Actor-update efficiency (Reviewer JiTK)

Reviewer JiTK raised concerns about whether critic-first training provides meaningful actor-update efficiency gains.

Both our original and newly added VLA experiments demonstrate improved actor-update efficiency, with Multi-head Freeze-then-Extract reaching the final mean performance of Alternating using fewer actor updates (Section 6.2, Fig. 8). We also added learning curves over 250k actor updates on six AntMaze and four locomotion tasks (Appendix B.2, Fig. 13), showing that Freeze-then-Extract with Independent Ensemble often achieves stronger early performance with fewer actor updates.

2. Broader VLA evaluation and end-to-end efficiency (Reviewers JiTK and EnKh)

Both reviewers expressed concerns about the limited VLA task coverage and the reliance on projected runtimes rather than complete training runs.

To address these concerns, we added experiments with π0.5\pi_{0.5} in MolmoSpaces on two language-conditioned tasks, Household Picking and Kitchenware Picking (Section 6.2, Fig. 8(e–h)). Together with GR00T N1.5 in RoboCasa, our evaluation now covers two VLA models, two simulation environments, and four task settings.

Unlike the projected GR00T runtimes, the π0.5\pi_{0.5} end-to-end times are based on actual wall-clock measurements from completed training jobs, rather than extrapolated estimates.

Multi-head Freeze-then-Extract exceeds the final mean success rate of Alternating after only 5k actor updates on both new tasks, reducing the end-to-end time required to reach this performance level by 74.4% on Household Picking and 60.7% on Kitchenware Picking.

3. Final task performance (Reviewers EnKh and u4Pn)

Reviewers EnKh and u4Pn questioned whether the efficiency gains could be achieved without sacrificing final task performance.

Multi-head achieves higher final mean success rates than Alternating at matched actor-update budgets on both additional tasks: 70.17% versus 60.67% on Household Picking and 50.89% versus 41.11% on Kitchenware Picking (Appendix D.1, Tables 20–21).

Nevertheless, to reiterate, our goal is not higher final performance than alternating training, but actor-update efficiency and critic reuse.

4. Computational costs and practical applicability (Reviewers EnKh and u4Pn)

Reviewer EnKh questioned the practicality of checkpoint tuning, while Reviewer u4Pn raised concerns about the additional computational cost of critic ensembles.

We clarified that checkpoint tuning is not required for our more practical Independent Ensemble variant. Building on this approach, we use Multi-head Ensemble, a lightweight shared-backbone variant of Independent Ensemble, in our VLA experiments.

For π0.5\pi_{0.5}, critic preparation, training, and advantage caching take only about 13 minutes, even with five critic heads, after feature-cache construction (Appendix D.2).

Furthermore, freezing enables fixed advantage caching and reuse across policy configurations, eliminating continued critic updates during actor training. In our GR00T experiments, reusing pretrained critics and cached advantages across five AWR temperatures reduces projected total training time by approximately 57% compared with independent alternating runs (Appendix D.3).

Summary

Overall, these revisions strengthen both the scientific and practical contributions of our work: identifying and analyzing the non-obvious degradation of frozen-critic extraction, investigating ensemble-based mitigation, and demonstrating substantial end-to-end efficiency gains with competitive final performance across different VLA models and simulation environments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.