- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper examines whether Q-value function pretraining is beneficial for online RL fine-tuning with a pretrained policy and proposes Initialization via Policy Ensemble (IPE) to improve state-action coverage for effective Q-value function. The empirical results show that pretraining the Q-value function through offline TD learning methods often provides little benefit over random initialization. The authors attribute this observation to a mismatch between the Q-value function of the pretrained policy and that of the policy obtained through online RL. Motivated by this, IPE trains multiple behavior-cloned policies on the same offline behavior distribution and collects their rollouts to initialize the replay buffer with more diverse transitions.
Critical strengths
S1 (Interesting observation). The paper investigates a question regarding the necessity of Q-value function pretraining for online RL fine-tuning. The experiments show that naive Q-value function pretraining provides little benefit over random initialization under both UTD = 1 (Figure 2) and UTD = 20 (Figure 10), offering useful empirical insights into fine-tuning pretrained policies.
S2 (Simple and intuitive approach). IPE provides a straightforward approach to improving initial state-action coverage by collecting rollouts from multiple independently trained behavior-cloned policies. The collected transitions are used to initialize the replay buffer for subsequent online RL fine-tuning with EXPO, without requiring additional Q-value function pretraining.
S3 (Empirical analysis). The paper provides several experiments examining Q-value function pretraining, action preference score, action diversity, and the number of policies used for IPE. These analyses help illustrate the motivation and empirical behavior of the proposed approach.
Critical weaknesses
W1 (Limited novelty and practical applicability). The proposed method primarily involves training multiple behavior-cloned (BC) policies on the same offline dataset and pooling their environment rollouts to initialize the replay buffer. While the paper shows that multiple BC policies can increase action coverage, this largely relies on differences arising from random initialization, rather than an explicit mechanism for encouraging diverse and useful behaviors. Consequently, it is unclear whether IPE can consistently improve state-action coverage, particularly when the offline dataset contains predominantly suboptimal behaviors. Moreover, independently trained BC policies may generate unsafe or undesirable behaviors during environment interaction, which could limit the applicability of the proposed method to real-world settings. Further discussion or illustrative experiments would help clarify the practical benefits and limitations of the proposed approach.
W2 (Insufficient comparisons with simpler data diversification strategies). The main motivation for the proposed method is that multiple BC policies can provide more diverse action coverage than a single pretrained policy. However, the paper does not compare IPE with simpler strategies, such as adding action noise to a single BC policy. Although Figure 9 controls the total number of collected trajectories across different ensemble sizes, it remains unclear whether training multiple BC policies is necessary to obtain these improvements. Comparisons with simpler exploration strategies under the same interaction rollouts would help clarify the effectiveness of the proposed method.
W3 (Insufficient justification for the Q-value function mismatch argument). The authors attribute the limited benefits of Q-value function pretraining to the mismatch between and . However, different policies generally induce different Q-value functions, and such a mismatch may not imply that Q-value function pretraining is ineffective for subsequent online RL. Although the preference score analysis shows differences in action preferences between the pretrained and reference critics, it may not establish that these differences are the primary reason for the limited benefits of Q-value function pretraining. Furthermore, the paper does not report preference scores for the Q-value function learned with IPE, making it difficult to determine whether the performance improvements arise from mitigating the identified mismatch or simply from increased state-action coverage in the initial replay buffer.
Questions for authors
Q1. How effective would IPE be when applied to large-scale pretrained policies, such as VLA models?
Q2. Could you provide additional explanations or experimental results addressing the concerns raised in the Weaknesses?
2 comments
I really appreciate the detailed explanation and additional experiments, which address my W2 concern. However, two concerns remain. W1: It remains unclear whether diversity from independently trained BC policies can consistently provide useful transitions when offline data is highly suboptimal. W3: The Q-function mismatch does not sufficiently explain why critic pretraining provides limited benefits. Therefore, I maintain my original rating.
Dear Reviewer p2ih,
Thank you for your thoughtful feedback! We provide clarifications to your questions below and will incorporate your feedback into the paper to enhance clarity.
IPE's diversity comes from independently trained BC policies, and we agree that the resulting coverage depends on the offline data and training setup. Empirically, Figures 6 and 8 show that multi-policy rollouts consistently yield broader action coverage and larger action-value variation than single-policy rollouts, and Figure 9 shows that these gains hold when the total rollout budget is fixed while the number of policies varies. A central finding of our work is that this simple source of diversity is sufficient: as shown in our response to W2 below, it substantially outperforms explicit noise-based diversification.
Regarding safety, each IPE policy is trained with the same BC objective on the same demonstrations as the single base policy that standard pipelines already roll out, so IPE does not introduce a qualitatively different type of behavior during data collection.
We also note the relation to PostBC [1], which modifies the pretrained policy itself, widening its action support based on uncertainty in the demonstrations. IPE instead modifies the initial replay data while keeping the same base policy for fine-tuning. The two methods act on different components of the fine-tuning pipeline and could be complementary.
[1] A. Wagenmaker et al., "Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning," ICML, 2026.
Thank you for pointing this out. We agree this comparison is important. We compare IPE against a single BC policy with Gaussian action noise (σ∈{0.05,0.1}) used to collect the initialization rollouts, under the same total rollout budget. IPE substantially outperforms both noise levels. This shows that training multiple BC policies provides benefits that simple exploration noise does not: noise perturbs actions around a single mode, whereas independently trained policies produce distinct, plausible behaviors that give the critic more informative action comparisons.
Mean success rate (%) over 3 seeds, at 100k online steps → end of training
We would like to clarify the argument in Section 4.2. Figure 3 tests what extensive offline pretraining gives the learner at initialization. Even with a near-100%-success base policy (a favorable setting rarely available in practice) and a critic pretrained to convergence, that critic only partially agrees with the reference critic on which actions are better. Fitting the critic on data concentrated around one base policy therefore does not provide the action comparisons that make online improvement faster, consistent with Figure 2, where Q-pretraining shows little or no advantage over random initialization.
Figures 4 and 5 further show that pretraining the critic with offline value-maximization objectives, which move the offline phase closer to online value maximization, improves neither downstream learning nor action-preference accuracy over naive pretraining. Figures 6 and 8 point to the reason: rollouts from a single BC policy provide few alternative actions to compare and little action-value variation. We will revise Section 4.2 to make this framing explicit.
IPE's improvements come from increased state-action coverage in the replay buffer. To isolate this, we evaluate IPE with and without critic pretraining. Both variants perform comparably and substantially outperform naive pretraining on single-policy data, indicating that the improvement comes from coverage.
Mean success rate (%) over 3 seeds, at 200k online steps → end of training
IPE makes no assumptions about the policy class and only changes the data used to initialize the replay buffer. For large pretrained policies such as VLAs, training several full models from scratch would be costly, but the same diversity can be obtained more cheaply by performing multiple supervised fine-tuning. Evaluating IPE in this regime is an exciting direction we will highlight as future work.