acceptodds
Under review as a conference paper at ICLR 2027

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

Abstract

Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function. In this paper, we systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. We find, surprisingly, that naive Q-function pretraining often provides little benefit over random initialization. We show this stems from a fundamental mismatch: the Q-function learned during pretraining targets the pretrained policy's Q-function, not the Q-function that online fine-tuning converges to, and this gap persists even after offline value maximization. Motivated by this finding, we propose Initialization via Policy Ensemble (IPE), a simple method that trains multiple diverse policies and uses their pooled rollouts to bootstrap the Q-function learning in online RL. Across a suite of challenging continuous control benchmarks, IPE yields an average 1.26x improvement in fine-tuning performance over naive Q-function pre-training.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Pretraining, Fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p2ihOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper examines whether Q-value function pretraining is beneficial for online RL fine-tuning with a pretrained policy and proposes Initialization via Policy Ensemble (IPE) to improve state-action coverage for effective Q-value function. The empirical results show that pretraining the Q-value function through offline TD learning methods often provides little benefit over random initialization. The authors attribute this observation to a mismatch between the Q-value function of the pretrained policy and that of the policy obtained through online RL. Motivated by this, IPE trains multiple behavior-cloned policies on the same offline behavior distribution and collects their rollouts to initialize the replay buffer with more diverse transitions.

Critical strengths

S1 (Interesting observation). The paper investigates a question regarding the necessity of Q-value function pretraining for online RL fine-tuning. The experiments show that naive Q-value function pretraining provides little benefit over random initialization under both UTD = 1 (Figure 2) and UTD = 20 (Figure 10), offering useful empirical insights into fine-tuning pretrained policies.

S2 (Simple and intuitive approach). IPE provides a straightforward approach to improving initial state-action coverage by collecting rollouts from multiple independently trained behavior-cloned policies. The collected transitions are used to initialize the replay buffer for subsequent online RL fine-tuning with EXPO, without requiring additional Q-value function pretraining.

S3 (Empirical analysis). The paper provides several experiments examining Q-value function pretraining, action preference score, action diversity, and the number of policies used for IPE. These analyses help illustrate the motivation and empirical behavior of the proposed approach.

Critical weaknesses

W1 (Limited novelty and practical applicability). The proposed method primarily involves training multiple behavior-cloned (BC) policies on the same offline dataset and pooling their environment rollouts to initialize the replay buffer. While the paper shows that multiple BC policies can increase action coverage, this largely relies on differences arising from random initialization, rather than an explicit mechanism for encouraging diverse and useful behaviors. Consequently, it is unclear whether IPE can consistently improve state-action coverage, particularly when the offline dataset contains predominantly suboptimal behaviors. Moreover, independently trained BC policies may generate unsafe or undesirable behaviors during environment interaction, which could limit the applicability of the proposed method to real-world settings. Further discussion or illustrative experiments would help clarify the practical benefits and limitations of the proposed approach.

W2 (Insufficient comparisons with simpler data diversification strategies). The main motivation for the proposed method is that multiple BC policies can provide more diverse action coverage than a single pretrained policy. However, the paper does not compare IPE with simpler strategies, such as adding action noise to a single BC policy. Although Figure 9 controls the total number of collected trajectories across different ensemble sizes, it remains unclear whether training multiple BC policies is necessary to obtain these improvements. Comparisons with simpler exploration strategies under the same interaction rollouts would help clarify the effectiveness of the proposed method.

W3 (Insufficient justification for the Q-value function mismatch argument). The authors attribute the limited benefits of Q-value function pretraining to the mismatch between QπbaseQ^{\pi_{base}} and QπRL∗Q^{\pi_{RL}^*}. However, different policies generally induce different Q-value functions, and such a mismatch may not imply that Q-value function pretraining is ineffective for subsequent online RL. Although the preference score analysis shows differences in action preferences between the pretrained and reference critics, it may not establish that these differences are the primary reason for the limited benefits of Q-value function pretraining. Furthermore, the paper does not report preference scores for the Q-value function learned with IPE, making it difficult to determine whether the performance improvements arise from mitigating the identified mismatch or simply from increased state-action coverage in the initial replay buffer.

Questions for authors

Q1. How effective would IPE be when applied to large-scale pretrained policies, such as VLA models?

Q2. Could you provide additional explanations or experimental results addressing the concerns raised in the Weaknesses?

2 comments

Dear Reviewer p2ih,

Thank you for your thoughtful feedback! We provide clarifications to your questions below and will incorporate your feedback into the paper to enhance clarity.

Novelty, consistency of coverage gains, and practical applicability

IPE's diversity comes from independently trained BC policies, and we agree that the resulting coverage depends on the offline data and training setup. Empirically, Figures 6 and 8 show that multi-policy rollouts consistently yield broader action coverage and larger action-value variation than single-policy rollouts, and Figure 9 shows that these gains hold when the total rollout budget is fixed while the number of policies varies. A central finding of our work is that this simple source of diversity is sufficient: as shown in our response to W2 below, it substantially outperforms explicit noise-based diversification.

Regarding safety, each IPE policy is trained with the same BC objective on the same demonstrations as the single base policy that standard pipelines already roll out, so IPE does not introduce a qualitatively different type of behavior during data collection.

We also note the relation to PostBC [1], which modifies the pretrained policy itself, widening its action support based on uncertainty in the demonstrations. IPE instead modifies the initial replay data while keeping the same base policy for fine-tuning. The two methods act on different components of the fine-tuning pipeline and could be complementary.

[1] A. Wagenmaker et al., "Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning," ICML, 2026.

Comparison with simpler data diversification strategies

Thank you for pointing this out. We agree this comparison is important. We compare IPE against a single BC policy with Gaussian action noise (σ∈{0.05,0.1}\sigma \in \{0.05, 0.1\}) used to collect the initialization rollouts, under the same total rollout budget. IPE substantially outperforms both noise levels. This shows that training multiple BC policies provides benefits that simple exploration noise does not: noise perturbs actions around a single mode, whereas independently trained policies produce distinct, plausible behaviors that give the critic more informative action comparisons.

Mean success rate (%) over 3 seeds, at 100k online steps → end of training

Task (horizon)IPE1 policy + noise (σ\sigma=0.1)1 policy + noise (σ\sigma=0.05)
Square (290k)62.2 → 88.317.3 → 40.613.7 → 53.1
Can (490k)47.7 → 84.030.9 → 51.326.0 → 49.1
Average55.0 → 86.224.1 → 46.019.9 → 51.1

Justification for the Q-function mismatch argument, and whether IPE's gains come from mitigating the mismatch or from coverage

We would like to clarify the argument in Section 4.2. Figure 3 tests what extensive offline pretraining gives the learner at initialization. Even with a near-100%-success base policy (a favorable setting rarely available in practice) and a critic pretrained to convergence, that critic only partially agrees with the reference critic on which actions are better. Fitting the critic on data concentrated around one base policy therefore does not provide the action comparisons that make online improvement faster, consistent with Figure 2, where Q-pretraining shows little or no advantage over random initialization.

Figures 4 and 5 further show that pretraining the critic with offline value-maximization objectives, which move the offline phase closer to online value maximization, improves neither downstream learning nor action-preference accuracy over naive pretraining. Figures 6 and 8 point to the reason: rollouts from a single BC policy provide few alternative actions to compare and little action-value variation. We will revise Section 4.2 to make this framing explicit.

IPE's improvements come from increased state-action coverage in the replay buffer. To isolate this, we evaluate IPE with and without critic pretraining. Both variants perform comparably and substantially outperform naive pretraining on single-policy data, indicating that the improvement comes from coverage.

Mean success rate (%) over 3 seeds, at 200k online steps → end of training

Task (horizon)IPE w/ random QIPE w/ pretrained Q1 BC policy w/ pretrained Q
Square (300k)71.0 → 92.078.8 → 89.456.0 → 70.7
Can MH (490k)41.3 → 53.744.2 → 60.020.7 → 28.7
Average56.2 → 72.961.5 → 74.738.4 → 49.7

Applicability to large-scale pretrained policies such as VLA models

IPE makes no assumptions about the policy class and only changes the data used to initialize the replay buffer. For large pretrained policies such as VLAs, training several full models from scratch would be costly, but the same diversity can be obtained more cheaply by performing multiple supervised fine-tuning. Evaluating IPE in this regime is an exciting direction we will highlight as future work.

I really appreciate the detailed explanation and additional experiments, which address my W2 concern. However, two concerns remain. W1: It remains unclear whether diversity from independently trained BC policies can consistently provide useful transitions when offline data is highly suboptimal. W3: The Q-function mismatch does not sufficiently explain why critic pretraining provides limited benefits. Therefore, I maintain my original rating.

Reviewer_GjoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Initialization via Policy Ensemble (IPE), which collects rollouts from multiple supervised policies to initialize the replay buffer for online RL fine-tuning. The method is motivated by an empirical study finding limited benefits from naive Q-function pretraining, which the authors attribute to differences between the pretrained and online Q-functions. Experiments on RoboMimic and OGBench report improved fine-tuning performance over both naive Q-function pretraining and random Q initialization.

Critical strengths
  • The paper addresses a practically relevant question: whether Q-function pretraining improves online fine-tuning from a pretrained policy. Comparisons use the same base policy, and additional experiments with RLPD and higher UTD provide evidence beyond the main EXPO setting.

  • IPE provides a simple approach to improving replay initialization through rollouts from multiple supervised policies. The ablation varying the number of policies while fixing the total number of trajectories provides useful evidence about the contribution of policy diversity.

Critical weaknesses
  • Section 4.2 attributes the limited benefit of pretraining to the difference between QπbaseQ^{\pi_{\mathrm{base}}} and QπRL∗Q^{\pi_{\mathrm{RL}}^*}. However, this difference alone does not explain the observed lack of advantage over random initialization. Moreover, preference accuracy (Figure 3 and Appendix B.1) cannot distinguish estimation error from differences in the policy being evaluated: an accurate QπbaseQ^{\pi_{\mathrm{base}}} need not prefer the reference action. Thus, the proposed explanation remains insufficiently supported.

  • Figure 9 controls the number of trajectories, but the comparisons in Figures 7 and 9 do not include simpler ways of increasing action coverage, such as action noise applied to a single policy. Given the related coverage perspective in PostBC [1], the additional benefit of IPE over simpler ways of increasing action coverage.

[1] A. Wagenmaker et al., "Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning," in Forty-third International Conference on Machine Learning, 2026.

Questions for authors
  • What evidence distinguishes the mismatch in Section 4.2 from critic estimation error or limitations of the pretraining procedure as explanations for Figure 2? Independent return estimates at the same states and actions, following the base and reference policies separately, would help distinguish estimation errors from actual policy value differences in Figure 3. Given the useful offline initializations reported by Cal-QL [2], CPQL [3], and Uni-RL [4], despite their different policy training, does the limited benefit persist with alternative critic objectives or backup operators while keeping the base policy, offline data, and online fine-tuning procedure fixed?

  • During naive Q-function pretraining, how is the next action in Equation (1) selected? Is it sampled from the base policy or selected using Equation (3)? Does IPE include separate Q-function pretraining before online fine-tuning? Please specify the backup policy and number of pretraining updates for each condition.

  • Could you compare IPE with collecting initialization rollouts from a single base policy with tuned action noise, using the same base policy for fine-tuning and the same total interaction budget? This would clarify the benefit of training multiple policies over a simpler way of increasing action coverage. How does IPE’s contribution relate to the coverage perspective in PostBC [1]?

  • Does the environment-step axis in Figure 7 include all initialization rollouts? Please clarify the total number of environment transitions used by each method, the additional policy training cost of IPE, and the formula used to calculate the reported 26% improvement.

  • Do the benefits of IPE extend to other methods for fine-tuning pretrained BC policies, such as DICE-RL [5]?

[2] Nakamoto, Mitsuhiko, et al. "Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning." Advances in Neural Information Processing Systems 36 (2023): 62244-62269.

[3] B. Kim and M. Oh, "Peng's Q($\lambda $) for Conservative Value Estimation in Offline Reinforcement Learning," in The Fourteenth International Conference on Learning Representations, 2026.

[4] Xu, Haoran, et al. "Uni-rl: Unifying online and offline rl via implicit value regularization." Advances in Neural Information Processing Systems 38 (2025): 133162-133187.

[5] Zhanyi Sun and Shuran Song. From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning. In Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:117175–117191, 2026.

2 comments

Dear Reviewer GjoN,

Thank you for your thorough and constructive review! We provide clarifications to your questions below, which will be incorporated into the paper to enhance clarity.

Distinguishing the mismatch in Section 4.2 from critic estimation error or limitations of the pretraining procedure

We would like to clarify the argument in Section 4.2. Figure 3 tests what extensive offline pretraining actually gives the learner at initialization. Even with a near-100%-success base policy (a favorable setting rarely available in practice) and a critic pretrained to convergence, the pretrained critic only partially agrees with the reference critic on which actions are better. We will revise Section 4.2 to make this framing explicit.

Regarding limitations of the pretraining procedure, Figures 4 and 5 show that pretraining the critic with offline value-maximization objectives, which bring the offline phase closer to the value maximization performed online, improves neither downstream learning nor action-preference accuracy over naive pretraining. Figures 6 and 8 point to the reason of rollouts from a single BC policy provide few alternative actions to compare and little action-value variation.

Backup policy and number of updates during Q-function pretraining; does IPE pretrain the critic?

During naive Q-function pretraining, the backup in Equation (1) samples one next action from the base policy. The number of offline Q-pretraining updates for each condition is shown on the negative x-axis of Figure 2 and listed for the relevant variants in Table 3; we will state these explicitly in the text.

The key of IPE is the data rather than pretraining versus no pretraining. As shown below, IPE with a random critic performs comparably to IPE with a pretrained critic, and both substantially outperform a pretrained critic on single-policy data.

Mean success rate (%) over 3 seeds, at 200k online steps → end of training

Task (horizon)IPE w/ random QIPE w/ pretrained Q1 BC policy w/ pretrained Q
Square (300k)71.0 → 92.078.8 → 89.456.0 → 70.7
Can MH (490k)41.3 → 53.744.2 → 60.020.7 → 28.7
Average56.2 → 72.961.5 → 74.738.4 → 49.7

Comparison with a single base policy with tuned action noise

Per the suggestion of the reviewer, we compare IPE against a single BC policy with Gaussian action noise (σ∈{0.05,0.1}\sigma \in \{0.05, 0.1\}) used to collect the initialization rollouts, keeping the base policy for fine-tuning and the total rollout budget fixed. IPE substantially outperforms both noise levels. Unstructured noise perturbs actions around a single mode, whereas independently trained BC policies produce distinct, plausible behaviors, yielding more informative action comparisons for the critic.

Mean success rate (%) over 3 seeds, at 100k online steps → end of training

Task (horizon)IPE1 policy + noise (σ\sigma=0.1)1 policy + noise (σ\sigma=0.05)
Square (290k)62.2 → 88.317.3 → 40.613.7 → 53.1
Can (490k)47.7 → 84.030.9 → 51.326.0 → 49.1
Average55.0 → 86.224.1 → 46.019.9 → 51.1

Relation to PostBC

PostBC modifies the pretrained policy itself, widening its action support based on uncertainty in the demonstrations so that its rollouts cover more actions. IPE instead modifies the initial replay data, pooling rollouts from several BC policies while keeping the same base policy for fine-tuning. The two methods therefore act on different components of the fine-tuning pipeline and could be complementary. We will add this discussion to the related work.

Environment transitions in Figure 7 and policy training cost

Figure 7's x-axis begins after initialization and shows the online RL phase. The number of initialization rollouts is held fixed across methods, and IPE divides this budget across its policies rather than collecting it per policy, so all methods use the same number of initialization transitions. The budgets are Square 200, Can 20, Lift 50, Cube 80, Scene 100, and Puzzle 30 trajectories.

As an example of training time, on an A4500 GPU, training one BC policy takes 2.80–4.39 hours across Square, Can, and Cube. Training three policies takes 8.50–12.96 hours in total (5.70–8.57 additional hours), and training five policies takes 20.48 hours on Square and 14.02 hours on Cube (16.32 and 11.22 additional hours, respectively). Since the policies are trained independently, this cost can be parallelized.

Thank you for the clarifications and additional experiments. The action noise comparison and the IPE results with random and pretrained critics provide useful evidence for the benefit of policy diversity. However, my main concern regarding Section 4.2 remains. The difference between the pretrained and online value functions alone does not explain the observed lack of advantage over random initialization. Moreover, preference accuracy still cannot distinguish critic estimation error from differences in the policy being evaluated, and training to convergence does not resolve this issue. Thus, while the additional experiments strengthen the empirical support for IPE, the proposed explanation remains insufficiently supported. For these reasons, I will maintain my current score.

Reviewer_iueVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the problem of assessing the benefits of pretraining a critic on online RL fine-tuning when a pretrained policy already available. The authors conduct experiments, and experimental results show marginal improvement achieved naïve Q-function pretraining. They claim that the reason is the mismatch between the base policy’s value function and the value function learning during subsequent RL. The paper presents IPE, which trains multiple imitation policies and pools their environment rollouts to initialize the replay buffer. Experiments across six manipulation tasks from RoboMimic and OGBench show performance improvement.

Critical strengths
  • The paper provides an approach IPE that is simple and shows promising improvements.
  • The paper presents preliminary experimental results to show little benefit of pretraining a critical on online RL fine-tuning.
  • The paper provides experimental results to validate the performance of the proposed IPE.
Critical weaknesses
  • Although the proposed IPE is promising in performance improvement, it demands to train a number of policies and utilize environment rollouts. Thus, the computational costs could be very high, which may compromise the practicability.
  • Q_{\pi_{base}} differing from Q_{\pi_{RL}} does not itself explain why pretraining provides no benefit, as adapting to a changing policy is inherent in actor-critic learning.
  • Algorithm 1 returns a replay buffer but does not specify a critic-pretraining procedure. Thus, it is not clear whether IPE improves critic initialization or mainly improves the data available during early online learning.
Questions for authors
  1. It would be better to add some experimental result showing the computational costs introduced by IPE.
  2. Figure 4 is not clear. Please redraw it.
  3. Are TD-backup actions sampled from the fixed base policy, or selected using EXPO’s value-maximizing candidate selection? Please clarify this.
  4. Does IPE start online training with pretrained critic weights or random weights?
2 comments

Dear Reviewer iueV,

Thank you for your thorough and constructive review! We provide clarifications to your questions below, which will be incorporated into the paper to enhance clarity.

Computational cost and practicality

IPE’s additional cost is confined to offline BC training and requires no additional environment interaction, since the total rollout budget used to initialize the replay buffer is held fixed and divided across policies. As an example of training time, on an A4500 GPU, training one BC policy takes 2.80–4.39 hours across Square, Can, and Cube. Training three policies takes 8.50–12.96 hours in total (5.70–8.57 additional hours), and training five policies takes 20.48 hours on Square and 14.02 hours on Cube (16.32 and 11.22 additional hours, respectively). We note that the A4500 is a relatively slow GPU by modern standards, so these timings are conservative. Moreover, because the BC policies are trained independently, this cost parallelizes trivially. With multiple GPUs available, the wall-clock overhead reduces to roughly that of training a single policy. We will include this analysis in the camera-ready version.

Why pretraining provides no benefit, given that actor-critic learning adapts to a changing policy

We agree that a critic can adapt as the policy changes. Figure 3 instead tests what extensive offline pretraining actually gives the learner at initialization. Even with a near-100%-success base policy (a favorable setting rarely available in practice) and a critic pretrained to convergence, the pretrained critic does not recover the reference critic's action preferences. Fitting the critic thoroughly on data concentrated around one base policy therefore does not provide the action comparisons that make online improvement faster. This is consistent with Figure 2, where Q-pretraining shows little or no advantage over random initialization. Moreover, Figures 4 and 5 show that pretraining the critic with offline value-maximization objectives, which match the critic's online updates, improves neither downstream learning nor action-preference accuracy. Figures 6 and 8 point to the underlying reason: rollouts from a single policy provide limited action coverage and little action-value variation, which motivates IPE's more diverse rollout data.

Does IPE improve critic initialization or the data available during early online learning?

IPE improves the data available to learn the critic and during online learning. The pooled multi-policy rollouts are added to the replay buffer and sampled during online RL with an offline-to-online ratio of 0.5, so the critic's TD updates learn from more diverse data from the very start of fine-tuning.

We additionally evaluate IPE with max-Q critic pretraining. IPE with a random critic performs comparably to IPE with a pretrained critic, and both substantially outperform a pretrained critic on single-policy data. This confirms that the improvement comes from the diversity of the data from IPE.

Mean success rate (%) over 3 seeds, at 200k online steps → end of training

Task (horizon)IPE w/ random QIPE w/ pretrained Q1 BC policy w/ pretrained Q
Square (300k)71.0 → 92.078.8 → 89.456.0 → 70.7
Can MH (490k)41.3 → 53.744.2 → 60.020.7 → 28.7
Average56.2 → 72.961.5 → 74.738.4 → 49.7

Are TD-backup actions sampled from the fixed base policy or selected using EXPO's candidate selection?

During online fine-tuning, TD-backup actions are selected using EXPO's value-maximizing candidate selection. EXPO draws a set of candidate actions from its policy and selects among them using their Q-values, with the number of candidates given by the "Number of action samples" entry in Appendix Table 2. We will state this explicitly in the main text.

Does IPE start online training with pretrained or random critic weights?

IPE start online training with randomly initialized critic or pretrained critic weights. As shown in the table above, using and not using critic pretraining in IPE yields comparable performance.

Figure 4 is not clear

Thank you for pointing this out. We will redraw Figure 4 in the camera-ready version to improve readability.

Thank you so much for your hard work on addressing my concerns. Some of my concerns have been addressed. However, the unresolved issues is IPE's performance gain. The results support replay buffer populated by multiple policies, but leave some other issues like the percentage of the gain coming from rollout diversity, critical pretraining help, etc. Thus, I would like to maintain my current rating.

Comments on the paper

1 comment
Area_Chair_1iV1OpenReview ↗

Please provide your feedback to the authors' response

Reviewers,

Please take a moment to respond to the rebuttal. Thank you!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.