acceptodds
Under review as a conference paper at ICLR 2027

Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation

Abstract

We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained -function, the learner aims to adapt it to the target environment using only a limited amount of online interaction. We first characterize the difficulty of this setting by establishing a minimax lower bound, showing that even when the pretrained -function is close to optimal , online adaptation can be no more efficient than pure online RL on certain hard instances. On the positive side, under a novel structural condition on the offline-pretrained value functions, we propose O2O-LSVI, an adaptation algorithm with problem-dependent sample complexity that provably improves over pure online RL. Finally, we complement our theory with neural-network experiments that demonstrate the practical effectiveness of the proposed method.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Value Adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Du8FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of online refinement of an imperfect Q-function pretrained offline. The authors first establish a minimax sample complexity lower bound for this problem. They then propose the O2O-LSVI algorithm and show that it achieves better sample complexity than purely online reinforcement learning methods under additional structural assumptions.

Critical strengths
  1. The lower bound provides a meaningful insight: even when the pretrained Q-function is sufficiently close to the optimal Q-function, Ω(H3d2/ϵ2)\Omega(H^3d^2/\epsilon^2) online interactions are still required in the worst case for a certain range of accuracy levels.

  2. The proposed algorithm establishes a regret bound under general function approximation that depends on the complexity of the function class. The paper also analyzes the additional regret caused by bounded misspecification of the reference function, making the theoretical analysis more comprehensive.

Critical weaknesses
  1. The β\beta-separable condition in Definition 5.1 and the bounded accuracy coverage assumption in Assumption 5.3 seem rather strong and may limit the applicability of the theoretical results under function approximation. Definition 5.1 requires the error of each reference Q-value to be either exactly zero or at least β>0\beta > 0. Assumption 5.3 further requires that, at every stage, the probability of visiting the inaccurate region under any policy is at most ρ\rho. Together, these assumptions imply Qref=Q∗Q_{ref}=Q^{*} when ρ<1\rho<1, provided that QQ is continuous and the state-action space is connected. In Remark 6.6, the authors emphasize that the tabular setting can be extended to continuous state-action spaces, where assuming continuity of QQ is natural.

Appendix A allows the observed reference function to have a uniform perturbation of at most τ\tau, which appears to relax the assumptions. However, it is not clear whether this extension fully addresses the applicability of the results to continuous state-action spaces. I remain concerned that these two assumptions may limit the relevance of the theoretical results to smooth continuous-control tasks.

  1. [1] has already studied how to leverage given predictions of optimal Q-values to improve regret in online RL. The current paper does not cite this directly relevant work, and its novelty relative to prior work needs to be further clarified.

[1] Golowich, N., & Moitra, A. (2022, June). Can Q-learning be improved with advice?. In Conference on Learning Theory (pp. 4548-4619). PMLR.

Questions for authors
  1. Regarding Weakness 1, could the authors further clarify the applicability of this assumption to continuous-control tasks, or discuss whether the relaxed assumption in Appendix A fully addresses this concern?

  2. Could the authors provide a more detailed comparison between the proposed work and [1]? Which part of the analysis is new beyond the extension to general function approximation?

5 comments

We thank the reviewer for the constructive feedback and support for acceptance. We address the reviewer's concerns below:

W1&Q1 How will structural condition be applied to continuous control setitng**

A. We thank the reviewer for raising this important point. We agree that the β\beta-separable condition and bounded inaccuracy coverage impose additional structure on the pretrained Q-function. However, we would like to clarify that these assumptions do not require discrete state-action spaces and can hold nontrivially in continuous-state, continuous-action MDPs. We provide an explicit construction motivated by Example 5.6.

1. Example: Continuous control with accurate and inaccurate trajectory regions.

Consider an episodic MDP with continuous action space A=[−1,1]\mathcal A=[-1,1] and state space S=0,1×[0,1]\mathcal S=\\{0,1\\}\times[0,1]. Each state is represented by s=(z,x)s=(z,x), where z∈0,1z\in\\{0,1\\} identifies one of two disconnected regions and x∈[0,1]x\in[0,1] is a continuous state variable.

At the beginning of each episode, let z∼Bernoulli(ρ)z\sim\mathrm{Bernoulli}(\rho). The region indicator remains unchanged throughout the trajectory, i.e., zh+1=zhz_{h+1}=z_h. Within each region, consider the smooth reward function

\in\left[\frac14,\frac34\right],$$ and action-dependent transition dynamics $$P(x_{h+1}\in du\mid z_h,x_h,a_h) =\left[1+\eta a_h(2u-1)\right]du,$$ where $0<\eta<1$. These dynamics are continuous in the state-action variables and have full support over $u\in[0,1]$. Suppose the offline dataset contains trajectories only from region $z=0$, with sufficient coverage of its continuous state-action space. Following the pessimistic construction in Example 5.6, define $$Q_{\mathrm{ref},h}(z,x,a)= \begin{cases} Q_h^\star(z,x,a), & z=0,\newline 0, & z=1. \end{cases}$$ Since the reward is uniformly lower bounded by $1/4$, we have $$Q_h^\star(z,x,a)\geq\frac{H-h+1}{4}.$$ Consequently, $$|Q_{\mathrm{ref},h}-Q_h^\star|= \begin{cases} 0, & z=0,\newline Q_h^\star\geq 1/4, & z=1. \end{cases}$$ Thus, Definition 5.1 is satisfied with $\beta=1/4$. Moreover, since the region indicator is preserved under every policy, $$\begin{aligned} &\sup_{\pi} \Pr_{(s,a)\sim d_h^\pi} \left(Q_{\mathrm{ref},h}(s,a)\neq Q_h^\star(s,a)\right)\\ &=\sup_\pi\Pr_\pi(z_h=1) =\rho,\qquad \forall h\in[H]. \end{aligned}$$ Therefore, Assumption 5.3 is satisfied for any $\rho\in[0,1]$, including arbitrarily small positive $\rho$. This example has a natural trajectory-level interpretation: with probability $1-\rho$, the entire trajectory lies in the accurately pretrained region, and $Q_{\mathrm{ref}}=Q^\star$ at every step. With probability $\rho$, the entire trajectory lies in the uncovered region, and the pretrained Q-function is inaccurate at every step. **2. Compatibility with continuous function approximation.** Importantly, this example is not restricted to tabular value functions. Both $Q^\star$ and $Q_{\mathrm{ref}}$ are continuous within each region and continuous on the disconnected state-action domain. Furthermore, the construction admits a finite-dimensional, Bellman-complete function class. For example, consider the feature map $$\phi(z,x,a)= \begin{bmatrix} \mathbf 1_{\\{z=0\\}}^\top r(x,a) & \mathbf 1_{\\{z=1\\}}^\top r(x,a)& \mathbf 1_{\\{z=0\\}}^\top & \mathbf 1_{\\{z=1\\}}^\top & a\mathbf 1_{\\{z=0\\}}^\top & a\mathbf 1_{\\{z=1\\}}^\top \end{bmatrix}^\top.$$ For any bounded continuation value function $V$, its Bellman backup satisfies $$(\mathcal T_hV)(z,x,a)=r(x,a)+c_{z,0}(V)+a \cdot c_{z,1}(V),$$ where $$c_{z,0}(V)=\int_0^1V(z,u)\,du,\qquad c_{z,1}(V)=\eta\int_0^1(2u-1)V(z,u)\,du.$$ Therefore, $\mathcal T_hV$ belongs to the linear span of $\phi$, verifying Bellman completeness. Both $Q^\star$ and $Q_{\mathrm{ref}}$ also belong to this finite-dimensional function class. Hence, the example demonstrates that our structural conditions can hold under continuous states, continuous actions, and general function approximation. We have added this example in Appendix B of our revised manuscript.

W2&Q2. Can you provde more detailed comparison with [1]?

A. We thank the reviewer for pointing out this closely related work. We agree that [1] is directly relevant and will include it in the related work. While both papers investigate how prior Q-value predictions can improve online RL, the key distinction lies in the structural assumptions under which pretrained Q-values provide improved sample efficiency.

Specifically, [1] introduces an approximate distillation condition, requiring that at every state and stage, there exists a nearly optimal action whose predicted Q-value does not significantly underestimate its true value:

∃a:Δh(s,a)+[Qh⋆(s,a)−Q~h(s,a)]+≤ϵ\exists a: \Delta_h(s,a) + [Q_h^\star(s,a)-\widetilde Q_h(s,a)]_{+} \le \epsilon where Δh(s,a)=Vh⋆(s)−Qh⋆(s,a)\Delta_h(s,a)=V_h^\star(s)-Q_h^\star(s,a). In contrast, our β\beta-separable value-gap condition and bounded inaccuracy coverage assumption do not require accurate predictions of near-optimal actions at every state. Instead, they characterize pretrained Q-functions that are accurate in some regions but substantially inaccurate in others, with the visitation probability of inaccurate regions bounded by ρ\rho under any policy.

For example, our Example 5.6 considers missing offline coverage, where Qref,h(s,a)=Qh⋆(s,a)Q_{\mathrm{ref},h}(s,a)=Q_h^\star(s,a) on covered state-action pairs but Qref,h(s,a)=0Q_{\mathrm{ref},h}(s,a)=0 on uncovered pairs. At a completely uncovered state ss, assuming rewards are bounded below by rmin⁡>0r_{\min}>0, we have Vh⋆(s)≥rmin⁡V_h^\star(s)\geq r_{\min} where

Vh⋆(s)=min⁡aΔh(s,a)+[Qh⋆(s,a)−Qref,h(s,a)]+V_h^\star(s) = \min_a\\{\Delta_h(s,a)+[Q_h^\star(s,a)-Q_{\text{ref},h}(s,a)]_{+}\\}

Therefore, the approximate distillation condition of [1] fails for ϵ<rmin⁡\epsilon<r_{\min}, whereas our β\beta-separable condition holds with β=rmin⁡\beta=r_{\min}, and our bounded inaccuracy coverage condition can hold with ρ<1\rho<1. This demonstrates that our framework provides improved sample-complexity guarantees for pretrained Q-functions that do not satisfy the distillation condition required for the improved guarantees in [1].

Conversely, the distillation condition in [1] permits substantial errors on suboptimal actions, without requiring bounded inaccuracy coverage. Thus, the two conditions characterize different regimes rather than one strictly generalizing the other.

Beyond this structural distinction, our work extends the analysis to general function approximation and establishes a minimax lower bound demonstrating that uniform closeness of pretrained Q-values alone is insufficient to guarantee improved online sample efficiency.

We have revised the related work to acknowledge [1] and explicitly clarify these differences.

[1] Golowich, N., & Moitra, A. (2022, June). Can Q-learning be improved with advice?. In Conference on Learning Theory (pp. 4548-4619). PMLR.

Dear Reviewer Du8F,

Thank you again for your detailed and constructive review and your support for our work. As we are about halfway through the discussion period, we would like to kindly follow up.

We would like to briefly summarize our main clarifications and revisions:

  • Applicability to continuous control (W1 & Q1). We provide an explicit continuous-state, continuous-action MDP construction satisfying the β\beta-separable value-gap and bounded inaccuracy coverage conditions, with β=1/4\beta=1/4 and arbitrarily small positive ρ\rho. The construction also admits a finite-dimensional Bellman-complete function class, demonstrating compatibility with continuous function approximation. We have added this example to Appendix B.

  • Comparison with prior Q-value advice methods (W2 & Q2). We expand the related work discussion to compare our assumptions with Golowich and Moitra (2022). Unlike their approximate distillation condition, which requires accurate predictions for a near-optimal action at every state, our conditions allow substantial inaccuracies in regions with bounded visitation probability. We provide an example where our assumptions hold while their condition fails, and clarify that the two frameworks characterize distinct regimes rather than one strictly generalizing the other. We additionally highlight our general function approximation results and minimax lower bound.

We would greatly appreciate it if you could let us know whether our responses address your concerns, and we would be happy to clarify any remaining questions.

Reviewer_YLRsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This theoretical paper studies offline-to-online RL where the only thing carried over from the pretraining is the Q-function with the goal of adapting it to online samples using general function approximation. The authors claim they have 3 main contributions:

  1. Formulate offline-to-online adaptation as a value-function adaptation framework with information-theoretic lower bound, showing that in the worst case, the offline pretraining does not improve the online-sample efficiency
  2. Propose a structured conditioning of using β\beta-separable value gaps, where Q_ref either equals exactly Q* or by at least β\beta
  3. Propose a new offline-to-online algorithm that use β\beta separable value gaps that is proved to be more efficient than purely online RL.
Critical strengths
  1. Strong mathematical rigor: The proofs are carefully written and consistent with the theoretical results.
  2. Simple mechanism: The exploration condition of: Q_ref +- β\beta is simple yet exact.
Critical weaknesses
  1. The authors claim (Section 5, opening sentence, and the abstract) that, by Theorem 4.1 “offline-to-online value adaptation cannot achieve improved sample efficiency in the most general setting.” The theorem shows something narrower: under the single assumption that Q_ref is within ζ\zeta H of Q*, improvement cannot be guaranteed, because one such Q_ref exists for which it fails. This is a worst-case statement about one assumption, not a statement that adaptation is hard in general, and the text should be revised to say so.
  2. Lack of empirical results: the AntMaze result table shows that it has a higher starting score with a comparable adapted score. This does not demonstrate improved online sample efficiency, which is central to the paper’s claim. Furthermore, there is a lack of an ablation study of different environments, β\beta, etc.
  3. Limited novelty for structural condition: β\beta-separatable value gap closely follows the β\beta-separatable dynamics shift of Qu et al. (2024). Instead of applying the dynamics shift in the tabular case, the authors apply it to value function/general function approximation
  4. Difficult to follow: authors lack to include crucial details about the paper. Including:
    1. the claim of “formulate offline-to-online adaptation as a value-function adaptation framework” but only gives a single sentence of “the agent’s only input from offline pretraining is a state-action value function Qref, and adaptation proceeds through online interaction” is given
    2. Main text relies heavily on the appendix. Such as: 1. Theorem 4.1 is stated with no description of the hard instance or proof intuition. 2. The experiments occupy two sentences with a single number (15.6% vs LSVI-UCB); the AntMaze baselines, Table 1, and Figure 1 are all in the appendix
Questions for authors
  1. Theorem 4.1 claims there exists a condition where sample efficiency is not improved by offline pretraining. In Section 5, the authors claim that this is true for the most general setting. Why would this be the case if the theorem is only for a specific case?
  2. The authors claim improved sample efficiency, but Table 1 only shows a higher starting score and a comparable final score, with no information on the rate of improvement or the number of online samples needed to reach it. How many online steps were used for adaptation, and can learning curves be provided?
  3. Appendix D.1 refers to a construction “already described in the lower-bound argument,” which does not appear in the paper. Was a description of the hard instance removed from the main text?
7 comments

We thank the reviewer for the constructive feedback. We address the reviewer's concerns below:

W1. Theorem 4.1 only shows that closeness of QrefQ_{\mathrm{ref}} to Q⋆Q^\star does not guarantee improved worst-case sample efficiency, rather than implying that offline-to-online adaptation is generally difficult.

A. We would like to emphasize that Theorem 4.1 establishes a minimax lower bound, which inherently characterizes the worst-case sample complexity through a constructed family of hard instances. Specifically, we construct a family of MDPs satisfying Eq. (2), for which any adaptation algorithm requires Ω(H3d2/ϵ2)\Omega(H^3d^2/\epsilon^2) online episodes, matching the lower bound for learning from scratch. Therefore, our result demonstrates that the closeness of QrefQ_{\mathrm{ref}} to Q⋆Q^\star alone is insufficient to guarantee improved worst-case sample efficiency, rather than suggesting that offline-to-online adaptation is generally difficult.

To summarize, letting T=KHT=KH denote the total number of online interaction steps, Theorem 4.1 implies a minimax regret lower bound of Ω(HdT)\Omega(Hd\sqrt{T}) in the corresponding parameter regime. Thus, any offline-to-online adaptation algorithm seeking a O~(T)\widetilde O(\sqrt{T}) regret guarantee cannot achieve a uniformly improved worst-case regret rate solely by leveraging a pretrained Q-function close to Q⋆Q^\star. Importantly, this does not contradict our positive results in Section 5. As we further clarify in our response to Q1, the constructed hard instances do not satisfy the favorable nontrivial (β,ρ)(\beta,\rho) regime required for improved sample efficiency: the separation parameter can be at most O(ϵ/H2)O(\epsilon/H^2), while the inaccurate region can have constant visitation probability. In contrast, Section 5 identifies additional structural conditions under which pretrained Q-functions can provably improve online sample efficiency. We will revise the corresponding statements to clarify this worst-case interpretation.


W3. Can you add more comments on the assumption for the novelty of the structural condition given Qu et al. (2024)

A. We would like to clarify that the settings are fundamentally different compared to Qu et. al.. In their work, the discrepancy between the two domains is modeled specifically as a shift in the transition dynamics. In contrast, our algorithm is not given an offline transition dataset; instead, it receives a pretrained but potentially imperfect reference QQ-function. Moreover, we do not restrict the discrepancy between the reference QQ-function and the optimal target-domain QQ-function to arise solely from a dynamics shift. Such an error may result from reward misspecification, transition-dynamics mismatch, or imperfect offline pretraining caused by insufficient data coverage.


W4. Can you add more details on Theorem 4.1 and experiments.

A. We thank the reviewer for highlighting these presentation issues and have made several updates in the revision, thanks to the additional 1-page limit in the rebuttal period.

First, regarding the offline-to-online value adaptation framework, Section 4 defines a pretraining-agnostic setting in which the learner receives an offline-pretrained Q-function QrefQ_{\mathrm{ref}} and adapts it through online interactions, without requiring access to the offline dataset, source transition dynamics, or behavioral policy. We further provide concrete examples in Section 5, particularly Example 5.6, illustrating how incomplete offline data coverage can lead to accurate Q-value estimates in covered regions but significant errors in uncovered regions. We agree that these elements could be presented more clearly and will expand the problem formulation and motivating examples in the revised manuscript.

Second, regarding the organization of the main text, we agree that several important details were deferred to the appendix, primarily due to space constraints and our initial emphasis on theoretical contributions. We improved the presentation in our revised manuscript. In particular, (1) For Theorem 4.1, we include the key hard-instance construction and proof intuition in the main text, explaining how a common pretrained Q-function can remain uninformative about the unknown transition parameters. (2) For the experiments, we move the neural network experimental results into the main text. We believe these revisions will make the problem formulation, theoretical insights, and empirical findings more accessible.

Q1. Theorem 4.1 claims there exists a condition where sample efficiency is not improved by offline pretraining. In Section 5, the authors claim that this is true for the most general setting. Why would this be the case if the theorem is only for a specific case?

A. We thank the reviewer for this question. We would like to clarify that Theorem 4.1 and the positive results in Section 5 are fully consistent, as they characterize the sample complexity of offline-to-online adaptation under different assumptions.

First, Theorem 4.1 establishes a worst-case lower bound under the general closeness assumption. Let MζM_{\zeta} denote the class of MDPs whose pretrained reference satisfies $ |Q_{\text{ref}}-Q^\star|_\infty \leq \zeta H$, Theorem 4.1 establishes that the minimax online sample complexity over this class is lower bounded by

=\Omega\left(\frac{H^3d^2}{\epsilon^2}\right).$$ This means that closeness to the optimal Q-function alone cannot guarantee improved sample efficiency uniformly over all instances in $\mathcal{M}_{\zeta}$. It does not imply that pretraining cannot benefit particular instances. **Second**, Section 5 introduces additional structural conditions that enable improved sample efficiency. To recap our description in main text, Definition 5.1 and Assumption 5.3 require the pretrained Q-function to satisfy $Q_{\mathrm{ref}}(s,a)=Q^\star(s,a)$ or $|Q_{\mathrm{ref}}(s,a)-Q^\star(s,a)|\geq\beta$ and \$ \sup_{\pi,h} \Pr_{(s,a)\sim d_h^\pi} \left(Q_{\mathrm{ref},h}(s,a)\neq Q_h^\star(s,a)\right) \le\rho.\$ Under these additional conditions, Corollary 6.3 establishes an upper bound of $$\widetilde{O}\left(\frac{H^4d^3}{\epsilon^2}\min\left[\tfrac{\epsilon}{\beta}+\rho,1\right]\right)$$ for linear MDPs. Thus, when $\beta$ is sufficiently large relative to $\epsilon$ and $\rho$ is small, the algorithm enjoys improved sample efficiency over standard online RL. With this in mind, we can show that these structural conditions do not contradict the lower bound. To see this explicitly, consider the hard instance constructed in Appendix D for Theorem 4.1. At stage $H-1$, the reference-value discrepancy satisfies \$ Q^\star_{H-1}(x,a)-Q_{\mathrm{ref},H-1}(x,a) =u\langle\theta_{H-1},a\rangle,\$ where $u=128\epsilon/((d-2)H^2)$. Choosing $a=\theta_{H-1}$ gives a nonzero discrepancy of $128\epsilon/H^2$. Consequently, any valid separation parameter for this hard instance must satisfy $\beta\leq128\epsilon/H^2$. Furthermore, the transient state $x$ is visited with probability at least $1/3$, and a policy choosing $a=\theta_{H-1}$ at that state necessarily encounters an inaccurate reference value. Therefore, the policy-uniform inaccuracy coverage must satisfy $\rho\geq 1/3$. Combining these observations yields $$\min\left[\frac{\epsilon}{\beta}+\rho,1\right] \geq \min\left[\frac{H^2}{128}+\frac13,1\right] =\Omega(1).$$ Hence, for the constructed hard instances, $\beta=O(\epsilon/H^2)$ and $\rho=\Omega(1)$, implying that the improvement factor $\min[\epsilon/\beta+\rho,1]=\Theta(1)$ rather than vanishing asymptotically. Consequently, the sample complexity bound in Corollary 6.3 remains of order $\widetilde{O}(H^4d^3/\epsilon^2)$, matching the dependence on $\epsilon$ of standard online RL. Therefore, **the positive guarantee in Section 5 does not yield an asymptotic sample-efficiency improvement on the hard instances constructed in Theorem 4.1**, demonstrating that the two theoretical results are fully consistent. **In summary**, the two results characterize different regimes: Theorem 4.1 demonstrates that pretrained-value accuracy alone is insufficient to guarantee improved worst-case sample efficiency, whereas Section 5 identifies additional sufficient structural conditions under which improvements can be established. We agree that the phrase "in the most general setting" could be misleading and have revised it to explicitly state "without additional structural assumptions beyond the closeness of the pretrained Q-function."

W2&Q2. Can you provide more ablation study of different environments and parameter β\beta in the experimental results?

A. We thank the reviewer for the suggestion. We provide additional results on the online sample complexity and performance of our neural-network experiments. The following table reports the approximate number of online environment steps required to achieve the corresponding performance.

DatasetSACCal-QL (Online Fine-tuning)O2O-LSVI (Online Fine-tuning)
AntMaze U-Maze1M: 0.0~250K: 99.8 ± 0.4~150K: 99.8 ± 0.4
AntMaze Medium-Play1M: 0.0~250K: 98.8 ± 1.6~200K: 99.3 ± 1.3
AntMaze Large-Play1M: 0.0~800K: 97.3 ± 1.8~900K: 98.5 ± 1.6

These results provide additional evidence of the online sample efficiency of O2O-LSVI. Compared with Cal-QL, O2O-LSVI achieves comparable or better performance with approximately 40% fewer online steps on AntMaze U-Maze and 20% fewer steps on AntMaze Medium-Play. On AntMaze Large-Play, O2O-LSVI achieves a slightly higher final score, although it requires approximately 100K additional online steps. In contrast, direct online RL (SAC) trained from scratch fails to achieve nonzero success within 1M environment steps across all three tasks.

We further conduct an ablation study on the effect of β\beta in the AntMaze U-Maze environment, considering β∈{10,50,100}\beta \in \{10, 50, 100\}. The results below report the final D4RL scores and the approximate number of online interaction steps required to achieve them.

β\betaOnline StepsFinal Score
10~250K99.6 ± 0.2
50~150K99.8 ± 0.4
100~100K95.4 ± 3.8

The results indicate that increasing β\beta reduces the required online interactions, but an excessively large β\beta may compromise final performance. In particular, β=50\beta=50 achieves the best final score while maintaining favorable sample efficiency.

Reviewer_oMhGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

As discussed in the introduction, do you think pretraining or posttraining is more helpful in RL setting?

Critical strengths

The proof is solid and analysis is good.

Empirical results to support analysis is great.

Critical weaknesses

The citation color is hard to recognize.

Questions for authors

As discussed in the introduction, do you think pretraining or posttraining is more helpful in RL setting?

4 comments

We thank the reviewer for the constructive feedback and support for acceptance. We address the reviewer's concerns below:

W1. The citation color is hard to recognize.

A. We thank the reviewer for pointing out the citation color issue. We have corrected the color from red to purple in the revised manuscript.


Q1. As discussed in the introduction, do you think pretraining or posttraining is more helpful in RL setting?

A. We believe that pretraining and post-training play complementary roles in RL, and their relative benefits depend on the quality and transferability of the pretrained knowledge.

Our theoretical results provide a more precise characterization of this trade-off. Theorem 4.1 demonstrates that even when a pretrained Q-function is close to the optimal Q-function, pretraining alone does not necessarily improve the worst-case online sample complexity. However, Theorem 6.2 and Corollary 6.3 establish that when the pretrained Q-function satisfies the β\beta-separable value-gap and bounded inaccuracy-coverage conditions, appropriately leveraging pretrained knowledge during online adaptation can substantially improve sample efficiency. In particular, the online sample complexity can improve by a factor of approximately min⁡[ϵ/β+ρ,1]\min[\epsilon/\beta+\rho,1] relative to standard online RL, up to logarithmic factors in the linear MDP setting.

Therefore, rather than viewing pretraining and post-training as competing approaches, our results suggest that their combination is particularly beneficial: pretraining provides reusable value information, while post-training corrects inaccuracies through targeted online exploration. The central contribution of our work is to theoretically characterize when and how such pretrained knowledge can accelerate subsequent online learning, rather than establishing that either stage is universally more effective.

Dear Reviewer oMhG,

Thank you again for your detailed and constructive review and your support for our work. As we are about halfway through the discussion period, we would like to kindly follow up.

We would like to briefly summarize our main clarifications and revisions:

  • Citation formatting (W1). We have corrected the citation color from red to purple in the revised manuscript to improve readability.

  • Pretraining versus post-training (Q1). We clarify that pretraining and post-training play complementary roles in RL. Theorem 4.1 establishes that pretrained Q-function accuracy alone does not guarantee improved worst-case sample efficiency, whereas Theorem 6.2 and Corollary 6.3 show that additional structural conditions enable improvements by a factor of approximately min⁡{ϵ/β+ρ,1}\min\{\epsilon/\beta+\rho,1\}. These results characterize when pretrained knowledge can effectively accelerate online adaptation.

We would greatly appreciate it if you could let us know whether our response addresses your question, and we would be happy to clarify any remaining points.

I plan to keep the score. Thanks for detailed reply!

Thank you for your response and for taking the time to consider our clarifications. We appreciate your careful evaluation and feedback.

Comments on the paper

3 comments

Dear Reviewers and Area Chair,

We sincerely thank you for your time and effort in reviewing our manuscript and for your valuable feedback. We have revised the manuscript to incorporate your suggestions and address the concerns raised during the review process. The key revisions are highlighted in red in the updated manuscript and summarized below:

  • Related Work: We have expanded the discussion of relevant literature suggested by the reviewers in the Related Work section and added a comparative remark in Section 6.1.
  • Empirical Results: We have moved the neural network experimental results to the main text and included additional results on the β\beta ablation study in Appendix H.2.
  • Hard Instance Construction: We have added a detailed description of the hard instance construction underlying the minimax lower bound in Section 4.
  • Continuous Control Justification: We have added an example illustrating separable value gaps in continuous control settings in Appendix B.

We appreciate your constructive feedback, which has helped us improve the clarity and completeness of our manuscript.

Best regards,

The Authors

Area_Chair_xjBNOpenReview ↗

Reviewer–Author Discussion Period Ending Soon

Dear Reviewers,

With the discussion period coming to a close, I encourage you to revisit the authors’ responses and let us know if you have any further comments.

Thank you for your careful reviews and engagement in the discussion!

Dear Reviewers and Area Chair,

We sincerely thank you for your constructive feedback and valuable suggestions. We have provided detailed responses, additional theoretical analyses, new experimental results, and substantial manuscript revisions to address the concerns raised. We briefly summarize the key updates below:

  1. Minimax lower bound and positive guarantees. We clarified that Theorem 4.1 establishes an Ω(H3d2/ϵ2)\Omega(H^3d^2/\epsilon^2) worst-case lower bound under uniform pretrained Q-function closeness, while our positive results establish improved sample efficiency under additional structural conditions. We further explained their consistency and expanded the hard-instance construction in Section 4.

  2. Structural assumptions and continuous control. We provided an explicit continuous-state, continuous-action MDP satisfying β\beta-separability and bounded inaccuracy coverage, together with a finite-dimensional Bellman-complete function class, now included in Appendix B.

  3. Novelty and related work. We clarified the distinctions from Qu et al. (2024) and Golowich and Moitra (2022), emphasizing our different assumptions and guarantees for imperfect pretrained Q-functions. We also expanded the related work discussion.

  4. Additional experiments and ablations. We provided further AntMaze comparisons, demonstrating improved online sample efficiency on U-Maze and Medium-Play, and added a β\beta ablation study. We also moved the neural-network experimental results into the main text.

  5. Manuscript improvements. We strengthened the theoretical exposition, clarified the complementary roles of pretraining and online adaptation, and improved the presentation and citation formatting.

We believe these additional analyses, experiments, and revisions substantially address the reviewers' principal concerns and strengthen the theoretical and empirical contributions of our work.

As the discussion period approaches its end, we would greatly appreciate it if the reviewers could revisit our detailed responses and revised manuscript and indicate whether their concerns have been adequately addressed or if any outstanding questions remain. We would also respectfully appreciate the Area Chair's assistance in encouraging further reviewer engagement.

Thank you again for your time, consideration, and constructive feedback.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.