acceptodds
Under review as a conference paper at ICLR 2027

Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation

Abstract

Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a practical paradigm that pre-trains the policy using static offline datasets and subsequently adapts the policy through online interactions. Existing O2O methods primarily address the transition through value calibration, while generally treating the offline-trained policy as a given initialization. We instead study O2O adaptation from the perspective of network plasticity, asking whether the offline-trained policy remains sufficiently adaptable for online learning. Controlled experiments show that prolonged optimization on static offline data progressively reduces network plasticity even after offline performance has largely saturated, and that lower plasticity is associated with weaker subsequent online improvement. Motivated by these observations, we propose REstoring plasticity via Fresh Initialization and policy Transfer (REFIT), a lightweight model-level method for the O2O transition. Before online fine-tuning, REFIT distills the offline policy into a freshly initialized student while temporarily freezing a random subset of student units, transferring the learned offline behavior to a more plastic policy initialization. Extensive experiments on D4RL and OGBench demonstrate that REFIT consistently achieves higher aggregate performance than existing O2O plug-in methods across both Cal-QL and IQL backbones, while plasticity diagnostics and ablations provide further evidence of restored network plasticity.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bzKSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates the performance issues of offline-to-online (O2O) RL from the perspective of plasticity, instead of the traditional focus on value calibration. The authors show that prolonged offline training hurts plasticity, and that lower plasticity is associated with a reduced ability to adapt during online training. With this insight, they propose REFIT, which distills the offline policy into a freshly initialized policy while temporarily freezing a random subset of its hidden units. After distillation, the units are unfrozen and standard online fine-tuning proceeds.

Critical strengths
  • The paper addresses a relevant performance issue in O2O RL and provides an interesting reframing of this issue from a plasticity perspective.
  • The paper's structure, analysis, proposed method, and visualizations are all straightforward and easy to follow.
  • The experiments are quite thorough, covering a variety of environments along with comparisons against existing O2O plug-in methods.
Critical weaknesses
  • Unclear why the proposed method was chosen
    • I would like clarification on why this distillation method in particular was settled upon when there are a wide variety of other general plasticity methods, some even cited within the paper itself (Sokar et al., 2023; Abbas et al., 2023; Nikishin et al., 2023). The proposed method also requires an additional, though potentially brief, training phase. Given that the goal is to maintain plasticity across the transition from offline to online training, it is unclear how traditional plasticity methods could slot into the offline training phase. For example, I wonder whether an architectural change like CReLU (Abbas et al., 2023), applied during offline training, could achieve similar results.
  • Lacking plasticity baseline comparisons for the proposed method
    • While the paper includes comparisons against existing value-estimation approaches (OPT and PARS), I do not believe these target plasticity. As mentioned, there are other, general methods for addressing plasticity. I would like to see some of these compared against the proposed method to further justify its design.
Questions for authors

My questions are listed below. I welcome the authors' clarification on these and on the weaknesses above, in case I have misunderstood any aspects of the work.

  • I'm curious whether the authors tried freezing neurons in specific layers instead of every layer (e.g., later layers versus early layers). If yes, what were the results? If no, what kind of impact do they believe it might have?
  • Is there any reason for focusing on older backbones and not more recent ones like Flow Q-Learning (Park et al., 2025) or other flow-based methods?
  • I wonder how the authors think their work relates to Li et al. (2026), "The Three Regimes of Offline-to-Online Reinforcement Learning"? It is only a workshop paper, but it seems to be one of the few other relevant works concerning plasticity in O2O.

References

S. Park, Q. Li, S. Levine. Flow Q-Learning. ICML 2025. https://openreview.net/forum?id=KVf2SFL1pi

L. Li et al. The Three Regimes of Offline-to-Online Reinforcement Learning. Workshop on Decision-Making from Offline Datasets to Online Adaptation, 2026. https://openreview.net/forum?id=IeZf5RvF2i

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We thank the reviewer for the careful review and constructive suggestions.

W1&W2. Design motivation and comparison with plasticity methods

Thanks for these helpful comments. REFIT is designed to improve policy adaptability at the O2O transition while transferring the offline behavioral prior. Fresh initialization supplies new student parameters, distillation transfers the teacher's behavior, and partial freezing retains an unoptimized parameter subset during this transfer. All student parameters are then unfrozen for online fine-tuning.

To further investigate the advantage of REFIT over other plasticity enhancing methods in the field of O2O RL, we additionally evaluate CReLU [1], ReDo [2], and Plasticity Injection [3]. CReLU addresses activation sparsity through positive and negative ReLU branches, ReDo periodically recycles low-activity neurons, Plasticity Injection introduces fresh residual parameters while preserving predictions at injection. REFIT instead transfers behavior into a student policy without retaining additional teacher branches during online learning.

Table R1. Plasticity baselines on six tasks.

EnvironmentIQLIQL+ReDoIQL+CReLUIQL + PlasticityInjectionIQL + REFIT
HalfCheetah medium48.1 ± 0.149.9 ± 0.150.0 ± 0.149.8 ± 0.150.0 ± 0.3
HalfCheetah random13.8 ± 3.943.2 ± 0.141.9 ± 3.739.9 ± 0.646.3 ± 1.4
AntMaze medium-play76.2 ± 1.586.3 ± 3.178.3 ± 7.686.7 ± 0.686.6 ± 4.8
AntMaze medium-diverse71.6 ± 2.785.3 ± 1.273.3 ± 10.488.3 ± 4.589.2 ± 2.7
AntMaze large-play46.0 ± 2.857.0 ± 1.748.3 ± 2.964.7 ± 2.163.8 ± 8.3
AntMaze large-diverse50.6 ± 5.759.0 ± 1.051.7 ± 7.664.3 ± 5.565.2 ± 4.0
Sum306.3380.7343.5393.7401.1

REFIT achieves the best or second-best mean score on all six tasks, demonstrating the advantage of our method specifically designed to preserve policy plasticity and improve online adaptation in O2O RL.

Q1. Layer-specific freezing

Thanks for this question. We additionally test applying partial freezing only to the first or last hidden layer, following the same strategy as REFIT.

Table R2. Layer-specific freezing. Early/Late denote partial freezing in the first/last hidden layer and report 3-seed means.

EnvironmentEarlyLateREFIT
HalfCheetah medium49.9 ± 0.449.8 ± 0.350.0 ± 0.3
HalfCheetah random44.6 ± 5.445.4 ± 1.046.3 ± 1.4
AntMaze medium-diverse85.0 ± 0.091.7 ± 5.889.2 ± 2.7
AntMaze medium-play85.0 ± 10.088.3 ± 7.686.6 ± 4.8
AntMaze large-play63.3 ± 10.461.7 ± 12.663.8 ± 8.3
AntMaze large-diverse68.3 ± 7.665.0 ± 5.065.2 ± 4.0

The layer-specific variants achieve mean scores close to the default on these tasks, with no consistent advantage of early over late freezing. We retain freezing across layers as a simple default that avoids task-specific layer selection.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

Q2. Evaluation on Flow Q-Learning

We thank the reviewer for this suggestion. Our original experiments focus on IQL and Cal-QL, two widely used baselines in O2O RL. Following this suggestion, we extend REFIT to FQL [4] and evaluate it on six tasks using three seeds.

Table R3. FQL extension (3 seeds).

EnvironmentFQLFQL + REFIT
AntMaze large-diverse65.0 ± 0.075.0 ± 8.7
AntMaze large-play56.7 ± 7.655.0 ± 5.0
AntMaze medium-diverse93.3 ± 2.9100.0 ± 0.0
AntMaze medium-play88.3 ± 7.693.3 ± 2.9
HalfCheetah medium69.6 ± 1.872.0 ± 0.9
HalfCheetah random49.1 ± 1.651.0 ± 2.2
Sum422.0446.3

As shown in Table R3, FQL+REFIT achieves clear performance gains on most evaluated tasks and a higher aggregate score. These results support the effectiveness of REFIT on FQL, extending its benefits to a more recent backbone.

Q3. Relationship to Li et al. (2026)

We thank the reviewer for highlighting this relevant work [5]. Li et al. study the stability–plasticity trade-off from a performance-based perspective, identifying three regimes based on the relative returns of the pretrained policy and offline dataset to guide fine-tuning strategies.

Our work takes a complementary network-level perspective. While Li et al. quantify plasticity through the range of returns observed during online fine-tuning, we focus on the intrinsic adaptability of the policy network, using FAU and other network-level metrics as diagnostic proxies. Specifically, we investigate how offline optimization degrades policy plasticity and propose REFIT to restore it at the O2O transition, thereby improving subsequent online adaptation.

References

[1] Abbas et al. (2023). Loss of Plasticity in Continual Deep Reinforcement Learning.

[2] Sokar et al. (2023). The Dormant Neuron Phenomenon in Deep Reinforcement Learning.

[3] Nikishin et al. (2023). Deep Reinforcement Learning with Plasticity Injection.

[4] Park et al. (2025). Flow Q-Learning.

[5] Li et al. (2026). The Three Regimes of Offline-to-Online Reinforcement Learning.

We thank the reviewer again for their constructive feedback. As the discussion period is approaching its end, we would greatly appreciate any thoughts on our responses and additional experimental results. We would be happy to provide further clarification or address any remaining concerns.

Reviewer_9b26OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In offline-to-online reinforcement learning, a policy is first trained on a fixed, previously collected dataset, then continues training through live interaction with the environment. Existing methods mostly correct the value estimates during this transition, treating the offline-trained policy itself as a good starting point. This paper instead asks whether the network is still able to change. Through controlled experiments, the authors show that training longer on the fixed dataset keeps reducing how many of the network's units are active, even after performance stops improving, and that networks with fewer active units improve less once online training begins. To address this, they propose REFIT. Before online training starts, the offline-trained policy serves as a teacher for a freshly initialized student network, and the student is trained to copy the teacher's behavior while a random portion of its units stay frozen at their initial, untrained values throughout this step. Across several simulated control and manipulation benchmarks, adding REFIT to two existing offline-to-online methods raises their overall scores by 35% and 20%, outperforming other methods built to fix the same transition.

Critical strengths
  • The paper identifies a clear and previously overlooked gap: existing O2O methods treat the offline-trained policy as a fixed, adequate starting point and focus only on correcting value estimates, without asking whether the policy network itself remains able to change.
  • Before proposing a fix, the authors run a controlled experiment that holds offline performance roughly constant across different training budgets and shows plasticity still declines, which isolates plasticity from policy quality as the likely explanation for weaker online improvement.
  • Each component of REFIT, fresh initialization, distillation, and partial freezing, has a clear and understandable role in restoring plasticity while preserving the learned offline behavior.
  • The main text includes ablations that isolate the contribution of each design component and test the freeze ratio and distillation target, supporting the specific design choices made.
Critical weaknesses
  • The paper does not analyze whether the plasticity findings or REFIT's benefit depend on the policy network's size or architecture, so it is unclear how far the central claim generalizes beyond the one configuration used.
  • The main comparison reports means and standard deviations but no significance test, and despite high variance on many tasks the headline conclusion comes from summing means across tasks of very different scales, which can let strong results on some tasks offset weak or noisy results on others without showing whether the net difference is reliable.
  • OPT and PARS, the two methods REFIT is compared against, are each described in a single clause, so the paper does not explain how they work or how their design relates to REFIT's plasticity-based approach.
Questions for authors
  1. Have you tested REFIT, or at least the plasticity diagnostics, on a different network width, depth, or parameter count, even on one or two tasks, to check whether the findings hold across architectures?
  2. Can you report a significance test on the paired per-seed differences, for example a paired t-test or a paired bootstrap confidence interval on the difference?
  3. Could you briefly explain the core mechanism of OPT and PARS and how each relates to or differs from REFIT's plasticity-based approach?
  4. Why does distilling the critic hurt performance? Is there a more specific mechanism behind this beyond added perturbation to the value function?
  5. Why does simply resetting the policy without any distillation already outperform the plain baseline? What does this imply about how much of REFIT's gain comes from the reset itself versus the distillation step?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We thank the reviewer for the careful review and constructive suggestions.

W1&Q1. Network width, depth, and plasticity diagnostics

We thank the reviewer for this suggestion. We evaluate five IQL actor configurations on AntMaze large-play and large-diverse, varying width from 256 to 1024 and depth from two to four hidden layers (Table R1).

Table R1. Actor network configurations.

SettingHidden-layer dimensionsWidthDepth
base[256, 256]2562
wide[512, 512]5122
wider[1024, 1024]10242
deep[256, 256, 256]2563
deeper[256, 256, 256, 256]2564

We evaluate both online performance and the policy's plasticity at the transition. Table R2 reports normalized scores after online fine-tuning, and Table R3 compares initial actor FAU with and without REFIT.

Table R2. Online normalized scores across network scales (3 seeds).

Settingantmaze-l-p IQLantmaze-l-p IQL + REFITantmaze-l-d IQLantmaze-l-d IQL + REFIT
wide55.0 ± 10.068.3 ± 2.958.3 ± 10.470.0 ± 0.0
wider45.0 ± 5.070.0 ± 8.756.7 ± 20.261.7 ± 17.6
deep61.7 ± 20.880.0 ± 17.348.3 ± 12.670.0 ± 13.2
deeper53.3 ± 7.666.7 ± 15.355.0 ± 8.770.0 ± 8.7

Table R3. Actor FAU (%) at the start of online fine-tuning.

Settingantmaze-l-p IQLantmaze-l-p IQL + REFITantmaze-l-d IQLantmaze-l-d IQL + REFIT
base24.32 ± 0.3350.73 ± 1.2224.21 ± 0.3151.01 ± 0.90
wide16.43 ± 0.2747.56 ± 0.6516.30 ± 0.2147.53 ± 0.19
wider10.14 ± 0.4442.49 ± 0.4310.16 ± 0.2442.27 ± 0.26
deep24.22 ± 0.0550.65 ± 1.7324.49 ± 0.2150.46 ± 1.96
deeper23.92 ± 0.1449.88 ± 1.5523.89 ± 0.2449.78 ± 1.60

REFIT increases initial FAU in all five configurations, alongside higher mean online scores across the width and depth variants on both tasks. These results show that REFIT's plasticity restoring effect persists across the tested network widths and depths, supporting its applicability across architectures.

W2&Q2. Statistical reliability and paired comparisons

We thank the reviewer for this comment. To assess task-level improvements, we analyze paired differences, di=REFITi−Baselineid_i = \mathrm{REFIT}_i - \mathrm{Baseline}_i , using pointwise 95% paired t confidence intervals.

Table R4. Summary of task-wise paired comparisons.

AlgorithmPositive mean differencesNegative mean differences95% CI > 095% CI < 095% CI includes 0
IQL16/160/1616/160/160/16
Cal-QL13/163/1611/162/163/16
Total29/323/3227/322/323/32

REFIT yields positive mean differences in 29/32 comparisons, with 27/32 CIs entirely above zero, including all 16 IQL tasks. These results show statistically detectable improvements on most evaluated tasks. For Cal-QL, two CIs are below zero and three include zero. The significance test supports that REFIT yields considerable performance gain on most of the tasks.

W3&Q3. Mechanisms of OPT and PARS and their relationship to REFIT

We thank the reviewer for this question. OPT and PARS primarily focus on correcting value estimates during the O2O transition.

OPT [1] addresses O2O value estimation mismatch by introducing a new critic trained on offline data and initial online rollouts. This new critic better adapts to the online data distribution, providing more reliable value estimates for subsequent policy improvement.

PARS [2] mitigates Q-value overestimation for OOD actions through reward scaling, layer normalization, and penalties on infeasible actions. By suppressing erroneous high Q-values outside the offline data distribution, it reduces the risk of misleading policy updates during online fine-tuning.

REFIT focuses on policy plasticity, asking whether the offline-trained policy remains adaptable for online learning. OPT and PARS primarily address value mismatch during the O2O transition.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

Q4. Why can critic distillation hurt performance?

We thank the reviewer for this question. In actor-critic methods, the critic provides value estimates that guide policy improvement. Distilling the critic may introduce additional approximation errors into these estimates, potentially leading to suboptimal policy updates during online fine-tuning. As discussed by Fujimoto et al. [3], inaccuracies in critic estimates can adversely affect policy optimization and may be further amplified through the interaction between actor and critic updates.

This provides a potential explanation for the performance degradation observed in our critic-distillation variants (Figure 6). In contrast, REFIT distills only the actor while retaining the original offline-trained critic and value networks, avoiding additional critic-distillation errors at the O2O transition.

Q5. Contributions of policy reset and distillation

We thank the reviewer for this question. Simply resetting the policy outperforms the plain baseline on aggregate because reinitialization restores network plasticity, allowing the policy to discard suboptimal behaviors learned offline and explore higher-return behaviors through online interaction.

However, a core motivation of O2O RL is to leverage the offline behavioral prior as a strong starting point, avoiding the need to learn from scratch during costly and potentially unsafe online interactions. Without distillation, both a full parameter reset and a random 50% parameter reset destroy the learned policy mapping, collapsing the initial online return at step 0.

Distillation resolves this issue by transferring the offline behavioral prior into the reset policy before online fine-tuning begins, preserving a strong starting point while retaining restored plasticity. We verify this across both full reset (Table R5) and 50% partial reset (Table R6) settings:

Table R5. Direct full reset versus REFIT at 300K online steps.

EnvironmentPure ResetREFIT
AntMaze medium-play71.2 ± 6.886.6 ± 4.8
AntMaze medium-diverse74.0 ± 5.589.2 ± 2.7
HalfCheetah medium47.3 ± 0.450.0 ± 0.3
HalfCheetah random38.6 ± 2.446.3 ± 1.4

Table R6. Direct partial reset versus partial reset with distillation at 300K online steps.

EnvironmentPartial ResetPartial Reset + Distillation
AntMaze medium-diverse70.0 ± 13.285.0 ± 5.0
AntMaze medium-play83.3 ± 5.885.0 ± 13.2
AntMaze large-play50.0 ± 5.060.0 ± 10.0
AntMaze large-diverse61.7 ± 7.665.0 ± 5.0
HalfCheetah medium49.7 ± 0.350.3 ± 0.1
HalfCheetah random42.5 ± 1.542.8 ± 0.9

Across both comparisons, direct resetting yields lower final performance, whereas adding distillation preserves the offline behavioral prior and improves final scores on the evaluated tasks. Reset and distillation thus serve complementary roles: reset restores plasticity for continued online improvement, and distillation preserves the offline behavioral prior so the agent does not have to learn from scratch online.

References

[1] Shin et al. (2025). Online Pre-Training for Offline-to-Online Reinforcement Learning. ICML.

[2] Kim et al. (2025). Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data. ICML.

[3] Fujimoto et al. (2018). Addressing Function Approximation Error in Actor-Critic Methods. ICML.

We thank the reviewer again for their constructive feedback. As the discussion period is approaching its end, we would greatly appreciate any thoughts on our responses and additional experimental results. We would be happy to provide further clarification or address any remaining concerns.

Reviewer_HbRtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that, in offline-to-online reinforcement learning, policies trained for longer during offline pretraining exhibit lower plasticity metrics and more limited improvement during online fine-tuning, despite achieving similar offline performance. To address this issue, the paper proposes REFIT, which transfers the offline policy’s behavior to a freshly initialized student policy before online fine-tuning. Specifically, REFIT randomly freezes a subset of neurons in each student layer at their initial values and trains the remaining parameters through policy distillation. All parameters are then unfrozen for online fine-tuning, while the offline-trained critic and value networks are retained. Experiments with IQL and Cal-QL on D4RL and OGBench demonstrate improved aggregate performance, supported by component ablations and plasticity diagnostics.

Critical strengths
  • The paper provides clear empirical motivation, showing that policies with similar offline performance can exhibit different levels of plasticity and subsequent online improvement.
  • The paper includes thorough ablations of its main components and hyperparameters, while also examining increased exploration as an alternative explanation for the gains.
Critical weaknesses
  • While the ablations support the benefit of partial freezing, its role in preserving plasticity remains unclear. Comparing variants with matched distillation accuracy and initial policy performance would help isolate this effect.
  • The practical advantage over early stopping remains unclear, as shorter offline training achieves comparable performance in Figure 8.
Questions for authors
  • Could you compare distillation with and without partial freezing while matching distillation accuracy and initial online policy performance, to clarify whether the gains come from preserving plasticity?
  • Given that shorter offline training achieves comparable performance in Figure 8, what practical advantages does REFIT offer over appropriate early stopping?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We thank the reviewer for the careful review and constructive suggestions.

W1&Q1. Partial freezing under matched distillation quality and initial performance

We thank the reviewer for this suggestion. We compare unfrozen distillation (UF, ρ=0) and partial freezing (PF, ρ=0.5) at closely matched distillation accuracy. We use the default PF checkpoint at 100K distillation updates as the reference and select the UF checkpoint within its 100K trajectory whose MSE and MAE closely match those of PF (Table R1). We then compare initial normalized scores and scores after 300K online steps (Table R2), alongside initial actor FAU (Table R3).

Table R1. Held-out distillation errors at the selected checkpoints.

EnvironmentUF / PF checkpointUF MSEPF MSEUF MAEPF MAE
HalfCheetah medium30K / 100K0.00316 ± 0.000230.00319 ± 0.000250.03407 ± 0.001010.03426 ± 0.00105
HalfCheetah random15K / 100K0.04266 ± 0.015850.04333 ± 0.016910.14973 ± 0.035490.15017 ± 0.03945
AntMaze medium-play30K / 100K0.07552 ± 0.018600.07864 ± 0.019240.18213 ± 0.021150.19046 ± 0.02285
AntMaze medium-diverse20K / 100K0.07150 ± 0.009560.07752 ± 0.007590.18012 ± 0.015480.18837 ± 0.00894

Table R2. Initial normalized score → score after 300K online fine-tuning steps.

EnvironmentUFPF
HalfCheetah medium47.4 ± 0.3 → 50.0 ± 0.247.4 ± 0.3 → 50.0 ± 0.3
HalfCheetah random16.3 ± 1.0 → 45.9 ± 3.316.5 ± 0.8 → 46.3 ± 1.4
AntMaze medium-play69.4 ± 4.6 → 80.0 ± 5.070.7 ± 6.8 → 86.6 ± 4.8
AntMaze medium-diverse63.0 ± 5.1 → 78.0 ± 7.662.3 ± 9.7 → 89.2 ± 2.7
Sum196.1 → 253.9196.9 → 272.1

Table R3. Initial actor FAU (%).

EnvironmentUFPF
HalfCheetah medium48.8 ± 1.151.6 ± 0.8
HalfCheetah random46.8 ± 1.951.1 ± 1.3
AntMaze medium-play48.5 ± 0.651.0 ± 1.5
AntMaze medium-diverse49.9 ± 0.851.2 ± 1.3

The selected policies have similar MSE/MAE, and their mean initial normalized scores differ by at most 1.3 points. PF retains higher FAU on all four tasks, and achieves higher aggregate online scores across four tasks. These results support the role of partial freezing in preserving policy plasticity and improving subsequent online adaptation under comparable distillation quality and initial performance.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

W2&Q2. Practical advantages over early stopping

First, we would like to clarify that REFIT and early stopping are complementary strategies. While early stopping aims to mitigate plasticity degradation by limiting offline optimization, REFIT instead seeks to restore plasticity already lost during offline training. Since plasticity loss can occur even with shorter offline training (Figure 2a), REFIT can also benefit early-stopped checkpoints by restoring their plasticity.

To verify this, we further evaluate three AntMaze tasks using checkpoints at 0.25M and 0.5M offline updates and at a saturation point. We retrospectively select the earliest checkpoint after which every subsequent 100K-update window has a score increase of at most two score points. The selected checkpoints are 340K, 440K, and 300K for large-diverse, large-play, and medium-play, respectively. Figure 15 in Appendix H.2 of the revised manuscript shows the offline learning curves and the selected saturation points for the three tasks.

Table R4. Scores after 300K online fine-tuning steps from early-stopping checkpoints.

EnvironmentOffline checkpointIQLIQL + REFIT
AntMaze large-diverse0.25M53.3 ± 5.863.3 ± 7.6
0.5M46.7 ± 7.660.0 ± 0.0
Saturation, 340K58.3 ± 10.468.3 ± 12.6
AntMaze large-play0.25M40.0 ± 26.045.0 ± 15.0
0.5M55.0 ± 15.060.0 ± 5.0
Saturation, 440K48.3 ± 5.850.0 ± 5.0
AntMaze medium-play0.25M85.5 ± 4.386.4 ± 2.7
0.5M86.2 ± 3.587.4 ± 4.0
Saturation, 300K86.7 ± 5.888.3 ± 5.8

REFIT achieves higher mean online scores in all nine comparisons, including all three saturation checkpoints, demonstrating that REFIT can consistently provide additional performance gains when combined with early stopping.

Second, from a practical perspective, selecting an appropriate offline stopping point across different environments involves a difficult trade-off. Because offline returns saturate while network plasticity continues to decline (Section 4.1, Figure 2a), offline performance alone does not reliably indicate when to stop. Stopping too early risks insufficient offline learning on more complex tasks, whereas stopping too late can lead to plasticity degradation. Both scenarios can hinder the efficiency of subsequent online fine-tuning. Rather than requiring per-task tuning of the offline stopping point, REFIT makes online adaptation more robust to the offline training budget: as shown in Figures 8(a) and 8(b), vanilla IQL degrades notably from 86.2 at 0.5M offline steps to 73.2 at 1M steps, whereas IQL + REFIT maintains stable online performance across all evaluated budgets. Moreover, REFIT introduces only 6.26% additional wall-clock time on a single A100 (Appendix C).

We thank the reviewer again for their constructive feedback. As the discussion period is approaching its end, we would greatly appreciate any thoughts on our responses and additional experimental results. We would be happy to provide further clarification or address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.