- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work investigates the performance issues of offline-to-online (O2O) RL from the perspective of plasticity, instead of the traditional focus on value calibration. The authors show that prolonged offline training hurts plasticity, and that lower plasticity is associated with a reduced ability to adapt during online training. With this insight, they propose REFIT, which distills the offline policy into a freshly initialized policy while temporarily freezing a random subset of its hidden units. After distillation, the units are unfrozen and standard online fine-tuning proceeds.
Critical strengths
- The paper addresses a relevant performance issue in O2O RL and provides an interesting reframing of this issue from a plasticity perspective.
- The paper's structure, analysis, proposed method, and visualizations are all straightforward and easy to follow.
- The experiments are quite thorough, covering a variety of environments along with comparisons against existing O2O plug-in methods.
Critical weaknesses
- Unclear why the proposed method was chosen
- I would like clarification on why this distillation method in particular was settled upon when there are a wide variety of other general plasticity methods, some even cited within the paper itself (Sokar et al., 2023; Abbas et al., 2023; Nikishin et al., 2023). The proposed method also requires an additional, though potentially brief, training phase. Given that the goal is to maintain plasticity across the transition from offline to online training, it is unclear how traditional plasticity methods could slot into the offline training phase. For example, I wonder whether an architectural change like CReLU (Abbas et al., 2023), applied during offline training, could achieve similar results.
- Lacking plasticity baseline comparisons for the proposed method
- While the paper includes comparisons against existing value-estimation approaches (OPT and PARS), I do not believe these target plasticity. As mentioned, there are other, general methods for addressing plasticity. I would like to see some of these compared against the proposed method to further justify its design.
Questions for authors
My questions are listed below. I welcome the authors' clarification on these and on the weaknesses above, in case I have misunderstood any aspects of the work.
- I'm curious whether the authors tried freezing neurons in specific layers instead of every layer (e.g., later layers versus early layers). If yes, what were the results? If no, what kind of impact do they believe it might have?
- Is there any reason for focusing on older backbones and not more recent ones like Flow Q-Learning (Park et al., 2025) or other flow-based methods?
- I wonder how the authors think their work relates to Li et al. (2026), "The Three Regimes of Offline-to-Online Reinforcement Learning"? It is only a workshop paper, but it seems to be one of the few other relevant works concerning plasticity in O2O.
References
S. Park, Q. Li, S. Levine. Flow Q-Learning. ICML 2025. https://openreview.net/forum?id=KVf2SFL1pi
L. Li et al. The Three Regimes of Offline-to-Online Reinforcement Learning. Workshop on Decision-Making from Offline Datasets to Online Adaptation, 2026. https://openreview.net/forum?id=IeZf5RvF2i
5 comments
Official Comment by Authors (Part 2/2)
Q2. Evaluation on Flow Q-Learning
We thank the reviewer for this suggestion. Our original experiments focus on IQL and Cal-QL, two widely used baselines in O2O RL. Following this suggestion, we extend REFIT to FQL [4] and evaluate it on six tasks using three seeds.
Table R3. FQL extension (3 seeds).
| Environment | FQL | FQL + REFIT |
|---|---|---|
| AntMaze large-diverse | 65.0 ± 0.0 | 75.0 ± 8.7 |
| AntMaze large-play | 56.7 ± 7.6 | 55.0 ± 5.0 |
| AntMaze medium-diverse | 93.3 ± 2.9 | 100.0 ± 0.0 |
| AntMaze medium-play | 88.3 ± 7.6 | 93.3 ± 2.9 |
| HalfCheetah medium | 69.6 ± 1.8 | 72.0 ± 0.9 |
| HalfCheetah random | 49.1 ± 1.6 | 51.0 ± 2.2 |
| Sum | 422.0 | 446.3 |
As shown in Table R3, FQL+REFIT achieves clear performance gains on most evaluated tasks and a higher aggregate score. These results support the effectiveness of REFIT on FQL, extending its benefits to a more recent backbone.
Q3. Relationship to Li et al. (2026)
We thank the reviewer for highlighting this relevant work [5]. Li et al. study the stability–plasticity trade-off from a performance-based perspective, identifying three regimes based on the relative returns of the pretrained policy and offline dataset to guide fine-tuning strategies.
Our work takes a complementary network-level perspective. While Li et al. quantify plasticity through the range of returns observed during online fine-tuning, we focus on the intrinsic adaptability of the policy network, using FAU and other network-level metrics as diagnostic proxies. Specifically, we investigate how offline optimization degrades policy plasticity and propose REFIT to restore it at the O2O transition, thereby improving subsequent online adaptation.
References
[1] Abbas et al. (2023). Loss of Plasticity in Continual Deep Reinforcement Learning.
[2] Sokar et al. (2023). The Dormant Neuron Phenomenon in Deep Reinforcement Learning.
[3] Nikishin et al. (2023). Deep Reinforcement Learning with Plasticity Injection.
[4] Park et al. (2025). Flow Q-Learning.
[5] Li et al. (2026). The Three Regimes of Offline-to-Online Reinforcement Learning.
We thank the reviewer again for their constructive feedback. As the discussion period is approaching its end, we would greatly appreciate any thoughts on our responses and additional experimental results. We would be happy to provide further clarification or address any remaining concerns.
Official Comment by Authors (Part 1/2)
We thank the reviewer for the careful review and constructive suggestions.
W1&W2. Design motivation and comparison with plasticity methods
Thanks for these helpful comments. REFIT is designed to improve policy adaptability at the O2O transition while transferring the offline behavioral prior. Fresh initialization supplies new student parameters, distillation transfers the teacher's behavior, and partial freezing retains an unoptimized parameter subset during this transfer. All student parameters are then unfrozen for online fine-tuning.
To further investigate the advantage of REFIT over other plasticity enhancing methods in the field of O2O RL, we additionally evaluate CReLU [1], ReDo [2], and Plasticity Injection [3]. CReLU addresses activation sparsity through positive and negative ReLU branches, ReDo periodically recycles low-activity neurons, Plasticity Injection introduces fresh residual parameters while preserving predictions at injection. REFIT instead transfers behavior into a student policy without retaining additional teacher branches during online learning.
Table R1. Plasticity baselines on six tasks.
REFIT achieves the best or second-best mean score on all six tasks, demonstrating the advantage of our method specifically designed to preserve policy plasticity and improve online adaptation in O2O RL.
Q1. Layer-specific freezing
Thanks for this question. We additionally test applying partial freezing only to the first or last hidden layer, following the same strategy as REFIT.
Table R2. Layer-specific freezing. Early/Late denote partial freezing in the first/last hidden layer and report 3-seed means.
The layer-specific variants achieve mean scores close to the default on these tasks, with no consistent advantage of early over late freezing. We retain freezing across layers as a simple default that avoids task-specific layer selection.