acceptodds
Under review as a conference paper at ICLR 2027

Do Not Disturb: When Plasticity Interventions Degrade Offline-to-Online Reinforcement Learning

Abstract

Maintaining neural network plasticity is widely regarded as an indispensable pre- requisite for continual reinforcement learning (RL). A vibrant literature advocates periodic parameter interventions—such as network resets, Shrink-and-Perturb, and dormant neuron recycling—to counteract capacity loss, dead units, and feature rank collapse. While effective in continual supervised learning and long-horizon online RL from scratch, these methods rest on an implicit assumption: that plasticity restoration is universally benign and can be scheduled unconditionally. In this work, we demonstrate that this assumption fails fundamentally in offline-to-online (O2O) continuous-control RL. When transitioning from static pretraining to online interaction, agents inherit structured, high-performing policy and value manifolds. Across standard D4RL continuous-control locomotion benchmarks, we show that unconditional peri- odic Shrink-and-Perturb induces severe intervention vulnerability, destabilizing converged locomotion policies and causing a catastrophic 68.3% collapse in ag- gregate Interquartile Mean (IQM) normalized return (38.78 → 12.30, paired Wilcoxon signed-rank W = 18.0, p = 1.44 × 10−11, mean paired difference +26.03 ± 19.09). To address this vulnerability, we articulate the “Do Not Dis- turb” principle: an agent should intervene only when representation capacity is demonstrably impaired. We introduce CAPACITYGATE, a closed-loop diagnostic framework that monitors feature effective rank and neuron dormancy using uni- form replay reservoir sampling and refractory cooldown control. In stable and moderately shifted transfer, CAPACITYGATE detects that representation capacity remains intact (ρ ≈ 1.0, d < 0.08) and safely abstains from intervention, preserv- ing baseline performance (38.78 IQM) and protecting converged policies against perturbation-induced collapse. Furthermore, under non-stationary physical stress (actuator crippling), we uncover a profound operator stability asymmetry: zero- functional-shift neuron recycling (ReDo) preserves bipedal balance manifolds (8.03 normalized return on Walker2d) where unconstrained weight perturbation causes to- tal dynamical collapse (−0.53). Our findings reframe plasticity restoration in O2O RL from an open-loop maintenance routine to a regime- and operator-dependent intervention that requires rigorous diagnostic gating.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, offline-to-online reinforcement learning, plasticity, representation learning, continual learning, continuous control, MuJoCo, adaptive intervention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Jdm5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors look at plasticity-maintenance mechanisms, and especially shrink-and-perturb methods. Where these are typically applied in an open-loop fashion, the authors argue and show that this does not work well for offline to online RL. They propose conditional mechanisms instead, such as capacity-gate, which abstains from intervention when the representation capacity is intact. It monitors relative effective rank and activation dormancy. They also introduce zero-functional-shift neuron recycling (ReDo) for actuator crippling.

Critical strengths
  • The central idea is powerful: not blindly applying the technique of shrink and perturb in the O2O scenario, as it may just destroy performance and behavior, impeding further learning. Proposing a conditional, closed-loop scheme is a good idea.
  • It is shown that on some tasks, capacitygate can maintain performance.
  • It is shown that on some tasks ReDo may be a good mechanism that does not perturb performance too much.
Critical weaknesses
  • In general, it seems that capacitygate can maintain performance by not applying shrink and perturb. In those cases, performance stays the same as before (a bit misleading to make it bold in Table 1, as the "none" method works just as well). It is less clear to me whether it can actually also allow interventions when it is necessary. Perhaps this is what is studied with the actuator crippling, but this is not clear to me. In table 2, capacitygate was only tested for one seed (why?) and has a low performance compared to fixed and redo.

  • Also the conclusion of fixed shrink and perturb vs redo is not clear to me: Figure 3 seems to say that sometimes redo is better, sometimes fixed. There is a mention of how important a stable gate is for bipedal walking, which may be an indication for a type of task where redo works better, but it seems hard to generalize that to any control task...

Questions for authors
  1. It would be great if the authors could address my uncertainties for the main weaknesses.

  2. The section on Relative Effective Rank Ratio is hard to follow. With penultimate, I guess the authors mean the penultimate layer (so last hidden layer). I could not find D anywhere, so I guess it is the dimension of that layer. My main question: What is the entropy of normalized singular values supposed to capture? Please explain it intuitively.

  3. Dormancy. What I understood is that according to the definition, neurons are dormant if they have all negative activations in the batch. This may give many dormant neurons when batches are small... Also, does it not assume a fixed activation function, like heaviside or ReLU? Also here, more explanation would be welcome.

  4. Why the hysteresis? Do the authors apply shrink and perturb when going over threshold? What happens when the value goes under the lower threshold, nothing? It is not clear if capacitygate is used with shrink and perturb or with redo or with both...

  5. What are "stress" tests? The actuator crippling?

3 comments

We thank the reviewer for recognizing the central motivation behind the Do Not Disturb principle and for identifying several places where the mechanism and experimental interpretation needed clarification. We have revised the manuscript and expanded the experimental evidence.

  1. Abstention versus active intervention

We agree that the original primary benchmark primarily demonstrated safe abstention: CAPACITYGATE preserved baseline performance by not applying Shrink-and-Perturb under stable and moderately shifted transfer. This should not be interpreted as evidence of active intervention benefit in those regimes.

We expanded the actuator-crippling evaluation to three seeds per environment. In HalfCheetah, CAPACITYGATE detects post-shift rank degradation and executes 6–7 Shrink-and-Perturb interventions, obtaining 8.35 ± 2.79. In Walker2d, it abstains across all three seeds and obtains 8.21 ± 2.38, matching the no-intervention baseline. These results illustrate task-dependent gating behavior; the HalfCheetah result is descriptive rather than statistically conclusive.

  1. Relative effective rank: intuition and dimensionality

We clarified that the penultimate representation is the final hidden-layer activation representation. Its dimension D is the number of units in that layer.

The normalized singular values describe how the representation's variance is distributed across feature directions. Their entropy measures the effective number of directions carrying substantial representation variance: a more concentrated spectrum corresponds to fewer dominant directions, while a more uniform spectrum corresponds to a richer distribution of directions. Effective rank therefore provides a summary of representational dimensionality rather than simply counting nonzero units.

The relative effective-rank ratio compares the online representation with its offline anchor, allowing the diagnostic to detect changes relative to the pretrained representation.

  1. Activation dormancy and architectural assumptions

We clarified the ReLU assumption underlying the dormancy definition. Under ReLU activations, outputs are non-negative, so a neuron with zero mean activation over the probe batch is inactive on that batch. We also clarify the operational threshold used in the implementation.

We agree that the estimate depends on the probe batch and its size. Dormancy is therefore treated as a diagnostic statistic over the specified probe distribution, not as proof that a neuron is permanently incapable of activation.

  1. Hysteresis and intervention behavior

Hysteresis uses separate activation and deactivation thresholds to avoid rapid switching when telemetry fluctuates near a single threshold. Once the gate activates, it remains active until the recovery criteria are satisfied. The refractory cooldown prevents repeated interventions at every diagnostic evaluation.

We clarified that the gate controls the configured intervention operator. In the reported actuator-crippling HalfCheetah experiment, the active operator is Shrink-and-Perturb; ReDo is evaluated separately as an unconditional operator baseline. The gate does not automatically switch between SP and ReDo.

  1. Stress-test terminology and operator comparisons

The stress tests refer to the non-stationary actuator-crippling experiments, in which a critical actuator is disabled during online adaptation. We have clarified the experimental context and the distinction between the operator comparisons and the gated intervention evaluation.

Finally, we added an unconditional ReDo stable-transfer comparison across 27 runs. ReDo obtains an IQM of 40.37 compared with 38.33 for the no-intervention baseline (paired W = 123.0, p = 0.117). This supports a more careful interpretation: intervention outcomes depend on the operator and task, rather than establishing that all periodic interventions are harmful.

We appreciate the reviewer's suggestions, which helped us make the diagnostic definitions, control logic, and scope of the conclusions more explicit.

Revised manuscript: Sections 3.2–3.4, 4, 5.1–5.4, and 6; Tables 2–3; Figures 3–4; Appendices C–E.

Reviewer_Jdm5OpenReview ↗

Acknowledgement of rebuttal and revision

The authors have answered my questions and updated the manuscript accordingly, which has become clearer. The results now show a small benefit to performing SP / ReDo when these are actually performed even with CapacityGate. However, these positive differences remain very subtle (e.g., 8.35 vs 6.23 on HalfCheetah, where the maximum is with a fixed procedure achieving 15.27).

I will slightly higher my score to "weak accept".

Thank you for the constructive feedback and for recognizing the clarifications in our previous response.

We have further strengthened the actuator-stress evaluation by adding CAPACITYGATE results across three paired seeds, giving a balanced four-arm comparison across both environments.

The results clarify two distinct behaviors:

On HalfCheetah, the gate detects persistent representation-rank collapse and activates with cooldown-spaced interventions.

On Walker2d, it abstains when capacity indicators remain within the calibrated bounds, preserving baseline performance while Fixed Shrink-and-Perturb causes substantial degradation.

We also expanded the operator-level analysis. The 37-checkpoint ReDo audit now reports policy and critic functional drift, and the manuscript explicitly describes ReDo at τ = 0.10 as approximate, low-disruption preservation rather than exact function preservation.

Finally, we clarified the effective-rank and dormancy definitions, probe-distribution choices, hysteresis thresholds, cooldown behavior, and terminal evaluation mechanics.

We appreciate your engagement with the revision and hope the additional evidence and clarifications make the method's operating conditions and limitations sufficiently clear.

Reviewer_wL6gOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper challenges a common implicit assumption in continual reinforcement learning: that periodic parameter interventions for restoring neural plasticity are always beneficial. Focusing on offline-to-online (O2O) continuous control RL, the authors empirically show that blindly applying periodic Shrink-and-Perturb can severely break well-trained policies, leading to a massive performance drop on D4RL locomotion tasks. To solve this problem, they propose the “Do Not Disturb” principle and a closed-loop framework called CAPACITYGATE. CAPACITYGATE monitors representation capacity and triggers parameter interventions only when the model’s representation is degraded. The paper also reveals that different plasticity repair operators behave drastically differently under non-stationary environments such as actuator failures. Overall, this work argues that plasticity restoration in O2O RL should no longer be a fixed open-loop schedule but a condition-based intervention controlled by real-time capacity diagnostics.

Critical strengths
  • Statistically rigorous headline result. The core negative result rests on a clean seed-matched paired design with rliable IQM, stratified bootstrap CIs, and paired Wilcoxon plus paired t-tests; all headline numbers are internally consistent and reproducible (68.3% = (38.78−12.30)/38.78; 42/45 = 93.3% = P(None > Fixed)).
  • Honest, complete disclosure. Per-seed results for all 135 primary runs are tabulated; CAPACITYGATE's stress-test results are explicitly labeled single-seed validation (hatched bars) rather than passed off as multi-seed means; appendices include a literature comparison matrix, full hyperparameters, and a step-by-step telemetry trace, plus reproducibility and AI-assistance statements.
Critical weaknesses

The paper suffers from poor writing quality and low-quality figures. Table 4 contains duplicated data. The experimental evaluation is limited, as the authors only use the D4RL benchmark. Additional benchmarks such as OGbench should be included to strengthen evaluation. Moreover, the work lacks comparisons against more state-of-the-art offline RL algorithms.

In the primary benchmark, CAPACITYGATE is indistinguishable from doing nothing. It fires zero interventions (0 out of 1,125 periodic evaluations) and all nine Table 1 cells are numerically identical to the "none" baseline. The main experiment therefore never exercises the method itself — the claimed "protection" is a trivial identity, and the framework's added value is untested in the confirmatory study.

Questions for authors

See Weaknesses

1 comment

We thank the reviewer for the detailed assessment. We took the concerns regarding the zero-intervention result, limited experimental scope, and missing comparisons against stronger O2O methods seriously. We have revised the manuscript and supplementary material to address these points.

  1. Active validation of CAPACITYGATE

We agree that the original primary benchmark, by itself, did not demonstrate active intervention efficacy. Its zero-intervention outcome demonstrates abstention under stable and moderately shifted transfer, rather than active intervention performance.

We corrected the diagnostic count from 1,125 to 1,080 evaluations (45 runs × 24 evaluations at steps 1,000–24,000). Step 25,000 is skipped by the terminal loop condition before final evaluation.

To test active behavior, we expanded the actuator-crippling experiments to three seeds per environment. The revised comparison includes 18 baseline runs and 6 newly executed CAPACITYGATE runs; six obsolete runs from an earlier un-cooldowned configuration are excluded.

  • HalfCheetah: CAPACITYGATE obtains 8.35 ± 2.79 (seeds: 8.58, 11.01, 5.44), compared with 6.23 ± 1.53 for no intervention. It detects post-shift rank degradation and executes 6–7 SP interventions.
  • Walker2d: CAPACITYGATE obtains 8.21 ± 2.38, matching the no-intervention baseline, and abstains on all three seeds. Fixed SP obtains −0.53 ± 0.33, while ReDo obtains 8.03 ± 1.57.

We report the HalfCheetah improvement as descriptive evidence only; three seeds do not establish statistical significance.

  1. Unconditional ReDo comparison

We added 27 unconditional ReDo runs across nine stable-transfer environment–shift cells, using three seeds per cell. The matched baseline and Fixed SP runs were reused from the original study.

The aggregate IQM results using task-stratified bootstrap 95% confidence intervals are:

  • None: 38.33 [36.73, 39.67].
  • ReDo: 40.37 [38.23, 42.29].
  • Fixed SP: 13.97 [10.94, 16.48].

ReDo versus None yields W = 123.0, p = 0.117 (not statistically significant). ReDo outperforms Fixed SP in 26 of 27 matched observations (W = 2.0, p = 4.47 × 10⁻⁸).

This narrows our original interpretation: the results do not establish that every open-loop intervention is harmful. The observed vulnerability is operator-dependent, with global weight perturbation behaving differently from low-disruption neuron recycling.

  1. Benchmark diversity and related methods

We expanded the related-work discussion and added a conceptual comparison with Policy Expansion (PEX), Cal-QL, and Robust Offline-to-Online RL (RO2O). We explicitly distinguish this conceptual comparison from direct empirical evaluation.

We have not conducted head-to-head experiments against these methods or added OGBench experiments. We acknowledge these as limitations and directions for future work rather than claiming that the experimental comparison request has been fully addressed.

  1. Presentation and reproducibility

We regenerated the figures, reconciled Section 5.2 telemetry with Figure 4 using the raw diagnostic records, corrected the run counts and ReDo preservation claims, and updated the supplementary reproduction pipeline.

The revised manuscript contains ten main-text pages, followed by references and appendices.

We appreciate the reviewer's central concern that abstention alone does not demonstrate active intervention utility. The additional multi-seed stress experiments address this distinction, while the revised manuscript makes the remaining limitations explicit.

Revised manuscript: Sections 5.1–5.4 and 7; Tables 1–3; Figures 1–4; Appendix A and supplementary materials.

Reviewer_NhQVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies plasticity interventions in offline-to-online continuous-control reinforcement learning and proposes CapacityGate, which gates interventions using effective rank and neuron dormancy. Across three D4RL locomotion tasks, the tested periodic Shrink-and-Perturb configuration substantially reduces returns, while CapacityGate avoids intervening and matches the baseline in the primary benchmark. Additional actuator-failure experiments show that intervention outcomes depend on both the task and the operator.

Critical strengths
  1. The paper provides useful negative evidence: periodic Shrink-and-Perturb reduces aggregate IQM from 38.78 to 12.30, with principal statistics consistent with the seed-level data in Table 4.
  2. The actuator-failure experiments provide informative boundary cases. Shrink-and-Perturb improves HalfCheetah returns but substantially harms Walker2d, while ReDo remains near the Walker2d baseline (Table 2), supporting a task- and operator-dependent interpretation within these experiments.
Critical weaknesses
  1. Section 3.4 claims exact function preservation under ReDo, but Table 5 uses a relative activation threshold of 0.10. Low-activity neurons can still contribute nonzero outputs, so clearing their outgoing weights does not generally preserve the function exactly.
  2. The diagnostic evidence needs reconciliation. Section 5.2 reports rank ratios between 0.99 and 1.03 and dormancy of at most 8.5 percent, whereas Figure 4 shows larger rank ratios and Hopper dormancy around 10 percent. Both remain within the triggering thresholds, so this discrepancy does not refute the decision to abstain from intervention, but it leaves the quantitative evidence for representation health unresolved.
Questions for authors
  1. What additional conditions would justify exact function preservation at the 0.10 threshold? If preservation is only approximate, are measurements of policy or critic output changes available to support the mechanism, or should the interpretation be narrowed?
  2. Do Section 5.2 and Figure 4 use identical runs, networks, and anchors? Please reconcile their ranges and clarify whether “intact capacity” refers to threshold compliance or to independently validated learning capacity.
3 comments

We thank the reviewer for the careful technical assessment. We particularly appreciate the identification of the distinction between exact functional preservation and approximate preservation under a relative activation threshold.

1. ReDo and functional preservation

We agree that the original exact-function-preservation wording was too strong for the implementation using a relative activation threshold of 0.10.

A neuron with low but nonzero activation can still contribute to the network output. Consequently, zeroing its outgoing weights does not generally guarantee exact preservation of the network function. Exact preservation requires the removed neuron's contribution to be identically zero over the inputs under consideration, or an equivalent condition ensuring that the removed contribution vanishes.

We have corrected the manuscript to distinguish the theoretical zero-contribution condition from the implemented relative-threshold ReDo operator. We no longer present the 0.10-threshold implementation as guaranteeing exact functional preservation.

We also added a 37-checkpoint functional-drift audit to the supplementary material. On held-out inputs, the policy action drift had a median of 0.00019, mean of 0.0345, 95th percentile of 0.1771, and maximum of 3.7568. The critic Q1 absolute drift had a median of 7.79, mean of 12.4636, 95th percentile of 38.14, and maximum of 447.7546. These measurements indicate that typical policy-output changes are small, but critic-output changes can be substantial. They characterize observed functional changes and do not establish exact equality of the network functions.

2. Reconciliation of Section 5.2 and Figure 4

We revisited the diagnostic records and updated the manuscript and supplementary material to align the reported telemetry with the underlying runs.

The reported ranges should be interpreted relative to the gate's activation and deactivation thresholds. The diagnostic decision is based on threshold compliance, not on an independent proof that the representation retains all learning capacity.

We have therefore narrowed the interpretation of “intact capacity”: it means that the monitored diagnostics did not cross the configured intervention thresholds on the specified probe distribution. It does not establish that the representation is universally optimal or that future adaptation is guaranteed.

3. Scope of the evidence

We agree that effective rank and dormancy are proxies for representation properties rather than complete measures of learning capacity. Their interpretation depends on the selected networks, anchor representation, probe distribution, and thresholds. We have made these qualifications more explicit.

The revised discussion also distinguishes the stable-transfer abstention results from the actuator-crippling experiments, where the diagnostics detect degradation and trigger interventions in HalfCheetah but abstain in Walker2d.

We appreciate the reviewer's emphasis on precise functional claims and internally consistent diagnostic reporting. These changes narrow the claims to what the implementation and measured evidence support.

Revised manuscript: Sections 3.2–3.4, 5.2, and 6; Figure 4; Appendix E and supplementary functional-drift audit.

Thank you for identifying the issue with the original ReDo function-preservation claim and the inconsistencies in the telemetry reporting.

We have made the following corrections:

Removed claims of exact function preservation for ReDo at τ = 0.10 and replaced them with approximate, low-disruption preservation.

Added a 37-checkpoint functional-drift audit. The median held-out policy action drift is 0.00019, while the median critic Q1 absolute drift is 7.79. We also report the corresponding tail and worst-case values to avoid implying that disruption is uniformly negligible.

Clarified that zero drift on a finite probe batch does not establish global function identity on unseen states.

Recomputed telemetry from the raw logs across all 1,080 diagnostic evaluations. The corrected ranges are ρ ∈ [0.9915, 1.6501] and dormancy d ∈ [2.34%, 11.91%], now consistent with Figure 4.

Operationalized "intact capacity" as compliance with calibrated effective-rank and dormancy thresholds, rather than a guarantee of future learning capacity.

We have also clarified the threshold-calibration scope and acknowledged that the checkpoint audit measures immediate post-intervention drift, not isolated long-term cumulative drift.

We appreciate the reviewer pointing out these important issues. The revision narrows the claims to what the empirical evidence supports.

Thank you for your detailed response, the functional-drift audit, and the recomputed telemetry aligning Section 5.2 with Figure 4. I appreciate that you narrowed the claims regarding approximate preservation under ReDo (τ=0.10\tau = 0.10) and properly operationalized ``intact capacity'' as threshold compliance. While the newly reported tail critic drift and the limited evaluation scope remain minor limitations, your revisions have satisfactorily addressed my primary soundness concerns. Note that although the revised main text currently exceeds the 9-page submission limit, this is acceptable given that the official camera-ready version allows up to 10 pages. Accordingly, I have raised my rating to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.