- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors look at plasticity-maintenance mechanisms, and especially shrink-and-perturb methods. Where these are typically applied in an open-loop fashion, the authors argue and show that this does not work well for offline to online RL. They propose conditional mechanisms instead, such as capacity-gate, which abstains from intervention when the representation capacity is intact. It monitors relative effective rank and activation dormancy. They also introduce zero-functional-shift neuron recycling (ReDo) for actuator crippling.
Critical strengths
- The central idea is powerful: not blindly applying the technique of shrink and perturb in the O2O scenario, as it may just destroy performance and behavior, impeding further learning. Proposing a conditional, closed-loop scheme is a good idea.
- It is shown that on some tasks, capacitygate can maintain performance.
- It is shown that on some tasks ReDo may be a good mechanism that does not perturb performance too much.
Critical weaknesses
-
In general, it seems that capacitygate can maintain performance by not applying shrink and perturb. In those cases, performance stays the same as before (a bit misleading to make it bold in Table 1, as the "none" method works just as well). It is less clear to me whether it can actually also allow interventions when it is necessary. Perhaps this is what is studied with the actuator crippling, but this is not clear to me. In table 2, capacitygate was only tested for one seed (why?) and has a low performance compared to fixed and redo.
-
Also the conclusion of fixed shrink and perturb vs redo is not clear to me: Figure 3 seems to say that sometimes redo is better, sometimes fixed. There is a mention of how important a stable gate is for bipedal walking, which may be an indication for a type of task where redo works better, but it seems hard to generalize that to any control task...
Questions for authors
-
It would be great if the authors could address my uncertainties for the main weaknesses.
-
The section on Relative Effective Rank Ratio is hard to follow. With penultimate, I guess the authors mean the penultimate layer (so last hidden layer). I could not find D anywhere, so I guess it is the dimension of that layer. My main question: What is the entropy of normalized singular values supposed to capture? Please explain it intuitively.
-
Dormancy. What I understood is that according to the definition, neurons are dormant if they have all negative activations in the batch. This may give many dormant neurons when batches are small... Also, does it not assume a fixed activation function, like heaviside or ReLU? Also here, more explanation would be welcome.
-
Why the hysteresis? Do the authors apply shrink and perturb when going over threshold? What happens when the value goes under the lower threshold, nothing? It is not clear if capacitygate is used with shrink and perturb or with redo or with both...
-
What are "stress" tests? The actuator crippling?
3 comments
Acknowledgement of rebuttal and revision
The authors have answered my questions and updated the manuscript accordingly, which has become clearer. The results now show a small benefit to performing SP / ReDo when these are actually performed even with CapacityGate. However, these positive differences remain very subtle (e.g., 8.35 vs 6.23 on HalfCheetah, where the maximum is with a fixed procedure achieving 15.27).
I will slightly higher my score to "weak accept".
Thank you for the constructive feedback and for recognizing the clarifications in our previous response.
We have further strengthened the actuator-stress evaluation by adding CAPACITYGATE results across three paired seeds, giving a balanced four-arm comparison across both environments.
The results clarify two distinct behaviors:
On HalfCheetah, the gate detects persistent representation-rank collapse and activates with cooldown-spaced interventions.
On Walker2d, it abstains when capacity indicators remain within the calibrated bounds, preserving baseline performance while Fixed Shrink-and-Perturb causes substantial degradation.
We also expanded the operator-level analysis. The 37-checkpoint ReDo audit now reports policy and critic functional drift, and the manuscript explicitly describes ReDo at τ = 0.10 as approximate, low-disruption preservation rather than exact function preservation.
Finally, we clarified the effective-rank and dormancy definitions, probe-distribution choices, hysteresis thresholds, cooldown behavior, and terminal evaluation mechanics.
We appreciate your engagement with the revision and hope the additional evidence and clarifications make the method's operating conditions and limitations sufficiently clear.
We thank the reviewer for recognizing the central motivation behind the Do Not Disturb principle and for identifying several places where the mechanism and experimental interpretation needed clarification. We have revised the manuscript and expanded the experimental evidence.
We agree that the original primary benchmark primarily demonstrated safe abstention: CAPACITYGATE preserved baseline performance by not applying Shrink-and-Perturb under stable and moderately shifted transfer. This should not be interpreted as evidence of active intervention benefit in those regimes.
We expanded the actuator-crippling evaluation to three seeds per environment. In HalfCheetah, CAPACITYGATE detects post-shift rank degradation and executes 6–7 Shrink-and-Perturb interventions, obtaining 8.35 ± 2.79. In Walker2d, it abstains across all three seeds and obtains 8.21 ± 2.38, matching the no-intervention baseline. These results illustrate task-dependent gating behavior; the HalfCheetah result is descriptive rather than statistically conclusive.
We clarified that the penultimate representation is the final hidden-layer activation representation. Its dimension D is the number of units in that layer.
The normalized singular values describe how the representation's variance is distributed across feature directions. Their entropy measures the effective number of directions carrying substantial representation variance: a more concentrated spectrum corresponds to fewer dominant directions, while a more uniform spectrum corresponds to a richer distribution of directions. Effective rank therefore provides a summary of representational dimensionality rather than simply counting nonzero units.
The relative effective-rank ratio compares the online representation with its offline anchor, allowing the diagnostic to detect changes relative to the pretrained representation.
We clarified the ReLU assumption underlying the dormancy definition. Under ReLU activations, outputs are non-negative, so a neuron with zero mean activation over the probe batch is inactive on that batch. We also clarify the operational threshold used in the implementation.
We agree that the estimate depends on the probe batch and its size. Dormancy is therefore treated as a diagnostic statistic over the specified probe distribution, not as proof that a neuron is permanently incapable of activation.
Hysteresis uses separate activation and deactivation thresholds to avoid rapid switching when telemetry fluctuates near a single threshold. Once the gate activates, it remains active until the recovery criteria are satisfied. The refractory cooldown prevents repeated interventions at every diagnostic evaluation.
We clarified that the gate controls the configured intervention operator. In the reported actuator-crippling HalfCheetah experiment, the active operator is Shrink-and-Perturb; ReDo is evaluated separately as an unconditional operator baseline. The gate does not automatically switch between SP and ReDo.
The stress tests refer to the non-stationary actuator-crippling experiments, in which a critical actuator is disabled during online adaptation. We have clarified the experimental context and the distinction between the operator comparisons and the gated intervention evaluation.
Finally, we added an unconditional ReDo stable-transfer comparison across 27 runs. ReDo obtains an IQM of 40.37 compared with 38.33 for the no-intervention baseline (paired W = 123.0, p = 0.117). This supports a more careful interpretation: intervention outcomes depend on the operator and task, rather than establishing that all periodic interventions are harmful.
We appreciate the reviewer's suggestions, which helped us make the diagnostic definitions, control logic, and scope of the conclusions more explicit.
Revised manuscript: Sections 3.2–3.4, 4, 5.1–5.4, and 6; Tables 2–3; Figures 3–4; Appendices C–E.