acceptodds
Under review as a conference paper at ICLR 2027

Reset Cost Does Not Determine Reset Value The Role of Post Reset Update Budgets

Abstract

Neuron resets aim to restore plasticity in deep reinforcement learning, and recent work frames the choice of which units to reset as a cost–benefit problem with an estimable immediate cost. Does that cost predict a reset's long-run value? From common SAC checkpoints we measure, for matched reset interventions, the immediate change in the critic's TD loss and the returns that follow. An operator-aligned sensitivity score ranks the held-out loss change of random masks accurately (Spearman ), where a signed first-order prediction does not. This local accuracy does not yield a reliable ordering of long-run value: within matched checkpoint–pressure regimes, pre-registered tests detect no monotonic cost–value relation that is stable across tasks and update pressures. Across six DeepMind Control tasks, repeatedly resetting the highest-scoring units is substantially less harmful under high update pressure than under low pressure on every task, despite several times the cumulative cost. A history recovery-budget factorial on two tasks separates the pre-reset training history from the post-reset update budget: in an algebraic decomposition the recovery budget accounts for of the pooled high-versus-low-pressure difference, and training history contributes a smaller, task-dependent share. Immediate cost is therefore measurable but insufficient to decide whether a reset is worthwhile; its value depends on the learning that follows. A cost estimator settles which reset is cheap, not which one is worth making.

Primary area:
reinforcement learning
Keywords:
deep reinforcement learning, plasticity loss, neuron reset, update-to-data ratio, reset cost estimation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HbcYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies whether the immediate increase in critic TD loss after a neuron reset predicts its effect on policy return, and finds that the training budget after the reset matters a lot. Through paired reset/no-reset comparisons on six DeepMind Control tasks, separate tests of pre- and post-reset update budgets on two tasks, and a TD3 replication, it shows that more post-reset updates make costly resets much less harmful.

Critical strengths
  1. The paper demonstrates its main claim within the settings studied, showing that post-reset update budgets affect the benefit of a reset. The experimental comparisons and controls provide reasonable support for this conclusion.
  2. The paper provides a more detailed empirical understanding of resets through controlled comparisons of different neuron-selection rules. These comparisons characterize how the choice of neurons to reset affects both immediate critic disruption and subsequent policy performance under different update budgets.
Critical weaknesses
  1. The benefits of resets at high UTD and their connection to plasticity loss have already been studied in prior work (Nikishin et al., 2022; Sokar et al., 2023). The paper basically adds two incremental observations: post-reset UTD has a greater influence than pre-reset UTD in the tested high-cost reset settings, and immediate reset cost does not consistently predict subsequent return. However, the practical usefulness of these observations remains unclear. It is not evident how they inform better decisions about which neurons to reset, when to reset them, or how to allocate training updates. Most other results reinforce existing understanding and offer limited additional practical guidance.
  2. The presentation of this paper is quite poor. Although I am quite familiar with resets and the related issue of plasticity loss, I still found this paper quite difficult to read: the authors’ explanations are hard to follow. Many paragraphs are overloaded with numerical results and statistical details, making the main findings and their implications difficult to follow. For example, in “An intermediate-cost reset” in Section 5, the detailed numerical comparisons could be moved to a table, allowing the main text to focus on the key findings and their interpretation. I would not recommend acceptance without a substantial rewrite.
Questions for authors
  1. Why were a 10% reset fraction and a 10k-step reset interval chosen for the main repeated-reset experiments? Both the disruption and the recovery budget depend on reset fraction, strength, and frequency—for example, one hard reset over 150k steps and soft resets every 10k steps may have very different effects.
  2. The main comparisons fix the interaction horizon and vary the number of learning updates. How should the practical conclusions be interpreted when collecting rollouts is itself expensive, as in real-world robotics or LLM reinforcement learning? Are the recommendations limited to a fixed interaction budget, or is there evidence that they also hold under a fixed total resource budget that includes both rollout collection and learning updates?
1 comment
AuthorsOpenReview ↗

Response to Reviewer HbcY

The central contribution is an experimental separation of pre-reset history and post-reset recovery budgets. The factorial shows that the latter accounts for most of the strong reset's UTD effect in this design (Table 5). Together with the compute-matched 2×22\times2, this yields concrete evaluation guidance: report the post-reset budget and horizon, and compare against no-reset at equal compute.

W1 — Incremental contribution and unclear practical value.

  • Prior findings and our contribution. Resets enabling high replay ratios and additional updates accelerating recovery are established findings. Elisii et al. also distinguish immediate cost from delayed benefit; Related Work acknowledges this. The factorial separates history from recovery budget (IˉR=+38.1%\bar I_R=+38.1\% versus IˉH=+6.6%\bar I_H=+6.6\%; Table 5). This is a decomposition specific to the design, not a general causal attribution. We also show that immediate reset cost alone is insufficient to judge long-run value (Table 3; Section 7).
  • Practical implications for evaluation (closing paragraph of Section 9). (1) Report the post-reset budget and evaluate more than one level: it accounts for most of the strong reset's UTD effect in the factorial (Table 5), and measured reset value differs between UTD 1 and 8 (Table 3). (2) Specify the evaluation horizon: the harm of a single strong reset at low UTD is transient, with recovery by about 300k steps (App. W, Table 21); trajectory and endpoint metrics can also differ in sign. (3) Include a no-reset control at equal compute, including when testing recovery procedures. In the 2×22\times2, R=[reset+re-fit−no-reset+re-fit]−[reset−no-reset]R=[\text{reset+re-fit}-\text{no-reset+re-fit}]-[\text{reset}-\text{no-reset}] subtracts the re-fit gain without a reset from the gain with a reset. It is positive on both tasks (+25.1+25.1, 7/8; +49.0+49.0, 7/8), but reset+re-fit remains below no-reset+re-fit (−77.6-77.6; −400.2-400.2; Table 22). This does not isolate update count, timing, or the restriction to critic-only updates. (4) Report the evaluation grid: our 10k grid was phase-locked to the reset schedule and understated within-cycle harm (Table 24). We removed practical recommendations that went beyond these findings.
  • Which neurons to reset, and when. Low-contribution, activation bottom-KK, and threshold-ReDo selectors yield small positive benefits at UTD 8; at UTD 1, their benefits are not distinguishable from zero (Table 4). Reducing the strength of highC resets removes most of their UTD-1 harm, but a constant small strength does so as well (App. H). The strength experiment therefore does not establish an advantage for adaptive scaling over a constant small strength.

W2 — Writing. We consolidated definitions, moved numerical details into tables, and clarified the contribution; the general response details these revisions.

Q1 — Why reset 10% of units every 10k steps? These values were fixed before the confirmatory runs (App. U); the repeated-reset design draws on Nikishin et al. and Sokar et al., while the specific values were ours. Fixing the intervention provides a common basis for comparing reset value across update budgets. An interval sweep would also change the number of resets within the fixed horizon and the recovery time between them. We focused on separating history and recovery budgets, complemented by strength variations (App. H), full critic/agent resets (Table 22), single versus repeated resets (App. O), a 600k-step horizon, and resets every 20k critic updates (Table 21). These test intervention extent and recovery conditions; sensitivity across reset intervals remains to be quantified (Section 8).

Q2 — Fixed interaction horizon versus fixed total resources. Our conclusion concerns regime dependence under a fixed interaction budget; it is not a recommendation to increase UTD.

What is matched. Appendices N/O match update counts, not interaction counts. From 60k updates onward, the single-reset recovery contrast is not detected (CIs include zero). The 2×22\times2 matches critic update counts at the same interaction budget. Reset and no-reset arms should receive the same updates (Section 9).

What is not matched. We did not hold the total resource budget—rollout collection plus learning—fixed. Neither comparison establishes optimality in total time or cost.

Supplementary check. Non-detection is not equivalence. Post hoc two one-sided tests (not in the PDF; both tasks pooled; margin ±5.15%\pm5.15\%, half the matched-step effect in Table 12) support equivalence at 80k (p<0.001p<0.001) and ≈150k (p=0.006p=0.006), but not at 60k (p=0.20p=0.20).

Remaining limitations. The results are consistent with post-reset update supply, but the mechanism is not identified. The factorial and 2×22\times2 use two tasks and reused trunks. The selector-ranking change comes from one set of runs in which both budgets differed; it was not reproduced at fixed history. The scope is state-based SAC on DMC.

Reviewer_D9RTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how the update-to-data (UTD) ratio affects neuron-reset effectiveness in deep reinforcement learning. It separates immediate reset cost from long-term learning benefits. Experiments on six DeepMind Control tasks show that higher UTD ratios reduce the negative effects of costly resets.

Critical strengths
  1. The paper uses matched checkpoints, no-reset controls, and multiple random seeds. The factorial experiments help distinguish pre-reset training history from post-reset recovery.
  2. The results show that immediate reset cost alone cannot reliably predict long-term performance. The analysis provides useful insights into the role of post-reset training budgets.
Critical weaknesses
  1. The benefit of higher UTD ratios for reset recovery is already known. The main contribution is a more detailed empirical analysis of this relationship.
  2. Changing UTD also changes actor updates, target updates, and replay dynamics (Sections 3 and 6). The experiments do not fully isolate the effect of additional gradient updates.
  3. The first-order sensitivity score does not accurately estimate the magnitude of reset loss. Joint reset costs are also strongly non-additive (Appendices A–C). A group-aware or second-order analysis could strengthen the technical justification.
  4. The main experiments rely heavily on deliberately destructive high-cost resets. Continual Backprop and ReGraMa are compared as scoring criteria rather than full reset methods. This limits conclusions about practical reset selection.
  5. Most experiments use critic-only resets in state-based SAC. The main recovery analysis covers only two tasks. The evaluation schedule also misses some within-cycle performance drops (Appendix W).
Questions for authors
  1. Can the authors isolate the effect of post-reset gradient updates using a fixed replay stream and matched computational budgets?
  2. Do the findings hold for standard implementations of ReDo, ReGraMa, and Continual Backprop under identical reset schedules and UTD settings?
Reviewer_MBZuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how neuron reset criteria used to improve plasticity impacts the immediate performance of the newtork and its longer-term performance. The authors find that a large negative initial impact may not be worse than smaller dips after updates are performed. The update-to-data ratio is found to be an important factor, with large UTD eliminating the effect of large dips. Experiments are conducted in continuous control RL environments to assess different choices.

Critical strengths

The broad research direction is interesting: Better understanding how neuron reset criteria can impact immediate and long-term performance and generally how neurons contribute to the overall network's learning and predictions throughout training.

Critical weaknesses

The quality of the writing is a large detriment. It's difficult to understand the experiments and findings. Many extraneous details are included which crowd out the main ideas. It's easy to get lost in many useless details. The notation and organization is lacking since definitions are often scattered (e.g. did_i is not defined in the experimental settings, which is a key quantity to measure neuron contributions). Another more minor aspect is that unusual terminology is used e.g. "update pressure" for the update-to-data ratio. Does high update pressure meanin that the update budget is strained, that is, "under pressure" or is high pressure referring to extracting more progress from each datapoint? I would just refer directly to the UTD ratio as being high or low. e.g.2 Using "sham" to mean the "no reset" baseline.

Also, the experiments are missing some important algorithms. Many resetting criteria from the existing literature are not chosen such as Continual backprop, ReDO, GraMa, etc (these are even cited in the paper). Investigating these would be scientifically valuable to better understand their properties and how they may make different tradeoffs in the short and long-term. The current findings are less interesting than they could be, with additional investigations.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer MBZu

Thank you for the detailed comments on presentation and comparison with existing reset criteria. The revision foregrounds the central question—how neuron selection affects immediate and long-term performance—and adds one of the requested comparisons: ReDo's dormant-unit criterion at both UTD levels. W2 below states the scope of the comparison.

W1 — Writing, scattered definitions, and terminology.

  • Table 1 (Section 3) now consolidates the definitions of the reset conditions (arms, contribution score cic_i, reset fraction KK, schedule), the cost and evaluation metrics (dTDd_{TD}, AUC, pre-registered VregV_{\mathrm{reg}}, UTD contrast I~\tilde I), and the update budgets (history hh, recovery rr).
  • We replaced "update pressure" and its abbreviated form "pressure" with "UTD ratio" or "post-reset update budget", as appropriate, and "sham" with "no-reset control". The figures were relabelled accordingly; none retains "pressure" or "sham".
  • Numerical details from Section 5 ("An intermediate-cost reset" and the six-task result) are now in Tables 3–4; those from Section 6 are in Table 5. The accompanying text states each finding and its interpretation.
  • Section 1 opens with the research question. Related Work includes "What this paper adds", and Section 9 ends with a paragraph specifying what to control when evaluating a selector.

W2 — Existing reset criteria: CBP, ReDo, and GraMa.

  • ReDo's dormant-unit criterion (τ=0.025\tau=0.025). We evaluate it using the Section 5 confirmatory checkpoints at UTD 1 and 8 (Section 5.1, Table 4; both metrics in Table 22). At UTD 8, it is a low-cost selector with a small positive benefit: walker +6.8+6.8 [+4.7,+9.0][+4.7,+9.0], 8/8; cheetah +6.6+6.6 [+0.1,+15.0][+0.1,+15.0], 6/8. At UTD 1, its benefit is not distinguishable from zero. The UTD contrast is positive under both metrics but detected only under VregV_{\mathrm{reg}}, not AUC. Its profile resembles that of our low-cost arm; these results do not establish moderation for threshold selectors in general.
  • Implementation and scope. We use normalised activation ≤τ\le\tau, with τ=0.025\tau=0.025 from Sokar et al.'s DQN setting; their SAC setting uses τ=0\tau=0. We apply this criterion on our schedule with the same reset operator as highC. The configuration was fixed before the runs and was not tuned. This is not a reproduction of their SAC configuration.
  • Activation bottom-KK (lowA). Table 4 also reports our fixed-KK variant of the ReDo activation statistic, explicitly distinguished from a threshold rule.
  • Wider resets. Full critic (K=100%K=100\%) and full-agent resets extend the comparison to larger interventions. They show UTD moderation in the same direction but remain harmful at UTD 8 on cheetah (Table 22).
  • CBP and GraMa. These were not run as reset arms; Section 8 states this. Appendix B (Table 6) compares their utility and gradient-magnitude scores and bottom-KK sets on the same networks, which describes selection overlap rather than long-term performance. During the discussion period, we prioritised the ReDo-criterion comparison and the compute-matched 2×22\times2; arm-level CBP/GraMa comparisons are left to future work.

Remaining limitations. ReDo-criterion and wider-reset experiments reuse the Section 5 confirmatory checkpoints on two tasks. They broaden the interventions tested but are not independent replications.

The revision consolidates definitions and numerical results and adds a direct comparison with the ReDo criterion at both UTD levels, distinguishing immediate reset cost from measured long-term benefit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.