- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies whether the immediate increase in critic TD loss after a neuron reset predicts its effect on policy return, and finds that the training budget after the reset matters a lot. Through paired reset/no-reset comparisons on six DeepMind Control tasks, separate tests of pre- and post-reset update budgets on two tasks, and a TD3 replication, it shows that more post-reset updates make costly resets much less harmful.
Critical strengths
- The paper demonstrates its main claim within the settings studied, showing that post-reset update budgets affect the benefit of a reset. The experimental comparisons and controls provide reasonable support for this conclusion.
- The paper provides a more detailed empirical understanding of resets through controlled comparisons of different neuron-selection rules. These comparisons characterize how the choice of neurons to reset affects both immediate critic disruption and subsequent policy performance under different update budgets.
Critical weaknesses
- The benefits of resets at high UTD and their connection to plasticity loss have already been studied in prior work (Nikishin et al., 2022; Sokar et al., 2023). The paper basically adds two incremental observations: post-reset UTD has a greater influence than pre-reset UTD in the tested high-cost reset settings, and immediate reset cost does not consistently predict subsequent return. However, the practical usefulness of these observations remains unclear. It is not evident how they inform better decisions about which neurons to reset, when to reset them, or how to allocate training updates. Most other results reinforce existing understanding and offer limited additional practical guidance.
- The presentation of this paper is quite poor. Although I am quite familiar with resets and the related issue of plasticity loss, I still found this paper quite difficult to read: the authors’ explanations are hard to follow. Many paragraphs are overloaded with numerical results and statistical details, making the main findings and their implications difficult to follow. For example, in “An intermediate-cost reset” in Section 5, the detailed numerical comparisons could be moved to a table, allowing the main text to focus on the key findings and their interpretation. I would not recommend acceptance without a substantial rewrite.
Questions for authors
- Why were a 10% reset fraction and a 10k-step reset interval chosen for the main repeated-reset experiments? Both the disruption and the recovery budget depend on reset fraction, strength, and frequency—for example, one hard reset over 150k steps and soft resets every 10k steps may have very different effects.
- The main comparisons fix the interaction horizon and vary the number of learning updates. How should the practical conclusions be interpreted when collecting rollouts is itself expensive, as in real-world robotics or LLM reinforcement learning? Are the recommendations limited to a fixed interaction budget, or is there evidence that they also hold under a fixed total resource budget that includes both rollout collection and learning updates?
Response to Reviewer HbcY
The central contribution is an experimental separation of pre-reset history and post-reset recovery budgets. The factorial shows that the latter accounts for most of the strong reset's UTD effect in this design (Table 5). Together with the compute-matched 2×2, this yields concrete evaluation guidance: report the post-reset budget and horizon, and compare against no-reset at equal compute.
W1 — Incremental contribution and unclear practical value.
W2 — Writing. We consolidated definitions, moved numerical details into tables, and clarified the contribution; the general response details these revisions.
Q1 — Why reset 10% of units every 10k steps? These values were fixed before the confirmatory runs (App. U); the repeated-reset design draws on Nikishin et al. and Sokar et al., while the specific values were ours. Fixing the intervention provides a common basis for comparing reset value across update budgets. An interval sweep would also change the number of resets within the fixed horizon and the recovery time between them. We focused on separating history and recovery budgets, complemented by strength variations (App. H), full critic/agent resets (Table 22), single versus repeated resets (App. O), a 600k-step horizon, and resets every 20k critic updates (Table 21). These test intervention extent and recovery conditions; sensitivity across reset intervals remains to be quantified (Section 8).
Q2 — Fixed interaction horizon versus fixed total resources. Our conclusion concerns regime dependence under a fixed interaction budget; it is not a recommendation to increase UTD.
What is matched. Appendices N/O match update counts, not interaction counts. From 60k updates onward, the single-reset recovery contrast is not detected (CIs include zero). The 2×2 matches critic update counts at the same interaction budget. Reset and no-reset arms should receive the same updates (Section 9).
What is not matched. We did not hold the total resource budget—rollout collection plus learning—fixed. Neither comparison establishes optimality in total time or cost.
Supplementary check. Non-detection is not equivalence. Post hoc two one-sided tests (not in the PDF; both tasks pooled; margin ±5.15%, half the matched-step effect in Table 12) support equivalence at 80k (p<0.001) and ≈150k (p=0.006), but not at 60k (p=0.20).
Remaining limitations. The results are consistent with post-reset update supply, but the mechanism is not identified. The factorial and 2×2 use two tasks and reused trunks. The selector-ranking change comes from one set of runs in which both budgets differed; it was not reproduced at fixed history. The scope is state-based SAC on DMC.