- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose a “training guard” that is informative of whether a training run of a SAC-like algorithm with high updates-to-data ratio will eventually collapse. They find empirically that a reasonable such “guard” is simply checking whether the Q values are growing exponentially over training, in which case the run can be terminated early.
Critical strengths
None
Critical weaknesses
It is unclear what problem the paper is trying to solve. Indeed, a large amount of compute is wasted on training runs that eventually collapse, and rapid |Q| growth may reasonably correlate with this collapse. However, there is no causal mechanism explaining why this growth happens under some architectures or algorithm configurations but not others, and there is no practical guidance toward how this can be used to design better configurations. There are a variety of other potential metrics that would correlate with collapse, such as the TD error on held-out samples [1] from various points in training, which may be used to better define a notion of “plasticity loss”. Indeed, I am not aware of prior work indicating that rapid |Q| growth itself constitutes plasticity loss, rather than being a diagnostic.
Experimentally, I suspect there may be various bugs in the implementation or limitations in the architecture, and it may be helpful to work with a more well-established architecture such as [2,3,4], which have already been shown to scale well to UTDs larger than 16. Indeed, as shown in FIgure 4, the runs are much worse than the typical Walker2d performance, with a return around 200 instead of a more typical 5000 [5].
[1] https://arxiv.org/abs/2508.14881 [2] https://arxiv.org/abs/2405.16158 [3] https://arxiv.org/abs/2502.15280 [4] https://openreview.net/forum?id=OpC-9aBBVJe [5] https://arxiv.org/abs/2101.05982
Questions for authors
None
2 comments
Reply to Reviewer QNzF (2/2)
There are a variety of other potential metrics that would correlate with collapse, such as the TD error on held-out samples [1] from various points in training
We ran your suggestion on the new cohort, and the result is in our general comment. Held-out TD error at 15k scores about the same as our growth measure for a flag by 80k (AUC 0.604 against 0.653, paired p = 0.464), and on this cohort even our measure cannot be told apart from chance.
Indeed, I am not aware of prior work indicating that rapid |Q| growth itself constitutes plasticity loss, rather than being a diagnostic.
Agreed, and keeping the two apart is the paper's first claim. The introduction now calls the growth of the value magnitude a diagnostic of divergence and no longer places it under plasticity loss.
there is no causal mechanism explaining why this growth happens under some architectures or algorithm configurations but not others, and there is no practical guidance toward how this can be used to design better configurations
We have no causal mechanism, and the end of Section 4.2 now says so; the rate model that read like one has moved to the appendix. What we can report is what removes the flags in our runs. A LayerNorm critic does (on Walker2d at UTD ratio 4 the guard flags none of 32 LayerNorm critics, against 17 of the 32 plain critics on the same seeds). Lowering the UTD ratio at step 15k does. Switching the six implementation details of part 1 together also did, in an exploratory run. The UTD drop is the closest we have to guidance. On the new cohort it removed every flag without lowering the median return, but that is 24 seeds of one task, and we do not claim it holds elsewhere.
I suspect there may be various bugs in the implementation or limitations in the architecture, and it may be helpful to work with a more well-established architecture such as [2,3,4]
We agree the measurements should be repeated on one of the architectures you cite, and we have not done that yet. Our SimBa runs failed the reproduction check we had set for them (reply to JTwE), so we do not use them as evidence. For our own trainer, part 1 has what we know: it passes our check on Walker2d and HalfCheetah, fails it on Ant, and splits its seeds where a CleanRL port does not, for a reason we have not yet narrowed to one implementation detail.
Reply to Reviewer QNzF (1/2)
Thank you for the review.
The summary reads the training guard as our proposal, which is our fault for not making it clear. The guard is the stop rule our runs already use. It stops a run once its loss or |Q| explodes, and it defines the outcome we forecast. The problem is the one your review names, the compute spent on runs that eventually collapse. The paper asks which symptom warns of that collapse and how early. Dormant units are not that symptom: on HalfCheetah at UTD ratio 16, critics with most units dormant keep improving through 60k steps and are never flagged. The value magnitude is, since by step 15k it ranks the runs the guard will later stop, and the paper turns that ranking into a stopping rule for seed sweeps (points (2) and (3) of our general comment). The rule does not improve any single run.
Figure 4 shows 80k-step Walker2d runs at UTD ratio 4 with 2048-wide critics, plain and with LayerNorm. The trainer itself learns Walker2d, reaching 0.983 of the released SAC median at standard settings (next paragraph). The low return in Figure 4 belongs to a configuration we chose because in our trainer its seeds split, with the guard flagging about half of them within 80k steps, and a per-seed forecast needs that split. The 5000 is also for a much longer budget. The released REDQ curves reach a median of 2684 at about 80k steps and 5209 only at 1 million, and at 80k our runs are far below even 2684, with a median best training return of 449. The panel plots the training-episode return of the stochastic policy rather than a deterministic evaluation, but that is not the cause either, since the median evaluation return of the untreated runs in our new cohort is 237.1.
We ran the same trainer at standard settings: 256x2 critic, batch 256, UTD ratio 1, 300k steps, 5 seeds, the -v4 tasks of the released curves. Its median deterministic evaluation return is 3211.3 on Walker2d, 7630.7 on HalfCheetah and 2000.2 on Ant, which is 0.983, 0.893 and 0.468 of the released SAC medians. Our rule passed a task at 0.7 of that median and failed it below 0.5. Walker2d, the task of Figure 4, and HalfCheetah pass. Ant fails, and with it the trainer as a whole (Appendix B), which the abstract now says. The check covers standard settings, not our wide configuration, so on its own it cannot rule out a fault that shows up only in that configuration. For that we compared our trainer with a port of CleanRL's SAC.
In our Walker2d configuration the port behaves differently: within 80k steps the guard flags none of its 16 seeds, against 31 of 64 runs of our trainer. So whether the seeds split depends on implementation details as well as on the configuration, and Section 5 now states the forecast and the stopping rule for our trainer only. In an exploratory follow-up on the 12 seeds where our trainer flags 5, we set six implementation details to the port's choices at once (TF32 matrix multiplies, how the log standard deviation is bounded, two details of the temperature loss, the determinism flags, and the step of the first update), and then no run is flagged. We have not yet changed them one at a time, so we cannot say which matters.