acceptodds
Under review as a conference paper at ICLR 2027

Rank Collapse Is Recoverable, Growing Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic

Abstract

Raising the update-to-data (UTD) ratio breaks off-policy critics in two ways grouped as "plasticity loss": collapsing representations and growing value magnitude . We separate them in Soft Actor-Critic (SAC) with scaled critics (width 2048, no normalization). Collapse is survivable: at UTD ratio 16, HalfCheetah critics with most units dormant keep learning, and the training guard, which stops runs whose loss or explodes, never flags them. Within one high-UTD SAC configuration, runs start close together, and how far a critic's has climbed by step 15k, its early growth, ranks the runs by how soon the guard flags them. At 15k, a flagged run's sits a median of over a hundredfold below its flag level, yet the climb's rate already orders the flags (Harrell's C and out-of-sample AUC 0.78 on Walker2d, 0.98 on Ant, at UTD ratio 4). Dormancy does not. Aborting on this rate saves about a tenth of held-out Walker2d compute and stays net-positive live. A LayerNorm critic lowers the rate, removes the flag on Walker2d at UTD ratio 4 and lowers the return.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, soft actor-critic, update-to-data ratio, value divergence, plasticity loss, representation collapse, early warning, training instability, compute efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QNzFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a “training guard” that is informative of whether a training run of a SAC-like algorithm with high updates-to-data ratio will eventually collapse. They find empirically that a reasonable such “guard” is simply checking whether the Q values are growing exponentially over training, in which case the run can be terminated early.

Critical strengths

None

Critical weaknesses

It is unclear what problem the paper is trying to solve. Indeed, a large amount of compute is wasted on training runs that eventually collapse, and rapid |Q| growth may reasonably correlate with this collapse. However, there is no causal mechanism explaining why this growth happens under some architectures or algorithm configurations but not others, and there is no practical guidance toward how this can be used to design better configurations. There are a variety of other potential metrics that would correlate with collapse, such as the TD error on held-out samples [1] from various points in training, which may be used to better define a notion of “plasticity loss”. Indeed, I am not aware of prior work indicating that rapid |Q| growth itself constitutes plasticity loss, rather than being a diagnostic.

Experimentally, I suspect there may be various bugs in the implementation or limitations in the architecture, and it may be helpful to work with a more well-established architecture such as [2,3,4], which have already been shown to scale well to UTDs larger than 16. Indeed, as shown in FIgure 4, the runs are much worse than the typical Walker2d performance, with a return around 200 instead of a more typical 5000 [5].

[1] https://arxiv.org/abs/2508.14881 [2] https://arxiv.org/abs/2405.16158 [3] https://arxiv.org/abs/2502.15280 [4] https://openreview.net/forum?id=OpC-9aBBVJe [5] https://arxiv.org/abs/2101.05982

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Reply to Reviewer QNzF (1/2)

Thank you for the review.

It is unclear what problem the paper is trying to solve.

The summary reads the training guard as our proposal, which is our fault for not making it clear. The guard is the stop rule our runs already use. It stops a run once its loss or |Q| explodes, and it defines the outcome we forecast. The problem is the one your review names, the compute spent on runs that eventually collapse. The paper asks which symptom warns of that collapse and how early. Dormant units are not that symptom: on HalfCheetah at UTD ratio 16, critics with most units dormant keep improving through 60k steps and are never flagged. The value magnitude is, since by step 15k it ranks the runs the guard will later stop, and the paper turns that ranking into a stopping rule for seed sweeps (points (2) and (3) of our general comment). The rule does not improve any single run.

Indeed, as shown in FIgure 4, the runs are much worse than the typical Walker2d performance, with a return around 200 instead of a more typical 5000 [5].

Figure 4 shows 80k-step Walker2d runs at UTD ratio 4 with 2048-wide critics, plain and with LayerNorm. The trainer itself learns Walker2d, reaching 0.983 of the released SAC median at standard settings (next paragraph). The low return in Figure 4 belongs to a configuration we chose because in our trainer its seeds split, with the guard flagging about half of them within 80k steps, and a per-seed forecast needs that split. The 5000 is also for a much longer budget. The released REDQ curves reach a median of 2684 at about 80k steps and 5209 only at 1 million, and at 80k our runs are far below even 2684, with a median best training return of 449. The panel plots the training-episode return of the stochastic policy rather than a deterministic evaluation, but that is not the cause either, since the median evaluation return of the untreated runs in our new cohort is 237.1.

We ran the same trainer at standard settings: 256x2 critic, batch 256, UTD ratio 1, 300k steps, 5 seeds, the -v4 tasks of the released curves. Its median deterministic evaluation return is 3211.3 on Walker2d, 7630.7 on HalfCheetah and 2000.2 on Ant, which is 0.983, 0.893 and 0.468 of the released SAC medians. Our rule passed a task at 0.7 of that median and failed it below 0.5. Walker2d, the task of Figure 4, and HalfCheetah pass. Ant fails, and with it the trainer as a whole (Appendix B), which the abstract now says. The check covers standard settings, not our wide configuration, so on its own it cannot rule out a fault that shows up only in that configuration. For that we compared our trainer with a port of CleanRL's SAC.

In our Walker2d configuration the port behaves differently: within 80k steps the guard flags none of its 16 seeds, against 31 of 64 runs of our trainer. So whether the seeds split depends on implementation details as well as on the configuration, and Section 5 now states the forecast and the stopping rule for our trainer only. In an exploratory follow-up on the 12 seeds where our trainer flags 5, we set six implementation details to the port's choices at once (TF32 matrix multiplies, how the log standard deviation is bounded, two details of the temperature loss, the determinism flags, and the step of the first update), and then no run is flagged. We have not yet changed them one at a time, so we cannot say which matters.

AuthorsOpenReview ↗

Reply to Reviewer QNzF (2/2)

There are a variety of other potential metrics that would correlate with collapse, such as the TD error on held-out samples [1] from various points in training

We ran your suggestion on the new cohort, and the result is in our general comment. Held-out TD error at 15k scores about the same as our growth measure for a flag by 80k (AUC 0.604 against 0.653, paired p = 0.464), and on this cohort even our measure cannot be told apart from chance.

Indeed, I am not aware of prior work indicating that rapid |Q| growth itself constitutes plasticity loss, rather than being a diagnostic.

Agreed, and keeping the two apart is the paper's first claim. The introduction now calls the growth of the value magnitude a diagnostic of divergence and no longer places it under plasticity loss.

there is no causal mechanism explaining why this growth happens under some architectures or algorithm configurations but not others, and there is no practical guidance toward how this can be used to design better configurations

We have no causal mechanism, and the end of Section 4.2 now says so; the rate model that read like one has moved to the appendix. What we can report is what removes the flags in our runs. A LayerNorm critic does (on Walker2d at UTD ratio 4 the guard flags none of 32 LayerNorm critics, against 17 of the 32 plain critics on the same seeds). Lowering the UTD ratio at step 15k does. Switching the six implementation details of part 1 together also did, in an exploratory run. The UTD drop is the closest we have to guidance. On the new cohort it removed every flag without lowering the median return, but that is 24 seeds of one task, and we do not claim it holds elsewhere.

I suspect there may be various bugs in the implementation or limitations in the architecture, and it may be helpful to work with a more well-established architecture such as [2,3,4]

We agree the measurements should be repeated on one of the architectures you cite, and we have not done that yet. Our SimBa runs failed the reproduction check we had set for them (reply to JTwE), so we do not use them as evidence. For our own trainer, part 1 has what we know: it passes our check on Walker2d and HalfCheetah, fails it on Ant, and splits its seeds where a CleanRL port does not, for a reason we have not yet narrowed to one implementation detail.

Reviewer_JTwEOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates two different phenomena that are typically associated with "plasticity loss" in deep reinforcement learning: collapsing representations and growing value magnitudes. The authors show that representation collapse, characterized by low effective rank and a high ratio of dormant neurons, does not necessarily lead to training failure, whereas rapid growth in ∣Q∣|Q| is strongly associated with subsequent critic divergence. Based on this observation, the authors propose an early-warning signal using the growth rate of the ∣Q∣|Q| during early training. This signal is used to predict future critic divergence and develop a simple early-stopping mechanism to reduce wasted computation.

Critical strengths

None

Critical weaknesses
  1. The empirical study mostly relies on large, unnormalized critics. However, a lot of recent high-UTD methods show that normalization is very helpful. For example, Simba [1] employs normalization and other stabilization techniques. This makes the findings and results questionable when applied to more recent and stronger methods.

[1] Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning. ICLR 2025.

  1. The proposed method is useful primarily when conducting large-scale experiments with many training runs. It does not improve the performance or stability of an individual training process.

  2. The proposed early-growth signal does not consistently outperform simpler alternatives. For example, in Table S6, on Walker2d, its AUC is 0.780, compared with 0.788 for the log⁡10\log_{10}|Q| and 0.792 for the post-learning effective-rank drop.

Questions for authors
  1. Can the proposed early-growth signal reliably predict divergence in stronger methods such as Simba?
  2. Instead of simply terminating runs with rapidly growing Q-values, could the proposed signal be used to trigger interventions?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer JTwE

Thank you for the review.

The proposed early-growth signal does not consistently outperform simpler alternatives. For example, in Table S6, on Walker2d, its AUC is 0.780, compared with 0.788 for the log10|Q| and 0.792 for the post-learning effective-rank drop.

You are right, and the revision no longer claims that the growth measure beats the level of |Q|, the training TD loss or the rank drop. The growth and the level of |Q| rank the runs about equally (Harrell's C 0.78 and 0.80 on Walker2d, 0.98 and 0.98 on Ant). What the revision keeps is that the value magnitude ranks the runs at step 15k while the dormant fraction does not, and that the magnitude is cheaper than the rank drop, since the trainer already logs it and the rank drop needs a probe batch and an SVD. The new cohort in our general comment points the same way as Table S6: a paired test does not separate held-out TD error from the growth measure.

Instead of simply terminating runs with rapidly growing Q-values, could the proposed signal be used to trigger interventions?

We tested this on the new cohort (general comment). A drop from UTD ratio 4 to 1 at step 15k removes every flag when all runs get it. When only the runs above our threshold get it, 3 of the 12 diverging seeds are missed. The signal did not earn a role as a trigger, and the revision keeps it for stopping runs in a sweep.

The empirical study mostly relies on large, unnormalized critics. However, a lot of recent high-UTD methods show that normalization is very helpful. For example, Simba [1] employs normalization and other stabilization techniques. This makes the findings and results questionable when applied to more recent and stronger methods. Can the proposed early-growth signal reliably predict divergence in stronger methods such as Simba?

We agree that normalization stabilizes high-UTD critics. In our trainer, on Walker2d at UTD ratio 4, the guard flagged none of 32 LayerNorm critics within 80k steps, against 17 of the 32 plain critics on the same seeds, so the forecast had nothing to catch. We have not run LayerNorm at higher ratios. The LayerNorm critic also ended at about a third of the plain critic's return on the pairs where neither was flagged (Section 4.4).

On SimBa we tried, and cannot answer yet. We ran SimBa at UTD ratio 32 with a positive control fixed in advance, and the control failed: SimBa at UTD ratio 8, on two seeds, had a median evaluation return of 403.7 at 50k steps, outside the range of the five released seeds at that point (754.9 to 2555.2). So our harness does not reproduce SimBa's released returns, and we draw no conclusion from the UTD 32 runs.

The abstract and Section 5 now limit the scope to wide unnormalized critics in our own trainer. A port of CleanRL's SAC at our Walker2d settings flags none of its 16 seeds within 80k steps, so the paper states the forecast for our trainer only.

The proposed method is useful primarily when conducting large-scale experiments with many training runs. It does not improve the performance or stability of an individual training process.

Agreed, and Section 4.3 now says the rule does not help any single run. Its use is across a seed sweep, where it stops early the runs the guard would stop later; point (3) of our general comment gives the saving. Fitting the threshold took 64 runs whose outcome is known, which the saving repays only after 584 further runs of the configuration (Section 4.3).

I think the current manuscript is not yet suitable for publication, particularly because the study based on unnormalized critics limits the informativeness of the findings. I encourage the authors to further strengthen the work and submit a revised version to a future venue.

AuthorsOpenReview ↗

The revision trains normalized critics too

Thank you for reading our reply. The revised PDF is no longer limited to unnormalized critics. Section 4.4 trains 32 seeds with a LayerNorm critic alongside the plain one. No LayerNorm run diverges, but where neither critic diverges, LayerNorm ends at about a third of the plain critic's return. Normalization is therefore not a free fix in this setting, and a forecast for the plain critic remains useful.

Reviewer_cHnvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that when a critic's average value estimates |Q| grows quickly, it flags divergent runs (unstable training) better than dormancy units (activated neutrons) and not consistency better than rank drop (critic's representation distinction). The authors also show that a LayerNorm reduces that growths, hence avoiding divergent runs, but the reduced the total returned reward (performance drop).

Critical strengths

The motivation is clear. There might be a case for why to use |Q| as a failure flagging signal over rank drop (cost / computations).

Many experiments were conducted to show why |Q| is a better failure flagging signal over activated neurons and critic's rank drop.

Critical weaknesses

The saved compute received by using LayerNorm critic is not studied well against the overall reduced return (total reward, performance drop), so not sure if LayerNorm is a valid proposed solution or not?

Figure 1 contains a lot of information, it takes so much time to understand the key message authors want to convey.

Even though the overall idea is simple, yet it was a bit hard to follow the paper, it took longer than it should be to understand.

Questions for authors

What are the other ways |Q| quick growth can be leveraged to save compute while maintain return cost?

Any future work , extension of this work?

If authors summarise their main findings and contributions, what it would be?

Any theoretical analysis for why LayerNorm would stabilise critic training?

With the LayerNorm integrated in the Critic, which signal is better for flagging failed runs, rank drop or |Q|?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer cHnv

Thank you for the review, and for saying where the paper lost you.

Figure 1 contains a lot of information, it takes so much time to understand the key message authors want to convey. Even though the overall idea is simple, yet it was a bit hard to follow the paper

Fair on both. Figure 1 is redrawn with larger text, and its caption now opens with its one message. The three result figures are redrawn too, the abstract, introduction and conclusion are rewritten, and the rate model has moved to the appendix.

If authors summarise their main findings and contributions, what it would be?

They are the three claims at the top of our general comment. Briefly: a critic can keep learning after its representation collapses, so collapse is not divergence. The value magnitude at step 15k ranks which seeds the guard will stop, and the dormant fraction does not. Stopping on that ranking saves about a tenth of the compute of a Walker2d seed sweep. The magnitude is also free to log, which, as your review suggests, is its advantage over the rank drop. The ranking scores lower in our newest cohort than in the earlier ones, and the general comment gives those numbers.

The saved compute received by using LayerNorm critic is not studied well against the overall reduced return (total reward, performance drop), so not sure if LayerNorm is a valid proposed solution or not?

We do not propose it. In this configuration LayerNorm trades return for stability, and Section 4.4 now reports both sides. It saves compute because the guard stops none of the LayerNorm runs, so a million environment steps buys 12.5 complete 80k-step runs with LayerNorm and 6.9 without. Each of those runs, though, learns less. On the seed pairs where the guard flags neither critic, the LayerNorm critic ends at about a third of the plain critic's return. If each flagged plain run is credited with its best 10k window before its flag, the median best-window return is 195.1 with LayerNorm and 307.0 without, and LayerNorm is higher in only 10 of the 32 pairs. We have not tested whether a tuned normalized agent gives up return in the same way.

With the LayerNorm integrated in the Critic, which signal is better for flagging failed runs, rank drop or |Q|?

Neither, in this configuration. Across the 32 seed pairs the guard flags no LayerNorm run and 17 plain runs, so with LayerNorm there is no failure for either signal to predict.

What are the other ways |Q| quick growth can be leveraged to save compute while maintain return cost?

We tried one on the new cohort in our general comment, lowering the UTD ratio from 4 to 1 at step 15k. Lowered on every run, it removed every flag at a higher median return and well under half the gradient updates. Lowered only on the runs our signal picks, it missed 3 of the 12 seeds that diverge and took more updates (4.3 million against 2.5 million), because the runs it left alone stayed at UTD ratio 4.

Any theoretical analysis for why LayerNorm would stabilise critic training?

The appendix gives one bound: with LayerNorm before each ReLU, the magnitude can grow only through the output layer and the last LayerNorm's gain and bias, while without it any layer can raise it. It is not a stability result, and it is loose at our numbers of updates. Empirically LayerNorm lowers the growth, the post-learning rank drop and dormancy together, so we cannot say which of them removes the flag.

Any future work , extension of this work?

First, finding out why seeds with identical settings climb at different speeds, which the paper can rank but does not explain. Second, the same measurements on a tuned normalized agent such as SimBa; our first attempt did not reproduce SimBa's released returns, so that question is still open.

AuthorsOpenReview ↗

The LayerNorm trade-off in the revised PDF

The revised PDF is up. Section 4.4 weighs what LayerNorm saves against what it costs, on 32 seeds trained with both critics. No LayerNorm run diverges, so a million environment steps buys 12.50 complete runs against 6.90 for the plain critic. Where neither critic diverges, though, LayerNorm ends at about a third of the plain critic's return. So LayerNorm pays for stability on every run, while Q-climb stopping pays only on the runs it stops.

Comments on the paper

2 comments
AuthorsOpenReview ↗

What the revision claims, and three new results

We thank the three reviewers. The revision states its main claims in plainer words and drops a comparative one. The claims are about SAC with wide unnormalized critics in our own trainer.

(1) A collapsing representation and a growing value magnitude are different failures. At UTD ratio 16 on HalfCheetah, critics with most units dormant keep raising their return through 60k steps, and the guard (the stop rule our runs already use) flags none of them.

(2) How far the value magnitude has climbed by step 15k ranks the runs by how soon the guard flags them (Harrell's C 0.78 on Walker2d and 0.98 on Ant at UTD ratio 4). The dormant fraction at that step forecasts no better than chance. At step 15k a flagged run's magnitude still has, at the median, more than a hundredfold to grow before its flag.

(3) Stopping runs on this ranking saves about a tenth of the compute of a Walker2d seed sweep on held-out seeds, with the threshold fit on other seeds whose outcome is known. With that threshold fixed in advance, it saved compute again live on fresh seeds.

The revision no longer says that this growth measure forecasts better than the level of |Q|, the training TD loss or the rank drop.

Two limits apply to all of this, and our reply to QNzF has both. Our trainer fails our check against the released SAC results because of Ant, where it reaches 0.468 of the released median (0.983 on Walker2d, 0.893 on HalfCheetah). And a port of CleanRL's SAC, run in our Walker2d configuration, has none of its 16 seeds flagged within 80k steps, so the seed split is so far specific to our trainer.

All three reviews found the paper hard to read in places, so the abstract, introduction and conclusion are rewritten, the rate model has moved to the appendix, Figure 1 is redrawn with larger text, and the three result figures are redrawn.

This week a further cohort of 24 Walker2d seeds at UTD ratio 4 finished, run to a plan we wrote before it started. It tests intervening instead of stopping (JTwE, cHnv) and held-out TD error as a competing metric (QNzF).

Each seed ran three times: untreated, with a critic reset at step 15k, and with a drop from UTD ratio 4 to 1 at step 15k. The guard flags 12 of the 24 untreated runs. With the UTD drop on every run it flags none, the median evaluation return goes from 237.1 to 263.7, and the cohort takes 2.5 million gradient updates instead of 6.3 million. The reset on every run also leaves none flagged, at a return of 186.5 and 7.2 million updates. Treating only the 14 runs above our threshold catches 9 of the 12 seeds that diverge and misses 3. Our plan allowed the targeted version no more flags than treating every run, so it fails, and the revision does not use the signal to choose which runs to treat (Section 4.3), only which runs of a sweep to stop. Stopping still has a place in a sweep over a fixed configuration, because it leaves the runs it keeps exactly as configured.

Held-out TD error, logged on 200 held-out transitions per run and read at 15k, has an AUC of 0.604 for a flag by 80k, against 0.653 for our growth measure, and a paired test does not separate them (p = 0.464). The 0.653 is below the 0.71 to 0.88 of our earlier Walker2d cohorts, and at 24 seeds it cannot be told apart from chance. These are AUCs without fitting. The paper's usual leave-one-seed-out fit is not informative at 24 seeds (a model with no feature scores 0.000 under it), so Section 4.2 reads this cohort without fitting, and Appendices E.4 and L print the fitted values, 0.160 for held-out TD error and 0.292 for our growth measure. We made that call after seeing these values.

Section 4.4 now weighs the compute a LayerNorm critic saves against the return it costs (cHnv), and the introduction treats the growth of the value magnitude as a diagnostic of divergence, not as a form of plasticity loss (QNzF).

The revised PDF will be uploaded within the next two days. Section numbers in our replies refer to it: the submitted Section 4.3 has moved to the appendix, so the submitted 4.4 and 4.5 are now 4.3 and 4.4.

AuthorsOpenReview ↗

Revised PDF uploaded

The revised PDF has a new title, "Dormant but Not Diverging: Early Value Growth Forecasts Which Seeds Fail in High-UTD Soft Actor-Critic". Dormancy does not predict divergence; the Q-climb, the rate at which |Q| grows over the first 10k steps of learning, does. At step 15k, long before any run nears divergence, it ranks Walker2d and Ant seeds by when they diverge, with Harrell's C of 0.78 and 0.98. Q-climb stopping saves 9.9% of a Walker2d sweep's compute under cross-validation and 8.8% live on new seeds. Its threshold fires on 45 of 84 new runs of our trainer but on only 2 of 72 where no seed diverges (Section 4.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.