acceptodds
Under review as a conference paper at ICLR 2027

Just Stack It: Deeper Reinforcement Learning with Near-Isometric Layers

Abstract

Deep networks have driven progress in vision and language, yet reinforcement learning (RL) agents still rely on networks a few layers deep, because adding layers degrades their performance. We show that this failure results from a loss of conditioning: during training, the Jacobian of a deep network's hidden layers moves away from an isometry, its features lose rank, and the deviations of the layers multiply with depth. Under the non-stationary, bootstrapped objective of RL, this degradation impairs the fitting of later targets. The distance of the Jacobian from an isometry and the rank of the features track performance across architectures and optimizers. We propose a simple residual layer that keeps deep networks close to an isometry. With it, deep networks train stably across value-based, actor-critic and goal-conditioned algorithms and optimizers, and their performance improves or remains stable with depth: they outperform the standard shallow networks and recent deep architectures at matched depth and parameters.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, neural networks, deep learning, isometry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ocLiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a residual layer that keeps deep networks close to an isometry, making deep networks more stable in reinforcement learning. The paper first analyzes why increasing network depth fails in RL and then proposes their residual layer. In the large-scale empirical study, the layer is shown to bring benefits across value-based, actor-critic, and goal-conditioned algorithms and optimizers when scale up the depth of networks.

Critical strengths
  1. The analysis of why increasing the depth of deep networks fails in RL is clear and insightful and could be helpful for future work on this topic.
  2. The empirical study is extensive, covering many baseline algorithms (PQN, SAC, TD3, and PPO) and different benchmarks (Atari, DM Control, MuJoCo Playground, Craftax-Symbolic, and JAXGCRL), showing the generality of the proposed layer, which can work in different setups.
Critical weaknesses
  1. Although the empirical study is extensive, most results use only three random seeds. I strongly encourage the authors to run more random seeds to ensure the reliability of the results.
  2. There is a highly relevant paper [1] about enforcing Isotropic Gaussian representations that is mentioned in the paper but is not directly compared in the empirical study. Although the authors of [1] did not try combining their method with deeper networks, I think it is worth a try. Could the authors provide a direct comparison?

[1] Stable Deep Reinforcement Learning via Isotropic Gaussian Representations. ICML 2026

Questions for authors

See Weaknesses.

2 comments

We thank the reviewer for the positive assessment of our analysis and experiments. We ran both requested experiments and report them below.

W1 (seeds). We added two seeds, so every main result now uses five seeds. The only exception is goal-conditioned RL, where we keep three seeds to match the protocol of Wang et al. (2025), whose published curves we compare to. Scores are IQM [95% CI]. There is no qualitative difference with the 3-seed results of the paper:

PublishedPrior deep (best depth)Ours, depth 32
Atari (HNS), PQN0.77 [0.72, 0.84]0.88 [0.85, 0.90]0.95 [0.91, 0.98]
Atari (HNS), PPO0.70 [0.68, 0.73]0.79 [0.74, 0.84]1.04 [0.96, 1.11]
DMC, SAC1.39 [1.30, 1.56]1.86 [1.80, 1.93]2.20 [2.09, 2.36]
DMC, PPO1.07 [1.01, 1.15]0.87 [0.77, 1.00]1.30 [1.18, 1.42]
Locomotion, SAC1.33 [1.29, 1.38]2.01 [1.95, 2.08]2.33 [2.27, 2.38]
DMC, TD31.01 [0.93, 1.12]–1.84 [1.70, 2.19]
Craftax, PQN16.3 [15.9, 17.8]–23.3 [23.2, 23.8]

The earlier submission prioritized breadth: five algorithms, five benchmark suites (72 task settings), three optimizers and a wide range of depths. Our network has the highest aggregate score at depth 32 on every benchmark suite and with every algorithm, which seed variation alone is unlikely to produce. All scores are IQM with stratified bootstrap confidence intervals (Agarwal et al., 2021), the recommended protocol for reliable comparisons in deep RL: on the multi-task suites each score pools all runs of all tasks (40 to 80 runs at five seeds), the IQM is robust to outlier runs, and the intervals account for the variation across seeds.

W2 (SIGReg). As suggested, we combined SIGReg with deeper networks: plain networks of depth 6, 10 and 32, a multi-skip network and ours at depth 32, on Atari-10 with PQN. We use the SIGReg loss from the authors' released code, unchanged, with their two regularization strengths. As in their code, the loss acts on the features that enter the final linear layer. Scores are IQM (HNS). The SIGReg columns use three seeds and the column without SIGReg five; the five-seed SIGReg version with the full table will be in the final paper.

NetworkNo SIGRegλ = 0.2λ = 1.0
Published0.770.660.51
Plain, depth 6 / 10 / 320.10 / 0.02 / −0.020.16 / 0.03 / −0.020.05 / 0.03 / −0.03
Multi-skip, depth 320.640.530.23
Ours, depth 320.950.700.60

The released code of Saheb Pasand et al. runs PQN with a plain network of 8 hidden layers, between our plain networks of depth 6 and 10. In their Atari results this network scores about 0.00 at the end of training and about 0.22 with SIGReg. We see the same on our plain networks: they score close to zero without SIGReg, and λ = 0.2 raises them slightly (0.10 to 0.16 at depth 6, 0.02 to 0.03 at depth 10), which supports that our implementation matches theirs. In their results and in our reimplementation, however, a plain network with SIGReg stays far below the published network with one hidden layer (0.77), and at depth 32 SIGReg changes nothing. On the three networks that train well without it (the published network, the multi-skip network and ours), SIGReg lowers the score at both strengths.

This is consistent with our analysis: SIGReg acts only on the features of the last hidden layer, whereas training a deep network also requires well-conditioned Jacobians in all hidden layers, which our layer provides. We thank the reviewer for suggesting this comparison; it makes the paper more complete.

We are adding both experiments to the revised paper. We hope this addresses the reviewer's concerns, and we would be grateful if they would consider updating their rating.

Thanks for the detailed rebuttal and clarification, which addressed and resolved my concerns. I will update the score accordingly.

Reviewer_aJTdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper attributes the failure of deep networks in RL to a loss of conditioning during training. The Jacobian of the hidden layers is not an isometry, which causes a loss of rank in the features, and the variances of each layer increase with depth. The authors analyze the tangent kernel across the layers to argue that both characteristics are needed and add a layer that combines a branch made of a single weight matrix, an RMS normalization and a shifted ReLU with weight 1/K into a pre-defined orthogonal skip connection. They prove that the singular values of the resulting network’s Jacobian lie within a range that does not depend on depth. Across PQN, PPO, SAC, TD3, and SAC+HER on Atari-10, DeepMind Control, Locomotion, Craftax, and JaxGCRL, with three optimizers, performance improves or remains stable with depth. At depth 32 and equal parameter count, the proposed network outperforms the published networks, SimbaV2, and the multi-skip network.

Critical strengths
  • The experiments are broad and the results are consistent in direction. Across five algorithms, five benchmarks, three optimizers, and depths from 1 to 64, the proposed network does not collapse with depth and outperforms SimbaV2 and the multi-skip network at equal parameter count.

  • The analysis and the method are closely aligned, and the claims are tested. The assumption of the bound (c_k ≤ K/2) is measured during training, and replacing the skip connection with fixed matrices of varying conditioning isolates the effect of the skip path’s conditioning on performance.

  • The paper is easy to follow. The diagnosis, analysis, layer design, and experiments follow a clear sequence, and each design choice is tied to the preceding analysis.

Critical weaknesses
  • The benefit of depth is not weighed against its cost. At depth 32, training time is about five times that of the published network for SAC and 1.5 to 2.0 times for Atari, and the only depth-versus-width comparison matches parameter counts rather than compute. The benefit also does not grow monotonically, since depth 16 outperforms depth 32 for PQN on Atari and PPO on DeepMind Control.

  • The main comparisons do not hold the optimizer and the tuning fixed. In Figure 2(d), the proposed network is trained with Muon while the baseline uses its published RAdam, and the baseline under the same optimizer appears only in the appendix. The proposed network uses settings chosen at depth 32, whereas prior networks use their published settings, and SimbaV2, tuned by its authors only up to depth 16, is evaluated at depth 32 without retuning.

  • The ablations and the goal-conditioned results support the claims only partly. The component ablation is run on Atari at depth 32 under Muon alone, and replacing the shifted ReLU changes the score by 0.04 to 0.05. In goal-conditioned RL, depth improves four of the eight tasks and leaves the others within 10%, and the comparison with contrastive RL selects the best depth of each method and relies on published numbers rather than reruns.

Questions for authors
  • Does the depth 32 network compare to wider, shallower networks (or those with higher update-to-data ratio) for the same wall clock time or FLOPs?

  • Could the main results be reported with the baseline and the proposed network under the same optimizer and SimbaV2 finetuned at depth 32?

  • Do the effects of the normalization and the shifted ReLU hold on benchmarks other than Atari, and how do the goal-conditioned results compare at a fixed depth rather than at the best depth of each method?

3 comments

We thank the reviewer for the positive remarks and the careful review. Several of the requested comparisons are already in the paper; we clarify and point to them below, and for the others we have added new results. (Scores are interquartile means with 95% bootstrap intervals, over three seeds unless stated.)

Q1. Depth at equal compute. Depth 32 costs about 5 times the wall-clock time of the published network for SAC, and 1.5 to 2 times on Atari (Table 12). The same compute spent on width or on more updates gives lower scores (SAC, DeepMind Control):

NetworkMatched to depth 32 onScore
Ours, depth 32, width 5122.24 [2.17, 2.38]
Ours, depth 8, width 1024parameters, FLOPs2.12 [2.07, 2.20]
Ours, depth 2, width 2048parameters, FLOPs1.72 [1.52, 2.02]
SimbaV2, 4 blocks, width 512parameters, FLOPs1.93 [1.86, 2.01]
Published critic, 15× updates per step (new)wall-clock time1.68 [1.45, 1.88]
Published critic1.51 [1.32, 1.80]
  • Equal FLOPs. The FLOPs of a dense network are proportional to its parameter count, so the equal-parameter comparison of the paper (Table 8; rows 2 to 4 here; two seeds for the wide versions of ours) is also an equal-FLOP comparison.
  • Equal wall-clock time (new). Against the published critic with 15 times as many updates, ours is higher on 6 of 8 tasks.
  • The cost comes from depth, not from our layer. At equal depth SimbaV2 takes 0.91 h per run against 0.95 h for ours.
  • Learning speed. On Atari, our network reaches the final score of the published network after about half of training (Figure 10).

Depth 16 against depth 32. The difference between these two depths is within seed noise. To check this we added two seeds on both benchmarks (five in total): the intervals of the two depths overlap on both (PQN on Atari: 0.98 [0.94, 1.03] at depth 16 and 0.95 [0.91, 0.98] at depth 32), and with PPO on DeepMind Control the order reverses within noise (1.28 [1.19, 1.41] at depth 16 and 1.30 [1.18, 1.42] at depth 32), indicating saturation rather than decline. In contrast, from depth 2 to 32 the comparison methods lose most of their score: the multi-skip network falls from 0.87 to 0.64 and the plain network to about 0. This is in fact what we claim (Section 5.1, "Performance improves with depth or remains stable"): our network can be made deeper, with one configuration, without losing performance, and on most benchmarks its score rises with depth. We will add this clarification to the paper.

Q2. Optimizer and tuning.

Optimizer. The headline table (Table 6) already compares all networks under the same optimizer. Figure 2(d) differs in one respect: its dashed line shows the published network under its published RAdam, because Muon is our addition (Appendix D.3; to our knowledge it had not been used to train deep RL agents before). We agree that the same-optimizer comparison should be visible in the figure itself, and we will add the published network under Muon to it. On Atari under the same optimizer:

Published networkOurs, depth 32
PPO, Kron0.81 [0.76, 0.86]1.09 [1.01, 1.16]
PQN, Muon0.90 [0.87, 0.93]0.98 [0.95, 1.01]
PQN, RAdam0.790.80

Our depth-32 network is at or above the published network under every optimizer. The other networks do not reach this: at depth 32, the plain network scores about 0, and the multi-skip network 0.47 (RAdam) and 0.64 (Kron). A better optimizer alone therefore does not make deep networks train.

Tuning:

  • We tuned our settings mostly on the deepest network as it is the most demanding case, and used its configuration at every depth (as stated in the paper): a layer that is meant to scale with depth ideally does not need strong retuning at all depths. If anything, this configuration understates our shallower networks, but as Appendix G shows (reporting the candidate settings at every depth), choosing the best one per depth changes only depth 2 (SAC: 1.70 instead of 1.55).
  • The compared networks were tuned as well: they use their authors' tuned settings, and we additionally swept the learning rate of the SAC baseline, of SimBa and of the multi-skip network, which did not change the comparison. SimbaV2's authors also scale its depth with one fixed tuned configuration, and we follow them.
  • As the reviewer asked, we also briefly checked whether tuning the learning rate of SimbaV2 at depth 32 helps. The published rate still gave the highest score, so we kept it (1e-5: 0.98 [0.93, 1.02]; 1e-4, published: 1.59 [1.36, 1.84]; 1e-3: 1.46 [1.30, 1.62]).
  • The headline compares against SimbaV2 at its best depth: 16 on DeepMind Control (2.24 against 1.92) and 32 on Locomotion (2.34 against 2.03).

Q3. Ablations and goal-conditioned RL.

Ablations We point out that the paper already removes the 1/K multiplier and the skip connection under all three optimizers (Table 14: without the multiplier, 0.54, 0.55 and 0.81 times the full layer under Muon, RAdam and Kron), and ablates the multiplier and the normalization on the continual supervised task at depths 2 to 32 (Table 18). In addition, as requested, we ran the ablation with SAC on DeepMind Control at depth 32 (wins–ties–losses over the 8 tasks against the full layer):

ChangeScoreW–T–L
Full layer2.24 [2.17, 2.38]
No normalization1.44 [1.21, 1.63]0–2–6
No 1/K multiplier1.26 [1.02, 1.65]0–3–5
ReLU for shifted ReLU2.18 [2.05, 2.29]0–6–2

As on Atari (Table 16), removing the normalization or the multiplier significantly lowers the depth-32 network below the published critic (1.51). The shifted ReLU is a minor component inside the full layer, as the paper already states ("a modest gain", Appendix J), but fits in the isometric narrative (here mainly for the branch throughput).

Goal-conditioned RL

  • For goal-conditioned RL, the paper already reports both agents at depths 4, 16 and 64 (Table 11 and Figure 8). At a fixed depth, our agent scores above contrastive RL on 7 of 8 tasks at depth 4, and on 5 of 8 at depths 16 and 64 (and within 10% on a sixth). The headline table (Table 6) is in fact stricter on our agent: it fixes ours at depth 64 and gives their agent its best depth on every task.
  • Depth does improve scores with our layer: from depth 4 to 64, our agent improves by 38% to 83% on four tasks, and on the other four, our agent at depth 4 already scores above contrastive RL at its best depth, which leaves little for depth to add. Moreover, our main aim was not merely to surpass Wang et al. (we used their learning rate as is; tuning it could perhaps have raised our scores further), but to test their conclusion: they found that standard TD agents saturate at depth 4 and concluded that "the self-supervised Contrastive RL algorithm is critical for scaling depth". With our layer this is no longer the case: a standard TD agent keeps improving up to depth 64 and reaches the level of their contrastive agent without any self-supervised objective. To our knowledge, this is the first time a TD agent benefits from such depth on these tasks.
  • We have run our goal-conditioned experiments with the published code base and settings of Wang et al., and have compared with their published results, as is standard practice.

We thank the reviewer again. The new results above (the equal wall-clock comparison, the additional seeds, the SimbaV2 learning-rate check and the SAC ablation) and the same-optimizer line in Figure 2(d) will be included in the revised paper.

Thank you for your detailed responses and the additional experiments and clarifications. They have addressed my concerns, and I sincerely appreciate the effort you have put into the rebuttal. I believe my current rating appropriately reflects my overall assessment of the paper, so I will maintain my score. I wish you the best of luck with your submission.

Reviewer_QAhmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates why deep MLPs fail in reinforcement learning, building on prior work around network conditioning and skip connections. Through a Neural Tangent Kernel (NTK) analysis, the authors show that under non-stationary, bootstrapped RL objectives, hidden-layer Jacobians drift away from dynamical isometry and feature representations lose rank, with deviations multiplying across layers. To keep the network well-conditioned throughout training, the authors propose a simple residual layer consisting of a fixed orthogonal/identity skip connection, parameter-free RMS normalization, a shifted ReLU activation, and a 1/K1/K depth multiplier. The method is validated across value-based, actor-critic, and goal-conditioned RL benchmarks (Atari, DM Control, MuJoCo Playground, Craftax, JaxGCRL) and multiple optimizers (Adam/RAdam, Muon, PSGD-Kron).

Critical strengths
  • Well-motivated and simple method: The paper is very well written and clearly connects the failure of deep RL networks to the singular values of the hidden-layer Jacobians and feature rank via NTK analysis. The resulting architectural solution—a residual layer with normalization, shifted ReLU, and a 1/K1/K depth scaling factor—is simple, clean, and directly motivated by the theory.
  • Broad empirical validation: The proposed layer is tested across a wide variety of RL algorithms (PQN, SAC, TD3, PPO, SAC+HER), domains, and optimizers, demonstrating that it reliably prevents the performance collapse seen in standard deep networks.
Critical weaknesses
  • Incomplete baseline comparisons: The strongest competing architectures are not evaluated uniformly across benchmarks. SimbaV2 performs very closely to the proposed method (and achieves a similar distance from isometry), yet it is only tested on SAC. Meanwhile, the Multi-skip baseline is only compared on PQN and PPO. Because SimbaV2 is a close competitor, omitting it from the PQN, PPO, and TD3 evaluations makes it difficult to assess the true advantage of the proposed layer.

  • Unexplored failure cases and selective presentation of results: Several unfavorable or flat scaling results are pushed to the appendix without sufficient analysis. In particular:

    • Degradation on hard tasks: In Table 22 (PPO on DM Control), performance on the most difficult tasks—HumanoidRun, HumanoidStand, and HumanoidWalk—consistently degrades as depth increases (e.g., HumanoidWalk drops from 1.63 at baseline to 0.46 at depth 4 and 0.16 at depth 32).
    • Flat scaling curves: Performance on Atari (Figure 2d–e) and PPO on Locomotion (Figure 11) is largely flat across depths.
    • Failure on policy networks: The authors briefly mention that deepening the PPO policy network did not work well and lowered scores (Section 5.1, Appendix F), but do not explain why.

    Since a primary contribution of the paper is diagnosing why depth fails in RL, the paper is weakened by not investigating why near-isometric layers fail to help policy networks and why they degrade with depth on complex humanoid tasks.

Questions for authors

See weaknessess

4 comments

We thank the reviewer for the thoughtful and positive assessment, and for the two concerns, which led to new experiments that improve the paper.

Lower scores on the humanoid tasks with PPO. Upon inspection, the three humanoid tasks differ enough from the other 13 tasks that they need their own set of hyperparameters. After tuning the value learning rate on these three tasks, our network no longer degrades with depth and now matches or exceeds the baseline on all three tasks (above it on all three at depths 16 and 32). We conclude that the drop came from the settings on this task family, not from a loss of conditioning of our layer with depth. Thank you for bringing this to our attention.

RunStandWalk
Baseline, best rate1.011.101.63
Ours, depth 4 / 16 / 321.19 / 1.17 / 1.241.04 / 1.16 / 1.141.71 / 1.63 / 1.73
Ours, depth 32, submitted setting0.320.270.16

Deep policies. Prior work already found little benefit in deepening the actor (Nauman et al., 2024; Lee et al., 2025a; 2025b), and we saw the same early in our own work. We agree that the paper passed over this too quickly and that it deserves more analysis. We have now tested deep policies with PPO on the 16 control tasks, with plain policy networks and policy networks of our layers at depths 16 and 32 (learning rate tuned for each, 3 seeds, about 600 runs in total). The results:

  • With the original baseline, replacing the small policy network by a deep network of our layers matches (but does not exceed) the original: 1.05 at depth 16 and 0.99 at depth 32, both within the confidence interval of the published agent's 1.08. Plain deep policy networks degrade severely: 0.73 (depth 16) and 0.44 (depth 32).
  • With our own agent (a depth-32 value network), a policy network of our layers scores 1.22 at depth 16 and 1.24 at depth 32, against 1.28 for the same agent with the small policy network of the PPO baseline, as used in the paper (again within the confidence interval). A deeper policy does not raise the score here either.

Our layer thus keeps deep policy networks stable, but a deeper policy does not raise the score, in contrast to a deeper value network. Both parts fit our analysis (Section 3). The policy objective also changes during training, so plain deep policies degrade as the analysis predicts, and our layer prevents this. Stability is what the layer provides; whether added depth then raises the score depends on whether the task needs the capacity, and on these tasks only the value network, which has to fit bootstrapped targets, does. We therefore keep the policy network small, which is also computationally cheaper and in line with prior work. The lower scores mentioned in the paper were at the learning rate of the small policy; with the rate tuned, the deep policy matches it. We will include a more thorough discussion and the new results in the revised paper.

SimbaV2 and the multi-skip network on other algorithms. We evaluate each baseline with the algorithms it was developed and tuned for, following the original publications, which we believe is the comparison most favourable to the baselines. SimbaV2 is a complete agent design for SAC (its network together with a distributional critic and reward scaling), published and tuned for SAC only. Porting it to PPO or PQN would effectively create a new method, with no published setting to start from: every design choice would be ours, and a weak result would be hard to tell apart from a weak port. The multi-skip network with the Kron optimizer (Castanyer et al.) was developed for PQN on Atari and also tested with PPO; we use both in our paper, porting directly from the authors' code. For TD3, no deep network had been published, so the submitted paper compared with the published critic only. Every baseline is thus evaluated with the algorithm and settings it was published for, over five algorithms, five benchmark suites and three optimizers, which (as the reviewer notes) is a broad evaluation, and broader than in prior work. We will state this reasoning more explicitly in the revised paper, as we agree this is indeed important to convey properly.

That said, we believe TD3 to be a natural extension for SimbaV2 (compared to e.g., PPO or PQN): like SAC, it is an off-policy actor-critic algorithm, so SimbaV2's critic network carries over to it without major redesign. It also fills a gap in the paper, which had no deep baseline for TD3. We have therefore implemented SimbaV2's network in TD3 and run it at depth 32 (learning rate 1e-4, its published rate) on the same 8 tasks (3 seeds). It scores 1.44 [1.25, 1.57], above the published critic (1.00 [0.93, 1.08]) but below our network at the same depth (2.05 [1.78, 2.41]). This is in line with the reported SAC results and supports the same conclusion with a second algorithm. Given the time available for the rebuttal period, we have run only depth 32 so far; we will run the other depths and add the full comparison to the revised paper.

SimbaV2 and isometry. That SimbaV2 is close to an isometry agrees with our analysis: its hyperspherical normalization keeps its layers close to an isometry (although the paper does not frame it in these terms), and we believe this is why it is the strongest prior network. However, our analysis (Section 3) gives two requirements: isometry of the layer Jacobians, and the rank of the features. The second requirement separates the two networks: SimbaV2's features have lower rank than those of our layer (Section 5.2), which therefore keeps a better-conditioned tangent kernel.

Flat curves. Our claim is that performance improves or remains stable with depth, whereas other deep networks often degrade. The flat curves are in the paper because we report every benchmark and depth we ran, including those where depth does not raise the score; some of them are in the appendix only because of the page limit. On these benchmarks, added depth has little to add under these algorithms, and our claim there is the absence of degradation: in the same panels (Figure 2d–e), the deep baselines degrade or collapse, so these results support the claim.

We hope the new experiments and clarifications address the reviewer's concerns, and we will include all of them in the revised paper. We thank the reviewer again for the useful comments.

We thank the reviewer again for their time. As the discussion period closes soon, we would like to share the full SimbaV2 results, which complete the preliminary comparison above. We have now run the network of SimbaV2 in TD3 at all remaining depths on the DeepMind Control tasks:

Depth02481632
Ours–1.091.621.401.572.05
SimbaV2 network–0.900.981.061.121.44
Baseline1.00–––––

The network of SimbaV2 also improves with depth in TD3, and our network scores above it at every depth, in line with the SAC results, and doubles the score of the baseline at depth 32. These results are now included, and a revised version of the paper has been uploaded.

We hope that these results, together with the new experiments on the humanoid tasks and on deep policy networks above, address the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating their assessment, and we remain happy to answer any further questions before the discussion closes.

Thank you for the thorough response and for running the additional experiments on the PPO humanoid tasks, deep policy networks, and SimbaV2 with TD3. I appreciate the work, and effort. This has addressed my questions, I believe the current rating of acceptance is appropriate.

Comments on the paper

1 comment

We are grateful to the reviewers for taking the time to read our paper, and for their suggestions. We have now uploaded a revised version that incorporates their requests.

Summary Our paper shows that depth fails in deep RL because networks lose isometry during training. From this, we derive a principled layer that stays close to isometry, with which deep networks outperform prior works across a wide range of tasks and algorithms. The revision strengthens these results by adding:

  • Five-seed results for the main comparison.
  • Compute-matched comparisons against wider networks and more updates.
  • SimbaV2 in TD3.
  • Ablations beyond atari.
  • Expanded experiments and analysis of deeper policy networks.
  • Comparisons with SIGReg and with contrastive RL at fixed depths.
  • Same-optimizer comparisons with the baselines.

The individual responses below give the details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.