- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a residual layer that keeps deep networks close to an isometry, making deep networks more stable in reinforcement learning. The paper first analyzes why increasing network depth fails in RL and then proposes their residual layer. In the large-scale empirical study, the layer is shown to bring benefits across value-based, actor-critic, and goal-conditioned algorithms and optimizers when scale up the depth of networks.
Critical strengths
- The analysis of why increasing the depth of deep networks fails in RL is clear and insightful and could be helpful for future work on this topic.
- The empirical study is extensive, covering many baseline algorithms (PQN, SAC, TD3, and PPO) and different benchmarks (Atari, DM Control, MuJoCo Playground, Craftax-Symbolic, and JAXGCRL), showing the generality of the proposed layer, which can work in different setups.
Critical weaknesses
- Although the empirical study is extensive, most results use only three random seeds. I strongly encourage the authors to run more random seeds to ensure the reliability of the results.
- There is a highly relevant paper [1] about enforcing Isotropic Gaussian representations that is mentioned in the paper but is not directly compared in the empirical study. Although the authors of [1] did not try combining their method with deeper networks, I think it is worth a try. Could the authors provide a direct comparison?
[1] Stable Deep Reinforcement Learning via Isotropic Gaussian Representations. ICML 2026
Questions for authors
See Weaknesses.
2 comments
Thanks for the detailed rebuttal and clarification, which addressed and resolved my concerns. I will update the score accordingly.
We thank the reviewer for the positive assessment of our analysis and experiments. We ran both requested experiments and report them below.
W1 (seeds). We added two seeds, so every main result now uses five seeds. The only exception is goal-conditioned RL, where we keep three seeds to match the protocol of Wang et al. (2025), whose published curves we compare to. Scores are IQM [95% CI]. There is no qualitative difference with the 3-seed results of the paper:
The earlier submission prioritized breadth: five algorithms, five benchmark suites (72 task settings), three optimizers and a wide range of depths. Our network has the highest aggregate score at depth 32 on every benchmark suite and with every algorithm, which seed variation alone is unlikely to produce. All scores are IQM with stratified bootstrap confidence intervals (Agarwal et al., 2021), the recommended protocol for reliable comparisons in deep RL: on the multi-task suites each score pools all runs of all tasks (40 to 80 runs at five seeds), the IQM is robust to outlier runs, and the intervals account for the variation across seeds.
W2 (SIGReg). As suggested, we combined SIGReg with deeper networks: plain networks of depth 6, 10 and 32, a multi-skip network and ours at depth 32, on Atari-10 with PQN. We use the SIGReg loss from the authors' released code, unchanged, with their two regularization strengths. As in their code, the loss acts on the features that enter the final linear layer. Scores are IQM (HNS). The SIGReg columns use three seeds and the column without SIGReg five; the five-seed SIGReg version with the full table will be in the final paper.
The released code of Saheb Pasand et al. runs PQN with a plain network of 8 hidden layers, between our plain networks of depth 6 and 10. In their Atari results this network scores about 0.00 at the end of training and about 0.22 with SIGReg. We see the same on our plain networks: they score close to zero without SIGReg, and λ = 0.2 raises them slightly (0.10 to 0.16 at depth 6, 0.02 to 0.03 at depth 10), which supports that our implementation matches theirs. In their results and in our reimplementation, however, a plain network with SIGReg stays far below the published network with one hidden layer (0.77), and at depth 32 SIGReg changes nothing. On the three networks that train well without it (the published network, the multi-skip network and ours), SIGReg lowers the score at both strengths.
This is consistent with our analysis: SIGReg acts only on the features of the last hidden layer, whereas training a deep network also requires well-conditioned Jacobians in all hidden layers, which our layer provides. We thank the reviewer for suggesting this comparison; it makes the paper more complete.
We are adding both experiments to the revised paper. We hope this addresses the reviewer's concerns, and we would be grateful if they would consider updating their rating.