acceptodds
Withdrawn from ICLR 2027

Rethinking Depth Scaling in Reinforcement Learning Through Horizon Generalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Scaling neural networks has driven major advances in supervised and self-supervised learning, yet the role of network depth in reinforcement learning remains unclear. Prior studies report limited gains from deeper networks, while a recent study finds substantial benefits only in self-supervised reinforcement learning. We reveal a missing piece of this puzzle: depth has a limited effect on in-distribution performance but substantially improves generalization to unseen goals. We study horizon generalization in locomotion, manipulation, and navigation tasks across self-supervised, off-policy, and on-policy reinforcement learning methods. In most settings, deeper policies maintain higher success rates as evaluation goals move farther beyond the training distribution, even when shallow and deep models perform similarly on training goals. Deeper networks also outperform wider, shallow networks with comparable parameter counts, showing that the gains are not due to model size alone. Mechanistic analyses further reveal depth-dependent differences in how policies represent and use relative goal information. Our results suggest that network depth primarily extends the range over which reinforcement learning policies generalize, and that standard in-distribution evaluations may underestimate its value.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, OOD generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5GyoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper demonstrates that scaling the architecture depth of (off and on-policy) actor-critic algorithms enables better generalization in out-of-distribution goal-conditioned RL.

Prior work on scaling contrastive reinforcement learning (CRL) algorithm showed that scaling matters for CRL when considering in-distribution training. This paper extends that line of work to other algorithms, showing that scaling doesn't help when goals are sampled in-distribution, but as goals are becoming more OOD, scaling improves generalization and therefore downstream performance.

Critical strengths
  • The paper presents a useful and relatively surprising result, which can in the future enable future research on scaling and generalization in reinforcement learning.
  • The sparse auto-encoder analysis is novel and insightful.
  • Comparing generalization across and network depth and width is a very useful result.
Critical weaknesses
  • The result is purely empirical, however the paper uses only four reinforcement learning environments to support its claims.
  • More generally some claims can be made more precise: generalization here is only in the context of goal-conditioned RL, not for instance in the context of offline RL or sim-to-real. The "horizon generalization" framing in the title is not as deep as the title may suggest. The above two points are discussed in the limitations.
  • The invariant representation is done only using SAC and Humanoid.
  • The description of Section 4.6 can be made clearer. For instance, terms like: "cumulative greedy ablation" tend to be somewhat vague.
Questions for authors

Minor comments:

  • You can cite BRO and Simba when presenting them in line 99.
  • When generating new OOD goals, how did you validate that these goals are sound and that there is a policy that can actually solve them?

I'm happy to increase my score if claims about generalization and "horizon generalization" are made more precise.

Or, even better, if these results extend beyond these four environments and not only within the context of goal-conditioned RL.

Reviewer_mVgHOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Scaling in RL is one of the major contemporary research directions. For many years, we have seen scaling work very well in supervised learning, but it did not translate easily to RL. In recent years, breakthroughs in scaling RL networks started to appear, promising the unlocking of the same benefits. The authors of the paper test how much scaling affects in-distribution and out-of-distribution performance across 3 major RL algorithms in 4 goal-conditioned environments. The key claim of the paper is that depth helps primarily in generalization to OOD goals, not necessarily in the ID performance.

Critical strengths
  • The paper is very well written, clear, informative, and easy to follow.
  • The topic is recent and important. We need more information on newly found scaling possibilities in RL, since we do not yet understand this topic well enough.
  • The observation that depth mainly helps with generalization has the potential to significantly impact the field and shape how we evaluate and talk about depth scaling.
  • The study itself is thorough and tests not only a binary ID/OOD regime, but also how much the goal is OOD.
  • The supplemented ablations on initial state being OOD and looped networks are interesting and relevant as well, and they further strengthen the claim.
  • I think that the breadth of methods, environments, and OOD regimes tested is enough to be informative and support the claims (but see Weaknesses).
Critical weaknesses
  • The major problem with the paper is scientific rigor. The authors do not report the number of seeds on which their experiments are carried out. On performance bar plots, there are no confidence intervals; are those results from a single seed?. This has been shown to be problematic and not enough for meaningful comparisons in RL. On the other hand, in training curve plots, there are some shaded regions, which I assume to be confidence intervals, but the authors do not say what they are, how many seeds were used, or anything else on the matter.
  • Another concern that I have is that the study does not fully disentangle horizon generalization from “mechanical” hardness of reaching a distant goal. For example, in a Humanoid environment, the authors place the goal in the disc around the starting point with radius in [1,5]. The OOD-4 regime is the disc with radius (8,9]. This means that the OOD-4 regime is hard for the agent, not only because it is OOD, but also because the goals sampled from it are much harder by their nature. This is especially true, given that success is defined as simply reaching a goal once. In particular, for closer goals the agent might with some frequency get to them, without developing stable locomotion. For further goals, this most likely is no longer true, the agent has to actually start walking. This could mean that the difference between shallow and deep networks is present also because deeper networks are simply better at stable walking, regardless of the goal placement. One ablation I would suggest is training a model on a full range of goals, and comparing its success rate on a particular OOD range. This could help disentangle whether further OOD goals are harder for the shallow agent because of horizon generalization, or because the shallow agent has trouble learning a sustained locomotion policy. This is only a possibility, and experiments on PointMaze, where low-level locomotion is much simpler, suggest that the results are still true. However, I think that a little more experiments helping to disentangle horizon generalization from sustained locomotion hardness could meaningfully strengthen the paper.
Questions for authors

I think that claims in the paper make sense and could be true. Furthermore, if they are indeed true, this has the potential to be a meaningful study, used in the future. However, in the current shape, I do not think that the presented experiments fully support those claims. This is both due to an unrigorous (or at least undocumented) experimental setup, as well as potential confounding factors that may obscure what is actually happening. Once those problems are resolved, I will happily update my rating, but in the current shape I think that this paper should not be accepted. Some additional questions to the authors:

  • I could not find a reproducibility statement. Do the authors plan on releasing the code?
  • How many seeds were used during the training and evaluation? What do shaded regions on curve plots represent?
  • Did the authors test models trained on a full range of goals to check what the reachability for those furthest ones is?
  • Have the authors tested other shapes of goal distribution in environments like Humanoid? While the study is mostly about horizon generalization, I think that it would be interesting to see how well the networks generalize, for example, when trained on a goal distribution being a 180-degree disc, and OOD regimes progressively adding the missing half of the disc.
Reviewer_ewv9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether making neural networks deeper in goal-conditioned RL enables them to reach goals farther away than those encountered during training. By keeping the environment’s dynamics and episode time constraints constant while varying the target location for evaluation, we report that the effect of depth is more pronounced for distant targets than for familiar ones across CRL, SAC, PPO, and four other environments. We compare our approach with models that increase the number of layers and models that repeat computations with the same weights, and we examine the possible reasons by analyzing or directly modifying the internal representations.

Critical strengths
  1. It is beneficial to distinguish and evaluate the performance of familiar targets from that of increasingly distant targets. The SAC and PPO results extend the relationship between depth and generalization observed by Wang et al. (2025) in CRL to other learning algorithms and provide evidence across various control tasks.

  2. The paper examine why depth is beneficial through a broad comparison with a similar number of parameters and a comparison involving repeated computations while sharing weights. The results also show that manipulating internal representations related to the target direction produces behavioral changes distinct from those caused by random feature manipulation, providing evidence that goes beyond the simple observation that internal information is easily interpretable.

Critical weaknesses
  1. The reliability of the depth trends is difficult to assess because independent training-run counts, aggregation across runs, and the meaning of shaded bands are not specified. Since the contribution is primarily empirical, this limits confidence in the consistency of the gains across independently trained models.

  2. The connection between the representation analysis and improved OOD success remains incomplete. Figures 5–6 show that goal direction is more readily decoded in deeper actors and that targeted interventions can steer behavior, but do not specify whether these analyses cover OOD goal distances. Steering between equal-distance goals does not by itself establish why deeper policies reach more distant goals. This leaves insufficient evidence for the stronger interpretation of invariant direction encoding as a key mechanism of horizon generalization, although it remains a plausible explanation.

Questions for authors
  1. Could you report the independent training-run counts per configuration and how the bars, curves and shaded bands were computed? Please distinguish variation across trained models from variation across evaluation episodes, and report the corresponding ID/OOD means and variability. If independent repeats are unavailable, a representative shallow/deep replication would help assess reliability.

  2. Wang et al. (2025), 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities (NeurIPS 2025, Figure 11), already reported depth-related unseen-goal generalization in CRL. To understand the additional representational explanation offered here, could you clarify whether the probes and interventions in Figures 5–6 include OOD goal distances? If limited to training distances, evaluating the existing direction probes at OOD distances would help connect this analysis to the observed generalization gains.

  3. Which environments were used for the width and repeated-computation comparisons in Figures 3–4? If these controls cover only one environment, a representative depth-versus-width comparison in another task domain, or an explicit limitation of the claim to the tested setting, would help clarify how broadly the architectural conclusion is supported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.