acceptodds
Withdrawn from ICLR 2027

Three Gains, One Loop: Diagnosing and Repairing the Bellman Bootstrap in High Dimensions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model-free reinforcement learning is widely believed to require world models or complex architectural innovations to scale to high-dimensional continuous control. We show that this difficulty can stem from a core structural instability: in high dimensions, the Bellman bootstrap forms a positive-feedback loop that can amplify noise faster than it accumulates signal. As the agent improves, its actions move into poorly covered regions, generating out-of-distribution temporal-difference errors that can corrupt the value landscape needed for continued learning. We identify three gains governing this feedback loop, corresponding to target signal, critic-gradient geometry, and error sensitivity, and show how each can be compressed by an architecture-preserving intervention: multi-step returns, LayerNorm, and Huber loss, respectively. We term this repair Gain-Compressed Repair (GCR). While the individual components are established techniques, the gain decomposition explains their complementary roles and predicts distinct failure signatures when individual gains are left uncontrolled. Together, these modifications add negligible parameter overhead yet enable standard TD3 and SAC to achieve nearly 800 reward on Dog-Run (38-D actions) within 2M steps and strong performance on 61-dimensional H1Hand tasks. Across 12 primary high-dimensional benchmarks spanning Dog, Humanoid, and H1Hand, GCR remains competitive with strong model-based baselines, suggesting that Bellman-loop instability is a key bottleneck for model-free learning in the studied state-based settings.

Primary area:
reinforcement learning
Keywords:
deep reinforcement learning, high-dimensional continuous control, Bellman bootstrap, temporal-difference learning, training stability, gain compression, model-free reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5jS5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates instability issues of model-free Reinforcement Learning for the high-dimensional case. Within a TD-learning context, the paper's main thesis is that three problems are at the root of instability: the signal gain of the bootstrap target, the gradient gain of the critic and the error gain of the self-referential update. Having singled-out such problems, the authors propose the "Gain-Compressed Repair (GCR)" architecture (making use of multi-step returns, LayerNorm, and Huber loss) with the aim to recover a stable training of the closed-loop system. The authors propose experiments on benchmarks along with a stability analysis to support their claims.

Critical strengths

The empirical results are strong and reach high returns on a couple of benchmarks. The proposed changes are simple, see Figure 1 and Table A.1

The paper claims that it can be transferred to SAC and the abstract suggests that the experimental results for TD3 would also apply for SAC.

Critical weaknesses

Novelty

(Fujimoto et al., ICLR 2025) introduced MR.Q and it already combines LayerNorm, Huber loss and 3-step returns, evaluated on DMC Dog and Humanoid. This paper points to this work in "components in isolation", but ignores the proposed MR.Q as obvious baseline for this submission.

Also, the bib ref of (Fujimoto et al., ICLR 2025) only points to the arxiv preprint, ignoring that it appeared at ICLR 2025. Please check all your refs for similar missing bib details, like doi, venue and the like.

In this light, the authors should make it clearer what the exact contribution beyond (Fujimoto et al., ICLR 2025) is.

Soundness

The core idea of this paper is the three-gain framework of the Bellman bootstrap in high dimensions. This narrative, however, can be questioned as follows.

Gain exceeding 1.

Following your narrative, the prior art fails to perform RL in high-dimensional control tasks for the gain to exceed 1 and hence form an escalating feedback loop. Hence, we would expect in Figure B.1 to see a gain to exceed the value of 10^0 in Figure B.1

In any case, can you provide an experiment where we see exactly what you claim is the reason for the prior art to fail?

In addition, can you provide an experiment where we see the exponential growth with MSE as discussed in your paper?

The small-gain theorem in the appendix.

A core contribution of this paper is the stability analysis. However, what type of stability is actually investigated, e.g. finite-gain L-stability? We ask the authors to be precise on the notion of stability, the involved spaces and norms.

Adaptive optimizers.

You use Adam as optimizer, which is an adaptive optimizer. This adds to the total training dynamics in different ways. First, in addition to the three gains you described you also have the dynamics of Adams. This should be investigated. In addition, a comparison with SGD could serve as baseline.

Secondly, given that Adam is adaptive, it is less surprising that the parameters of the Huber loss have little effect, but you attribute it to the stability analysis in paragraph 4.3.

Why high dimensions?

It is unclear what exactly the impact of the dimension of the control task is in your paper. You argue that this is relevant for higher dimensions only, but you also do not investigate. So what exactly can we say about this?

Impact of neural net

RL literature on various control tasks also investigate alternative policy approximators to neural nets. In particular, for mountain car it is known that neural net-based policies show a high regret for various RL algorithms (e.g., SAC, PPO, ARS). This raises two questions: (i) is GCR also beneficial for low-dimensional control and (ii) what is the impact of the policy approximator?

Necessity for all three factors.

You argue that all three improvements of GCR are necessary, but your ablation study shows in a sense the opposite: Removing any factor makes the ablated GCR still perform significantly better than the baseline TD3. Hence, we shall ask the reverse question: Which of the three factors added to TD3 causes the largest improvement?

Also, we should treat results within a confidence interval as equal. For instance, 622.9 ± 89.9 (GCR) and 563.5 ± 49.6 in Table 2. This actually puts the necessity of Huber loss under question mark.

Furthermore, how is the constant gradient of the Huber loss (beyond beta) being self-limiting?

I have not had the time to check the math in detail. Maybe one detail that is interesting in eq (19) is the factor (1 + γⁿ τ) due to the multi-step return. Can you evaluate this term (and the other factors) for the experiments, in particular the ablation study, since this analysis explains the experimental results here, right?

Presentation

I got a bit confused in what experiments GCR is GCR with TD3 or GCR with SAC. It would help to call GCR with TD3 then GCR-TD3, and it would also help to add matching baselines in the baselines, like GCR-TD3 versus TD3, GCR-SAC versus SAC.

Questions for authors

In addition to the points raised above:

Clarifications:

  1. Where do the baseline numbers in Table 1 come from (re-run or taken from prior papers)? Please be precise on the evaluation protocol and number of seeds.

  2. How does TD3 "implicitly address aspects of the instability" and how can you provide a reference for "where the instability remains subcritical"?

Suggestions:

  • A gradient-free, critic-free reference such as ARS, and an on-policy method such as PPO, would help to test whether the difficulty really lies in the Bellman loop.

  • Fig. 2: do the baselines learn, just more slowly? Longer curves for the baselines would clarify this.

  • How is "overwriting well-learned regions" related to catastrophic forgetting in RL?

  • References: please check all references and their context. For instance, MR.Q appeared at ICLR 2025; Huber loss goes back to Huber (1964); Scannell et al. (2024) is representation learning for TD-based RL.

Reviewer_uBtbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the difficulty of applying model-free reinforcement learning to high-dimensional continuous control. It proposes a three-gain framework addressing target signal, critic-gradient sensitivity, and error amplification through multi-step returns, critic LayerNorm, and Huber loss, respectively. The resulting method, GCR, is implemented within TD3 and SAC and evaluated through high-dimensional control benchmarks, component ablations, and supplementary experiments. The main contribution is an explanation of how established components complement one another, together with evidence that simple model-free methods can perform well in these settings.

Critical strengths
  1. The method is simple and practically useful. GCR uses small networks and readily implementable components, achieving strong results across several high-dimensional control tasks with shared hyperparameters.
  2. The evaluation includes useful supplementary analyses. Component ablations, comparisons between LayerNorm and gradient clipping, longer training runs, and network-width experiments examine the method from several perspectives. The SAC results also show that the benefits are not limited to TD3.
Critical weaknesses
  1. The comparisons omit the closest related method and more recent world model approaches. The paper acknowledges that MRQ also uses Huber loss, multi-step returns, and LayerNorm, but does not compare against it directly. Since the individual components are not claimed as novel, this omission leaves the contribution over existing methods unclear. The world model baselines mainly comprise TD-MPC2 and DreamerV3, omitting more recent relevant methods such as BMPC and BOOM, both discussed in the paper. The supplementary comparisons with BRO, SimBaV2, and XQC cover only the 19-dimensional H1 tasks, rather than the primary 38- and 61-dimensional settings. The results therefore do not sufficiently establish competitiveness with recent state-of-the-art world model methods.
  2. The theoretical analysis has two substantive gaps. (1) The connection between “signal gain” and learning quality is unclear. Equation (4) defines signal gain as the ratio of bootstrap variance to accumulated reward variance. Proposition B.1 derives a decrease in this ratio with the return horizon under assumptions including approximately uncorrelated rewards and approximately constant bootstrap variance. However, bootstrap variance can reflect genuine differences in value, while reward variance can arise from stochastic noise. A smaller ratio therefore does not necessarily imply a more accurate or informative target, or more stable learning. The paper neither sufficiently establishes this connection nor validates the assumptions in the evaluated tasks. Furthermore, the implementation explicitly uses uncorrected off-policy multi-step targets, but the analysis does not account for the bias arising from differences between the behavior and target policies. Observed rewards being free of critic approximation error does not make them an unbiased return estimate for the target policy. This limits the analysis’s applicability to the implemented algorithm. (2) The local LayerNorm bound is extended to an unsupported network-level bound. Proposition C.1 bounds the Jacobian of LayerNorm with respect to its own input. A complete network layer also includes the preceding weight matrix, as shown in Eq. (27), but these weight-matrix factors are omitted from the network-level gradient bound in Eq. (29). The argument that activation standard deviation grows with the weights does not generally justify this cancellation: the activation standard deviation at a particular input does not control weight amplification in every input direction. Consequently, Eq. (29) and the claimed approximately 25-fold reduction in the gradient bound are not adequately justified.
Questions for authors
  1. What specifically distinguishes GCR from MRQ in algorithm design and training procedure? Are there direct comparisons showing the benefits of these differences?
  2. Why are BMPC and BOOM omitted from the evaluation? Are there results under matching tasks and interaction budgets that support the paper’s performance claims?
  3. What precise relationship connects the variance ratio in Eq. (4) to target estimation error or learning stability? How does it distinguish useful reward variation from stochastic noise, and genuine value variation from bootstrap estimation error?
  4. Have reward temporal correlations, bootstrap variance, and G_{sig} been measured across return horizons to validate the assumptions and approximate 1/n1/n scaling in Proposition B.1?
  5. How does policy mismatch bias in the off-policy multi-step target affect the proposed signal-improvement explanation?
  6. What justifies dropping the weight-matrix factors between Eqs. (27) and (29)? What additional assumptions are required?
Reviewer_rBZKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This submission focuses on training instability problems of continuous action space reinforcement learning. The submission claims to identify three critical problems with temporal difference learning and proposes LayerNorm, Huber loss and multi-step update to resolve these issues.

Critical strengths

Please see below.

Critical weaknesses

The writing of this submission is not clear and some of the claims are simply incorrect. Some of these claims even appear in the abstract and do not make much sense. For instance:

“As the agent improves, its actions move into poorly covered regions, generating out-of-distribution temporal-difference errors that can corrupt the value landscape needed for continued learning.”

As the agent improves the opposite of this claim would be true. Furthermore, the submission consists of incorrect statements backed by irrelevant references. I will not go into each instance of this issue, but I will leave one example. The submission states:

“As the agent improves, its policy produces actions in regions of the high-dimensional action space that the critic has never evaluated.”

And cites (Lyle et al., 2023). This cited paper does not have a claim about this. The statement above is also not clear that it is true.

The paper is written as an incoherent stream of thoughts with adjacent claims. In its current form, this submission is not ready to be published just solely even how it is written.

The ways that temporal difference learning can fail has already been known and also has been referred to as the deadly triad including function approximation, bootstrapping, and off-policy learning (Sutton and Barto 2018).

Beyond all the writing issues, this paper does not have any novel contributions. LayerNorm, multi step return already known to improve performance. Using Huber loss instead of MSE is also already known and even implemented as an option in major baselines such as stablebaselines. Neither the issues identified in this submission nor the solutions are novel.

All this submission does is to take very well known fundamental reinforcement learning concepts and rephrase and rebrand them in I believe to be control and signal processing terminology to present it as if now this is a novel discovery. I find this kind of approach to research truly problematic.

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT press, Cambridge MA, 2018.

Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.