- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates instability issues of model-free Reinforcement Learning for the high-dimensional case. Within a TD-learning context, the paper's main thesis is that three problems are at the root of instability: the signal gain of the bootstrap target, the gradient gain of the critic and the error gain of the self-referential update. Having singled-out such problems, the authors propose the "Gain-Compressed Repair (GCR)" architecture (making use of multi-step returns, LayerNorm, and Huber loss) with the aim to recover a stable training of the closed-loop system. The authors propose experiments on benchmarks along with a stability analysis to support their claims.
Critical strengths
The empirical results are strong and reach high returns on a couple of benchmarks. The proposed changes are simple, see Figure 1 and Table A.1
The paper claims that it can be transferred to SAC and the abstract suggests that the experimental results for TD3 would also apply for SAC.
Critical weaknesses
Novelty
(Fujimoto et al., ICLR 2025) introduced MR.Q and it already combines LayerNorm, Huber loss and 3-step returns, evaluated on DMC Dog and Humanoid. This paper points to this work in "components in isolation", but ignores the proposed MR.Q as obvious baseline for this submission.
Also, the bib ref of (Fujimoto et al., ICLR 2025) only points to the arxiv preprint, ignoring that it appeared at ICLR 2025. Please check all your refs for similar missing bib details, like doi, venue and the like.
In this light, the authors should make it clearer what the exact contribution beyond (Fujimoto et al., ICLR 2025) is.
Soundness
The core idea of this paper is the three-gain framework of the Bellman bootstrap in high dimensions. This narrative, however, can be questioned as follows.
Gain exceeding 1.
Following your narrative, the prior art fails to perform RL in high-dimensional control tasks for the gain to exceed 1 and hence form an escalating feedback loop. Hence, we would expect in Figure B.1 to see a gain to exceed the value of 10^0 in Figure B.1
In any case, can you provide an experiment where we see exactly what you claim is the reason for the prior art to fail?
In addition, can you provide an experiment where we see the exponential growth with MSE as discussed in your paper?
The small-gain theorem in the appendix.
A core contribution of this paper is the stability analysis. However, what type of stability is actually investigated, e.g. finite-gain L-stability? We ask the authors to be precise on the notion of stability, the involved spaces and norms.
Adaptive optimizers.
You use Adam as optimizer, which is an adaptive optimizer. This adds to the total training dynamics in different ways. First, in addition to the three gains you described you also have the dynamics of Adams. This should be investigated. In addition, a comparison with SGD could serve as baseline.
Secondly, given that Adam is adaptive, it is less surprising that the parameters of the Huber loss have little effect, but you attribute it to the stability analysis in paragraph 4.3.
Why high dimensions?
It is unclear what exactly the impact of the dimension of the control task is in your paper. You argue that this is relevant for higher dimensions only, but you also do not investigate. So what exactly can we say about this?
Impact of neural net
RL literature on various control tasks also investigate alternative policy approximators to neural nets. In particular, for mountain car it is known that neural net-based policies show a high regret for various RL algorithms (e.g., SAC, PPO, ARS). This raises two questions: (i) is GCR also beneficial for low-dimensional control and (ii) what is the impact of the policy approximator?
Necessity for all three factors.
You argue that all three improvements of GCR are necessary, but your ablation study shows in a sense the opposite: Removing any factor makes the ablated GCR still perform significantly better than the baseline TD3. Hence, we shall ask the reverse question: Which of the three factors added to TD3 causes the largest improvement?
Also, we should treat results within a confidence interval as equal. For instance, 622.9 ± 89.9 (GCR) and 563.5 ± 49.6 in Table 2. This actually puts the necessity of Huber loss under question mark.
Furthermore, how is the constant gradient of the Huber loss (beyond beta) being self-limiting?
I have not had the time to check the math in detail. Maybe one detail that is interesting in eq (19) is the factor (1 + γⁿ τ) due to the multi-step return. Can you evaluate this term (and the other factors) for the experiments, in particular the ablation study, since this analysis explains the experimental results here, right?
Presentation
I got a bit confused in what experiments GCR is GCR with TD3 or GCR with SAC. It would help to call GCR with TD3 then GCR-TD3, and it would also help to add matching baselines in the baselines, like GCR-TD3 versus TD3, GCR-SAC versus SAC.
Questions for authors
In addition to the points raised above:
Clarifications:
-
Where do the baseline numbers in Table 1 come from (re-run or taken from prior papers)? Please be precise on the evaluation protocol and number of seeds.
-
How does TD3 "implicitly address aspects of the instability" and how can you provide a reference for "where the instability remains subcritical"?
Suggestions:
-
A gradient-free, critic-free reference such as ARS, and an on-policy method such as PPO, would help to test whether the difficulty really lies in the Bellman loop.
-
Fig. 2: do the baselines learn, just more slowly? Longer curves for the baselines would clarify this.
-
How is "overwriting well-learned regions" related to catastrophic forgetting in RL?
-
References: please check all references and their context. For instance, MR.Q appeared at ICLR 2025; Huber loss goes back to Huber (1964); Scannell et al. (2024) is representation learning for TD-based RL.