- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether encouraging low-dimensional representations through an explicit learning objective can improve the sample efficiency of reinforcement learning. Motivated by the Neural Regression Collapse (NRC) phenomenon, the authors propose NRC2Reg, a representation regularizer that encourages intermediate network features to align with the row space of the network's final linear readout. The authors apply NRC2Reg to contrastive RL, regularizing both the critic's state-action and goal encoders and the actor network. Experiments across nine goal-conditioned RL environments at multiple network depths demonstrate substantial improvements in challenging tasks.
Critical strengths
- Using Neural Regression Collapse as a training regularizer is an interesting idea. NRC2Reg requires no additional learnable parameters and can be integrated into existing residual network architectures through an auxiliary loss.
- NRC2Reg provides significant improvements on some difficult goal-conditioned control tasks, especially Humanoid U-Maze, Humanoid Big Maze, and Arm Push Hard.
- The authors investigate effective rank, intrinsic dimension, readout conditioning, and estimated feature information. The comparison between fixed orthogonal bottlenecks and NRC2Reg provides useful insight into how different approaches to dimensionality reduction affect representation learning.
Critical weaknesses
- Although the experiments include nine environments and several network depths, most results use only three random seeds. I strongly encourage the authors to run more random seeds to ensure the reliability of the results.
- The experiments show that NRC2Reg substantially reduces the condition number of the readout matrix, while its effective rank remains almost unchanged. This raises an alternative hypothesis that the observed performance improvements may be partly attributable to improved optimization and conditioning rather than low-dimensional representation learning itself.
Questions for authors
- Table 5 shows that NRC2Reg decreases AUC across all AntMaze settings. Could the authors investigate why the regularizer improves learning on challenging Humanoid and manipulation tasks but not on AntMaze?
- NRC2Reg is motivated by a general representational property of networks, but all experiments use contrastive RL. Does NRC2Reg work beyond contrastive RL?
5 comments
Author Rebuttal (Part 2/3)
Q1: Why does NRC2Reg not help on the ant mazes?
We wondered about the same question and investigated it in detail! In short, learning on the ant mazes is limited by a weakness of CRL itself in reaching far goals. It affects both methods, and a regularizer of the representation is not meant to fix it. A small change to how the actor's goals are sampled addresses it, as we will show below.
A failure case shared by both methods. Our starting point was Ant U5 Maze, where there were runs of both CRL and NRC2Reg that ended with zero time at goal. This maze is evaluated on far goals only, which lie behind a wall, so the ant must first move away from the goal to reach it. The actor, however, is trained only on hindsight goals, that is, on positions the agent has already visited, so a far goal that has not been reached yet is a goal the actor was never trained on. Our trained checkpoints confirm this: at the far goals of U5, the critic prefers about 40% of uniformly sampled actions to the action chosen by the actor, for CRL and NRC2Reg alike. The first far-goal success therefore depends on how the actor extrapolates, and a run in which it does not happen stays at zero. The two methods differ in this extrapolation: the actor trained with NRC2Reg tends to remain near the start for longer, so its first far-goal success can come later, which is what the overall lower AUC in Table 5 reflects.
Why NRC2Reg helps on the humanoid mazes. The two task families are limited by different things. An untrained humanoid falls within a few dozen steps, which ends the episode, so it first has to learn to walk. By contrast, an untrained ant already survives most of the 1000-step episode. NRC2Reg makes this high-dimensional control easier to learn: in Figure 4(c, d) the humanoid stays upright longer, and in Figure 4(a, b) its first far-goal success arrives earlier. Interestingly, Figure 3 of the scaled CRL paper shows that, with a sufficiently deep network, the humanoid learns to vault over the wall of the U-Maze. This suggests that the humanoid may not have solved the problem of far goals either: it may simply be able to get over the wall, a shortcut that the body of the ant does not allow, so the ant stays stuck behind it. The ant also needs little help with locomotion, so the regularizer has less to gain there.
A minimal remedy. Since the far goals are missing from the actor's objective, we put them there: half of each actor batch is trained on the goal that was commanded when the data was collected, and the other half on hindsight goals as before. The critic, the regularizer and all hyperparameters are unchanged, and the change is applied to both methods.
Table R2: Ant U5 Maze at depth 32 with hindsight goals only and with commanded goals in half of the actor batch.
| Actor goals | Final, CRL | Final, NRC2Reg | AUC, CRL | AUC, NRC2Reg | |
|---|---|---|---|---|---|
| Ant U5 Maze | hindsight | 83 | 90 | 0.037 | 0.041 |
| hindsight + commanded | 160 | 194 | 0.083 | 0.126 |
With commanded goals, the final time at goal and the AUC rise for both methods, no run ends at zero, and NRC2Reg has the higher AUC.
We note that this remedy concerns how CRL samples the goals of its actor and is independent of NRC2Reg, so it does not change the method or the central conclusions of the paper. It is an addition for antmaze navigation tasks with far goals. The results of the paper stay under the original protocol, and the new Appendix F gives the full analysis.
Q2: Does NRC2Reg work beyond contrastive RL?
Yes! The regularizer itself needs only a linear readout and an intermediate feature, and its motivation carries over to other problems: neural regression collapse [3] describes regression networks whose features settle into the row space of the readout, and NRC2Reg asks an intermediate layer to do the same. We therefore ran preliminary experiments in three other settings, where we added only the NRC2Reg term to a published setup with everything else unchanged.
Visual behavior cloning. We train a visual behavior-cloning policy on the expert Cheetah-run data of the Visual D4RL datasets, following the setup of [4]. The network consists of a convolutional encoder followed by a three-layer policy head, whose linear readout outputs the 6-dimensional action. NRC2Reg aligns one hidden layer of the policy head with that readout.
Table R3: Test MSE and normalized return on Cheetah-run. Return is averaged over 5 evaluation seeds with 100 episodes each.
| Test MSE | Return | |
|---|---|---|
| Baseline | 3.00 | 33.0 |
| NRC2Reg (λ = 1), hidden layer 1 / 2 / 3 | 2.56 / 2.82 / 2.92 | 42.3 / 40.8 / 33.7 |
NRC2Reg lowers the test error at all three layers, and the return is at least that of the baseline at all three. The gain in test error is largest at the first hidden layer and smallest at the last one. This is the same preference for an intermediate feature that we see in CRL, where aligning the last block learns the slowest (Section 5).
Author Rebuttal (Part 3/3)
Deep regression. We use the synthetic benchmark of [5], which tests whether a regularizer helps a network recover a low-dimensional target structure. The inputs have 100 dimensions, of which 96 are pure noise. The targets are 3-dimensional points that lie on a low-dimensional shape: a Swiss roll, a torus, a circle, or a mammoth-shaped point cloud. We keep their network, which has two hidden layers.
Table R4: Test MSE on the synthetic regression benchmark of [5], mean over 10 runs.
| Swiss Roll | Torus | Circle | Mammoth | |
|---|---|---|---|---|
| Baseline | 3.22 | 3.05 | 0.156 | 137 |
| NRC2Reg | 0.13 | 1.16 | 0.030 | 60 |
NRC2Reg lowers the test error on all four datasets. On Swiss Roll, the regularized feature falls from rank 16.5 to 3.0, the dimension of the target, which is what neural regression collapse predicts [3]. As in behavior cloning, the layer matters: on Swiss Roll the gain comes from the intermediate layer, and the same term on the last hidden layer does not help.
Quasimetric RL. Within RL, we also tried NRC2Reg on quasimetric RL (QRL) [6], a goal-conditioned method that is not contrastive. QRL learns a quasimetric in a learned representation space, so we expected that shaping this representation would help there as well. On the online FetchPush task, NRC2Reg on the first hidden layer of the encoders and the actor raises the final success rate from 0.906 to 0.955 with λ = 0.1 and to 0.923 with λ = 1.
Why contrastive RL. Among online RL algorithms, we chose CRL because it is the one that rests most directly on representation learning. Its critic is the distance between two learned embeddings, so the algorithm succeeds or fails with the quality of these representations, and a regularizer of the representation acts exactly where it matters. CRL is also the algorithm for which very deep networks have been shown to pay off. It is therefore the setting that offers the intermediate features of a deep network to regularize.
We regard the results outside CRL as preliminary, and CRL remains the focus of the paper. The new Appendix G of the revised paper reports these experiments in full and contains more settings.
If the above resolves your concerns, we kindly ask you to reconsider the overall evaluation. We are happy to discuss any follow-up questions.
References:
[1] Agarwal et al. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021.
[2] Palenicek et al. XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning. ICLR, 2026.
[3] Andriopoulos et al. The Prevalence of Neural Collapse in Neural Multivariate Regression. NeurIPS, 2024.
[4] Dong et al. Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension. NeurIPS, 2026.
[5] Zhang et al. Deep Regression Representation Learning with Topology. ICML, 2024.
[6] Wang et al. Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning. ICML, 2023.
Author Rebuttal (Part 1/3)
We thank the reviewer for the clear review, and for recognizing the idea of the regularizer, its gains on the hard humanoid and manipulation tasks, and the comparison with fixed bottlenecks. Below, we address each of the points raised.
W1: Number of seeds
We followed the experimental setup of scaled CRL and ran three seeds for most configurations, given the cost of training such deep networks. We agree, however, that more seeds present the results more faithfully. We therefore extended our main experiments of the paper to six seeds. Following [1], we now report 95% bootstrap confidence intervals over seeds, so the intervals reflect the variation across independent runs. To compare the two methods, we compute the confidence interval of their difference (NRC2Reg minus CRL) in the same way, resampling the runs of each method independently. When this interval lies entirely above zero, the gain cannot be explained by which seeds happened to be run. Three observations stand out:
The conclusions of the paper are unchanged. Across the ten humanoid and arm settings, NRC2Reg has the higher AUC in all ten and the higher final performance in nine, and its gains remain largest on the hard tasks. For example, the final time at goal rises from 378 to 515 on Arm Push Hard (depth 32) and from 21 to 135 on Humanoid Big Maze (depth 64).
Some gains are now more pronounced. With six seeds, the confidence interval of the gain in final performance excludes zero on Arm Push Hard (depth 32), Humanoid U-Maze (depth 32) and Humanoid Big Maze (depth 32/64), and the interval of the gain in AUC excludes zero on six of the ten settings. For example, on Humanoid Big Maze (depth 64), the gain itself grows from +62 to +114, and its interval from [−24, +137] to [+31, +179].
The estimates are more precise where the claims rest. For example, on Humanoid, the interval of the final performance narrows from 119 to 76 points for CRL and from 148 to 96 points for NRC2Reg. On Arm Push Hard (depth 32), the interval of the gain narrows from [+101, +236] to [+93, +186].
The revised Table 5 (Appendix D) reports every setting with these intervals, Appendix B describes how they are computed, and Figures 2 to 4 are redrawn with six seeds.
W2: Better conditioning rather than a low-dimensional representation?
Recent work [2] does link good conditioning to sample-efficient off-policy RL: critics with a better-conditioned loss landscape reach higher returns, and a low condition number is desirable in general. However, this evidence is mostly correlational. We therefore tested it directly.
A control with the same conditioning. We trained CRL without NRC2Reg and added to the critic loss a term that holds the condition number of both critic readouts, Wϕ and Wψ, at 170, the average value that NRC2Reg reaches on this task. For a readout W with singular values σ1≥⋯≥σn, the term is
Lcond(W)=n1∑i=1nmax(0, 1−sg(σ1)170,σi)2,
where sg(⋅) is the stop-gradient operator. The term is zero once the condition number is at most 170, and otherwise it raises only the singular values that are too small.
Table R1: Humanoid (depth 32) at the end of training (Wϕ: critic readout; H3: block-3 feature).
The control has the same readout condition number and the same readout effective rank as NRC2Reg, but it does not reproduce the gain of NRC2Reg. Neither its final performance nor its AUC is above that of CRL. What separates the two lies upstream of the readout. The block-3 feature of the control stays as high-dimensional as in CRL (rank 73 against 71), while NRC2Reg reduces it to rank 10. This is also why the effective rank that the reviewer refers to is almost unchanged: it is measured at the readout, where it is set by the task (about 12 with or without NRC2Reg), whereas the reduction caused by NRC2Reg happens in the block-3 feature. We therefore read the better conditioning of the readout as a by-product of aligning that feature with the readout, not as the source of the gain.