acceptodds
Under review as a conference paper at ICLR 2027

Representational Manifold Hypothesis Enables Sample-efficient Online Contrastive Reinforcement Learning

Abstract

General-purpose mechanisms from supervised deep learning, such as residual blocks, normalization, and depth, have recently been shown to transfer to online reinforcement learning (RL) and to improve its sample efficiency. One further mechanism, the manifold hypothesis, has been observed in RL but used only architecturally, through bottlenecks whose fixed rank must be chosen before training. We instead make the representation low-dimensional by learning. Building on neural regression collapse, we propose NRC2Reg, a regularizer that adds no parameters and pulls intermediate features toward the row space of the network's own readout, with a single coefficient and no rank to tune. On Contrastive RL, the features settle near task intrinsic dimension at effective rank ten of a 64-dimensional budget, a regime no swept hard bottleneck reaches, while NRC2Reg improves sample efficiency in all seven environment and depth settings and final performance in six, with the largest gains on maze tasks where the unregularized agent barely learns.

Primary area:
reinforcement learning
Keywords:
online reinforcement learning, manifold hypothesis, representation learning, contrastive RL, sample efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GYK5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether encouraging low-dimensional representations through an explicit learning objective can improve the sample efficiency of reinforcement learning. Motivated by the Neural Regression Collapse (NRC) phenomenon, the authors propose NRC2Reg, a representation regularizer that encourages intermediate network features to align with the row space of the network's final linear readout. The authors apply NRC2Reg to contrastive RL, regularizing both the critic's state-action and goal encoders and the actor network. Experiments across nine goal-conditioned RL environments at multiple network depths demonstrate substantial improvements in challenging tasks.

Critical strengths
  1. Using Neural Regression Collapse as a training regularizer is an interesting idea. NRC2Reg requires no additional learnable parameters and can be integrated into existing residual network architectures through an auxiliary loss.
  2. NRC2Reg provides significant improvements on some difficult goal-conditioned control tasks, especially Humanoid U-Maze, Humanoid Big Maze, and Arm Push Hard.
  3. The authors investigate effective rank, intrinsic dimension, readout conditioning, and estimated feature information. The comparison between fixed orthogonal bottlenecks and NRC2Reg provides useful insight into how different approaches to dimensionality reduction affect representation learning.
Critical weaknesses
  1. Although the experiments include nine environments and several network depths, most results use only three random seeds. I strongly encourage the authors to run more random seeds to ensure the reliability of the results.
  2. The experiments show that NRC2Reg substantially reduces the condition number of the readout matrix, while its effective rank remains almost unchanged. This raises an alternative hypothesis that the observed performance improvements may be partly attributable to improved optimization and conditioning rather than low-dimensional representation learning itself.
Questions for authors
  1. Table 5 shows that NRC2Reg decreases AUC across all AntMaze settings. Could the authors investigate why the regularizer improves learning on challenging Humanoid and manipulation tasks but not on AntMaze?
  2. NRC2Reg is motivated by a general representational property of networks, but all experiments use contrastive RL. Does NRC2Reg work beyond contrastive RL?
5 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1/3)

We thank the reviewer for the clear review, and for recognizing the idea of the regularizer, its gains on the hard humanoid and manipulation tasks, and the comparison with fixed bottlenecks. Below, we address each of the points raised.


W1: Number of seeds

We followed the experimental setup of scaled CRL and ran three seeds for most configurations, given the cost of training such deep networks. We agree, however, that more seeds present the results more faithfully. We therefore extended our main experiments of the paper to six seeds. Following [1], we now report 95% bootstrap confidence intervals over seeds, so the intervals reflect the variation across independent runs. To compare the two methods, we compute the confidence interval of their difference (NRC2Reg minus CRL) in the same way, resampling the runs of each method independently. When this interval lies entirely above zero, the gain cannot be explained by which seeds happened to be run. Three observations stand out:

  • The conclusions of the paper are unchanged. Across the ten humanoid and arm settings, NRC2Reg has the higher AUC in all ten and the higher final performance in nine, and its gains remain largest on the hard tasks. For example, the final time at goal rises from 378 to 515 on Arm Push Hard (depth 32) and from 21 to 135 on Humanoid Big Maze (depth 64).

  • Some gains are now more pronounced. With six seeds, the confidence interval of the gain in final performance excludes zero on Arm Push Hard (depth 32), Humanoid U-Maze (depth 32) and Humanoid Big Maze (depth 32/64), and the interval of the gain in AUC excludes zero on six of the ten settings. For example, on Humanoid Big Maze (depth 64), the gain itself grows from +62 to +114, and its interval from [−24, +137] to [+31, +179].

  • The estimates are more precise where the claims rest. For example, on Humanoid, the interval of the final performance narrows from 119 to 76 points for CRL and from 148 to 96 points for NRC2Reg. On Arm Push Hard (depth 32), the interval of the gain narrows from [+101, +236] to [+93, +186].

The revised Table 5 (Appendix D) reports every setting with these intervals, Appendix B describes how they are computed, and Figures 2 to 4 are redrawn with six seeds.


W2: Better conditioning rather than a low-dimensional representation?

Recent work [2] does link good conditioning to sample-efficient off-policy RL: critics with a better-conditioned loss landscape reach higher returns, and a low condition number is desirable in general. However, this evidence is mostly correlational. We therefore tested it directly.

A control with the same conditioning. We trained CRL without NRC2Reg and added to the critic loss a term that holds the condition number of both critic readouts, WϕW_\phi and WψW_\psi, at 170, the average value that NRC2Reg reaches on this task. For a readout WW with singular values σ1≥⋯≥σn\sigma_1 \geq \dots \geq \sigma_n, the term is

Lcond(W)=1n∑i=1nmax⁡(0, 1−170,σisg(σ1))2,\mathcal{L}_{\mathrm{cond}}(W) = \frac{1}{n} \sum_{i=1}^{n} \max\left(0,\ 1 - \frac{170\\,\sigma_i}{\mathrm{sg}(\sigma_1)}\right)^2,

where sg(⋅)\mathrm{sg}(\cdot) is the stop-gradient operator. The term is zero once the condition number is at most 170, and otherwise it raises only the singular values that are too small.

Table R1: Humanoid (depth 32) at the end of training (WϕW_\phi: critic readout; H3H_3: block-3 feature).

FinalAUCκ(Wϕ)\kappa(W_\phi)rank⁡99(Wϕ)\operatorname{rank}_{99}(W_\phi)rank⁡99(H3)\operatorname{rank}_{99}(H_3)
CRL6510.25810^4.71271
CRL +Lcond+ \mathcal{L}_{\mathrm{cond}}6320.20010^2.21173
NRC2Reg6730.28210^2.21210

The control has the same readout condition number and the same readout effective rank as NRC2Reg, but it does not reproduce the gain of NRC2Reg. Neither its final performance nor its AUC is above that of CRL. What separates the two lies upstream of the readout. The block-3 feature of the control stays as high-dimensional as in CRL (rank 73 against 71), while NRC2Reg reduces it to rank 10. This is also why the effective rank that the reviewer refers to is almost unchanged: it is measured at the readout, where it is set by the task (about 12 with or without NRC2Reg), whereas the reduction caused by NRC2Reg happens in the block-3 feature. We therefore read the better conditioning of the readout as a by-product of aligning that feature with the readout, not as the source of the gain.

AuthorsOpenReview ↗

Author Rebuttal (Part 2/3)

Q1: Why does NRC2Reg not help on the ant mazes?

We wondered about the same question and investigated it in detail! In short, learning on the ant mazes is limited by a weakness of CRL itself in reaching far goals. It affects both methods, and a regularizer of the representation is not meant to fix it. A small change to how the actor's goals are sampled addresses it, as we will show below.

A failure case shared by both methods. Our starting point was Ant U5 Maze, where there were runs of both CRL and NRC2Reg that ended with zero time at goal. This maze is evaluated on far goals only, which lie behind a wall, so the ant must first move away from the goal to reach it. The actor, however, is trained only on hindsight goals, that is, on positions the agent has already visited, so a far goal that has not been reached yet is a goal the actor was never trained on. Our trained checkpoints confirm this: at the far goals of U5, the critic prefers about 40% of uniformly sampled actions to the action chosen by the actor, for CRL and NRC2Reg alike. The first far-goal success therefore depends on how the actor extrapolates, and a run in which it does not happen stays at zero. The two methods differ in this extrapolation: the actor trained with NRC2Reg tends to remain near the start for longer, so its first far-goal success can come later, which is what the overall lower AUC in Table 5 reflects.

Why NRC2Reg helps on the humanoid mazes. The two task families are limited by different things. An untrained humanoid falls within a few dozen steps, which ends the episode, so it first has to learn to walk. By contrast, an untrained ant already survives most of the 1000-step episode. NRC2Reg makes this high-dimensional control easier to learn: in Figure 4(c, d) the humanoid stays upright longer, and in Figure 4(a, b) its first far-goal success arrives earlier. Interestingly, Figure 3 of the scaled CRL paper shows that, with a sufficiently deep network, the humanoid learns to vault over the wall of the U-Maze. This suggests that the humanoid may not have solved the problem of far goals either: it may simply be able to get over the wall, a shortcut that the body of the ant does not allow, so the ant stays stuck behind it. The ant also needs little help with locomotion, so the regularizer has less to gain there.

A minimal remedy. Since the far goals are missing from the actor's objective, we put them there: half of each actor batch is trained on the goal that was commanded when the data was collected, and the other half on hindsight goals as before. The critic, the regularizer and all hyperparameters are unchanged, and the change is applied to both methods.

Table R2: Ant U5 Maze at depth 32 with hindsight goals only and with commanded goals in half of the actor batch.

Actor goalsFinal, CRLFinal, NRC2RegAUC, CRLAUC, NRC2Reg
Ant U5 Mazehindsight83900.0370.041
hindsight + commanded1601940.0830.126

With commanded goals, the final time at goal and the AUC rise for both methods, no run ends at zero, and NRC2Reg has the higher AUC.

We note that this remedy concerns how CRL samples the goals of its actor and is independent of NRC2Reg, so it does not change the method or the central conclusions of the paper. It is an addition for antmaze navigation tasks with far goals. The results of the paper stay under the original protocol, and the new Appendix F gives the full analysis.


Q2: Does NRC2Reg work beyond contrastive RL?

Yes! The regularizer itself needs only a linear readout and an intermediate feature, and its motivation carries over to other problems: neural regression collapse [3] describes regression networks whose features settle into the row space of the readout, and NRC2Reg asks an intermediate layer to do the same. We therefore ran preliminary experiments in three other settings, where we added only the NRC2Reg term to a published setup with everything else unchanged.

Visual behavior cloning. We train a visual behavior-cloning policy on the expert Cheetah-run data of the Visual D4RL datasets, following the setup of [4]. The network consists of a convolutional encoder followed by a three-layer policy head, whose linear readout outputs the 6-dimensional action. NRC2Reg aligns one hidden layer of the policy head with that readout.

Table R3: Test MSE and normalized return on Cheetah-run. Return is averaged over 5 evaluation seeds with 100 episodes each.

Test MSEReturn
Baseline3.0033.0
NRC2Reg (λ = 1), hidden layer 1 / 2 / 32.56 / 2.82 / 2.9242.3 / 40.8 / 33.7

NRC2Reg lowers the test error at all three layers, and the return is at least that of the baseline at all three. The gain in test error is largest at the first hidden layer and smallest at the last one. This is the same preference for an intermediate feature that we see in CRL, where aligning the last block learns the slowest (Section 5).

AuthorsOpenReview ↗

Author Rebuttal (Part 3/3)

Deep regression. We use the synthetic benchmark of [5], which tests whether a regularizer helps a network recover a low-dimensional target structure. The inputs have 100 dimensions, of which 96 are pure noise. The targets are 3-dimensional points that lie on a low-dimensional shape: a Swiss roll, a torus, a circle, or a mammoth-shaped point cloud. We keep their network, which has two hidden layers.

Table R4: Test MSE on the synthetic regression benchmark of [5], mean over 10 runs.

Swiss RollTorusCircleMammoth
Baseline3.223.050.156137
NRC2Reg0.131.160.03060

NRC2Reg lowers the test error on all four datasets. On Swiss Roll, the regularized feature falls from rank 16.5 to 3.0, the dimension of the target, which is what neural regression collapse predicts [3]. As in behavior cloning, the layer matters: on Swiss Roll the gain comes from the intermediate layer, and the same term on the last hidden layer does not help.

Quasimetric RL. Within RL, we also tried NRC2Reg on quasimetric RL (QRL) [6], a goal-conditioned method that is not contrastive. QRL learns a quasimetric in a learned representation space, so we expected that shaping this representation would help there as well. On the online FetchPush task, NRC2Reg on the first hidden layer of the encoders and the actor raises the final success rate from 0.906 to 0.955 with λ = 0.1 and to 0.923 with λ = 1.

Why contrastive RL. Among online RL algorithms, we chose CRL because it is the one that rests most directly on representation learning. Its critic is the distance between two learned embeddings, so the algorithm succeeds or fails with the quality of these representations, and a regularizer of the representation acts exactly where it matters. CRL is also the algorithm for which very deep networks have been shown to pay off. It is therefore the setting that offers the intermediate features of a deep network to regularize.

We regard the results outside CRL as preliminary, and CRL remains the focus of the paper. The new Appendix G of the revised paper reports these experiments in full and contains more settings.


If the above resolves your concerns, we kindly ask you to reconsider the overall evaluation. We are happy to discuss any follow-up questions.

References:

[1] Agarwal et al. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021.

[2] Palenicek et al. XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning. ICLR, 2026.

[3] Andriopoulos et al. The Prevalence of Neural Collapse in Neural Multivariate Regression. NeurIPS, 2024.

[4] Dong et al. Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension. NeurIPS, 2026.

[5] Zhang et al. Deep Regression Representation Learning with Topology. ICML, 2024.

[6] Wang et al. Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning. ICML, 2023.

Reviewer_toNcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NRC2Reg, which aligns intermediate features with the row space of each network’s own readout. It encourages low-dimensional representations through a training objective rather than a fixed architectural bottleneck and evaluates the approach on several goal-conditioned control tasks using Contrastive RL.

Critical strengths

S1. The method is simple and introduces no additional trainable parameters.

S2. It produces substantial improvements on challenging Humanoid mazes and some manipulation tasks.

S3. The paper supplements performance evaluations with analyses of feature rank, intrinsic dimension, and readout conditioning to investigate the mechanism.

Critical weaknesses

W1. The seed count and statistical reliability are insufficient. The three-seed evaluation described in Section 2.2 provides limited evidence for the robustness of the improvements, and some configurations in Table 5 use only two seeds. Even the cited work by Agarwal et al. (2021) reports that bootstrap confidence intervals can underestimate uncertainty with three runs. Pooling the last five evaluation epochs when computing IQM does not increase the number of independent runs. Furthermore, the standard error of the retained values after trimming does not adequately account for temporal dependence within runs and variability across runs.

W2. The use of absolute block index 3 is insufficiently justified. The paper uses (X=3) at every depth, although this corresponds to the third of four blocks at depth 16 and the third of sixteen blocks at depth 64. The rationale for using the same absolute index despite these substantially different relative positions is unclear.

W3. Potential interference with early representation expansion and the choice of (X=3) as a compromise remain insufficiently examined. The rank of (h_X) expands early in training, raising the question of whether NRC2Reg interferes with this process. The particularly high AUC under earlier regularization could reflect rapid convergence to a local optimum within a restricted representation. In contrast, the configurations with higher final performance can be interpreted as placing the direct alignment constraint later, allowing representations to develop through more blocks before combining the regularized representation with subsequent residual transformations that are not directly regularized. These observations raise doubts about whether (X=3) is an appropriate compromise.

W4. The hyperparameter choices may not transfer across tasks. Detailed ablations are restricted to Humanoid. It is therefore unclear whether the optimal coefficient and placement vary across tasks or can be fixed independently of the task. Figure 5(a) suggests that (\lambda=0.1) may be more favorable during parts of training, but quantitative comparisons supporting the default choice of (\lambda=1) are missing.

W5. The manuscript’s completeness and reporting require improvement. Section 3 ends with the incomplete phrase “in practice, this.” The abstract’s seven environment–depth settings do not match the fourteen settings in Table 5. The nine environments in Table 3 and the “remaining eight” in Section 5.1 can be understood as excluding Humanoid from the latter, but the experimental scope should be reported consistently. Some configurations also have lower AUC, contradicting claims of improved sample efficiency in every setting. Together with the limited seed count and detailed ablations on only one environment, these writing and reporting issues give the impression of a manuscript prepared in haste, before sufficient experimental validation and editorial review.

W6. The suitability of AUC as a central evaluation metric is questionable. Rapid convergence to a local optimum early in training can produce high AUC despite lower final performance. Higher AUC therefore does not necessarily indicate a better final policy or a more desirable learning process. This is particularly relevant here because regularization placement affects AUC and final performance differently. The use of AUC as a central justification for improved sample efficiency and the default configuration requires further explanation.

Questions for authors

Q1. Please address the weaknesses raised above.

6 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1/3)

We thank the reviewer for the detailed and specific review. The weaknesses concern two topics: the reliability of our statistics (W1, W6), and how the regularized block and the coefficient are chosen (W2 to W4). We ran new experiments for each of them, and we have corrected the writing issues raised in W5.


W1: Number of seeds and statistical method

We followed the experimental setup of scaled CRL and ran three seeds for most configurations, given the cost of training such deep networks. We agree, however, that more seeds present the results more faithfully. We therefore extended our main experiments of the paper to six seeds. Following [1], we now report 95% bootstrap confidence intervals over seeds, so the intervals reflect the variation across independent runs. To compare the two methods, we compute the confidence interval of their difference (NRC2Reg minus CRL) in the same way, resampling the runs of each method independently. When this interval lies entirely above zero, the gain cannot be explained by which seeds happened to be run. Three observations stand out:

  • The conclusions of the paper are unchanged. Across the ten humanoid and arm settings, NRC2Reg has the higher AUC in all ten and the higher final performance in nine, and its gains remain largest on the hard tasks. For example, the final time at goal rises from 378 to 515 on Arm Push Hard (depth 32) and from 21 to 135 on Humanoid Big Maze (depth 64).

  • Some gains are now more pronounced. With six seeds, the confidence interval of the gain in final performance excludes zero on Arm Push Hard (depth 32), Humanoid U-Maze (depth 32) and Humanoid Big Maze (depth 32/64), and the interval of the gain in AUC excludes zero on six of the ten settings. For example, on Humanoid Big Maze (depth 64), the gain itself grows from +62 to +114, and its interval from [−24, +137] to [+31, +179].

  • The estimates are more precise where the claims rest. For example, on Humanoid, the interval of the final performance narrows from 119 to 76 points for CRL and from 148 to 96 points for NRC2Reg. On Arm Push Hard (depth 32), the interval of the gain narrows from [+101, +236] to [+93, +186].

The revised Table 5 (Appendix D) reports every setting with these intervals, Appendix B describes how they are computed, and Figures 2 to 4 are redrawn with six seeds.

We note that averaging the last five evaluations and reporting a standard error follow the protocol of scaled CRL, where the window smooths out the fluctuations of the learning curves from one evaluation to the next. We kept them to stay aligned with that work. With six seeds, the bootstrap intervals above become more reliable, and because they resample whole runs, they also alleviate the other two concerns: the pooled IQM now serves only as the point estimate, and all uncertainty is measured across independent runs.


W2 & W3: The choice of the block index X = 3

Section 5.2 does not present X = 3 as the best index. It states that (1) regularizing early speeds up learning while regularizing late raises the ceiling; (2) the first and the last block are never good targets, and (3) block 3 is chosen to balance sample efficiency and final performance. The results below agree with each of these statements. The real research question behind this position ablation was never which block is best, but how to turn the raw input into a feature that is low-dimensional (which NRC2Reg provides), informative (without the information loss of a bottleneck) and refined (by enough computation before it is compressed). The position trades these properties against each other. Regularizing early makes the feature easy to optimize and speeds up learning, but the input has passed through few blocks by then. Regularizing late acts on a more refined feature, but learning takes off late. Before submission we tried to remove this trade-off by architecture, by refining the input further before the regularized block without moving that block later. We tried to prepend a weight-shared block that is applied to the input recursively [2], or a mixture-of-experts module [3], and then to regularize its output. Neither gave a satisfactory solution. We see this as an open direction and will discuss it in the revised paper. X = 3 is thus a default that gives a gain in most settings, and the experiments below address your specific concerns.

AuthorsOpenReview ↗

Author Rebuttal (Part 2/3)

Does NRC2Reg interfere with the early expansion of the rank? It changes the scale of the expansion rather than its shape. In Figure 1(a), the rank of the block-3 feature still rises in the early epochs and then falls, as in CRL, but it peaks near 30 instead of 156. To see whether the larger expansion matters for the final policy, we let it happen first and switched NRC2Reg on only from epoch 10 or 25.

Table R1: Delayed start of NRC2Reg on Humanoid (depth 32); H3H_3: block-3 feature.

NRC2Reg activefrom the startfrom epoch 10from epoch 25
rank⁡99(H3)\operatorname{rank}_{99}(H_3) at epoch 530157156
Final673669679

The final performance is the same, so limiting the early expansion from the start does not cost anything at the end.

Is the fast early learning a convergence to a restricted solution? If it were, CRL should overtake with more training. For a stable estimate, we pool the position sweep of Section 5 with a second sweep of the same setting, which ran for 150M steps.

Table R2: Humanoid at depth 32, 150M steps

CRLX = 1X = 3X = 5X = 7X = 8
At 50M1231271711214631
At 75M543507585604542393
At 100M650613686656699629
At 150M716720772768811767

The gain of X = 3 is still present at 150M (772 against 716), so the fast early learning does not come at the price of a worse solution.

Do later positions end higher, and is the same absolute index appropriate at every depth? We tested the two depths at which the relative position of X = 3 differs most.

Table R3: Humanoid at depth 64 (16 blocks), 100M steps.

CRLX = 3X = 6X = 12
At 50M9536417070
At 75M557631631549
Final637682757720

Table R4: Humanoid at depth 16 (4 blocks), 100M steps

CRLX = 1X = 2X = 3X = 4
At 50M3325306031
At 75M229120160283147
Final522309488521411

At depths 16 and 32, the first and the last block are the weakest positions. At depths 32 and 64, an earlier interior block learns faster, while a later one takes off later but tends to end higher, as stated in Section 5.2. An absolute index of 3 therefore stays away from both ends at every depth we use, and at depths 32 and 64 it lies on the side of fast learning with a clear gain at the end of training. This is why we use it as the default, while a later block can be preferable when only the final performance matters. The paper claims no more than this about X = 3. A new appendix section of the revised paper shows the learning curves of the runs in Tables R1 to R3.


W4: Do the coefficient and the placement transfer across tasks?

We again do not claim that λ = 1 and X = 3 are the best values for every task: a grid search on a given task will always find a better setting. Our claim is that they are a safe default, which we fixed once and did not tune per task. To test this, we repeated both ablations on a second task family.

Table R5: Placement on Arm Push Hard (depth 32).

CRLX = 1X = 3X = 5X = 7X = 8
At 50M221354450467345335
At 75M362355464515439421
Final378419515552491439

Table R6: Performance at 50M / 75M / 100M steps for three coefficients (depth 32).

CRLλ = 0.1λ = 1λ = 10
Arm Push Hard221 / 362 / 378338 / 405 / 456450 / 464 / 515419 / 423 / 459
Arm Binpick Hard83 / 141 / 17696 / 144 / 197125 / 150 / 198108 / 125 / 159
  • Placement. On Arm Push Hard every position ends above CRL, and the pattern is the one of Humanoid: the first and the last block are the weakest, the early interior blocks learn fastest, and the late blocks take off later and are still rising at the end. X = 3 is thus again on the side of fast learning, with a clear gain at the end.

  • Coefficient. On the two arm tasks, λ = 1 is the best of the three values. The best coefficient may vary between tasks, but λ = 1 is above CRL on both tasks at every point of training.

The default also carries over beyond CRL. In the preliminary experiments that we add in a new appendix, λ = 1 without tuning lowers the test error in visual behavior cloning and on three of four deep regression datasets, and it raises the success rate of quasimetric RL. The learning curves of the runs above are in the new appendix section.

AuthorsOpenReview ↗

Author Rebuttal (Part 3/3)

W5: Completeness and consistent reporting

Thank you for the careful reading. We acknowledge them and have corrected these points in the revised paper:

  • Incomplete sentence. The sentence at the end of Section 3 is completed.
  • Seven and fourteen settings. The abstract counted each environment family once per depth, whereas Table 5 lists every configuration separately. This compact way of counting was indeed confusing, and both now use the same count.
  • Nine and eight environments. Table 3 and Section 5.1 now describe the experimental scope in the same way.
  • Settings with a lower AUC. We have modified the claim to "NRC2Reg raises the AUC in every setting outside the ant mazes and remains competitive with the baseline on the ant mazes."

W6: The role of AUC in our evaluation

We agree that a high AUC does not by itself imply a better final policy, and we were aware of this when writing the paper. For this reason AUC never appears alone. It was used only in Table 1, Table 5, Figure 5, and Figure 6 of the original submission, as a compact summary where space is limited, and is always next to the final performance. Section 5.2 makes the very point you raise: it states that final performance and sample efficiency favor different positions.

AUC is also not our central evidence for sample efficiency. For the main experiments on which the claims of the paper rest, we show the learning curves (Figures 2 to 4 and Figure 5(a)), where the sample efficiency is directly visible. In the revised paper, a new appendix section adds the learning curves of all ablations, including the settings of Figure 5(b, c).

On the specific risk of a fast convergence to a poor solution, we have discussed that under W2 & W3: the gain of the default setting is still present at 150M steps, and switching the regularizer on later leads to the same final performance.


If the above resolves your concerns, we kindly ask you to reconsider the overall evaluation. We are happy to discuss any follow-up questions.

References:

[1] Agarwal et al. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021.

[2] Ghugare et al. On the Role of Computation in Reinforcement Learning. ICML, 2026.

[3] Willi et al. Mixture of Experts in a Mixture of RL Settings. RLC, 2024.

Reviewer_qgQ5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to add a regularizer inspired by neural regression collapse throughout training in constrastive, goal-conditioned RL tasks. This regularizer encourages learning a bottleneck in a flexible manner, which may help the network exclusively keep relevant information to the task. Experiments showcase the benefits of this approach compared to baselines, such as imposing a hard bottleneck using narrow layers. Additional investigations are done to study where the regularizer is most beneficial and how it reduces the rank of the intermediate representations.

Critical strengths

The main idea is novel and interesting. Incorporating an observation from deep learnign theory (neural collapse) into the training process is an intriguing direction, especially as it appears to give meaningful benefits to the algortihms. The experiments, including deeper investigations, chosen are appropriate and yield interesting results.

The writing and presentation are clear, ideas and contributions are well-explained and easy to follow.

Critical weaknesses

No major weaknesses.

Some of the learning curves (Fig.4, a and b plots) are very noisy, and it's difficult to say if the results are statistically significant. Running more random seeds could be necessary to properly assess the porposed method.

Questions for authors

The suggested experiments in the following section would be a great adddition to the paper, although not absolutely necessary. With satisfactory responses, I would be willing to increase my score.

  • The proposed regularizer seems agnostic to the dataset/application. Constrastive goal-conditioned RL is one particular choice but, in principle, the regularizer could be applied to other settings. Would this method give benefits in another context or is there a reason the method would be particlarly effecrive in constrastive RL? For example, classification seems like a natural fit where the number of classes is often much lower than the dimension of the hidden layers. Can you test a different domain?

  • Ablations were conducted using the regularizer on different blocks of the network. Based on the motivation, a natural choice would be to apply the regularizer to all blocks or many blocks to preserve only task-relevant information. e.g. an alternative is to apply to all blocks past block X. Could this be succesful? How important is it to maintain larger dimensions for some blocks?

  • It's interesting that, even with a large regularization coefficient, the performance doesn't dip below the baseline (Fig.5). Is this to be expected? I would guess that once it is large enough, it would contribute large gradients which would destabilize the entire learning process. What if we enforced the NRC2 condition separately? For example, by interleaving a separate optimization process which reduces the NRC2 loss exclusively until it becomes quite small. This could be run at regular intervals. This would separate the gradients of the regularizer from the main loss but maintain the desired NRC2 constraint.

4 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1/3)

We thank the reviewer for the encouraging assessment and for the very concrete suggestions! We ran each of the suggested experiments during the rebuttal and address the points one by one below.


W1: Noisy curves and the number of seeds

We followed the experimental setup of scaled CRL and ran three seeds for most configurations, given the cost of training such deep networks. We agree, however, that more seeds present the results more faithfully. We therefore extended our main experiments of the paper to six seeds. Following [1], we now report 95% bootstrap confidence intervals over seeds, so the intervals reflect the variation across independent runs. To compare the two methods, we compute the confidence interval of their difference (NRC2Reg minus CRL) in the same way, resampling the runs of each method independently. When this interval lies entirely above zero, the gain cannot be explained by which seeds happened to be run. Three observations stand out:

  • The conclusions of the paper are unchanged. Across the ten humanoid and arm settings, NRC2Reg has the higher AUC in all ten and the higher final performance in nine, and its gains remain largest on the hard tasks. For example, the final time at goal rises from 378 to 515 on Arm Push Hard (depth 32) and from 21 to 135 on Humanoid Big Maze (depth 64).

  • Some gains are now more pronounced. With six seeds, the confidence interval of the gain in final performance excludes zero on Arm Push Hard (depth 32), Humanoid U-Maze (depth 32) and Humanoid Big Maze (depth 32/64), and the interval of the gain in AUC excludes zero on six of the ten settings. For example, on Humanoid Big Maze (depth 64), the gain itself grows from +62 to +114, and its interval from [−24, +137] to [+31, +179].

  • The estimates are more precise where the claims rest. For example, on Humanoid, the interval of the final performance narrows from 119 to 76 points for CRL and from 148 to 96 points for NRC2Reg. On Arm Push Hard (depth 32), the interval of the gain narrows from [+101, +236] to [+93, +186].

The revised Table 5 (Appendix D) reports every setting with these intervals, Appendix B describes how they are computed, and Figures 2 to 4 are redrawn with six seeds.

Fig. 4(a, b). These two panels are noisier than the rest of the paper for a specific reason: they show time at goal on the far-goal layout of the humanoid mazes, and far goals are reached rarely. At the end of training, the time at goal on this layout is at most 55 of 1000 steps, and many runs stay at zero, so each point of the curves rests on a few successful episodes. With six seeds, NRC2Reg remains ahead in all four maze settings, and more of its runs reach a far goal at some point of training (20 of 24 against 9 of 24 for CRL).


Q1: Does NRC2Reg help outside contrastive RL?

Thank you for this suggestion! We agree that the regularizer itself needs only a linear readout and an intermediate feature, and its motivation carries over to other problems: neural regression collapse describes regression networks whose features settle into the row space of the readout, and NRC2Reg asks an intermediate layer to do the same. We therefore ran preliminary experiments in three other settings, where we added only the NRC2Reg term to a published setup with everything else unchanged.

Visual behavior cloning. We train a visual behavior-cloning policy on the expert Cheetah-run data of the Visual D4RL datasets, following the setup of [2]. The network consists of a convolutional encoder followed by a three-layer policy head, whose linear readout outputs the 6-dimensional action. NRC2Reg aligns one hidden layer of the policy head with that readout.

Table R1: Test MSE and normalized return on Cheetah-run. Return is averaged over 5 evaluation seeds with 100 episodes each.

Test MSEReturn
Baseline3.0033.0
NRC2Reg (λ = 1), hidden layer 1 / 2 / 32.56 / 2.82 / 2.9242.3 / 40.8 / 33.7

NRC2Reg lowers the test error at all three layers, and the return is at least that of the baseline at all three. The gain in test error is largest at the first hidden layer and smallest at the last one. This is the same preference for an intermediate feature that we see in CRL, where aligning the last block learns the slowest (Section 5).

Deep regression. We use the synthetic benchmark of [3], which tests whether a regularizer helps a network recover a low-dimensional target structure. The inputs have 100 dimensions, of which 96 are pure noise. The targets are 3-dimensional points that lie on a low-dimensional shape: a Swiss roll, a torus, a circle, or a mammoth-shaped point cloud. We keep their network, which has two hidden layers.

Table R2: Test MSE on the synthetic regression benchmark of [3], mean over 10 runs.

Swiss RollTorusCircleMammoth
Baseline3.223.050.156137
NRC2Reg0.131.160.03060
AuthorsOpenReview ↗

Author Rebuttal (Part 2/3)

NRC2Reg lowers the test error on all four datasets. On Swiss Roll, the regularized feature falls from rank 16.5 to 3.0, the dimension of the target, which is what neural regression collapse predicts. As in behavior cloning, the layer matters: on Swiss Roll the gain comes from the intermediate layer, and the same term on the last hidden layer does not help.

Quasimetric RL. Within RL, we also tried NRC2Reg on Quasimetric RL (QRL) [4], a goal-conditioned method that is not contrastive. QRL learns a quasimetric in a learned representation space, so we expected that shaping this representation would help there as well. On the online FetchPush task, NRC2Reg on the first hidden layer of the encoders and the actor raises the final success rate from 0.906 to 0.955 with λ = 0.1 and to 0.923 with λ = 1.

Why contrastive RL. Among online RL algorithms, we chose CRL because it is the one that rests most directly on representation learning. Its critic is the distance between two learned embeddings, so the algorithm succeeds or fails with the quality of these representations, and a regularizer of the representation acts exactly where it matters. CRL is also the algorithm for which very deep networks have been shown to pay off. It is therefore the setting that offers the intermediate features of a deep network to regularize.

We regard the results outside CRL as preliminary, and CRL remains the focus of the paper. The new Appendix G of the revised paper reports these experiments in full and contains more settings.

For classification. The corresponding phenomenon is neural collapse; neural regression collapse is its extension to regression, where the targets are continuous. Ideas that are similar in spirit have been explored for classification. For example, [5] and [6] show that the last-layer classifier need not be learned and can be fixed to a simplex equiangular tight frame (NC2). These works prescribe the classifier of a classification network. NRC2Reg is different in kind: it is a loss term that aligns an intermediate feature with a learned readout, and it is derived for regression-type objectives. The works closest to ours are therefore those on regression [2, 3], which is why we took our additional tasks from them.


Q2: Regularizing many blocks, or all blocks past X

We ran both variants on Humanoid (depth 32): NRC2Reg on blocks 3 to 8, and on all eight blocks, with λ = 1 on each block and everything else unchanged.

Table R3: Humanoid (depth 32) at the end of training (H3H_3, H8H_8: features of block 3 and of the last block; ϕ\phi: state-action encoder output; ID: intrinsic dimension).

NRC2RegFinalAUCrank⁡99(H3)\operatorname{rank}_{99}(H_3)rank⁡99(H8)\operatorname{rank}_{99}(H_8)ID(H3H_3)ID(H8H_8)ID(ϕ\phi)
none (CRL)6510.2587120311.18.13.9
block 3 (paper)6730.282101799.07.03.9
blocks 3 to 86090.1854118.26.53.5
all blocks6050.1887207.56.13.4

Could it be successful? Both variants still train, but they learn more slowly than NRC2Reg on a single block, and their final performance is not higher than that of CRL.

Why? The features are over-compressed. The rank is at most 20 in every regularized block, and at block 3 it is 4 and 7, below even the frozen bottleneck with k=8k=8 of Figure 1. The intrinsic dimension shows where the two cases differ. NRC2Reg on a single block lowers the intrinsic dimension of the intermediate features from 11.1 to 9.0, but leaves that of the output ϕ(s,a)\phi(s,a) at 3.9: it removes degrees of freedom that do not reach the output. With many regularized blocks, the intrinsic dimension of the output falls as well, to 3.5 and 3.4, toward the 2.9 of the k=8k=8 bottleneck in Figure 1(c). Intrinsic dimension that is lost cannot be restored by later blocks [2]. Regularizing more blocks therefore does not keep only the task-relevant information, but removes part of it.

How important is it to keep larger dimensions in some blocks? What has to be kept is the intrinsic dimension of the output rather than the rank of the blocks. A low rank is costly at one place, the last feature, which enters the linear readout. In the literature on plasticity in deep RL, a low rank of this feature goes with a reduced ability to fit subsequent target values [7], and reduced plasticity shows as slower training progress rather than a higher plateau [8]. We observe the same: with many regularized blocks, the contrastive accuracy of the critic is 0.79 at 60M steps against 0.87 with one block, and it catches up by the end of training. The placement study of Section 5 locates the block that matters: NRC2Reg on the last block alone learns equally slowly (AUC 0.184), while NRC2Reg on block 7 alone, which leaves one free block before the readout, does not (final 732, AUC 0.264). A higher rank is not better in itself either: [7] report that raising the rank of the last feature reduced the performance of online DQN.

AuthorsOpenReview ↗

Author Rebuttal (Part 3/3)

Q3: Large coefficients, and enforcing NRC2 by a separate optimization

Is it expected that a large coefficient does not hurt? Yes. NRC2Reg is trained jointly with the critic. The NRC2 loss drops early and keeps decreasing over the whole training. A larger coefficient mainly makes the NRC2 loss proportionally smaller, so that its weighted share of the critic loss stays at about 0.1 to 0.2% throughout training. To test an extreme case, we ran λ = 100 on Humanoid (Table R4): the NRC2 loss is about 100 times smaller than with λ = 1, and the final performance stays at the same level.

What if NRC2 is enforced by a separate optimization? We implemented this suggestion: the training losses carry no NRC2 term, and an optimizer that minimizes only the NRC2 loss is interleaved with training, in three schedules.

Table R4: Humanoid (depth 32) at the end of training (H3H_3: block-3 feature).

NRC2 enforced byFinalrank⁡99(H3)\operatorname{rank}_{99}(H_3)NRC2 loss
not enforced (CRL)651710.55
joint training, λ = 1 (paper)673104e-4
joint training, λ = 10066884e-6
separate: one step after every training step112e-7
separate: same, only while the NRC2 loss exceeds 0.0034623e-3
separate: every 800 training steps, until the NRC2 loss is below 0.01465119e-3

The separate optimization does maintain the constraint, but all three schedules end below CRL. A step that minimizes only the NRC2 loss does not take the critic into account, and the easiest way to lower the loss is to collapse the feature. In the first two schedules, the block-3 feature falls to rank 1 to 2 and does not recover, although in the second schedule the separate steps make up only about 1% of all updates late in training. Once a feature is compressed this far, it is hard to restore. The third schedule, the closest to your description, avoids the collapse but still ends below CRL: between two phases the NRC2 loss drifts back to about 0.1, and each phase has to pull the feature back. Under joint training, both gradients enter the same update, so the feature that satisfies the constraint is also one that serves the critic.


We hope that our responses answer your questions, and we are happy to discuss any follow-up.

References:

[1] Agarwal et al. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021.

[2] Dong et al. Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension. NeurIPS, 2026.

[3] Zhang et al. Deep Regression Representation Learning with Topology. ICML, 2024.

[4] Wang et al. Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning. ICML, 2023.

[5] Yang et al. Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network? NeurIPS, 2022.

[6] Zhu et al. A Geometric Analysis of Neural Collapse with Unconstrained Features. NeurIPS, 2021.

[7] Kumar et al. Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning. ICLR, 2021.

[8] Lyle et al. Understanding Plasticity in Neural Networks. ICML, 2023.

AuthorsOpenReview ↗

Kind reminder that the author-reviewer discussion ends in 24 hours.

Dear Reviewer qgQ5,

We truly appreciate your thoughtful and encouraging reviews. Before the discussion period concludes in the next 24 hours, we wanted to check whether our rebuttal addressed your remaining points. If everything looks good on your end, we would be very thankful if you might consider increasing your score. Happy to provide any additional clarifications if needed!

Best,

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response Facilitating Reviewer-AC Discussion

We thank all reviewers for their thoughtful and constructive feedback. We have incorporated the suggested experiments, discussions, and analyses into the revised manuscript, with revisions highlighted in blue. Below, we summarize our core contributions and the major updates introduced during the rebuttal to facilitate any future discussion.


Core contributions of NRC2Reg

  1. A simple regularizer motivated by neural regression collapse (highlighted by qgQ5, toNc, GYK5): NRC2Reg aligns an intermediate feature of the CRL encoders with the row space of the network's own linear readout through an auxiliary loss. It adds no trainable parameters and has a single coefficient.
  2. Large gains on hard tasks (highlighted by toNc, GYK5): NRC2Reg improves the sample efficiency and the final performance of scaled CRL, most of all on Humanoid U-Maze, Humanoid Big Maze and Arm Push Hard, for example from 21 to 135 in time at goal on Humanoid Big Maze (depth 64).
  3. How learned compression differs from a fixed bottleneck (highlighted by qgQ5, toNc, GYK5): Through rank, intrinsic dimension, readout conditioning and mutual information, we show that a frozen orthogonal bottleneck removes task information along with rank, whereas NRC2Reg compresses the feature without this loss.

Major updates of experimental results during rebuttal

  • Table 5, Appendices B and D, Figures 2 to 4: Increased to six seeds for main experiments, with 95% bootstrap confidence intervals over seeds for each method and for their difference. The conclusions are unchanged, and the interval of the gain in final performance excludes zero on Arm Push Hard, Humanoid U-Maze and Humanoid Big Maze at depth 32 and on Humanoid Big Maze at depth 64 (addressing qgQ5's W1, toNc's W1 and GYK5's W1).
  • Appendix F: Analysis of the ant mazes. Learning there is limited by how CRL trains its actor on far goals, which affects both methods alike. Training half of each actor batch on the commanded goal raises the final performance of both on Ant U5 Maze, from 83 to 160 for CRL and from 90 to 194 for NRC2Reg (addressing GYK5's Q1).
  • Appendix G: Preliminary results beyond contrastive RL. NRC2Reg lowers the test error in visual behavior cloning and on a deep regression benchmark, and raises the success rate of quasimetric RL (addressing qgQ5's Q1 and GYK5's Q2).
  • Appendix H: Learning curves for the position of the regularizer at depth 64 and over 150M steps, its delayed start, the position and the coefficient on two manipulation tasks, and all settings of Figure 5(b, c) (addressing toNc's W2, W3, W4 and W6).

Additional experiments reported in the individual responses

  • A control that gives CRL the same readout conditioning as NRC2Reg but does not reproduce its gain (addressing GYK5's W2).
  • Regularizing many or all blocks, which over-compresses the features (addressing qgQ5's Q2).
  • Very large coefficients, and enforcing the NRC2 condition by a separate optimization (addressing qgQ5's Q3).
  • The position of the regularizer at depth 16 (addressing toNc's W2 & W3).

Major updates of discussions and clarifications during rebuttal

  • Section 5.2 and Conclusion: X = 3 and λ = 1 are presented as defaults that give a gain in most settings, together with the trade-off between learning speed and final performance across positions. The conclusion discusses, as a future direction, how to design the part of the network before the regularized block so that this trade-off disappears (addressing toNc's W2, W3 and W4).
  • Abstract, Sections 3 and 5.1, Table 3: The incomplete sentence is completed, the settings and environments are counted consistently, the claim on sample efficiency is calibrated (addressing toNc's W5 and GYK5's W1).

We believe these revisions address the reviewers' concerns and strengthen our manuscript. The individual responses below provide the full tables and details for each point.

AuthorsOpenReview ↗

Clarification on review's follow-up comment

Thank you for the follow-up comment and for your overall positive feedback on the paper! It gives us the chance to state our position more clearly. We would like to add a short clarification on how we read the two results.

Enforcing the constraint more strongly within NRC2Reg works. Within NRC2Reg, the constraint is tightened by the coefficient. With λ = 100, the NRC2 loss is about 100 times smaller than with λ = 1, and the final performance stays at the same level (668 and 673, against 651 for CRL, Table R4). This shows that, among the representations that satisfy the constraint closely, there are solutions at least as good as that of CRL, and that NRC2Reg finds them. The two variants depart from NRC2Reg in ways that can lose such solutions.

One regularized block is a bottleneck at one place; regularizing every block reduces the rank everywhere. NRC2Reg is the learned counterpart of a bottleneck at one place (Section 3), like the bottleneck of the image patch embedding in [1], where the rest of the network keeps its width. The reviewer's aim of regularizing more blocks was to keep only the task-relevant information. One low-dimensional feature already serves this purpose, since all later blocks compute from it, and these blocks keep their full width to process it. When every block is regularized, each regularized feature of the 256-wide network has a rank of at most 20 (Table R3), so the network behaves like a much narrower one. A narrower network is a smaller class of representations, which may not contain a solution as good as that of CRL, so a lower performance is expected. For scaled CRL itself, "wider networks consistently outperform narrower networks" [2], and smaller actors also degrade actor-critic methods [3]. Table R3 agrees: the output of the encoder loses intrinsic dimension.

A separate optimization no longer searches among these solutions. NRC2Reg looks for a solution of the CRL objective among the representations whose feature at block X is low-dimensional, and both enter the same update. A step that minimizes the NRC2 loss alone accepts any feature that satisfies the constraint, including a collapsed one, which is what we observe (rank 1 to 2).

We therefore do not read the two results as a sensitivity of NRC2Reg. They show what compressing the representation requires: a low-dimensional feature at one place, learned jointly with the CRL objective. NRC2Reg is one reasonable way to do this. The same block and coefficient (X = 3, λ = 1) works and is used in all main experiments of the paper, which makes them a robust default. Thanks again for your engagement in the discussion!

References:

[1] Li and He. Back to Basics: Let Denoising Generative Models Denoise. CVPR, 2026.

[2] Wang et al. 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities. NeurIPS, 2025.

[3] Mastikhina et al. Optimistic Critics Can Empower Small Actors. RLC, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.