acceptodds
Withdrawn from ICLR 2027

Do Hyper-Connections Transfer to Reinforcement Learning? Sphere-Constrained Mixing and the Geometry of Depth.

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hyper-connections replace residual connections with learned mixtures of all preceding layer outputs and accelerate language model pretraining, but the mixing weights must be constrained to remain stable at scale. We ask whether both the benefit and the instability transfer to on-policy reinforcement learning. We first diagnose the naive transfer: hyper-connections degrade PPO on the highest-dimensional continuous control tasks, precisely those where the gradient-routing argument predicts the largest gain, while the mixing norm drifts in every run and per-layer hidden state norms diverge at the one layer where mixing is active. To address this, we propose SHC-PPO, Sphere-Constrained Hyper connections, which keeps the mixing vector on the unit sphere via Riemannian optimization applied after each Adam step, and situate it within a design space of constraints organized by the invariant maintained, whether it holds exactly, and whether it is enforced in the forward pass or the update rule. Evaluating on MuJoCo continuous control, we find that in shallow networks every constraint holding the norm near one recovers parity with the baseline. Because shallow networks contain a single actively mixing layer, we extend the analysis to six layers, where four layers mix. There the constraints separate, and they separate on optimization geometry rather than on return: SHC-PPO preserves gradient-norm anisotropy two orders of magnitude better than unconstrained mixing, an order of magnitude better than forward renormalization maintaining the same invariant, and an order of magnitude better than a network with no mixing at all, while returns across the four conditions remain within each other's confidence intervals. Where a constraint is enforced is therefore measurable in the optimization geometry at depths where it is invisible in the reward.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Proximal Policy Optimization, Hyper-connections, Residual Connections, Riemannian optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dqrfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether hyper-connections (HC) transfer to on-policy RL. Using a single-stream HC reduction in PPO's MLPs, the authors report that unconstrained mixing matches the MLP baseline on three MuJoCo tasks but degrades returns on Ant-v4 and Humanoid-v4, with the mixing-vector norm drifting in every run. They propose SHC-PPO, which keeps each mixing vector on the unit sphere via Riemannian projection and retraction. They compare it against forward renormalization, a simplex constraint, weight decay, and fixed residual skips, organized by invariant, exactness, and enforcement site. At L=3, all norm-preserving constraints recover baseline parity and are indistinguishable, while weight decay collapses training. At L=6, returns remain indistinguishable, but a gradient-anisotropy measure separates the conditions. The authors interpret this as evidence that enforcement site matters in optimization geometry before it matters in return.

Critical strengths

1. Clean diagnostic design for localizing the failure. Using the non-mixing layers of a shallow network as within-network controls is a clever way to attribute hidden-state divergence to the mixing operation rather than to generic architectural change. This matters because it is the paper's most convincing evidence that unconstrained mixing causes the degradation. It is weakened, however, by being based on a single seed.

2. Transparent and honest reporting. The authors report negative and null results openly, including indistinguishable returns at both depths and the single-run origin of SHC-PPO's nominal lead at L=6. They also scope their claims carefully and acknowledge that Lemma 1 does not explain the main separation. This makes the paper useful as a reference for others attempting the transfer. Still, these strengths support the paper's value as an empirical note rather than its central methodological claims.

Critical weaknesses
  1. The central comparison is confounded. SHC-PPO updates α\alpha with a separate fixed-step Riemannian SGD outside Adam and gradient clipping, while HC-Renorm uses Adam on α\alpha. In addition, HC is initialized at ∥α∥2≈0.707\|\alpha\|_2 \approx 0.707 rather than unit norm (contradicting §4.1), so SHC-PPO also differs in initial scale. The 20×20\times "enforcement-site" separation could therefore come from optimizer or initialization differences, which undermines the paper's main claim.

  2. The key metric is not scale-invariant. Gradient anisotropy is the raw variance of per-tensor gradient norms, so it shrinks quadratically with overall gradient magnitude and mixes tensors of very different sizes. The reported 1313–87×87\times differences may reflect gradient scale rather than conditioning.

  3. No performance benefit, and weak statistics. Returns never improve over baselines. Results rely on 3 seeds and IQM aggregated over unnormalized returns across environments.

Questions for authors

Clarifications:

  1. Optimizer confound. α\alpha is updated by fixed-step Riemannian SGD outside Adam and gradient clipping under SHC-PPO, but by Adam under HC-Renorm. Can you report a variant that applies Adam to α\alpha and then retracts onto the sphere (or HC-Renorm with the same SGD step)? If the 20×20\times anisotropy gap persists, the enforcement-site claim would be much more convincing.

  2. Initialization. D.3 implies HC starts at ∥α∥2=1/2≈0.707\|\alpha\|_2 = 1/\sqrt{2} \approx 0.707, yet §4.1 describes a "unit initialization." Which is correct? If HC starts at 0.7070.707, how much of the HC vs. SHC-PPO gap comes from the 2\sqrt{2} difference in initial input scale?

  3. Metric scale-dependence. Does the anisotropy ordering hold under a scale-invariant measure (e.g., coefficient of variation or variance of log-norms)? Please also report the mean gradient norm per condition, so that dispersion can be separated from overall magnitude.

  4. Aggregation. Were returns normalized per environment before computing IQM in Table 3? If not, do the conclusions of §6.1 survive normalized aggregation?

Suggestion:

  1. Hidden-state norms at the mixing layer approach the tanh saturation bound (≈7.1\approx 7.1 of a maximum of 88 for 6464 units). Have you tested whether saturation alone explains the degradation, for example with a non-saturating activation?
Reviewer_4ssrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether hyper-connections (HC), which allow each layer to consume a learnable mixture of outputs from preceding layers, can be effectively transferred to on-policy reinforcement learning.

Using CleanRL PPO with a three-layer, 64-unit Tanh MLP trained for 3 million environment steps, the authors evaluate the approach on five MuJoCo tasks. They find that unconstrained mixing vectors (α(\alpha) drift from their initialization across all environments and random seeds, reaching final L2 norms between 0.72 and 1.89. This is accompanied by substantial performance degradation on Ant-v4 and Humanoid-v4, where returns decrease by approximately one-third and one-quarter, respectively. A single-seed analysis of layer-wise hidden-state norms shows that the norm increases primarily at the second layer, the only layer with active mixing.

To address this issue, the authors propose SHC-PPO, which constrains (α(\alpha) to the unit sphere through tangent-space gradient projection followed by normalization-based retraction. The paper organizes its baselines along three design dimensions: the enforced invariant, whether enforcement is exact, and where the constraint is applied (during the forward pass or at the parameter-update stage). The compared methods include HC-RENORM (forward normalization), HC-SIMPLEX (softmax parameterization), HC-DECAY (weight decay), unconstrained HC, MLP, and RESMLP.

For (L=3(L=3), all methods that maintain mixing-vector norms near 1 achieve interquartile mean (IQM) returns with overlapping confidence intervals relative to MLP (1435, 1330, and 1388 versus 1401). In contrast, HC-DECAY reduces the mixing-vector norm to approximately 0.006 and suffers a performance collapse.

For (L=6(L=6), with four mixing layers, the authors conduct experiments on Ant and Humanoid using three seeds. They report nearly two orders of magnitude of variation in "gradient anisotropy," defined as the variance of gradient norms across parameter tensors: SHC-PPO ((4.1×10−4(4.1\times10^{-4})) < MLP ((5.5×10−3(5.5\times10^{-3})) < HC-RENORM ((8.0×10−3(8.0\times10^{-3})) < HC ((3.6×10−2(3.6\times10^{-2}). However, all return confidence intervals overlap, and SHC-PPO's nominal performance advantage is largely driven by a single seed (Ant: 1879 versus 858/872).

Critical strengths
  1. The experimental reporting is relatively careful and transparent. The authors provide IQM estimates with stratified bootstrap confidence intervals, per-seed results (Tables 6–7), explicit identification of three-seed experiments, publicly available code, and architectural unit tests. I appreciate the willingness to report unsuccessful transfer results rather than presenting inconclusive improvements as definitive gains.

  2. The single-stream reduction provides a controlled setting in which the mixing coefficients are the primary architectural variable across conditions. In particular, the (L=3(L=3) construction, where only the second layer performs mixing while layers 0 and 1 serve as within-network controls, is a useful way to localize the effects of the proposed mechanism. SHC-PPO itself is also straightforward to implement, introduces less than 2%.

Critical weaknesses
  1. The proposed method has no statistically improvement, while its main positive result relies on a problematic diagnostic metric.

    For L=3L=3, the confidence intervals of SHC-PPO and the MLP baseline overlap. The same holds for (L=6(L=6), where the reported results are 706 [664, 1209] versus 619 [497, 1042]. Furthermore, the nominal advantage at (L=6(L=6) is driven by a single random seed. With no convincing improvement in returns, the paper shifts its emphasis to "gradient anisotropy." However, this metric, defined in Appendix F as the variance of gradient norms across parameter tensors, has three important limitations.

(a) A uniform reduction in gradient magnitude by a factor of 3.6 would reduce (Var⁡(ni)(\operatorname{Var}(n_i)) by approximately a factor of 13, even if the relative distribution of gradient norms remained unchanged. Gradient magnitude can vary with Tanh saturation, advantage scaling, and policy entropy, none of which directly establishes improved conditioning. A more meaningful comparison would use the coefficient of variation or the variance of log gradient norms.

(b) The set of parameter tensors differs across methods. The HC-based methods introduce four additional low-dimensional (α(\alpha) tensors (with dimensions between 2 and 5) that are absent from the MLP baseline. Their gradient norms may naturally differ from the overall mean (nˉ(\bar n), mechanically affecting the reported variance. Consequently, the claimed approximately 13-fold improvement of SHC-PPO over MLP may partly reflect the contribution of these additional tensors rather than better conditioning of the backbone network.

(c) Gradient anisotropy is not a direct measure of condition number. The paper provides neither Hessian/Fisher spectral analyses nor the layer-wise gradient norm profiles used by Creus Castanyer et al. Finally, the reported increase in the second-layer hidden-state norm to approximately 7.1 may have a much simpler explanation. For a 64-dimensional Tanh output, the maximum possible L2 norm is 8. Thus, a norm near 7.1 indicates substantial activation saturation. This could be a direct consequence of mixing coefficients with (∥α∥>1(\|\alpha\|>1) amplifying the pre-activations, rather than evidence of a more subtle optimization pathology. The paper does not explicitly investigate this explanation or report the corresponding pre-activation statistics.

  1. The supposedly controlled comparisons confound the optimizer, and the simplified architecture does not preserve the defining mechanism of hyper-connections. The paper emphasizes that its comparisons isolate individual design factors. However, the central comparison between HC-RENORM and SHC-PPO changes not only where the constraint is enforced but also the optimizer and learning rate. For HC-RENORM, the effective mixing coefficients are defined as $ \alpha_{\mathrm{eff}} = \frac{\alpha_{\mathrm{raw}}} {|\alpha_{\mathrm{raw}}|},$ Thus, the gradient is projected onto the tangent space and subsequently processed by Adam's coordinate-wise adaptive normalization, using a learning rate of (3×10−4(3\times10^{-4}).

In contrast, Algorithm 1 excludes (α(\alpha) from Adam in SHC-PPO and instead updates it using ordinary Riemannian SGD with ηS=10−2\eta_S=10^{-2}.

Thus, these methods differ in at least three respects: the constraint enforcement site, the optimizer (Adam versus SGD), and the learning rate ((3×10−4)versus(10−2(3\times10^{-4}) versus (10^{-2})), with no reported learning-rate tuning.

As a result, the approximately 20-fold anisotropy difference attributed to the enforcement site in Section 6.2 could instead arise from the optimizer or step-size differences. Without an optimizer-matched comparison, the claimed causal interpretation is not supported.

There is also a forward-scale confound at initialization. SHC-PPO retracts the mixing coefficients onto the unit sphere, resulting in a coefficient sum of (ℓ(\sqrt{\ell}) under uniform mixing (approximately 2.24 when (ℓ=5(\ell=5)). In contrast, MLP and HC-SIMPLEX use coefficients that sum to 1. This difference changes the scale of the forward activations independently of the proposed optimization constraint.

  1. Finally, the cross-environment IQM is computed without normalizing returns separately for each environment, unlike the procedure advocated by Agarwal et al. As a result, the aggregate is heavily influenced by HalfCheetah, whose reported returns are approximately (2249±9722249\pm972).
Questions for authors
  1. Optimizer confounding in the comparison between SHC-PPO and HC-RENORM. Please confirm whether SHC-PPO updates (α\alpha) using Riemannian SGD with (ηS=10−2\eta_S=10^{-2}), without passing these parameters through Adam, while HC-RENORM updates (αraw\alpha_{\mathrm{raw}}) using Adam with a learning rate of (3×10−43\times10^{-4}).

  2. Does the reported anisotropy metric include the gradients of the (α\alpha) tensors? Please report the metric after excluding these tensors and recompute it using a scale-invariant alternative, such as the coefficient of variation or variance of log gradient norms, with confidence intervals across seeds. Additionally, please provide the layer-wise gradient norm profiles for each (L=6L=6) condition to support the interpretation in terms of network conditioning.

  3. Which initialization is actually used: the "unit initialization" described in Section 4.1 or the (1/ℓ1/\ell) initialization in Appendix D.3, which has L2 norm (1/ℓ1/\sqrt{\ell})? Does HC start with an L2 norm of approximately 0.707 when (L=3)?

  4. When the second-layer hidden-state norm reaches approximately 7.1, what fraction of the Tanh units are saturated? Could the observed failure mechanism be explained simply by amplified pre-activations causing Tanh saturation? Please provide activation or pre-activation statistics to distinguish this explanation from a more general optimization-conditioning issue.

  5. What exactly is the RESMLP baseline? Does it use standard residual connections or a fixed (1/\ell) dense mixture of preceding hidden states?

Reviewer_7uTSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the use of hyperconnection, and its possible impact on RL. They proposes SHC-PPO, which constrains mixing coefficients to the unit sphere. In three-layer MuJoCo policies, several constraints recover performance broadly comparable to a plain MLP. At six layers, SHC reduces the reported variance of gradient norms across parameter tensors, without a clear return improvement. The authors attribute this difference to enforcing the constraint during optimization. But my big concern would be we are evaluating hyperconnection which is introduced for LLM on shallow network, which may make the conlucsion not pratical.

Critical strengths
  1. The negative result is useful: unconstrained mixing can hurt PPO, and comparing several constraints helps characterize this failure. The paper is also candid about the absence of a clear return improvement.

  2. The method is simple and theortically solid.

Critical weaknesses
  1. The evaluation is too limited to support the transfer and depth claims. The paper motivates its study through HC/mHC’s behavior at depth, but evaluates only 3- and 6-layer MLPs, with one and four active mixing layers respectively. This is my biggest concern as HC/mHC are mostly for deep LLMs, not for shallow networks. Can the author explains more why this setup make sense to make sure the connection is transferable?
Questions for authors
  1. The experiments provide a narrow observation about shallow PPO networks, but leave the main transfer question largely unanswered. This is my main question, and make other trivials vacuous.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.