- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether hyper-connections (HC) transfer to on-policy RL. Using a single-stream HC reduction in PPO's MLPs, the authors report that unconstrained mixing matches the MLP baseline on three MuJoCo tasks but degrades returns on Ant-v4 and Humanoid-v4, with the mixing-vector norm drifting in every run. They propose SHC-PPO, which keeps each mixing vector on the unit sphere via Riemannian projection and retraction. They compare it against forward renormalization, a simplex constraint, weight decay, and fixed residual skips, organized by invariant, exactness, and enforcement site. At L=3, all norm-preserving constraints recover baseline parity and are indistinguishable, while weight decay collapses training. At L=6, returns remain indistinguishable, but a gradient-anisotropy measure separates the conditions. The authors interpret this as evidence that enforcement site matters in optimization geometry before it matters in return.
Critical strengths
1. Clean diagnostic design for localizing the failure. Using the non-mixing layers of a shallow network as within-network controls is a clever way to attribute hidden-state divergence to the mixing operation rather than to generic architectural change. This matters because it is the paper's most convincing evidence that unconstrained mixing causes the degradation. It is weakened, however, by being based on a single seed.
2. Transparent and honest reporting. The authors report negative and null results openly, including indistinguishable returns at both depths and the single-run origin of SHC-PPO's nominal lead at L=6. They also scope their claims carefully and acknowledge that Lemma 1 does not explain the main separation. This makes the paper useful as a reference for others attempting the transfer. Still, these strengths support the paper's value as an empirical note rather than its central methodological claims.
Critical weaknesses
-
The central comparison is confounded. SHC-PPO updates with a separate fixed-step Riemannian SGD outside Adam and gradient clipping, while HC-Renorm uses Adam on . In addition, HC is initialized at rather than unit norm (contradicting §4.1), so SHC-PPO also differs in initial scale. The "enforcement-site" separation could therefore come from optimizer or initialization differences, which undermines the paper's main claim.
-
The key metric is not scale-invariant. Gradient anisotropy is the raw variance of per-tensor gradient norms, so it shrinks quadratically with overall gradient magnitude and mixes tensors of very different sizes. The reported – differences may reflect gradient scale rather than conditioning.
-
No performance benefit, and weak statistics. Returns never improve over baselines. Results rely on 3 seeds and IQM aggregated over unnormalized returns across environments.
Questions for authors
Clarifications:
-
Optimizer confound. is updated by fixed-step Riemannian SGD outside Adam and gradient clipping under SHC-PPO, but by Adam under HC-Renorm. Can you report a variant that applies Adam to and then retracts onto the sphere (or HC-Renorm with the same SGD step)? If the anisotropy gap persists, the enforcement-site claim would be much more convincing.
-
Initialization. D.3 implies HC starts at , yet §4.1 describes a "unit initialization." Which is correct? If HC starts at , how much of the HC vs. SHC-PPO gap comes from the difference in initial input scale?
-
Metric scale-dependence. Does the anisotropy ordering hold under a scale-invariant measure (e.g., coefficient of variation or variance of log-norms)? Please also report the mean gradient norm per condition, so that dispersion can be separated from overall magnitude.
-
Aggregation. Were returns normalized per environment before computing IQM in Table 3? If not, do the conclusions of §6.1 survive normalized aggregation?
Suggestion:
- Hidden-state norms at the mixing layer approach the tanh saturation bound ( of a maximum of for units). Have you tested whether saturation alone explains the degradation, for example with a non-saturating activation?