- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces N-GLU, a network block that applies normalization to both branches of a multiplicative gate to improve representation capacity and training stability in deep RL. The authors evaluate N-GLU with PQN on Atari-57 and FastTD3 on 27 HumanoidBench tasks and four MuJoCo Playground tasks, reporting improved performance under fixed wall-clock budgets. Ablations examine feature rank, feature norms, and dormant neurons. The main concerns are the contribution relative to existing normalized multiplicative architectures, the provenance of baseline timing results, ambiguous interaction counts, and validation on stronger contemporary algorithms.
Critical strengths
-
Practically relevant objective. The paper studies performance under fixed training-time budgets, addressing the trade-off between representation capacity and throughput in parallel simulation.
-
Simple and accessible modification. N-GLU can replace linear or convolutional hidden blocks with limited implementation changes. The continuous-control experiments modify only the critic, providing a focused evaluation.
-
Broad evaluation and useful diagnostics. The experiments cover both discrete and continuous control. Comparisons with parameter-matched wider networks, unnormalized GLU, and Pre-LN GLU help assess the roles of capacity, multiplicative interactions, and normalization placement.
Critical weaknesses
-
The incremental contribution over closely related architectures is insufficiently established. Hadamax already normalizes two branches separately before multiplying their activated outputs and analyzes effective rank. Therefore, normalized multiplicative interactions and their representation benefits have direct precedents in RL. A controlled comparison between Hadamax-style blocks and N-GLU, using the same backbone and matched parameter budgets, is needed to isolate the contributions of asymmetric branch activation, activation choice, and normalization configuration.
-
The Hadamax wall-clock comparison lacks a traceable timing protocol. The paper uses publicly released Hadamax score records that describe sample efficiency rather than wall-clock learning curves. Although the appendix reports training time and SPS, it does not explain how these measurements were obtained or mapped to the released scores. The implementation, timing procedure, and conversion method must be specified. This omission directly affects the central claim of superior compute efficiency.
-
The interaction-count convention is ambiguous and potentially misleading. Figure 11 labels an axis ending at as “Environment steps,” while the experiments use 128 parallel environments. If this follows the official FastTD3 counting convention, vectorized collection steps correspond to transitions. The authors should explicitly distinguish vectorized steps from total environment interactions and use consistent units for sample-efficiency comparisons.
-
Evaluation on stronger contemporary algorithms is missing. FastTD3 provides a reasonable controlled testbed, but improvements on this backbone alone do not establish broader compute-efficiency advantages. The evaluation should include FlashSAC and WarpSAC, both as competitive baselines and as backbones for testing N-GLU. Comparisons against their original networks under controlled capacity, training settings, and hardware would establish whether the gains persist alongside stronger representation and stabilization mechanisms.
Questions for authors
See weaknesses. I would consider raising my score if the authors demonstrate that adding N-GLU improves FlashSAC or WarpSAC under matched wall-clock budgets and otherwise identical training settings.
3 comments
Rebuttal (2/2)
Evaluation on stronger contemporary algorithms is missing. [...] and stabilization mechanisms.
Thank you for your suggestion. Based on your suggestion, we incorporated N-GLU into FlashSAC and evaluated whether it yields further improvements on top of this advanced architecture. We compared Base FlashSAC and N-GLU FlashSAC in matched parameter and wall-clock budget.
Setup. We replace the first MLP layer of each FlashSAC residual block with an N-GLU block, adjusting the hidden dimension to so that both models have the same parameter budget (). Since BatchNorm and LayerNorm both serve to stabilize training, and FlashSAC already uses BatchNorm throughout its network, we normalize each branch with BatchNorm instead of LayerNorm. We evaluate on six IsaacLab tasks (two dexterous hand manipulation and four humanoid locomotion tasks) with 5 seeds each. All agents are trained for the same wall clock time that FlashSAC requires for 50M environment steps. Scores are normalized by the final mean score of FlashSAC on each task.
Results.
| Method | Final IQM | Final 95% CI | AUC IQM | AUC 95% CI |
|---|---|---|---|---|
| FlashSAC | 1.001 | [0.993, 1.007] | 0.660 | [0.654, 0.664] |
| N-GLU FlashSAC | 1.006 | [0.999, 1.012] | 0.674 | [0.669, 0.679] |
N-GLU FlashSAC achieves a slight improvement in final performance and a modest but clear improvement in AUC, with confidence intervals that do not overlap. This trend is consistent across tasks: N-GLU FlashSAC achieves a higher AUC and final score in four of the six tasks. These results suggest that N-GLU remains beneficial even when combined with an architecture that already employs advanced normalization techniques.
We have added this experiment to Appendix A of the revised manuscript, including the architecture illustration, per task learning curves, and full experimental details.
Thank you for the thorough rebuttal — raising score to 4
Thank you for the thorough and honest rebuttal, as well as the substantial revisions to the manuscript (including the timing protocol, the interaction-count axis fix, the parameter-matched Base2X runs, and the new FlashSAC experiments).
While the improvement on top of FlashSAC (+0.014 in AUC IQM and +0.005 in final IQM) is modest rather than dramatic, I sincerely appreciate the authors' effort in running this matched-parameter and matched-wall-clock evaluation and reporting the results candidly. I also tried N-GLU in my own pixel-based RL experiments and observed modest improvements as well. Overall, I like this work for its simplicity, clarity, and practical value, and I am raising my rating from 3 to 4.
Rebuttal (1/2)
We deeply apologize that, due to limited computational resources, we were unable to complete the controlled comparison between Hadamax style blocks and N-GLU on the same backbone with matched parameter budgets during the rebuttal period. We will include this comparison in the camera ready version.
In the meantime, we would like to clarify the key novelty of our work relative to Hadamax.
(1) Generality. Hadamax is an encoder designed specifically for pixel input on Atari, with Hadamard products only in its convolutional blocks. N-GLU is a general block that replaces any hidden linear or convolutional layer. This is why it applies directly to the FastTD3 critic MLP on HumanoidBench and MuJoCo Playground, a setting that Hadamax does not consider.
(2) Compute efficiency. Hadamax replaces strided downsampling with stride 1 convolutions followed by max pooling. This makes the network very large and each update expensive. Under equal computation budget, N-GLU reaches an IQM of 3.4 versus 1.6 for Hadamax in Atari-57.
(3) Expressivity claim. Hadamax reports empirically higher feature rank, but its architecture combines multiplicative gating with max pooling in a much larger network than the vanilla PQN network, so its results cannot tell which component causes the gain. In particular, the ablation study in the Hadamax paper (Figure 10) shows that removing max pooling hurts performance the most, which suggests that the main improvement may come from max pooling together with the increased capacity. In contrast, we isolate the multiplicative interaction under a matched parameter budget and show through a principled theoretical analysis that it raises the upper bound on feature rank.
(4) Necessity of normalization. Hadamax attributes the stability of its products to PQN itself and does not analyze the role of normalization. We identify feature norm explosion as the failure mode of multiplicative gating under nonstationarity. Through the GLU, Pre-LN GLU, and random label MNIST experiments, we show that normalizing both branches before the product is necessary. Because FastTD3 uses a replay buffer and target networks, our results also show that this stability holds beyond the PQN setting.
Thank you for pointing this out. We measured the wall clock time of Hadamax PQN in our hardware setup, using the official PyTorch implementation (link) and running it for 200M frames, which matches the training budget of the officially reported scores. We then mapped the released scores to wall clock time by assuming a constant throughput (SPS) over training, so that each score recorded at frame t corresponds to time t/SPS. All baselines were run on the same codebase, and all wall clock times were measured with the same timing protocol. We have updated Appendix E to include this information.
Thank you for pointing this out. We have revised the x axis of Figure 11 (now Figure 14 in the revised manuscript) to report total environment interactions (1.28×107) instead of vectorized steps (105).