acceptodds
Under review as a conference paper at ICLR 2027

Compute Efficient Deep Reinforcement Learning via Normalized GLU Block

Abstract

Massively parallel simulators in robotics and game environments have shifted reinforcement learning (RL) into a wall-clock-constrained regime, where agents must achieve strong performance within limited compute budgets. This creates a fundamental trade-off in network design: larger models provide greater expressive capacity but incur higher per-step computational cost, whereas smaller models train faster but often saturate early. Addressing this challenge requires architectures that achieve higher representational capacity per parameter. We revisit Gated Linear Units (GLUs), which have shown strong parameter efficiency in vision and language models, and investigate their effectiveness in RL. We find that naively applying GLUs to RL leads to unstable training because their multiplicative interactions amplify activation variance, causing feature norm explosion. To overcome this limitation, we propose N-GLU, a simple architectural modification that combines GLUs with pre-activation layer normalization. N-GLU stabilizes feature magnitudes while preserving the expressive benefits of multiplicative gating. We provide both theoretical motivation and empirical evidence that N-GLU produces higher-rank feature representations than standard ReLU blocks, thereby alleviating plasticity loss, a common failure mode in RL. Across a diverse set of benchmarks, including Atari 57, HumanoidBench, and MuJoCo Playground, replacing ReLU blocks with N-GLU in PQN and FastTD3 consistently improves compute efficiency over standard MLP-based architectures.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Compute Efficiency

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_MCu9OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces N-GLU, a network block that applies normalization to both branches of a multiplicative gate to improve representation capacity and training stability in deep RL. The authors evaluate N-GLU with PQN on Atari-57 and FastTD3 on 27 HumanoidBench tasks and four MuJoCo Playground tasks, reporting improved performance under fixed wall-clock budgets. Ablations examine feature rank, feature norms, and dormant neurons. The main concerns are the contribution relative to existing normalized multiplicative architectures, the provenance of baseline timing results, ambiguous interaction counts, and validation on stronger contemporary algorithms.

Critical strengths
  • Practically relevant objective. The paper studies performance under fixed training-time budgets, addressing the trade-off between representation capacity and throughput in parallel simulation.

  • Simple and accessible modification. N-GLU can replace linear or convolutional hidden blocks with limited implementation changes. The continuous-control experiments modify only the critic, providing a focused evaluation.

  • Broad evaluation and useful diagnostics. The experiments cover both discrete and continuous control. Comparisons with parameter-matched wider networks, unnormalized GLU, and Pre-LN GLU help assess the roles of capacity, multiplicative interactions, and normalization placement.

Critical weaknesses
  • The incremental contribution over closely related architectures is insufficiently established. Hadamax already normalizes two branches separately before multiplying their activated outputs and analyzes effective rank. Therefore, normalized multiplicative interactions and their representation benefits have direct precedents in RL. A controlled comparison between Hadamax-style blocks and N-GLU, using the same backbone and matched parameter budgets, is needed to isolate the contributions of asymmetric branch activation, activation choice, and normalization configuration.

  • The Hadamax wall-clock comparison lacks a traceable timing protocol. The paper uses publicly released Hadamax score records that describe sample efficiency rather than wall-clock learning curves. Although the appendix reports training time and SPS, it does not explain how these measurements were obtained or mapped to the released scores. The implementation, timing procedure, and conversion method must be specified. This omission directly affects the central claim of superior compute efficiency.

  • The interaction-count convention is ambiguous and potentially misleading. Figure 11 labels an axis ending at 10510^5 as “Environment steps,” while the experiments use 128 parallel environments. If this follows the official FastTD3 counting convention, 10510^5 vectorized collection steps correspond to 1.28×1071.28\times10^7 transitions. The authors should explicitly distinguish vectorized steps from total environment interactions and use consistent units for sample-efficiency comparisons.

  • Evaluation on stronger contemporary algorithms is missing. FastTD3 provides a reasonable controlled testbed, but improvements on this backbone alone do not establish broader compute-efficiency advantages. The evaluation should include FlashSAC and WarpSAC, both as competitive baselines and as backbones for testing N-GLU. Comparisons against their original networks under controlled capacity, training settings, and hardware would establish whether the gains persist alongside stronger representation and stabilization mechanisms.

Questions for authors

See weaknesses. I would consider raising my score if the authors demonstrate that adding N-GLU improves FlashSAC or WarpSAC under matched wall-clock budgets and otherwise identical training settings.

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

The incremental contribution over closely related architectures is insufficiently established. [...], and normalization configuration.

We deeply apologize that, due to limited computational resources, we were unable to complete the controlled comparison between Hadamax style blocks and N-GLU on the same backbone with matched parameter budgets during the rebuttal period. We will include this comparison in the camera ready version.

In the meantime, we would like to clarify the key novelty of our work relative to Hadamax.

(1) Generality. Hadamax is an encoder designed specifically for pixel input on Atari, with Hadamard products only in its convolutional blocks. N-GLU is a general block that replaces any hidden linear or convolutional layer. This is why it applies directly to the FastTD3 critic MLP on HumanoidBench and MuJoCo Playground, a setting that Hadamax does not consider.

(2) Compute efficiency. Hadamax replaces strided downsampling with stride 1 convolutions followed by max pooling. This makes the network very large and each update expensive. Under equal computation budget, N-GLU reaches an IQM of 3.4 versus 1.6 for Hadamax in Atari-57.

(3) Expressivity claim. Hadamax reports empirically higher feature rank, but its architecture combines multiplicative gating with max pooling in a much larger network than the vanilla PQN network, so its results cannot tell which component causes the gain. In particular, the ablation study in the Hadamax paper (Figure 10) shows that removing max pooling hurts performance the most, which suggests that the main improvement may come from max pooling together with the increased capacity. In contrast, we isolate the multiplicative interaction under a matched parameter budget and show through a principled theoretical analysis that it raises the upper bound on feature rank.

(4) Necessity of normalization. Hadamax attributes the stability of its products to PQN itself and does not analyze the role of normalization. We identify feature norm explosion as the failure mode of multiplicative gating under nonstationarity. Through the GLU, Pre-LN GLU, and random label MNIST experiments, we show that normalizing both branches before the product is necessary. Because FastTD3 uses a replay buffer and target networks, our results also show that this stability holds beyond the PQN setting.

The Hadamax wall-clock comparison lacks a traceable timing protocol. [...] compute efficiency.

Thank you for pointing this out. We measured the wall clock time of Hadamax PQN in our hardware setup, using the official PyTorch implementation (link) and running it for 200M frames, which matches the training budget of the officially reported scores. We then mapped the released scores to wall clock time by assuming a constant throughput (SPS) over training, so that each score recorded at frame tt corresponds to time t/SPSt / \text{SPS}. All baselines were run on the same codebase, and all wall clock times were measured with the same timing protocol. We have updated Appendix E to include this information.

The interaction-count convention is ambiguous and potentially misleading. [...] for sample-efficiency comparisons.

Thank you for pointing this out. We have revised the x axis of Figure 11 (now Figure 14 in the revised manuscript) to report total environment interactions (1.28×1071.28 \times 10^7) instead of vectorized steps (10510^5).

AuthorsOpenReview ↗

Rebuttal (2/2)

Evaluation on stronger contemporary algorithms is missing. [...] and stabilization mechanisms.

Thank you for your suggestion. Based on your suggestion, we incorporated N-GLU into FlashSAC and evaluated whether it yields further improvements on top of this advanced architecture. We compared Base FlashSAC and N-GLU FlashSAC in matched parameter and wall-clock budget.

Setup. We replace the first MLP layer of each FlashSAC residual block with an N-GLU block, adjusting the hidden dimension to h=8d/3h = 8d/3 so that both models have the same parameter budget (8d28d^2). Since BatchNorm and LayerNorm both serve to stabilize training, and FlashSAC already uses BatchNorm throughout its network, we normalize each branch with BatchNorm instead of LayerNorm. We evaluate on six IsaacLab tasks (two dexterous hand manipulation and four humanoid locomotion tasks) with 5 seeds each. All agents are trained for the same wall clock time that FlashSAC requires for 50M environment steps. Scores are normalized by the final mean score of FlashSAC on each task.

Results.

MethodFinal IQMFinal 95% CIAUC IQMAUC 95% CI
FlashSAC1.001[0.993, 1.007]0.660[0.654, 0.664]
N-GLU FlashSAC1.006[0.999, 1.012]0.674[0.669, 0.679]

N-GLU FlashSAC achieves a slight improvement in final performance and a modest but clear improvement in AUC, with confidence intervals that do not overlap. This trend is consistent across tasks: N-GLU FlashSAC achieves a higher AUC and final score in four of the six tasks. These results suggest that N-GLU remains beneficial even when combined with an architecture that already employs advanced normalization techniques.

We have added this experiment to Appendix A of the revised manuscript, including the architecture illustration, per task learning curves, and full experimental details.

Reviewer_MCu9OpenReview ↗

Thank you for the thorough rebuttal — raising score to 4

Thank you for the thorough and honest rebuttal, as well as the substantial revisions to the manuscript (including the timing protocol, the interaction-count axis fix, the parameter-matched Base2X runs, and the new FlashSAC experiments).

While the improvement on top of FlashSAC (+0.014 in AUC IQM and +0.005 in final IQM) is modest rather than dramatic, I sincerely appreciate the authors' effort in running this matched-parameter and matched-wall-clock evaluation and reporting the results candidly. I also tried N-GLU in my own pixel-based RL experiments and observed modest improvements as well. Overall, I like this work for its simplicity, clarity, and practical value, and I am raising my rating from 3 to 4.

Reviewer_DsYXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes N-GLU, a simple architectural modification that combines Gated Linear Units with layer normalization. The authors provide a theoretical analysis of how N-GLU can improve expressive power, followed by a large-scale empirical study across discrete and continuous control benchmarks using several common baseline algorithms. By incorporating N-GLU into PQN and FastTD3, the authors demonstrate improvements over both original baselines, as well as better computational efficiency than some competing methods.

Critical strengths
  1. The paper provides both theoretical and empirical evidence supporting the effectiveness of N-GLU. The empirical study is extensive, covering different baseline algorithms (PQN and FastTD3) and benchmarks (Atari, HumanoidBench, and MuJoCo Playground).

  2. The paper is clearly written and easy to follow.

Critical weaknesses
  1. The paper lacks comparisons with recent work. For example, FlashSAC [1] is a strong recent baseline for continuous control tasks that also emphasizes architectural design as a key contribution. The lack of comparison with such methods is a notable weakness.

  2. The improvements from incorporating N-GLU are relatively modest, particularly in the aggregated results across the two continuous control benchmarks.

[1] FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control.

Questions for authors

Could the authors provide a comparison with FlashSAC? Alternatively, could they investigate whether incorporating N-GLU into FlashSAC yields further improvements?

2 comments

Could the authors provide a comparison with FlashSAC? Alternatively, could they investigate whether incorporating N-GLU into FlashSAC yields further improvements?

Thank you for your suggestion. Based on your suggestion, we incorporated N-GLU into FlashSAC and evaluated whether it yields further improvements on top of this advanced architecture.

Setup. We replace the first MLP layer of each FlashSAC residual block with an N-GLU block, adjusting the hidden dimension to h=8d/3h = 8d/3 so that both models have the same parameter budget (8d28d^2). Since BatchNorm and LayerNorm both serve to stabilize training, and FlashSAC already uses BatchNorm throughout its network, we normalize each branch with BatchNorm instead of LayerNorm. We evaluate on six IsaacLab tasks (two dexterous hand manipulation and four humanoid locomotion tasks) with 5 seeds each. All agents are trained for the same wall clock time that FlashSAC requires for 50M environment steps. Scores are normalized by the final mean score of FlashSAC on each task.

Results.

MethodFinal IQMFinal 95% CIAUC IQMAUC 95% CI
FlashSAC1.001[0.993, 1.007]0.660[0.654, 0.664]
N-GLU FlashSAC1.006[0.999, 1.012]0.674[0.669, 0.679]

N-GLU FlashSAC achieves a slight improvement in final performance and a modest but clear improvement in AUC, with confidence intervals that do not overlap. This trend is consistent across tasks: N-GLU FlashSAC achieves a higher AUC and final score in four of the six tasks. These results suggest that N-GLU remains beneficial even when combined with an architecture that already employs advanced normalization techniques.

We have added this experiment to Appendix A of the revised manuscript, including the architecture illustration, per task learning curves, and full experimental details.

Thanks for the detailed rebuttal and clarification, which addressed and resolved my concerns. I will update the score accordingly.

Reviewer_zDefOpenReview ↗
Rating
4(+1)
Confidence
2(−1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes N-GLU, a normalized GLU architecture designed to improve representational expressivity and compute efficiency in RL. By applying LayerNorm to both GLU branches before their multiplicative interaction, the method aims to stabilize training while preserving the expressive benefits of gating. Experiments on Atari-57, HumanoidBench, and MuJoCo Playground demonstrate promising improvements over existing architectural baselines under fixed wall-clock budgets.

Critical strengths
  • Strong empirical results. N-GLU demonstrates substantial performance improvements across a diverse set of discrete and continuous control tasks.

  • Simple and practical architecture. The proposed block can be integrated into existing RL algorithms with relatively little modification.

  • Interesting investigation of training stability. The paper identifies feature-norm explosion as a potential issue with applying GLUs to RL and proposes a simple normalization strategy to address it.

Critical weaknesses
  1. Limited clarity regarding the novelty of N-GLU

GLUs and multiplicative feature interactions have already been explored in prior work, including RL architectures such as Hadamax. While the proposed normalization strategy appears useful, the paper could better distinguish its contribution from existing multiplicative architectures.

  1. The stability analysis deserves more emphasis in the main text

Since the proposed normalization strategy is arguably the central novelty of the paper, I believe the analysis of its stabilizing effects deserves greater emphasis.

While Section 4.3 demonstrates that N-GLU reduces feature-norm explosion and improves training performance, Appendix F provides further evidence connecting feature-norm growth to gradient instability and learning failure. These findings are important for understanding the motivation and effectiveness of the proposed architecture.

I suggest incorporating the key findings from Appendix F into the main text, particularly the relationship between activation norms, gradient variance, and training stability.

  1. Insufficiently detailed ablation analysis (Section 4.3)

Figure 6 provides evidence that normalization improves GLU performance, particularly on HBench-Hard. However, the presentation leaves several questions:

It is unclear whether returns and representation metrics are aggregated across tasks using the mean, median, or another statistic.

The normalization of returns and interpretation of uncertainty bands are not specified.

The analysis is limited to Atari-10 and a selected subset of five HumanoidBench environments.

Per-environment ablations would help establish whether the stabilization benefits are consistent across tasks.

  1. Missing parameter-matched comparison in the main experiments (Section 4.1)

N-GLU has approximately twice as many parameters as the standard PQN architecture. Although the authors include Base2X, a parameter-matched ReLU network, in Section 4.3, it is absent from the full Atari-57 evaluation in Section 4.1.

I suggest including Base2X in the full Atari-57 benchmark under the same wall-clock training budget, to more convincingly demonstrate that N-GLU's performance improvements arise from architectural advantages rather than simply increased parameter count.

  1. Limitations of the theoretical analysis (Section 3.2)

The feature-rank analysis provides useful intuition, but its novelty appears limited, as it primarily follows from a known property of Hadamard products. Moreover, it establishes a larger upper bound relative to linear layers rather than the ReLU networks used as experimental baselines.

Since the theoretical argument applies broadly to multiplicative architectures, its connection to the specific proposed normalization strategy is also limited. The authors could clarify the scope of the theoretical claims.

Questions for authors

What distinguishes N-GLU from previous multiplicative-gating approaches in RL, particularly Hadamax?

Could the authors bring the key stability analysis from Appendix F into the main text to better motivate and explain the proposed normalization strategy?

Could the authors clarify the aggregation and normalization procedures in Figure 6 and provide per-environment ablation results?

Could the authors include a parameter-matched Base2X baseline in the full Atari-57 experiments, evaluated under the same wall-clock budget?

Could the authors provide a more direct theoretical comparison against ReLU networks or clarify the limited scope of the current theoretical claims?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

What distinguishes N-GLU from previous multiplicative-gating approaches in RL, particularly Hadamax?

We thank the reviewer for this comment. There are four main differences between Hadamax and N-GLU.

(1) Generality. Hadamax is an encoder designed specifically for pixel input on Atari, with Hadamard products only in its convolutional blocks. N-GLU is a general block that replaces any hidden linear or convolutional layer. This is why it applies directly to the FastTD3 critic MLP on HumanoidBench and MuJoCo Playground, a setting that Hadamax does not consider.

(2) Compute efficiency. Hadamax replaces strided downsampling with stride 1 convolutions followed by max pooling. This makes the network very large and each update expensive. Under equal computation budget, N-GLU reaches an IQM of 3.4 versus 1.6 for Hadamax in Atari-57.

(3) Expressivity claim. Hadamax reports empirically higher feature rank, but its architecture combines multiplicative gating with max pooling in a much larger network than the vanilla PQN network, so its results cannot tell which component causes the gain. In particular, the ablation study in the Hadamax paper (Figure 10) shows that removing max pooling hurts performance the most, which suggests that the main improvement may come from max pooling together with the increased capacity. In contrast, we isolate the multiplicative interaction under a matched parameter budget and show through a principled theoretical analysis that it raises the upper bound on feature rank.

(4) Necessity of normalization. Hadamax attributes the stability of its products to PQN itself and does not analyze the role of normalization. We identify feature norm explosion as the failure mode of multiplicative gating under nonstationarity. Through the GLU, Pre-LN GLU, and random label MNIST experiments, we show that normalizing both branches before the product is necessary. Because FastTD3 uses a replay buffer and target networks, our results also show that this stability holds beyond the PQN setting.

Could the authors bring the key stability analysis from Appendix F into the main text to better motivate and explain the proposed normalization strategy?

Thank you for your suggestion. Following it, we have moved the stability analysis to the main text as a new section (Section 4.4, Stability Analysis).

Could the authors clarify the aggregation and normalization procedures in Figure 6 and provide per-environment ablation results?

Thank you for pointing this out. Based on your suggestion, we have added per-environment ablation result in Appendix H (Figure 15, 16). Per-environment results shows identical trends with aggregated results.

We explain the aggregation and normalization procedures below.

For HBench-Hard, we normalize each task's return as (R−Rrandom)(Rtarget−Rrandom)\frac{(R − R_{random})}{(R_{target} − R_{random})}, where RrandomR_{random} is the return of a random policy and RtargetR_{target} is the HumanoidBench success threshold. A score of 1.0 therefore means that the task is solved. Returns are aggregated with the interquartile mean (IQM) over all seed×taskseed \times task runs, and the shaded region shows the 95% confidence interval.

For Atari, we first compute each game's human-normalized score, HNS=R−RrandomRhuman−Rrandom\mathrm{HNS} = \frac{R - R_{random}}{R_{human} - R_{random}}, and average it over seeds. We then aggregate the 10 games into the Atari-10 score [1], which predicts the median HNS over the full 57-game suite. The shaded region is obtained by applying the same formula to the per-game mean and standard error across seeds.

The representation metrics (feature rank, feature norm, and dormant neuron ratio) are averaged over all seed×taskseed \times task runs, and their shaded regions show the standard deviation.

We have updated the paper to describe detailed aggregation and normalization procedure for all figures in Appendix E.

AuthorsOpenReview ↗

Rebuttal (2/2)

Could the authors include a parameter-matched Base2X baseline in the full Atari-57 experiments, evaluated under the same wall-clock budget?

Thank you for pointing this out. Based on your suggestion, we evaluated Base2X baselines in full Atari-57 and also HumanoidBench (27 tasks), under the same wall-clock training budget. We have updated Figure 1 to include these results (2X PQN and 2X FastTD3). On both benchmarks, 2X PQN and 2X FastTD3 yields no meaningful gain over the base network, while N-GLU clearly outperforms both.

Could the authors provide a more direct theoretical comparison against ReLU networks or clarify the limited scope of the current theoretical claims?

We would like to clarify a possible misunderstanding. A larger upper bound is the desired result, since it means that N-GLU can theoretically attain a higher feature rank than a vanilla ReLU network. We acknowledge that the analysis builds on a known property of Hadamard products. However, we extend this property to the stable feature rank [2], a widely used measure of expressive capacity. The purpose of the theoretical claim is not to justify the specific normalization strategy, but to establish the expressive benefit of GLU in a principled way, which to our knowledge has not been done in prior work.

References

[1] Aitchison, Matthew, Penny Sweetser, and Marcus Hutter. "Atari-5: Distilling the arcade learning environment down to five games." International Conference on Machine Learning. PMLR, 2023.

[2] Kumar, Aviral, et al. "Implicit under-parameterization inhibits data-efficient deep reinforcement learning." arXiv preprint arXiv:2010.14498 (2020).

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the full-benchmark Base2X comparisons, the per-environment ablations, and the expanded stability analysis. These additions substantially strengthen the paper and address most of my concerns.

One minor theoretical concern remains: the current analysis establishes a larger feature-rank upper bound for GLUs compared with linear layers, but does not establish the same advantage over ReLU networks, which can also increase feature rank through nonlinear transformations. Nevertheless, I do not consider this a major limitation given the strength of the empirical results.

Overall, I find the rebuttal convincing. Since it is late in the discussion period, I will raise my score to 4 while lowering my confidence to reflect the remaining uncertainty in my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.