- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ReLog, a novel hardware-driven activation function designed from an engineering perspective using the IEEE 754 floating-point bit layout, moving away from traditional abstract continuous mathematical curves. By isolating the exponent and mantissa bits, ReLog enables fast, continuous, and sublinear growth using standard Arithmetic Logic Unit (ALU) instructions rather than expensive Special Function Units (SFUs), providing an intrinsic per-node damping mechanism that ensures robust training stability and prevents activation explosion under recurrent stress. While empirical evaluations across Diffusion Transformers and image classification tasks demonstrate that ReLog significantly improves performance and stability without diverging, it exhibits higher wall-clock computational costs on standard GPUs due to memory bandwidth constraints.
Critical strengths
- ReLog's forward-pass log-compression property provides a provable bounded-orbit mechanism. Across a spectral-radius sweep up to in recurrent networks, ReLog achieved zero observed divergences, whereas traditional gradient clipping failed to rescue unclipped baseline instabilities.
- In Diffusion Transformers (DiT) and CIFAR-10 image classification tasks, ReLog significantly outperformed ReLU, GELU, and SiLU across FP32, BF16, and FP16 precisions.
- ReLog completely avoids computationally expensive Special Function Units (SFUs) and is constructed entirely using standard Arithmetic Logic Unit (ALU) bit operations.
Critical weaknesses
- On standard GPUs (such as NVIDIA B200 / RTX 4090), ReLog's compiled kernel is slower than standard activations (approximately slower in forward pass and slower in forward-plus-backward). Because element-wise activations are bandwidth-bound rather than compute-bound on GPUs, the hardware ALU advantage fails to translate into wall-clock speedups.
- ReLog closely resembles a piecewise-linear approximation to a logarithmic activation. However, the experiments primarily compare it against ReLU, GELU, and SiLU. Without comparisons against log1p-based activations and generic piecewise-linear approximations, it is difficult to determine whether the observed improvements originate from the proposed floating-point construction or simply from the sublinear growth behavior.
- The improvements in the diffusion transformer experiments are relatively small in absolute magnitude. Although statistical significance is reported, the paper does not demonstrate corresponding improvements in generation quality or performance under matched training-time budgets.
- The experimental evaluation mainly involves small-scale or customized architectures. In particular, the CIFAR-10 classification experiments use a relatively weak CNN baseline, and the paper does not demonstrate improvements on stronger and more representative architectures.
Questions for authors
- What empirical evidence supports the ALU-native efficiency motivation when the proposed implementation is slower than standard activations on GPU? Can the authors provide measurements on hardware where special-function operations are a significant bottleneck?
- How does ReLog compare against log1p-based activations and generic piecewise-linear logarithmic approximations under identical training settings?
2 comments
Response to Reviewer mfSb (2/2)
Response to Reviewer mfSb (part 2 of 2)
3. The DiT improvements, generation quality and training budgets (Weakness 3). Your concern was well founded, and computing FID changed our claim.
- The DiT in the submission cannot generate. Each 16×16 patch holds 768 values but the model width is 384, so half of the noise in every patch is invisible to the network and the loss can never go below 0.50. All four activations produce noise images, with FID around 445. ReLog's advantage in the DiT table is statistically real, but small and confined to this bottleneck. (That table reports best-epoch validation loss; our caption mislabeled it as training loss, which we fixed.)
- In a DiT that works, ReLog is at parity. Retraining the same architecture on 64×64 images with 4×4 patches removes the bottleneck. FID over 10,000 samples: ReLog 145.3 ± 4.7, ReLU 144.9 ± 13.4, SiLU 141.4 ± 1.5, GELU 133.5 ± 3.0, with no difference significant after Holm correction (3 seeds per activation).
- Training budgets. We matched training steps (200 epochs for every run), not wall-clock time. On our GPU, ReLog runs took roughly 8,800 s against about 7,000 s for the baselines, because that machine uses our slower pure-PyTorch implementation. Under a matched time budget ReLog would have had fewer steps, so we would not expect it to gain ground.
We no longer claim that ReLog outperforms the baselines in a DiT. Section 5.1, the contribution list and the Conclusion say so, and the new Appendix D gives the details, with sample images from both models.
4. Small or custom architectures (Weakness 4). The paper tests ReLog across six model families: MLPs, a vanilla RNN, a selective state-space model, a CNN, a Transformer language model, and a DiT at two input resolutions. The outcomes are mixed, and the revised paper says so: ReLog leads on CIFAR-10 but is at parity on WikiText-2 and in the working DiT. We deliberately did not use a heavily tuned architecture such as ResNet for CIFAR-10. Its initialization, learning-rate schedule and block design have been tuned around ReLU over years of community work, so substituting a new activation there measures how well it fits ReLU-shaped assumptions rather than the activation itself. Our CNN is untuned toward any activation, and each activation gets its own learning-rate sweep, so the comparison is fair at the scale we tested. We agree this does not show how ReLog behaves in larger, stronger models.
Response to Reviewer mfSb (1/2)
Response to Reviewer mfSb (part 1 of 2)
Thank you for the review. Your questions go to the center of the paper, and answering them honestly changed how we state its contribution. Part 1 covers efficiency and the log1p comparison; part 2 covers the DiT and the choice of architectures. Our general comment lists every change in the revised PDF.
1. Evidence for the ALU-native motivation (Question 1, Weakness 1). You are right that on GPUs a standalone elementwise activation is bandwidth-bound, so ALU savings do not appear in wall-clock time: our compiled kernel is slower than ReLU (Section 5.4), and even a torch.compile-fused version only matches the baselines' forward time on an RTX 4090 (0.29 ms against 0.30-0.34 ms). We have now measured on hardware where special functions are a real bottleneck, as you asked:
The comparison that matters most is ReLog against the exact logarithm: the same shape, at about a quarter of the cost on the microcontroller. Section 5.4 and Appendix C (F10, F11) give the details. These are microbenchmarks of single activations, not full networks, and ReLU stays faster.
2. log1p and piecewise-linear baselines (Question 2, Weakness 2). Your suspicion is correct, and we now say so in the paper. We had run a log1p-based activation, exact log2(1+x) = log1p(x)/ln 2 (zero for x ≤ 0, like ReLog), alongside ReLog under identical training settings (matched learning rate and precision, 30 seeds) on both synthetic regression anchors:
So the accuracy and stability gains come from sublinear growth, not from the floating-point construction. The revised contribution list and Conclusion report this comparison and restate the contribution accordingly: ReLog reaches that shape with ALU-only operations (shift, mask, add), which is what makes it cheap on hardware without fast transcendentals. We did not test a generic piecewise-linear approximation with other knot placements; ReLog is itself the piecewise-linear interpolant of log2(1+x) with knots at powers of two (Mitchell, 1962), and comparing knot placements remains open. These comparisons cover the two regression anchors only, not the larger models.