acceptodds
Under review as a conference paper at ICLR 2027

PRECISION ENGINEERING AN ACTIVATION FUNCTION

Abstract

Activation functions are typically designed as mathematical abstractions and evaluated independently of their underlying hardware representations. In this work, we approach the activation function as an engineering problem constrained by three requirements: (1) sublinear growth that mirrors the diminishing marginal impact of large-magnitude features while remaining unbounded above, (2) execution via standard Arithmetic Logic Unit (ALU) instructions rather than Special Function Units (SFUs), and (3) continuity and differentiability almost everywhere, sufficient for stable gradient-based optimization. To satisfy these constraints simultaneously, we introduce ReLog, an engineered activation, constructively defined by the IEEE 754 floating-point bit layout. By isolating the exponent and mantissa, ReLog replaces the linear positive domain of ReLU with a piecewise-linear behavior. This hardware-driven construction yields several testable geometric and optimization properties, which we verify empirically rather than assert. Topologically, ReLog possesses a finite number of piecewise “kinks,” bounded by for exponent field width (128 for FP32/BF16, 16 for FP16/FP8-E5M2, 8 for FP8-E4M3), with strictly linear sub-regions between them. Its gradient evaluates to exactly for exponent field and bias – a magnitude-dependent decay we show acts as an intrinsic per-node damping mechanism: across a spectral-radius sweep up to (240 trained configurations), ReLog shows zero observed divergence (Wilson 95% CI ), while standard clipping norms fail to reliably rescue an unclipped baseline at the same instability levels. Because ReLog is defined by the bit pattern, numerical precision governs its own stability: we find a clean, monotonic training-instability dose-response throughout the precision formats. We do not position ReLog as a universal replacement for all architectures, but rather test and report which domains benefit from it and which suffer.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
activation functions, hardware-aware neural network design, IEEE 754 floating-point, low-precision training, quantization, piecewise-linear networks, training stability, gradient clipping, recurrent neural networks, diffusion transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mfSbOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReLog, a novel hardware-driven activation function designed from an engineering perspective using the IEEE 754 floating-point bit layout, moving away from traditional abstract continuous mathematical curves. By isolating the exponent and mantissa bits, ReLog enables fast, continuous, and sublinear growth using standard Arithmetic Logic Unit (ALU) instructions rather than expensive Special Function Units (SFUs), providing an intrinsic per-node damping mechanism that ensures robust training stability and prevents activation explosion under recurrent stress. While empirical evaluations across Diffusion Transformers and image classification tasks demonstrate that ReLog significantly improves performance and stability without diverging, it exhibits higher wall-clock computational costs on standard GPUs due to memory bandwidth constraints.

Critical strengths
  1. ReLog's forward-pass log-compression property provides a provable bounded-orbit mechanism. Across a spectral-radius sweep up to ρ=5.0\rho = 5.0 in recurrent networks, ReLog achieved zero observed divergences, whereas traditional gradient clipping failed to rescue unclipped baseline instabilities.
  2. In Diffusion Transformers (DiT) and CIFAR-10 image classification tasks, ReLog significantly outperformed ReLU, GELU, and SiLU across FP32, BF16, and FP16 precisions.
  3. ReLog completely avoids computationally expensive Special Function Units (SFUs) and is constructed entirely using standard Arithmetic Logic Unit (ALU) bit operations.
Critical weaknesses
  1. On standard GPUs (such as NVIDIA B200 / RTX 4090), ReLog's compiled kernel is slower than standard activations (approximately 2×2\times slower in forward pass and 1.6–1.9×1.6\text{--}1.9\times slower in forward-plus-backward). Because element-wise activations are bandwidth-bound rather than compute-bound on GPUs, the hardware ALU advantage fails to translate into wall-clock speedups.
  2. ReLog closely resembles a piecewise-linear approximation to a logarithmic activation. However, the experiments primarily compare it against ReLU, GELU, and SiLU. Without comparisons against log1p-based activations and generic piecewise-linear approximations, it is difficult to determine whether the observed improvements originate from the proposed floating-point construction or simply from the sublinear growth behavior.
  3. The improvements in the diffusion transformer experiments are relatively small in absolute magnitude. Although statistical significance is reported, the paper does not demonstrate corresponding improvements in generation quality or performance under matched training-time budgets.
  4. The experimental evaluation mainly involves small-scale or customized architectures. In particular, the CIFAR-10 classification experiments use a relatively weak CNN baseline, and the paper does not demonstrate improvements on stronger and more representative architectures.
Questions for authors
  1. What empirical evidence supports the ALU-native efficiency motivation when the proposed implementation is slower than standard activations on GPU? Can the authors provide measurements on hardware where special-function operations are a significant bottleneck?
  2. How does ReLog compare against log1p-based activations and generic piecewise-linear logarithmic approximations under identical training settings?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mfSb (1/2)

Response to Reviewer mfSb (part 1 of 2)

Thank you for the review. Your questions go to the center of the paper, and answering them honestly changed how we state its contribution. Part 1 covers efficiency and the log1p comparison; part 2 covers the DiT and the choice of architectures. Our general comment lists every change in the revised PDF.

1. Evidence for the ALU-native motivation (Question 1, Weakness 1). You are right that on GPUs a standalone elementwise activation is bandwidth-bound, so ALU savings do not appear in wall-clock time: our compiled kernel is slower than ReLU (Section 5.4), and even a torch.compile-fused version only matches the baselines' forward time on an RTX 4090 (0.29 ms against 0.30-0.34 ms). We have now measured on hardware where special functions are a real bottleneck, as you asked:

  • A microcontroller with no FPU (Arduino Uno, ATmega328P), where every transcendental runs in software. Forward pass per call, positive inputs: ReLU 4.59 µs, ReLog 40.76 µs, exact log2(1+x) 169.65 µs, SiLU 186.19 µs, GELU 263.29 µs. ReLog is 4.2-6.5× faster than the exact logarithm, SiLU and GELU, and 8.9× slower than ReLU.
  • CPU (no SFUs; 4096×4096, FP32). Forward-only, ReLog is 10-13% faster than GELU and SiLU and 14% slower than ReLU; against an exact log2(1+x), it needs 0.54× the forward time.

The comparison that matters most is ReLog against the exact logarithm: the same shape, at about a quarter of the cost on the microcontroller. Section 5.4 and Appendix C (F10, F11) give the details. These are microbenchmarks of single activations, not full networks, and ReLU stays faster.

2. log1p and piecewise-linear baselines (Question 2, Weakness 2). Your suspicion is correct, and we now say so in the paper. We had run a log1p-based activation, exact log2(1+x) = log1p(x)/ln 2 (zero for x ≤ 0, like ReLog), alongside ReLog under identical training settings (matched learning rate and precision, 30 seeds) on both synthetic regression anchors:

  • Weierstrass target: exact log2 is significantly better than ReLog (mean loss 0.198 vs. 0.221, Welch p < 0.00001).
  • Chaotic logistic-map target: the two are statistically indistinguishable (0.077 vs. 0.084, p = 0.069).
  • Both beat ReLU, GELU and SiLU by roughly 2× on both tasks.

So the accuracy and stability gains come from sublinear growth, not from the floating-point construction. The revised contribution list and Conclusion report this comparison and restate the contribution accordingly: ReLog reaches that shape with ALU-only operations (shift, mask, add), which is what makes it cheap on hardware without fast transcendentals. We did not test a generic piecewise-linear approximation with other knot placements; ReLog is itself the piecewise-linear interpolant of log2(1+x) with knots at powers of two (Mitchell, 1962), and comparing knot placements remains open. These comparisons cover the two regression anchors only, not the larger models.

AuthorsOpenReview ↗

Response to Reviewer mfSb (2/2)

Response to Reviewer mfSb (part 2 of 2)

3. The DiT improvements, generation quality and training budgets (Weakness 3). Your concern was well founded, and computing FID changed our claim.

  • The DiT in the submission cannot generate. Each 16×16 patch holds 768 values but the model width is 384, so half of the noise in every patch is invisible to the network and the loss can never go below 0.50. All four activations produce noise images, with FID around 445. ReLog's advantage in the DiT table is statistically real, but small and confined to this bottleneck. (That table reports best-epoch validation loss; our caption mislabeled it as training loss, which we fixed.)
  • In a DiT that works, ReLog is at parity. Retraining the same architecture on 64×64 images with 4×4 patches removes the bottleneck. FID over 10,000 samples: ReLog 145.3 ± 4.7, ReLU 144.9 ± 13.4, SiLU 141.4 ± 1.5, GELU 133.5 ± 3.0, with no difference significant after Holm correction (3 seeds per activation).
  • Training budgets. We matched training steps (200 epochs for every run), not wall-clock time. On our GPU, ReLog runs took roughly 8,800 s against about 7,000 s for the baselines, because that machine uses our slower pure-PyTorch implementation. Under a matched time budget ReLog would have had fewer steps, so we would not expect it to gain ground.

We no longer claim that ReLog outperforms the baselines in a DiT. Section 5.1, the contribution list and the Conclusion say so, and the new Appendix D gives the details, with sample images from both models.

4. Small or custom architectures (Weakness 4). The paper tests ReLog across six model families: MLPs, a vanilla RNN, a selective state-space model, a CNN, a Transformer language model, and a DiT at two input resolutions. The outcomes are mixed, and the revised paper says so: ReLog leads on CIFAR-10 but is at parity on WikiText-2 and in the working DiT. We deliberately did not use a heavily tuned architecture such as ResNet for CIFAR-10. Its initialization, learning-rate schedule and block design have been tuned around ReLU over years of community work, so substituting a new activation there measures how well it fits ReLU-shaped assumptions rather than the activation itself. Our CNN is untuned toward any activation, and each activation gets its own learning-rate sweep, so the comparison is fair at the scale we tested. We agree this does not show how ReLog behaves in larger, stronger models.

Reviewer_BRimOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReLog, a piecewise-linear activation function constructed from the exponent and mantissa of floating-point numbers. The paper studies its mathematical properties, training stability, and behavior under different numerical precisions, and benchmark its GPU implementation. The paper evaluates ReLog against standard activation functions on synthetic recurrent systems, C-MAPSS sequence modeling, image generation on Imagenette, and image classification on CIFAR-10 . The paper reports improved stability in synthetic recurrent systems and in some C-MAPSS experiments using a custom TinySSM with an S6 selective-scan block. They also report lower noise-prediction training loss for aa Diffusion Transformer with adaLN-conditioned Transformer blocks on Imagenette, and higher classification accuracy for a custom CNN with three convolutional blocks on CIFAR-10.

Critical strengths
  1. ReLog is an activation function that compresses large positive values inside a network, aiming to reduce the risk of these values growing out of control during training. It constructs this behavior directly from the way computers store numbers.

  2. In the controlled stability tests, the authors report no training failures in 240 runs using ReLog, compared with 107 using ReLU. The image generation and classification experiments are also repeated multiple times, providing further support for the reported improvements.

Critical weaknesses

The image generation results are incomplete: The DiT experiments only report training loss. Lower training loss does not necessarily reflect better generated images. Metrics such as FID would help establish whether the improvement reflects better image quality.

The practical efficiency benefit remains unclear: The reported FP32 ReLog kernel takes about 2× the forward-pass time of ReLU and 1.6–1.9× the forward-plus-backward time. The authors acknowledge this slowdown and suggest that edge devices may benefit, but provide no experiments on those devices. The practical efficiency advantage therefore remains unverified.

Questions for authors
  1. The CIFAR-10 experiment uses a small, custom CNN with three convolutional blocks. Results with another established architecture or model size would help show whether the gains extend beyond this setup. Multiple random seeds are useful, but do not establish this broader applicability.

  2. The explicitly listed DiT settings give 4 activations × 3 precisions × 20 seeds = 240 runs, while the text reports 720 cells. For CIFAR-10, the listed settings give 4 × 3 × 30 = 360 runs, while the text reports 1080 cells. It is not clear that if these total include additional experimental settings that are not described.

2 comments
AuthorsOpenReview ↗

Response to Reviewer BRim

Response to Reviewer BRim

Thank you for the review. Your two weaknesses led to the two most important changes in the revised paper, so we take them first, then your questions. Our general comment lists every change in the revised PDF.

1. DiT: training loss and FID (Weakness 1). You were right to ask for FID, and computing it changed our claim.

  • The metric was validation loss, mislabeled. The DiT table reports best-epoch noise-prediction MSE on a held-out validation split; our caption wrongly called it training loss. We fixed the label and recomputed the table from the final run logs. Baseline entries changed by up to 0.0005; ReLog is still lowest in every row, and all nine comparisons remain significant.
  • Lower loss did not mean better images, as you suspected. Computing FID showed that this DiT cannot generate at all. Each 16×16 patch holds 768 values but the model width is 384, so half of the noise in every patch is invisible to the network and the loss can never go below 0.50. All four activations produce noise images, with FID around 445. ReLog's lower loss is statistically real, but small and confined to this bottleneck.
  • In a DiT that works, ReLog is at parity. We retrained the same architecture on 64×64 images with 4×4 patches, which removes the bottleneck (3 seeds per activation). This model generates structured images. FID over 10,000 samples: ReLog 145.3 ± 4.7, ReLU 144.9 ± 13.4, SiLU 141.4 ± 1.5, GELU 133.5 ± 3.0. No difference is significant after Holm correction.

We therefore no longer claim that ReLog outperforms the baselines in a DiT. Section 5.1, the contribution list and the Conclusion now say this, and the new Appendix D gives the details, with sample images from both models.

2. Efficiency on edge devices (Weakness 2). We have now run the experiment you asked for, on a microcontroller with no floating-point unit (Arduino Uno, ATmega328P), where every transcendental function runs in software. Forward pass per call, positive inputs: ReLU 4.59 µs, ReLog 40.76 µs, exact log2(1+x) 169.65 µs, SiLU 186.19 µs, GELU 263.29 µs. ReLog is 4.2-6.5× faster than GELU, SiLU and the exact logarithm, but 8.9× slower than ReLU. On CPU, ReLog's forward pass is 10-13% faster than GELU and SiLU and 14% slower than ReLU. So the efficiency advantage is real against the smooth activations, but not against ReLU. Section 5.4 and Appendix C (F10, F11) give the details. These are microbenchmarks of single activations, not full networks.

3. Beyond the three-block CNN (Question 1). We agree that the CIFAR-10 result alone does not establish broader applicability, which is why the paper tests ReLog across six model families: MLPs, a vanilla RNN, a selective state-space model, this CNN, a Transformer language model, and a DiT at two input resolutions. The gains do not carry over uniformly: ReLog leads on CIFAR-10 but is at parity on WikiText-2 and in the working DiT, and the revised paper says so. We did not add a second, established CIFAR architecture, for a methodological reason. Architectures such as ResNet have had their initialization, learning-rate schedule and block design tuned around ReLU over years of community work, so substituting a new activation there measures how well it fits ReLU-shaped assumptions rather than the activation itself. This is the confound our per-activation learning-rate sweep avoids at the hyperparameter level, and the reason we built the CNN untuned toward any activation. On model size, the same logic holds: at the size we tested, the comparison is apples to apples, with an identical network in which only the activation changes and each activation gets its own learning-rate sweep. That makes it a fair test at that size. We do not claim the gap is the same at every size, and our own DiT results show why we are careful: there, ReLog's advantage depended on the model configuration.

4. The 240/720 and 360/1080 counts (Question 2). The totals include one more setting that our text failed to state: a 3-point learning-rate grid. DiT: 4 activations × 3 learning rates × 3 precisions × 20 seeds = 720. CIFAR-10: 4 × 3 × 3 × 30 = 1080 (a 10-seed pilot on the same grid was 360). There are no other settings. Sections 5.1 and 5.2 now state all four factors, and the run logs confirm 720/720 and 1080/1080 completed runs.

I thank the authors for their detailed response. The rebuttal has adequately addressed my concerns, and I have no further questions. I will maintain my original rating. I encourage the authors to incorporate the clarifications and additional results into the final version.

Reviewer_JegKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReLog, an activation function constructed from the IEEE 754 bit layout. For x > 0, it computes f(x) = (unbiased exponent of x+1) + (mantissa fraction of x+1). This is the classical Mitchell (1962) linear-mantissa approximation of log₂(x+1), used here as a training-time activation.

The authors pose three design constraints: sublinear growth, ALU-only execution without SFU calls, and almost-everywhere differentiability. The contributions are:

(i) a proof that the forward pass bounds recurrent state for any ‖W‖∞ (Proposition 1); (ii) a synthetic recurrent stress test in which ReLog shows 0/240 divergences versus 107/240 for ReLU, plus a C-MAPSS SSM experiment; (iii) an FP32-FP8 precision sweep showing training instability increasing as mantissa width shrinks, with FP4/NF4 declared structurally infeasible; (iv) improvements over ReLU/GELU/SiLU on a small DiT (Imagenette) and a small CIFAR-10 CNN.

A throughput benchmark shows ReLog is about 2× slower than ReLU (forward) and 1.6–1.9× slower (forward+backward) on GPU.

Critical strengths
  1. Well-motivated first-principles design. The paper derives the activation directly from the IEEE 754 format: the exponent/mantissa layout already approximates log₂, so the resulting function is non-linear and sublinear, and its forward pass provably bounds recurrent state, which mitigates explosion. The approximation itself is classical (Mitchell, 1962). Turning the storage format into a training-time activation is nonetheless a coherent idea that could inspire further hardware-aware activation design.

  2. Careful, candid stability evaluation. The recurrent stress test reports 0/240 divergences for ReLog versus 107/240 for ReLU, with confidence intervals and significance tests. The paper also openly reports negative results such as the GPU slowdown, which makes its claims easy to audit.

Critical weaknesses
  1. The hardware-efficiency contribution is not supported by evidence, and the only measurement contradicts it. ALU-native execution is one of the three core design constraints and the main reason for building the activation from the bit layout. However:

    • The only performance measurement (Section 5.3) shows ReLog is about 2× slower than ReLU in the forward pass and 1.6–1.9× slower in forward+backward. It reports no absolute times or separate GELU/SiLU numbers.
    • The compiled kernel exists only for FP32. The BF16/FP16 experiments run through a pure-PyTorch simulation path whose cost is not reported.
    • The ALU-level efficiency argument is purely conceptual, with no hardware-level measurement.
    • The suggested benefit on SFU-free edge hardware is not evaluated.

    As a result, a central design goal remains an argument rather than a demonstrated property.

  2. Several claims are overstated or not matched by experiments.

    • The introduction claims stress tests across CV, NLP and DSP, and Section 5.3 is titled "...in NLP tasks", yet no NLP experiment appears in the paper.
    • The stability comparison is mainly against ReLU (with gradient clipping). Bounded or compressive alternatives that also act on the forward pass, such as tanh, LayerNorm, or spectral normalization, are deferred to future work. It is therefore unclear whether the advantage is specific to ReLog.
    • The DiT result is reported only on training loss, with very small absolute differences (e.g., 0.5060 vs 0.5071), and without validation loss or FID.
    • The paper also contains draft artifacts, such as a note that kernel fixes have "not yet been re-validated on real hardware" and references to undefined or missing content (e.g., "Weierstrass anchor"). This makes it unclear which code produced the reported results.
Questions for authors

Clarifications:

  1. Where are the NLP experiments referred to in the introduction and in the title of Section 5.3?
  2. Were all reported training runs produced with the kernel version shown in Appendix A, or with the earlier version that the revision note says was fixed?
  3. How does ReLog compare with tanh, LayerNorm, or spectral normalization on the recurrent stress test, as planned in Appendix C (F2)?

Would change my assessment:

  1. Real-hardware benefit. Can you demonstrate a measurable efficiency gain (latency, throughput, or energy) of ReLog over ReLU/GELU/SiLU on real hardware? This could be a fused GPU kernel or SFU-free hardware such as an MCU or FPGA, as suggested in the paper.
  2. Performance comparison across models. Can you report task performance together with runtime cost for ReLog versus the baselines on a broader set of models, including the BF16/FP16 settings used in the paper? Validation metrics (e.g., FID for DiT) should be included rather than training loss only.

I would consider raising my score if the authors can show concrete hardware gains together with a convincing performance comparison across models, since this would substantiate the paper's central efficiency claim.

3 comments
AuthorsOpenReview ↗

Response to Reviewer JegK (1/2)

Response to Reviewer JegK (part 1 of 2)

Thank you for a careful review. We answer the two points you said would change your assessment first, and the weaknesses and clarification questions in part 2. Our general comment lists every change in the revised PDF.

1. A measurable efficiency gain on real hardware (Critical Weakness 1). We can now show one on an MCU against GELU and SiLU, though not against ReLU. All of the following is new in Appendix C (F10, F11), and Section 5.4 and the Conclusion now report it:

  • Microcontroller, as you suggested (Arduino Uno, ATmega328P, 16 MHz, no FPU; forward pass of one element, 5,120 calls per measurement, median of five, call overhead subtracted). For positive inputs: ReLU 4.59 µs, ReLog 40.76 µs, exact log2(1+x) 169.65 µs, SiLU 186.19 µs, GELU 263.29 µs. ReLog is 4.2-6.5× faster than exact log2, SiLU and GELU, and 8.9× slower than ReLU. Its outputs match our training code bit for bit on a self-test. A cycle-accurate simulation of the chip agrees with the board to within 1%.
  • CPU, which has no Special Function Units (4096×4096, FP32, 200 iterations; each baseline's native eager kernel against ReLog compiled with torch.compile). Forward-only, the number relevant to inference on an edge device: ReLog 5.82 ms, against ReLU 5.12, GELU 6.68 and SiLU 6.48 ms. ReLog is 10-13% faster than GELU and SiLU and 14% slower than ReLU. Forward-plus-backward: ReLog 16.34 ms, against 9.66, 18.32 and 12.78 ms.
  • Against a transcendental with the same shape. An exact log2(x+1), also compiled, takes 10.70 ms forward and 23.01 ms forward-plus-backward, so ReLog needs 0.54× and 0.71× its time. This is the comparison the ALU-native design is about: the same logarithmic shape without a transcendental call.
  • A fused GPU version. Compiling ReLog's bitwise formulation with torch.compile on an RTX 4090 laptop GPU gives a forward time of 0.29 ms, the same as the baselines (0.30-0.34 ms). At this size a standalone elementwise operation is memory-bandwidth bound, so ALU savings do not show in wall-clock time. Fusion removes our hand-written kernel's slowdown, but it does not produce a gain.
  • Absolute times and GELU/SiLU on the B200. The Section 5.4 kernel measurement was against ReLU only, and our cluster allocation expired before we could rerun it with all baselines. The CPU and RTX 4090 numbers above are the absolute times we can report.

Limits: the MCU test is a scalar microbenchmark of float inference, not an end-to-end network; we have no energy or FPGA measurements; and MCU deployments are often int8-quantized, where activations become lookup tables and all of them are cheap. ReLU is faster in every measurement except the bandwidth-bound GPU forward pass.

2. Performance across models, BF16/FP16 cost, and validation metrics.

  • The DiT metric is validation loss; our caption mislabeled it. The DiT table reports best-epoch noise-prediction MSE on a held-out validation split, not training loss. We corrected the caption and text. Recomputing the table from the final run logs changed some baseline entries by up to 0.0005: ReLog is still lowest in every row, all nine comparisons remain significant after Holm correction, and the effect sizes are now d = 0.82-1.70 (previously reported as 0.82-1.95).
  • FID, and a correction. Computing FID showed that this DiT cannot generate: each 16x16 patch holds 768 values but the model width is 384, which provably bounds the loss at 0.50. In a DiT without the bottleneck, ReLog is at parity with the baselines on validation loss and FID (145.3 ± 4.7 against 133.5-144.9, none significant after Holm correction). We no longer claim that ReLog outperforms in a DiT; the general comment and the new Appendix D give the details.
  • The BF16/FP16 path is expensive, and it was unreported. These formats run ReLog through the pure-PyTorch bit simulation, in eager mode in our training runs. On the RTX 4090 that path is 10-17× slower than ReLU forward (3.6-5.0 ms against 0.30-0.34 ms) and 9-12× slower forward-plus-backward. Compiled, it matches the baselines forward and runs at 0.9-1.4× their time forward-plus-backward. This is now in Appendix C.
  • Runtime per task model. We did not measure end-to-end runtime for each task model, so we cannot report task performance and runtime side by side per model.
AuthorsOpenReview ↗

Response to Reviewer JegK (2/2)

Response to Reviewer JegK (part 2 of 2)

3. Overstated claims and draft artifacts.

  • The NLP experiments (Question 1). They were run in full (WikiText-2 language modeling, 1800/1800 cells) but cut from the 9-page submission for space. Section 5.3 now reports them: under a per-activation learning-rate sweep, ReLU reaches 77.53 perplexity, ReLog 77.86, GELU 78.12 and SiLU 78.29 (FP32, 30 seeds each), with no significant difference between ReLog and any baseline after Holm correction. Restoring this section moved the throughput section to 5.4.
  • tanh, LayerNorm and spectral normalization (Question 3). We have not run these baselines; they remain follow-up F2 in Appendix C. Without them, the stress test cannot show that the stability comes from ReLog's log-compression specifically rather than from any activation that keeps the recurrent state bounded. Tanh would plausibly avoid divergence too, by a different route: it saturates, whereas ReLog is unbounded and Proposition 1 bounds its orbit through logarithmic growth. We revised Section 3, the contribution list and the Conclusion to limit the claim to the comparison we ran (against ReLU, zero observed divergences across 240 configurations) and to say that saturating activations may be similarly stable for a different reason. We also removed an introduction sentence that described this experiment as a comparison against classical stabilizers.
  • Draft artifacts. Appendix A now states plainly which kernel produced the results (next point), "Weierstrass anchor" is defined in Section 2, and we removed one more draft note we found in Appendix B.

4. Which kernel produced the reported results (Question 2). None of the three fixes described in Appendix A (two int64_t indexing fixes and an exp < 255 Inf/NaN guard in the backward kernel) was ever committed to the kernel repository; the FP32 kernel code has not changed since its initial commit. Every experiment in Sections 3-5 therefore ran on the unfixed kernel. The indexing fixes only matter for tensors near 2^31 elements, far beyond anything in this paper. The missing Inf/NaN guard is a real open gap for Section 3's stress test, which deliberately pushes toward divergence: we have no evidence it affected the 0/240 result, and we do not claim the reverse. Appendix A now says this directly, and a footnote in Section 3 flags it.

Separately, every cluster run before 2026-07-04 used the slower pure-PyTorch path instead of the compiled kernel, because of a missing install step in the training container. Accuracy results are unaffected, since the two paths are verified bit-for-bit identical wherever both apply. Only wall-clock time differs, and the Section 5.4 timings were measured after the fix, with the compiled kernel confirmed active.

Thank you for the more detailed explanation and response. The results provided indicate that ReLog’s efficiency advantage over GELU/SiLU on CPUs and MCUs is limited to isolated microbenchmarks for these activation functions. In all other scenarios—with the exception of bandwidth-constrained GPU forward passes, where performance is comparable—ReLog is slower than ReLU; furthermore, it is significantly slower when using the BF16/FP16 precision formats employed during training. Based on these findings, I do not believe your argument regarding an "efficiency advantage" holds true. I will maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and a correction to the DiT result

Summary of revisions, and a correction to the Diffusion Transformer result

We thank all three reviewers. We have uploaded a revised PDF. This comment lists what changed and explains the one change that reverses a claim; each reviewer's questions are answered under their review.

Changes in the revised PDF

  • Diffusion Transformer claim corrected (Section 5.1, contribution list, Conclusion) and a new Appendix D with the evidence. Explained below.
  • DiT table corrected: its caption said training loss, but the values are best-epoch validation MSE on a held-out split. We fixed the label and recomputed the table from the final run logs; some baseline entries changed by up to 0.0005, ReLog is still lowest in every row, all nine comparisons remain significant, and the effect sizes are now d = 0.82-1.70 (previously 0.82-1.95).
  • Section 3, recurrent stress test: a new scope paragraph states that the comparison is against ReLU only and that tanh, LayerNorm and spectral normalization were not tested; a footnote flags the missing backward-kernel Inf/NaN guard as an open gap for the 0/240 result.
  • Contribution list and Conclusion: report that an exact log2(x+1) matches or beats ReLog on both regression anchors, so ReLog's accuracy gains come from its logarithmic shape; its contribution is reaching that shape with ALU-only operations.
  • Section 5.3: WikiText-2 results restored (cut from the 9-page submission for space).
  • Section 5.4 and Appendix C (F10): CPU timings with absolute times, a timing comparison against an exact log2(x+1), and the cost of the BF16/FP16 path on GPU.
  • Section 5.4, Conclusion and Appendix C (F11): a microcontroller benchmark (Arduino Uno, no FPU). ReLog's forward pass is 4.2-6.5× faster than exact log2(1+x), SiLU and GELU, and 8.9× slower than ReLU.
  • Sections 5.1 and 5.2: sweep sizes spelled out on all four axes (720 and 1080 cells).
  • Appendix A: states that the three kernel fixes it describes were never merged.
  • Smaller changes: Weierstrass anchor defined (Section 2); C-MAPSS implementation footnote; roadmap paragraph and a one-row WikiText table removed for space; a leftover draft note removed (Appendix B); the AI use statement now discloses AI assistance during this revision.

The Diffusion Transformer correction

While computing FID at Reviewers' request, we found a flaw in our own Diffusion Transformer (DiT) experiment. It changes what that experiment shows, so we are reporting it to all three reviewers and not only in the reply where it came up.

The DiT in the submission cannot generate. It splits a 224x224 image into 16x16 RGB patches, so each patch holds 768 values, but the model width is 384. The network sees each patch through a rank-384 linear map, so half of each patch's noise is invisible to it. Noise-prediction MSE is provably at least 0.50 at every timestep, for any activation and any amount of training. A model that predicts zero scores 1.0.

We confirmed this by measurement. Across all 80 FP32 runs at the reported learning rate (4 activations x 20 seeds), every activation sits at 0.502 to 0.503 for timesteps t >= 300. Samples from these models are noise, and FID is about 445 for all four activations.

The reported advantage is real but narrow. The losses in the DiT table (about 0.506 to 0.509) are the 0.50 floor plus a small excess. ReLog's lower loss is statistically robust: significant against all three baselines at every timestep we tested (Holm-corrected p <= 0.021, Cohen's d from 0.76 to 2.0, 20 seeds). But it is concentrated at the lowest noise levels (t = 1: 0.731 for ReLog against 0.760 to 0.808) and is at most 0.1% of the loss for t >= 300.

A DiT without the bottleneck shows parity. We retrained the same architecture on 64x64 inputs with 4x4 patches (patch dimension 48, well below the width of 384), with 3 seeds per activation and equal training length. This model denoises (validation MSE 0.025 to 0.028) and generates structured images. FID over 10,000 samples: ReLog 145.3 +/- 4.7, ReLU 144.9 +/- 13.4, SiLU 141.4 +/- 1.5, GELU 133.5 +/- 3.0. No comparison is significant after Holm correction. GELU is nominally best on FID and on validation loss in all three seeds.

What we changed in the paper. We no longer claim that ReLog outperforms the baselines in a Diffusion Transformer. Section 5.1, the contribution list and the Conclusion now say: parity in a working generative model, and a significant advantage only in a capacity-bottlenecked model, whose mechanism we have not established. A new Appendix D gives the bound, the per-timestep tables, the control experiment, the sampling and FID protocol, and sample grids from both models.

What the control does not show. It uses three seeds at one learning rate and one precision, at a different resolution from the main sweep, without weight averaging. It can rule out a large ReLog advantage, not a small one in either direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.