- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper the authors propose SQUAF, a trainable activation function shaped like a smooth step function. Each layer has one such function, and the step heights, spacing and sharpness are learned during training. Before training, each SQUAF is fitted to the network's usual activation (e.g. ReLU or GELU), so the network starts close to the original. The paper proves that SQUAF can approximate any continuous function of one variable. It reports gains on fitting simple functions, image classification (CIFAR-100, ImageNet), knowledge distillation, physics-informed networks for solving PDEs, and fitting single images and 3D shapes.
Critical strengths
- The method is simple and has an intuitive reading as a soft, learnable step function. Fitting it to an existing activation before training is a practical way to initialize it.
- Breadth of evaluation covers CNNs, ViTs, PINNs and INRs, and shows reported gains.
- SQUAF is smooth with closed-form derivatives, so unlike piecewise activations it can be used where higher-order derivatives are needed, as in PINNs.
Critical weaknesses
- The novelty of the method is limited. SQUAF is essentially a kernel activation function (KAF) with normalized weights and a learned grid, in an already crowded area (rational/PAU activations, KANs). KAF is tested only on a toy fit, where it does worse than all methods and looks untuned, and is then dropped. Rational and spline activations are never compared. The universality theorem holds for any such family.
- Most examples are toy examples with PINN examples covering 10 of the 20 datasets (and skip many of the hard data), INR only on image fitting or shape fitting. No NERF or Blender Bot data. All models considered are quite small.
- There are no error bars anywhere, and margins are often under 1 point. ImageNet has no learnable baselines, and on ViT-T every flexible-shape baseline is below GELU (which feels surprising). The gains are not significant as this method adds more latency.
- It is not clear where the gains are coming from : on CIFAR, all but the first SQUAF is trained. No classification experiment compares frozen and trained SQUAFs. All models are trained from scratch. Finally the KD claim that students cannot beat their teachers is contradicted by standard KD methods on the same teacher–student pairs [1].
- Page 7 with all the tables and figures is very hard to read and table 4 is almost incomprehensible.
[1] Decoupled Knowledge Distillation Zhao et al. CVPR 2022
Questions for authors
- Does learning the activation matter? On CIFAR-100, how do all-frozen and all-trained SQUAFs compare? Alternatively, what happens when a pretrained ViT is fine-tuned with a GELU-fitted SQUAF vs. with GELU?
- How does SQUAF compare with a tuned KAF and with rational (PAU) activations on ImageNet, with error bars over 3 seeds?
- Does SQUAF help in more realistic settings, e.g. pretraining a language model such as GPT-2, or distilling from a strong teacher with the DeiT recipe [1]? Also what does the SQUAF student network get when trained normally?
- Why do the Burgers 2d-C baselines in Table 4 differ from PINNacle's, and why does DiTAC collapse on ViT-T?
[1] Training data-efficient image transformers & distillation through attention Touvron et al. ICML 2021
10 comments
Response to Reviewer VhJb (part 2/6)
Weakness 2: The evaluation mainly covers toy or small-scale cases, omits harder PINNacle problems, and does not include NeRF.
Response: In our original submission, the experimental evaluation includes tasks covering function fitting, image classification on CIFAR-100 and ImageNet-1K (including knowledge distillation and model depth reduction), ten PDE problems from PINNacle, standard 2D/3D INR tasks following the established benchmarks, with numerous evaluated model architectures and compared baseline AFs. It also includes preliminary results for GPT-2 fine-tuning and KAN architectures in appendices. To the best of our knowledge, no other work on AFs has demonstrated comparable experimental breadth to us. There are AF papers dedicated to regression, classification, INRs, KANs, PDE solving, or LLMs, while our evaluation covers all of these. We believe this breadth is important evidence for SQUAF's effectiveness as a general-purpose AF and should not be undervalued. As for the scale of individual experiments, we'd like to clarify that a single ImageNet-1K training run for the larger models considered in this paper (ViT-T and Swin-T) could easily take hundreds of GPU hours, and we believe that they should be considered as large-scale instead of toy experiments. It's also worth noting that since the results of baseline AFs are not generally available in the literature, we have had to reproduce many of them by ourselves, and this also considerably increased the scale of our experimentation. Overall, we believe the breadth and scale of our experiments provide a substantial evaluation of SQUAF’s effectiveness.
Nonetheless, as suggested by the reviewer, we now evaluate SQUAF on the remaining 12 PDE problems from PINNacle, extending beyond the ten cases considered in Sec. 4.3. Table 31 in Appendix A.21 (also shown below) reports the L2RE results for baselines Tanh, LAAF and GAAF (cited from the original PINNacle paper), and compares them with SQUAF results averaged over three runs. Note that given the limited time of the rebuttal period, we were unable to complete experiments for other ten baseline AFs in Table 4, so we only compare with the three baselines that have available results in the literature.
| PDE | Tanh | LAAF | GAAF | SQUAF |
|---|---|---|---|---|
| Poisson 2d-CG | 6.36E-1 | 4.80E-1 | 8.71E-1 | 3.62E-1 |
| Poisson 2d-MS | 6.30E-1 | 5.93E-1 | 9.31E-1 | 5.74E-1 |
| Heat 2d-CG | 3.64E-2 | 2.39E-2 | 4.61E-1 | 1.87E-2 |
| Heat 2d-LT | 9.99E-1 | 9.99E-1 | 9.99E-1 | 9.99E-1 |
| NS 2d-LT | 9.96E-1 | 9.98E-1 | 9.99E-1 | 9.93E-1 |
| Wave 1d-C | 5.88E-1 | 4.54E-1 | 6.77E-1 | 4.24E-1 |
| Wave 2d-CG | 1.84E0 | 8.19E-1 | 7.94E-1 | 7.99E-1 |
| Wave 2d-MS | 1.34E0 | 1.06E0 | 1.06E0 | 1.04E0 |
| Chaotic GS | 3.19E-1 | 9.47E-2 | 9.46E-2 | 9.41E-2 |
| Chaotic KS | 1.01E0 | 1.01E0 | 1.00E0 | 1.00E0 |
| High dim PNd | 3.04E-3 | 4.14E-3 | 7.75E-3 | 2.41E-3 |
| High dim HNd | 3.61E-1 | 5.22E-1 | 5.21E-1 | 2.68E-1 |
It's shown that SQUAF improves over the best listed baseline on 9 of the 12 cases and matches it on two at the reported precision. This shows that SQUAF is also effective on many of the harder PDE problems.
Response to Reviewer VhJb (part 3/6)
Moreover, per your suggestion, we extend the INR evaluation to novel-view synthesis with NeRF on eight scenes from the Blender dataset, following FINER's experimental setup [1]. The SQUAF initialization follows exactly the same as our 2D INR experiments. Table 32 in Appendix A.22 (also shown below) reports the results. All baseline results are cited from FINER's Table 3.
PSNR :
| Methods | Chair | Drums | Ficus | Hotdog | Lego | Materials | Mic | Ship |
|---|---|---|---|---|---|---|---|---|
| PEMLP | 31.32 | 20.18 | 24.49 | 30.59 | 25.90 | 25.16 | 26.38 | 21.46 |
| Gauss | 32.68 | 23.16 | 26.10 | 32.17 | 28.29 | 26.19 | 33.59 | 22.28 |
| SIREN | 33.31 | 24.89 | 27.26 | 32.85 | 29.60 | 27.13 | 33.28 | 22.25 |
| WIRE | 29.31 | 22.22 | 25.91 | 30.11 | 25.76 | 25.05 | 32.35 | 21.15 |
| FINER | 27.05 | |||||||
| SQUAF | 34.01 | 25.20 | 28.72 | 33.16 | 30.44 | 34.10 | 22.64 |
SSIM :
| Methods | Chair | Drums | Ficus | Hotdog | Lego | Materials | Mic | Ship |
|---|---|---|---|---|---|---|---|---|
| PEMLP | 0.960 | 0.814 | 0.914 | 0.945 | 0.904 | 0.909 | 0.960 | 0.754 |
| Gauss | 0.967 | 0.883 | 0.933 | 0.956 | 0.932 | 0.916 | 0.980 | 0.782 |
| SIREN | 0.971 | 0.947 | 0.948 | 0.932 | 0.979 | 0.788 | ||
| WIRE | 0.938 | 0.858 | 0.931 | 0.938 | 0.886 | 0.898 | 0.978 | 0.734 |
| FINER | 0.911 | 0.959 | ||||||
| SQUAF | 0.974 | 0.913 | 0.961 | 0.961 | 0.952 | 0.983 | 0.804 |
LPIPS :
| Methods | Chair | Drums | Ficus | Hotdog | Lego | Materials | Mic | Ship |
|---|---|---|---|---|---|---|---|---|
| PEMLP | 0.026 | 0.185 | 0.056 | 0.037 | 0.070 | 0.050 | 0.057 | 0.190 |
| Gauss | 0.019 | 0.082 | 0.055 | 0.040 | 0.037 | 0.013 | 0.136 | |
| SIREN | 0.017 | 0.033 | 0.030 | 0.028 | 0.014 | 0.116 | ||
| WIRE | 0.035 | 0.106 | 0.045 | 0.048 | 0.075 | 0.058 | 0.014 | 0.172 |
| FINER | 0.052 | 0.033 | 0.032 | |||||
| SQUAF | 0.014 | 0.050 | 0.022 | 0.028 | 0.022 | 0.009 | 0.096 |
It's shown that SQUAF performs the best on 7 of the 8 scenes, and is the runner-up on the remaining one. These results extend SQUAF's effectiveness on INR beyond fitting images and SDFs.
Response to Reviewer VhJb (part 1/6)
We thank the reviewer for taking time to review our paper and provide valuable feedbacks. Please find our point-by-point responses below. References to paper contents in our responses correspond to the revised manuscript unless otherwise specified.
Response: Our contribution is a general construction based on learnable soft quantization, together with its theoretical properties, closed-form derivatives, and effectiveness as an AF across a variety of model architectures and tasks. The SQUAF formulation is defined through a general CPMF satisfying the stated differentiability and localization conditions, with Gaussian and Laplacian constructions as examples. Theorem 1 establishes sufficient conditions for this construction to approximate continuous functions on a closed interval. Indeed, the universal approximation capability is shared by other function families. However, our universal approximation theorem (Theorem 1) does not automatically hold for any such family. For example, KAF generally fails the localization property P2, and therefore falls outside the assumptions of our theorem. In contrast, the Gaussian and Laplacian-based SQUAF constructions are compliant with our Theorem 1.
Nonetheless, we agree that more comparison with KAF, PAU and splines could better showcase the benefits of our proposed parameterization. To this end, we have added KAF, PAU and B-spline comparisons on all four 3D shape representation tasks in Appendix A.14. All methods use the same PEMLP backbone, one AF parameter set per layer shared across neurons, and initialization to approximate ReLU over [−0.8,0.8]. All Gaussian-kernel variants use the same initial grid A^={0,±q,±2q,…,±kq} as SQUAF, with initial q=0.05 and k=16. Full SQUAF learns (z,q,α) with normalized weights as in Eq. (7). Unnormalized SQUAF removes the normalization denominator while retaining these trainable parameters. KAF corresponds to unnormalized SQUAF with fixed (q,α); it trains only z. We also consider an input-scaled version of KAF ϕscaled(x)=∑i=−1616ziexp[−α(sx−iq)2] as suggested by Reviewer DYwK, where q and α are fixed while (z,s) are trainable.
Tables 17–18 in Appendix A.14 (also shown below) show that full SQUAF achieves the best average Chamfer distance and IoU.
Chamfer distance (lower is better):
IoU (higher is better):
We also incorporate SQUAF's results with only z trainable from Table 9 of Appendix A.5 into the tables above, in order to present the following observations. Jointly learning the grid, smoothness and amplitudes is particularly important for the normalized SQUAF: its average Chamfer distance improves from 3.227×10−6 with only z trained to 2.542×10−6 with all parameters trained. For the unnormalized family, the corresponding improvement is smaller, from KAF's 2.776×10−6 to 2.728×10−6 of the unnormalized SQUAF. Overall, the combination of normalization and full parameter adaptation, i.e., our original SQUAF, is shown to be the best design.